Cfgadm SCSI-Plugin MPxIO Support [PSARC/2008/764 Self Review]
Jerry Gilliam
jg at jurassic.sfbay.Sun.COM
Thu Dec 11 13:57:33 PST 2008
I am sponsoring this case as a fast-track for Hyon Kim. No
interfaces are introduced or modified but the team desires to
record this information. The project team believes this case
qualifies for self-review with automatic approval, so I've
introduced it as such. Patch/micro release binding is requested.
----------
Cfgadm SCSI plugin support on a MPxIO enabled controller
1. Introduction
1.1. Project/Component Working Name:
Cfgadm SCSI plugin support on an MPxIO enabled controller
1.2. Name of Document Author/Supplier:
Author: Hyon Kim
1.3 Date of This Document:
Dec 1, 2008
4. Technical Description
4.1 Background
The cfgadm SCSI plugin checks only the existence and state of devinfo
nodes when probing a SCSI controller apid for its child thus fails to
report pathinfo based configuration on a MPxIO enabled SCSI controller.
Currently mpt(7D) driver supports MPxIO and CR 6569367 addresses the
missing cfgadm apids once an mpt based SAS controller is enabled with MPxIO.
This fasttrack proposes updates to the SCSI plugin to make it recognize
MPxIO enabled controller by probing pathinfo nodes and report pathinfo
associated apids. The type field of the apid carries path indication as
shown in the example below. This fasttrack doesn't introduce any change
to the existing cfgadm/libcfgadm interfaces.
Note that this proposal affects only the SCSI devices that are connected
to an MPxIO enabled SCSI controller and are supported by scsi_vhci(7D)
driver.
4.2 Proposed Solution
4.2.1 Listing operation
With the proposed updates, the cfgadm SCSI plugin will construct pathinfo
associated apids with the unit-address of the pathinfo nodes when probing
an MPxIO enabled controller. The unit address typically contains target
and logical unit number(LUN) information and uniquely identifies a device
on a SCSI controller.
The following cfgadm output shows MPxIO enabled mpt.7D based controller and
its child apids. The disk and tape devices that are connected on
controller c1 are configured as scsi_vhci.7D child devices so its dynamic
component of the apids consists of the unit address. The ses and
smp target devices still has /dev link based dynamic component which
is constructed through the existing SCSI plugin implementation. The type
field will indicates if an apid is configured as a path to a device
through suffix 'path'.
# cfgadm -la
Ap_Id Type Receptacle Occupant Condition
c1 scsi-sas connected configured unknown
c1::0,0 disk-path connected configured unknown
c1::1,0 disk-path connected configured unknown
c1::2,0 disk-path connected configured unknown
c1::3,0 disk-path connected configured unknown
c1::4,0 disk-path connected configured unknown
c1::5,0 disk-path connected configured unknown
c1::6,0 disk-path connected configured unknown
c1::7,0 disk-path connected configured unknown
c1::8,0 disk-path connected configured unknown
c1::9,0 disk-path connected configured unknown
c1::a,0 tape-path connected configured unknown
c1::es/ses1 ESI connected configured unknown
c1::smp/expd0 smp connected configured unknown
Without MPxIO enabled on c1, the apid for c1::0,0 and c1::a,0 would be
c1::dsk/c1t0d0 disk connected configured unknown
c1::rmt/1 tape connected configured unknown
Note that apid c1 has type scsi-sas. With the fix of CR 6762621 and
associated mpt.7D change of the SCSI plugin will append the initiator
interconnect type as defined in uts/common/sys/scsi/impl/services.h for
non SPI SCSI controller apids. The controller type will remain same whether
MPxIO is enabled on the controller or not. The 'scsi-bus' type will be used
for SPI controllers as it does today.
In order to help the administrator map the pathinfo related apid with
an associated scsi_vhci client device, the Information field will show
the /dev link of the client devices as below.
# cfgadm -lav c1::0,0
Ap_Id Receptacle Occupant Condition Information
When Type Busy Phys_Id
c1::0,0 connected configured unknown Client
Device: /dev/dsk/c5t5000C5000B1F49DBd0s0(sd10)
unavailable disk-path n
/devices/pci at 0,0/pci10de,375 at f/pci1000,3150 at 0:scsi::sd17 at 0,0
# cfgadm -lav c1::a,0
Ap_Id Receptacle Occupant Condition Information
When Type Busy Phys_Id
c3::a,0 connected configured unknown Client
Device: /dev/rmt/1(st2)
unavailable tape-path n /devices/pci at 0,0/pci10de,375 at f/pci1000,3150 at 0
:scsi::st2 at a,0
Note that the -s option can be used to conveniently capture the path apid and
its associated client device as shown example below.
# cfgadm -la -s "select=type(disk-path),cols=ap_id:type:info"
Ap_Id Type Information
c1::0,0 disk-path Client Device:
/dev/dsk/c5t5000C5000B1F49DBd0s0(sd42)
c1::1,0 disk-path Client Device:
/dev/dsk/c5t5000C5000B1E9213d0s0(sd43)
c1::2,0 disk-path Client Device:
/dev/dsk/c5t5000C5000B1F3923d0s0(sd44)
c1::3,0 disk-path Client Device:
/dev/dsk/c5t5000C5000B1EA087d0s0(sd45)
c1::4,0 disk-path Client Device:
/dev/dsk/c5t5000C5000C759D9Fd0s0(sd46)
c1::5,0 disk-path Client Device:
/dev/dsk/c5t5000C5000C7498BBd0s0(sd47)
c1::6,0 disk-path Client Device:
/dev/dsk/c5t5000C5000C7479CFd0s0(sd48)
c1::7,0 disk-path Client Device:
/dev/dsk/c5t5000C5000827AB03d0s0(sd49)
c1::8,0 disk-path Client Device:
/dev/dsk/c5t5000C5000C749403d0s0(sd50)
c1::9,0 disk-path Client Device:
/dev/dsk/c5t5000C5000C7413EBd0s0(sd51)
4.2.2 State Change operation
The configure/unconfigure state change operations will be supported on
pathinfo based apids. The operations will trigger
mdi_pi_online()/mdi_pi_offline() on the associated pathinfo nodes
respectively.
For the unconfigure operation the SCSI plugin which check if the
path that is associated with the given path apid is the last available
path to the client scsi_vhci device. If it is, the SCSI plugin goes through
RCM check before initiating path offline. If the associated client device is
indeed in use, the SCSI plugin will be pass back the error message as it does
for the devinfo node based apid.
cfgadm: Component system is busy, try again: failed to offline the client
device:
Resource Information
------------------ -------------------------
/dev/dsk/c5t5000C5000B1F49DBd0s0 mounted filesystem "/mnt"
4.4 Man Pages
See attached draft of cfgadm_scsi.1M revision.
4.5 Release Binding
Micro/patch binding is requested.
4.6 References
PSARC/1998/424 scsi hotplug cfgadm plugin
http://sac.sfbay/PSARC/1998/424
PSARC/1998/424 cfgadm extensions
http://sac.sfbay/PSARC/1998/423
6. Resources and Schedule
6.4. Steering Committee requested information
6.4.1. Consolidation C-team Name:
ON
6.5. ARC review type: FastTrack
6.6. ARC Exposure: open
System Administration Commands cfgadm_scsi(1M)
NAME
cfgadm_scsi - SCSI hardware specific commands for cfgadm
SYNOPSIS
/usr/sbin/cfgadm [-f] [-y | -n] [-v] [-o hardware_option]
-c function ap_id...
/usr/sbin/cfgadm [-f] [-y | -n] [-v] [-o hardware_option]
-x hardware_function ap_id...
/usr/sbin/cfgadm [-v] [-a] [-s listing_option] [-o hardware_option]
[-l [ap_id | ap_type ... ]]
/usr/sbin/cfgadm [-v] [-o hardware_option] -t ap_id...
/usr/sbin/cfgadm [-v] [-o hardware_option] -h [ap_id]...
DESCRIPTION
The SCSI hardware specific library /usr/lib/cfgadm/scsi.so.1
provides the functionality for SCSI hot-plugging through the
cfgadm(1M) command. cfgadm operates on attachment points,
which are locations in the system where hardware resources
can be dynamically reconfigured. Refer to cfgadm(1M) for
information regarding attachment points.
For SCSI hot-plugging, each SCSI controller is represented
by an attachment point in the device tree. In addition, each
SCSI device is represented by a dynamic attachment point.
Attachment points are named through ap_ids. Two types of
ap_ids are defined: logical and physical. The physical ap_id
is based on the physical pathname, whereas the logical ap_id
is a shorter more user-friendly name. For SCSI controllers,
the logical ap_id is usually the corresponding disk con-
troller number. For example, a typical logical ap_id would
be c0.
SCSI devices are named relative to the controller ap_id.
Thus if a disk device is attached to controller c0, its
ap_id can be:
c0::dsk/c0t0d0
where dsk/c0t0d0 identifies the specific device. In general,
the device identifier is derived from the corresponding log-
ical link for the device in /dev. For example, a SCSI tape
drive logical ap_id could be c0::rmt/0. Here c0 is the logi-
cal ap_id for the SCSI controller and rmt/0 is derived from
the logical link for the tape drive in /dev/rmt. If an iden-
tifier can not be derived from the link in /dev, a unique
identifier will be assigned to it. For example, if the tape
device has no link in /dev, it can be assigned an ap_id of
the form c0::st3 where st3 is a unique internally generated
identifier.
| When a controller is capable of supporting Solaris IO
| multipathing(also known as MPxIO and and Sun StorEdge
| Traffic Manager (STMS)) and is enabled, the controller
| provides a path to a scsi_vhci(7D)
| multipath device. If a device attached such controller is
| supported by scsi_vhci(7D) its ap_id can be
|
| c0::0,0
|
| where 0,0 uniquely identifies the target and logical unit
| information. The Type field for a path associated
| ap_ids indicates if it represent a path to the scsi_vhci(7D)
| multipath devices along with the type of device that is connected
| to through the path.
|
A simple listing of attachment points in the system will
include attachment points at SCSI controllers but not SCSI
devices. Use the -a flag to the list option (-l) to list
SCSI devices as well. For example:
# cfgadm -l
Ap_Id Type Receptacle Occupant Condition
c0 scsi-bus connected configured unknown
sysctrl0:slot0 cpu/mem connected configured ok
sysctrl0:slot1 sbus-upa connected configured ok
To list SCSI devices in addition to SCSI controllers:
# cfgadm -al
Ap_Id Type Receptacle Occupant Condition
c0 scsi-bus connected configured unknown
c0::dsk/c0t14d0 disk connected configured unknown
c0::dsk/c0t11d0 disk connected configured unknown
c0::dsk/c0t8d0 disk connected configured unknown
c0::dsk/c0t0d0 disk connected configured unknown
c0::rmt/0 tape connected configured unknown
sysctrl0:slot0 cpu/mem connected configured ok
sysctrl0:slot1 sbus-upa connected configured ok
| If the controller c0 was enabled with Solaris IO multipathing and
| the connected disk and tape devices is supported by Solaris
| IO multipathing the list would be:
|
| # cfgadm -al
|
| Ap_Id Type Receptacle Occupant Condition
| c0 scsi-bus connected configured unknown
| c0::11,0 disk-path connected configured
unknown
| c0::14,0 disk-path connected configured
unknown
| c0::8,0 disk-path connected configured unknown
| c0::0,0 disk-path connected configured unknown
| c0::a.0 tape-path connected configured unknown
| sysctrl0:slot0 cpu/mem connected configured ok
| sysctrl0:slot1 sbus-upa connected configured ok
|
|
Refer to cfgadm(1M) for more information regarding listing
attachment points. The receptacle and occupant state for
attachment points at the SCSI controller have the following
meanings:
empty not applicable
disconnected bus quiesced (I/O activity on bus is
suspended)
connected bus active
configured one or more devices on the bus is configured
unconfigured no device on the bus is configured
The corresponding states for individual SCSI devices are:
empty not applicable
disconnected bus to which the device is attached is
quiesced
connected bus to which device is attached is active
| configured device or path to a multipath SCSI device is
| configured
|
| unconfigured device or path to a multipath SCSI device is
| not configured
OPTIONS
cfgadm defines several types of operations besides listing
(-l).These operations include testing, (-t), invoking confi-
guration state changes, (-c), invoking hardware specific
functions (-x), and obtaining configuration administration
help messages (-h).
-c function The following generic commands are
defined for the SCSI hardware
specific library:
For SCSI controller attachment
points, the following configuration
state change operations are sup-
ported:
connect Unquiesce the SCSI
bus.
disconnect Quiesce the bus
(suspend I/O activity
on bus).
Incorrect use of this
command can cause the
system to hang. See
NOTES.
configure Configure new devices
on SCSI bus.
unconfigure Unconfigure all dev-
ices connected to
bus.
The following generic commands are
| defined for SCSI devices and path to a multipath
| SCSI device:
configure configure a specific
| device or a specific path to
| a multipath SCSI device.
| unconfigure unconfigure a specific
| device or a specific path to
| a multipath SCSI device.
-f When used with the disconnect com-
mand, forces a quiesce of the SCSI
bus, if supported by hardware.
Incorrect use of this command can
cause the system to hang. See NOTES.
-h ap_id SCSI specific help can be obtained
by using the help option with any
SCSI attachment point.
-o hardware_option No hardware specific options are
currently defined.
-s listing_option Attachment points of class scsi can
be listed by using the select sub-
option. Refer to the cfgadm(1M) man
page for additional information.
-t ap_id No test commands are available at
present.
-x hardware_function Some of the following commands can
only be used with SCSI controllers
and some only with SCSI devices.
In the following, controller_ap_id
refers to an ap_id for a SCSI con-
troller, for example, c0.
device_ap_id refers to an ap_id for
a SCSI device, for example:
c0::dsk/c0dt3d0.
The following hardware specific
functions are defined:
insert_device controller_ap_id
Add a new device to the SCSI
controller, controller_ap_id.
This command is intended for
interactive use only.
remove_device device_ap_id
Remove device device_ap_id.
This command is intended for
interactive use only.
replace_device device_ap_id
Remove device device_ap_id and
replace it with another device
of the same kind.
This command is intended for
interactive use only.
reset_device device_ap_id
Reset device_ap_id.
reset_bus controller_ap_id
Reset bus controller_ap_id
without resetting any devices
attached to the bus.
reset_all controller_ap_id
Reset bus controller_ap_id and
all devices on the bus.
locator [=on|off] device_ap_id
Sets or gets the hard disk loca-
tor LED, if it is provided by
the platform. If the [on|off]
suboption is not set, the state
of the hard disk locator is
printed.
led[=LED,mode=on|off|blink]
device_ap_id
If no sub-arguments are set,
this function print a list of
the current LED settings. If
sub-arguments are set, this
function sets the mode of a
specific LED for a slot.
EXAMPLES
Example 1 Configuring a Disk
The following command configures a disk attached to con-
troller c0:
# cfgadm -c configure c0::dsk/c0t3d0
Example 2 Unconfiguring a Disk
The following command unconfigures a disk attached to con-
troller c0:
# cfgadm -c unconfigure c0::dsk/c0t3d0
Example 3 Adding a New Device
The following command adds a new device to controller c0:
# cfgadm -x insert_device c0
The system responds with the following:
Adding device to SCSI HBA: /devices/sbus at 1f,0/SUNW,fas at e,8800000
This operation will suspend activity on SCSI bus c0
Continue (yes/no)?
Enter:
y
The system responds with the following:
SCSI bus quiesced successfully.
It is now safe to proceed with hotplug operation.
Enter y if operation is complete or n to abort (yes/no)?
Enter:
y
Example 4 Replacing a Device
The following command replaces a device attached to con-
troller c0:
# cfgadm -x replace_device c0::dsk/c0t3d0
The system responds with the following:
Replacing SCSI device: /devices/sbus at 1f,0/SUNW,fas at e,8800000/sd at 3,0
This operation will suspend activity on SCSI bus: c0
Continue (yes/no)?
Enter:
y
The system responds with the following:
SCSI bus quiesced successfully.
It is now safe to proceed with hotplug operation.
Enter y if operation is complete or n to abort (yes/no)?
Enter:
y
Example 5 Encountering a Mounted File System While Unconfi-
guring a Disk
The following command illustrates encountering a mounted
file system while unconfiguring a disk:
# cfgadm -c unconfigure c1::dsk/c1t0d0
The system responds with the following:
cfgadm: Component system is busy, try again: failed to offline:
/devices/pci at 1f,4000/scsi at 3,1/sd at 1,0
Resource Information
------------------ --------------------------
/dev/dsk/c1t0d0s0 mounted filesystem "/mnt"
Example 6 Display the Value of the Locator for a Disk
The following command displays the value of the locator for
a disk. This example is specific to the SPARC Enterprise
Server family:
# cfgadm -x locator c0::dsk/c0t6d0
The system responds with the following:
Disk Led
c0t6d0 locator=on
Example 7 Set the Value of the Locator for a Disk
The following command sets the value of the locator for a
disk. This example is specific to the SPARC Enterprise
Server family:
# cfgadm -x locator=off c0::dsk/c0t6d0
| Example 8 Configuring a path to a multipath SCSI disk
|
| The following command configures a path connected through con-
| troller c0:
|
| # cfgadm -c configure c0::2,0
|
|
| Example 9 Unconfiguring a path to a multipath SCSI disk
|
|
| The following command unconfigures a path connected through con-
| troller c0:
|
|
| # cfgadm -c unconfigure c0::2,0
|
FILES
/usr/lib/cfgadm/scsi.so.1
hardware-specific library for generic SCSI hot-plugging
/usr/platform/SPARC-Enterprise/lib/cfgadm/scsi.so.1
platform-specific library for generic SCSI hot-plugging
ATTRIBUTES
See attributes(5) for descriptions of the following attri-
butes:
____________________________________________________________
| ATTRIBUTE TYPE | ATTRIBUTE VALUE |
|_____________________________|_____________________________|
| Availability | SUNWcsl |
|_____________________________|_____________________________|
SEE ALSO
cfgadm(1M), luxadm(1M), config_admin(3CFGADM),
| libcfgadm(3LIB), scsi_vhci(7D), attributes(5)
NOTES
The disconnect (quiesce) operation is not supported on con-
trollers which control disks containing critical partitions
such as root (/), /usr, swap, or /var. The disconnect opera-
tion should not be attempted on such controllers. Incorrect
usage can result in a system hang and require a reboot.
When a controller is in the disconnected (quiesced) state,
there is a potential for deadlocks occurring in the system.
The disconnect operation should be used with caution. A con-
troller should be kept in the disconnected state for the
minimum period of time required to accomplish the DR opera-
tion. The disconnect command is provided only to allow the
replacement of the SCSI cables while the system is running.
It should not be used for any other purpose. The only fix
for a deadlock (if it occurs) is to reboot the system.
Hotplugging operations are not supported by all SCSI con-
trollers.
WARNINGS
The connectors on some SCSI devices do not conform to SCSI
hotplug specifications. Performing hotplug operations on
such devices can cause damage to the hardware on the SCSI
bus. Refer to your hardware manual for additional informa-
tion.
More information about the opensolaris-arc
mailing list