2008/242 Data Fast-Path for Softmac
Darren Reed
Darren.Reed at Sun.COM
Tue Apr 8 15:04:28 PDT 2008
Garrett D'Amore wrote:
> As a general architectural comment here, I want to point out that I
> feel that this case introduces additional complexity, that should be
> unnecessary -- performance critical NICs should be written to GLDv3.
> (The fact that GLDv3's API is still private is another matter
> altogether -- but I believe introducing this code to address a
> "performance issue" with a certain NIC product, is the wrong way to go
> here.)
>
> Far far better, IMO, is to "fix" the main driver this case is for
> (cassini), which architecturally could easily be converted to GLDv3
> and bundled into Nevada, and then spend effort to wrap up GLDv3 enough
> to publish it for 3rd parties.
And this would be the second or third "hack" (with PSARC case) to
support cassini with special code because it isn't a GLDv3 driver?
You've got to wonder if/when the rot will stop at the management
(or business) level that fails to understand what the correct direction
is here.
Darren
>
>
> -- Garrett
>
> Sebastien Roy wrote:
>
>> I'm submitting this fast-track for Cathy Zhou. The timer is set for
>> April 14th, 2008.
>>
>> Data Fast-Path for Softmac
>> ==========================
>>
>> Release binding: Patch
>>
>> Summary
>> =======
>>
>> This case introduces a GLDv3 fast-path architecture for softmac
>> devices.
>> The softmac architecture defined by PSARC/2006/499 initially
>> contained
>> a fast-path which was not delivered with Clearview UV as
>> discussed in
>> section 2.1 of PSARC/2008/002. PSARC/2008/002 states that an
>> alternative
>> fast-path architecture would be defined in a future case, and
>> that is
>> this case.
>>
>> Details
>> =======
>>
>> - Data fast-path and slow-path
>>
>> Currently, the softmac module discovers each legacy network device
>> and registers (soft)MACs to the GLDv3 framework on behalf of these
>> legacy devices. Whenever a softMAC is used, the softmac module
>> opens
>> a DLIOCRAW stream over the legacy device, and enables
>> DL_PROMISC_SAP
>> promiscuous mode on the stream. This DLIOCRAW stream is then
>> shared
>> by all data packets sent and received over this softMAC. We
>> call this
>> DLIOCRAW stream "shared lower stream".
>>
>> Although the above is consistent with the existing GLDv3 model and
>> requires less changes to the GLDv3 framework, it introduces
>> unacceptable performance regression over legacy devices,
>> because of
>> the extra data demultiplexing and filtering processing in the
>> GLDv3
>> DLS/MAC layer.
>>
>> Therefore, this case proposes a softmac data fast-path
>> architecture
>> which is less intrusive to the current Solaris network
>> architecture
>> and will address the performance issue of the above approach:
>>
>> Specifically, when the fast-path is used, a dedicated lower-stream
>> will be opened over the legacy device for each IP/ARP
>> (upper-)stream
>> over the softMAC, and all DLPI messages (including control
>> messages
>> and data messages) will be exchanged between the upper-stream and
>> the corresponding lower-stream directly. Therefore, the data
>> demultiplexing, filtering and classification processing will be
>> done
>> by the lower-stream, and the GLDv3 DLS/MAC layer processing
>> will be
>> no longer needed.
>>
>> Note that some GLDv3 features requires the GLDv3 DLS/MAC layer
>> processing to not be bypassed to assure its function correctness.
>> For example, softmac fast-path must be disabled to support GLDv3
>> VNIC functionality. As the result, we propose: By default, the
>> softmac fast-path mode will be used to assure the performance; MAC
>> clients will be able to request to disable the softmac fast-path
>> mode to support certain features, and if that succeeds, the system
>> will fallback to the existing softmac data-path model (called
>> slow-path).
>>
>> The details of the new softmac data fast-path model is stated
>> as below
>> (note that the softmac fast-path will only be used for the IP/ARP
>> streams over softMACs, which is the case performance matters
>> most):
>>
>> 1. When a stream (including a VLAN stream) is opened on a softMAC,
>> the softmac module will takes over the DLPI processing on this
>> stream;
>>
>> 2. For IP/ARP streams over a softMAC, softmac data fast-path
>> will be
>> used by default, unless fast-path is disabled by any MAC client
>> explicitly. The softmac module first identifies an IP/ARP
>> stream
>> by seeing whether there is a SIOCSLIFNAME ioctl sent from
>> upstream,
>> if there is one, this stream is either an IP or an ARP stream
>> and will use fast-path potentially;
>>
>> 3. When the softmac fast-path is used, an dedicated
>> lower-stream will
>> be setup for each IP/ARP stream (1-1 mapping). From that
>> point on,
>> all control and data messages will be exchanged between the
>> IP/ARP
>> upper-stream and the legacy device through this dedicated
>> lower-stream. As a result, the DLS/MAC layer processing in
>> GLDv3
>> will be skipped, and this greatly improves the performance;
>>
>> 4. When the softmac data fast-path is disabled by a MAC client
>> (e.g.,
>> by a VNIC), all the IP/ARP upper streams will try to switch
>> from
>> the fast-path to the slow-path. The dedicated lower-stream
>> will be
>> destroyed, and all the control and data-messages will go
>> through
>> the existing GLDv3 code path and (in the end) the shared
>> lower-stream;
>>
>> 5. On the other hand, when the last MAC client cancels its
>> fast-path
>> disable request, all the IP/ARP streams will try to switch
>> back to
>> the fast-path mode;
>>
>> Step 5 and 6 both rely on the data-path mode switching process
>> described below:
>>
>> 1) To switch the softmac data-path mode (between fast-path and
>> slow-path), softmac will first send a DL_NOTE_REPLUMB
>> DL_NOTIFY_IND message upstream over each IP/ARP streams that
>> needs data-path mode switching;
>>
>> 2) When IP receives this DL_NOTE_REPLUMB message, it will bring
>> down all the IP interfaces on the corresponding ill (IP
>> Lower
>> level structure), and bring up those interfaces over again;
>> this will in turn cause the ARP to "replumb" the interface.
>>
>> During the replumb process, both IP and ARP will send
>> downstream the necessary DL_DISABMULTI_REQ and DL_UNBIND_REQ
>> messages and cleanup the old state of the underlying
>> softMAC,
>> following with the necessary DL_BIND_REQ and
>> DL_ENABMULTI_REQ
>> messages to setup the new state. Between the cleanup and
>> re-setup
>> process, IP/ARP will also send down a DL_NOTE_REPLUMB_DONE
>> DL_NOTIFY_CONF messages to the softMAC to indicate the
>> *switching point*;
>>
>> 3) When softmac receives the DL_NOTE_REPLUMB_DONE message, it
>> either creates or destroys the dedicated lower-stream
>> (depending
>> on which data-path mode the softMAC switches to), and change
>> the softmac data-path mode. From then on, softmac will
>> process
>> all the succeeding control messages (including the
>> DL_BIND_REQ and
>> DL_ENABMULTI_REQ messages) and data messages based on new
>> data-path mode.
>>
>> Interfaces
>> ==========
>>
>> - mac_fastpath_disable()/mac_fastpath_enable() (Consolidation
>> Private)
>>
>> Two MAC client interfaces will be added:
>>
>> int mac_fastpath_disable(mac_handle_t);
>>
>> This function will be called when a mac client requires to disable
>> the softmac data fast-path over a specific mac. It will return
>> errno
>> in case that the fast-path is failed to be disabled. If the
>> given mac
>> is a mac over a GLDv3 device, 0 (success) will be directly
>> returned.
>>
>> void mac_fastpath_enable(mac_handle_t);
>>
>> The mac_fastpath_enable() function will be called when a mac
>> client
>> cancels its softmac fast-path disabling requests over a
>> specific mac.
>> When the last fast-path disabling request is canceled, the
>> underlying
>> softmac will try to reenable the data fast-path. It is not
>> considered
>> as a failure if fast-path fails to be reenabled, as that will only
>> have performance impact.
>>
>> - dld_str_open()/dld_str_close()/dld_str_private() (Project Private)
>>
>> Three new dls interfaces will be added:
>>
>> int dld_str_open(queue_t *, dev_t *, void *opaque);
>>
>> This function will be called by softmac to setup the dld stream
>> related structures, which will be used when softmac fast-path is
>> not used. An opaque pointer will be passed into this function
>> to keep softmac specific state associated with the given stream.
>>
>> int dld_str_close(queue_t *);
>>
>> This function will be called by softmac to finish up the dld
>> stream
>> operation and destroy dld related structures.
>>
>> void *dld_str_private(queue_t *);
>>
>> This function will be called to query the opaque pointer
>> associated
>> with the given stream.
>>
>> - the mac_capab_legacy_t structure (Project Private)
>>
>> The mac_capab_legacy_t structure was introduced by PSARC/2008/002,
>> it is the data pointer of the MAC_CAPAB_LEGACY MAC capability.
>>
>> This case proposes to add the following fields to the data
>> structure:
>>
>> /*
>> * Info and callbacks of legacy devices.
>> */
>> typedef struct mac_capab_legacy_s {
>> ...
>> int (*ml_fastpath_disable)(void *);
>> void (*ml_fastpath_enable)(void *);
>> boolean_t (*ml_active_set)(void *);
>> void (*ml_active_clear)(void *);
>> } mac_capab_legacy_t;
>>
>> The ml_fastpath_enable() and ml_fastpath_disable() callbacks
>> will be
>> called as the result of first mac_fastpath_disable() and last
>> mac_fastpath_enable() request on a particular softMAC.
>>
>> Further, to assure there is at most one exclusive active MAC
>> consumer
>> at a time (see [3]), we must prevent other exclusive active MAC
>> consumers when there is already IP/ARP stream on a softMAC, and
>> prevent
>> IP/ARP streams if there is already exclusive active MAC consumers.
>> This is done by the mac module calling the ml_active_set() and
>> ml_active_clear() callbacks to inform softmac the existence and
>> disappearance of an exclusive active MAC consumer.
>>
>> - the DL_NOTIFY_CONF primitive (M_PROTO type) (Project Private)
>>
>> The DL_NOTIFY_CONF primitive is similar to DL_NOTIFY_IND, but
>> it is
>> notification sent from upstream (from a DLPI consumer to a DLPI
>> provider).
>>
>> DL_NOTIFY_CONF is a M_PROTO message with the following payload:
>>
>> typedef struct {
>> t_uscalar_t dl_primitive; /* set to DL_NOTIFY_CONF */
>> uint32_t dl_notification; /* Which notification? */
>> } dl_notify_conf_t;
>>
>> The only dl_notification type we will support is
>> DL_NOTE_REPLUMB_DONE.
>>
>> - the DL_NOTE_REPLUMB DL_NOTIFY_IND message (Project Private)
>>
>> As described above, the DL_NOTE_REPLUMB message is sent by softmac
>> to inform the IP/ARP upstream to "replumb" the stream.
>>
>> References
>> ==========
>>
>> [1] PSARC 2006/499 Clearview Nemo unification and vanity naming -
>> http://sac.sfbay/PSARC/2006/499/
>> [2] PSARC 2008/002 Clearview UV Updates -
>> http://sac.sfbay/PSARC/2008/002/
>> [3] PSARC 2005/207 Clearview for Nemo -
>> http://sac.sfbay/PSARC/2005/207/
>
>
More information about the opensolaris-arc
mailing list