[crossbow-discuss] Comments on updated arch document
Narayan Venkat
narayan.venkat at sun.com
Sat Sep 8 11:12:08 PDT 2007
Please see attached document for a list of comments/questions related
to the latest version of Crossbow arch. document. Specifically, it calls
out issues/questions with the mac_xxx interfaces specified in sec 5.x.
Thanks
Narayan
This message posted from opensolaris.org
-------------- next part --------------
1) MAC client open/close related:
(Crossbow-virt.pdf Section 5.2.2 Pg 36)
int mac_client_open(mac_handle_t *mh,
mac_client_handle_t *mchp, mac_bind_cpus
void mac_client_close(mac_client_handle_t mch);
typedef struct mac_bind_cpus_s {
uint_t mbc_ncpus;
uint32_t *mbc_cpus;
} mac_bind_cpus_t;
Q1.1) The mac_client_open() interface definition line in the document
is abruptly cut. It seems like there are additional arguments
such as flags etc.
Q1.2) On pg 38, there is a reference to the following flags, but which
interface takes them as an argument?
MAC_OPEN_FLAGS_FORCE_MULTI_RINGS
MAC_OPEN_FLAGS_FORCE_ONE_RING
It seems like these are an argument to mac_client_open(),
but there is a reference mac_open() in the description see below:
"If MAC_OPEN_FLAGS_FORCE_MULTI_RINGS flag is set and it is not
possible to allocate mbc_ncpus hardware rings, the mac_open()
call will fail, otherwise the MAC layer will attempt to reserve
one hardware ring for the MAC client."
Q1.3) Are there any other flags other than the following ones?
MAC_OPEN_FLAGS_FORCE_MULTI_RINGS
MAC_OPEN_FLAGS_FORCE_ONE_RING
- Is there a way to force a software ring?
Q1.4) Is the mbc_cpus in mac_bind_cpus_t an array of CPU ids?
Q1.5) The following description of mbc_cpus on pg 37 is not clear,
especially for the non-NULL case.
"If mbc_cpus is NULL, the MAC layer will pick the CPUs.
If mbc_cpus is non-NULL, the MAC layer will chose the CPUs.".
Q1.6) What is the relationship between Unicast addresses(multiple
unicast set via mac_unicast_add()), Rings and CPUs?
- Is there a 1:1 relation between a unicast address and a ring?
- Is there a 1:1 relation between a ring and CPU?
- The Rings and CPUs are tightly coupled in this interface.
How can allocate multiple rings even when there is one CPU(or less
number of CPUs).
- When there are multiple CPUs and multiple unicast addresses,
is there address fanout per CPU?
Q1.7) How is the binding of CPUs via mac_bind_cpus_t is co-ordinated
with CPU DR(on the platforms that support them)?
NOTE: CPU DR is already a supported feature on LDoms.
Q1.8) LDoms requires the CPU binding to be changed dynamically,
how can this be accomplished ?
Q1.9) The following XXX on pg 37. When are the interface changes for
priority and bandwidth specification available?
"XXX We still need to add the priority and bandwidth
limit as argument to mac_open(). We also need an entry
point to change the set of CPUs."
Q1.10) Can the mac client interface be extended to support creating
a client based on ether_type? This is required for mac clients
like fiberchannel over ethernet.
2) MAC Unicast address related:
(Crossbow-virt.pdf Section 5.2.4 Pg 38)
mac_unicast_handle_t mac_unicast_add(mac_client_handle_t mch,
mac_addr_type_t addr_type, int *addr_slot, uint_t prefix_len,
uchar_t *mac_addr, uint32_t flags);
void mac_unicast_unset(mac_unicast_handle_t);
void mac_unicast_get(mac_unicast_handle_t mah, uchar_t *mac_addr);
void mac_unicast_update(mac_unicast_handle_t mah,
mac_addr_type_t addr_type, int *addr_slot, uint_t prefix_len,
uchar_t *mac_addr);
QUESTIONS:
Q2.1) The section 4.5 describes "By value" type which is used
to set a specific MAC address by the MAC client. But there
is no equivalent addr_type definition under mac_unicast_add()
interface.
NOTE: LDoms requires the MAC addresses that are allocated
by LDom manager be used by the network device. So, LDoms
will not use any other addr_type other than "By value" type.
Q2.2) Is there an impact to the multiaddress_capab_t.maddr_add()/
maddr_remove() interfaces? Are these being obsoleted or
going away?
Q2.3) A system with many domains (aka LDoms) with virtual network
devices, it requires the use of a large number layer2 addresses,
this will exhaust h/w slots available on most standard NICs.
How can a client take advantage of layer2 filtering provided by
NICs like NII-NIU/Neptune. Specifically, this will help in
avoiding the programming of the device into PROMISCous mode
etc. Currently there are no interfaces that seem to provide
such ability.
Q2.4) Clients will need the ability to specify if mac_unicast_add()
is allowed it to go into promiscous mode or not. An error return
value is required if no h/w mac address slot is available.
Q2.5) On pg 40, the follow description still pointing to the
rings argument even though it has been removed from
mac_unicast_add() interface.
"The rings argument specifies the list of rings to
associate with the specified unicast MAC address.
If it is NULL, the MAC layer allocates a set of rings
according to those available to the MAC client, see
Section ringselection."
Q2.6) Can it be assumed that every address added to a client is
processed in a separate ring (either h/w ring or s/w ring)?
Q2.7) How are the multiple addresses per client maintained, is it done
in the MAC layer or does it bybpass the MAC layer and passed
to h/w directly.
Q2.8) Can unlimited number of mac addresses be assigned to a MAC
client? What are the software/hardware features that limit this?
3) Rings related:
(Crossbow-virt.pdf Section 5.3 Pg 43)
mac_rint_t *mac_rings_list_get(mac_client_handle_t mch,
uint_t nrings);
void mac_rings_list_free(mac_rings_t *rings, uint_t nrings);
uint16_t mac_ring_get_flags(mac_ring_t ring);
QUESTIONS:
Q3.1) All of these interfaces are now categorized as project-private
API. What motivated this change. These interfaces need to be
more open.
Q3.2) The mac_rings_list_get() is only for h/w rings, is there
an equivalent interface to obtain s/w ring information.
Or this interface can be extended return both h/w ring
or s/w ring information.
Q3.3) Are the mac_resource_set() and mac_resources() interfaces
going away?
Q3.4) What is the action taken when no free h/w ring available.
As per the documentation of mac_rings_list_get(), if no h/w
ring available, it returns NULL. In such case, how does
mac_unicast_add() behave when NULL is passed for rings?
Q3.5) Are there any interfaces other than the above mac_rings_xxx
interfaces that are available to deal with MAC rings?
Q3.6) Is the mac_rings_list_get() returns the list of mac rings
assigned to the client at the time of client open. How can
this be changed after the client is open.
Q3.7) Assigning h/w rings to a specific MAC address limits the
bandwidth to the number of rings that are assigned to that
address. Is there a way to not to bind h/w rings specific
to MAC address so that the bandwidth could be used by
any mac client depending on the traffic?
4) Receive callback related:
(Crossbow-virt.pdf Section 5.2.5 Pg 40)
int mac_rx_set(mac_client_handle_t mch, mac_rx_fn_t rx_fn,
void *arg);
int mac_rx_clear(mac_client_handle_t mch);
QUESTIONS:
Q4.1) How can a client get rx callback per ring that is assigned
to the mac client? This will allow parallel processing
and improve the performance. Such a feature is already
being used in the current implementation of LDoms vSwitch
driver and the mac_xxx interfaces should support such an
ability.
Q4.2) How can a client get a separate callback for a defined type of
traffic, such as different SAP numbers etc. This will
be useful to provide out of the band type packet processing
or related services.
Q4.3) There is a reference mac_addr_set(), should it be
mac_unicast_add()?
5) Transmit related:
(Crossbow-virt.pdf Section 5.2.7 Pg 41)
mblk_t *mac_tx(mac_client_handle_t *mch, mblk_t *mp, uint64_t hint);
QUESTIONS:
Q5.1) What are the valid values for the 'hint' argument?
From the description on pg 42, NULL seems to be
a valid value. Is it safe to assume that the 'hint' is a
ring-id, if so, a NULL value of 0 will conflict with a
ring-id of 0.
Q5.2) If NULL specified as a 'hint', how is the tx ring
selected?
Q5.3) The 'hint' argument description says the following.
What is the meaning of a connection in this context and
how to identify this?
"The hint must be the same for packets of the same connection."
6) Multicast addresses related:
(Crossbow-virt.pdf Section 5.2.6 Pg 41)
int mac_multicast_add(mac_client_handle_t mch, const uint8_t *addr);
int mac_multicast_remove(mac_client_handle_t mch, const uint8_t *addr);
No comments at this point.
7) Promiscous mode realted:
(Crossbow-virt.pdf Section 5.2.8 Pg 42)
Its not clear if the above interface will be available or not,
but two new intefaces are added:
int mac_promisc_add(mac_client_handle_t mch, mac_promisc_type
promisc_type, mac_promisc_fn_t promisc_fn, void *arg,
mac_promisc_handle_t *php);
int mac_promisc_remove(mac_client_handle_t mch,
mac_promisc_handle_t *ph);
MAC_PROMISC_ALL - send all packets
MAC_PROMISC_MULTI - only broadcast and multicast
May be the mac_promisc_add(MAC_PROMISC_ALL) will force device
to operate in the promiscous mode.
QUESTIONS:
Q7.1) According to the section 4.6, the promiscuous mode operates
in the layer2 switch model. When choosing the promiscuous mode
model can it be either layer2 switch model or shared
ethernet model?
Q7.2) From the explanation of mac_promisc_add(), it seems like
the mac_promisc_add() could be called without setting
MAC address via mac_unicast_add(). Is this correct?
If so, what is the expected behaviour?
8) Statistics related:
Q8.1) Is the mac_stat_get() interface being obsoleted or changed?
If so, what is the new equivalent interface?
GENERAL QUESTONS:
================
Qg.1) Are there any GLDv3 MAC client interfaces that are being
obsoleted(provided by the Nemo framework) but not documented
in this doc?
Qg.2) Are there any changes to the MAC driver interfaces or being
obsoleted?
Qg.3) There are no MAC client interfaces to specify bandwidth
attributes. From the section 4.7, it seems like they are
implemented as part of VNIC and not as MAC client interfaces.
If this is the case, how can the bandwidth attributes be
specified?
Qg.4) When will the classification interface be fully documented
for review?
Qg.5) In the future it will be great if the document can include
version info and change bars.
More information about the crossbow-discuss
mailing list