PSARC 2007/229 add afe driver to Solaris (update)

[Garrett D'Amore]

Darren Reed wrote:
> Garrett D'Amore wrote:
>> ...
>>   rx_max_loanup - maximum number of outstanding packets that will be 
>> loaned up to the upper stack without bcopy'ing (default = 128)
>>
>> Tuning these would be either for performance tweaking (which we don't 
>> expect to be necessary), or (in the case of rx_max_loanup) to prevent 
>> the upper layers from holding mblks.  (Setting rx_max_loanup == 0 may 
>> facilitate DDI detach, since the driver will refuse to detach while 
>> the upper layers are holding message blocks that have been loaned 
>> up... as the device's instance is required to properly tear down the 
>> DMA bindings for those buffers.  Note that the device is normally DR 
>> safe, although IPv6 in.ndpd seems to like to hold on to received 
>> mblks perhaps longer than it should.)
>
>
> Reading what you're saying about rx_max_loanup here, are there 
> implications
> for modunload'ing if rx_max_loanup is non-0?

If the upper layer stacks "hold" a loaned up mblk, then yes, you will be 
unable to modunload, or even DR detach, the device.

Generally I've not found this to impact IPv4 usage at all.  For IPv6 
I've noticed that sometimes I have to pkill -1 in.ndpd after unplumbing 
in order to permit the detach to complete.  (I suspect that if I wait 
long enough for in.ndpd to realize that the interface is no longer 
active, it might release the associated mblk.)

>
> Is there any way to interrogate the driver to find out how many 
> buffers have
> been loaned out and are outstanding?  Maybe a read-only rx_loaned?
> (Trying to think how you trouble shoot a problem when the driver doesn't
> unload and you need to dettermine "why not".)

Possibly I could add that.  Right now it just says "driver busy" or 
somesuch.  When you find no open streams to the device, its a pretty 
good sign that the NIC is not in use anymore.

There are some longer term solutions to this problem in general that I 
want to work on, but they require larger changes, and I'm not confident 
enough to include them in the initial release.  (It should be able to 
unbind the buffer from DMA and release the DMA handle without doing the 
ddi_dma_mem_free()...  and the acch associated with the DMA region 
should be able to outlive the devinfo originally used to allocate it.

Of course, that solves *detach*, but it doesn't do much for modunload.  
(You absolutely cannot modunload if you desballoc'd buffers outstanding!)

    -- Garrett