2009/235 dladm Possible Values List

[Garrett D'Amore]

Mike Shapiro wrote:
> On Tue, Apr 14, 2009 at 11:42:42AM -0700, Garrett D'Amore wrote:
>   
>> Mike Shapiro wrote:
>>     
>>
[snip!]

>>
>> What I think you're asking for, "optimal_jumbo", is really 
>> "minimum_dma_overhead", and is inherently very specific to the 
>> implementation -- its just one aspect of choosing the "optimum" MTU. 
>> Calling it "optimum" is a misnomer that I think will lead to great 
>> confusion. And, furthermore, I expect that it will lead to a greatly 
>> increased number of service calls.
>>     
>
> Naming aside, if you set the network at 9000 and then the driver
> at something smaller which is better, then you get the benefit.
>   

Huh? Either I'm not parsing this correctly, or you're missing a critical 
point.

If the rest of the network is using 9000 bytes, and your device driver 
uses a smaller value, then you'll have fragmentation or other network 
breakage. Fragmentation leads to poor performance. (Far worse than other 
situations.) Path MTU discovery helps here, but you can't *rely* on it. 
(Or at least you're not *supposed* to.)


>  
>   
>> Overall, this feels really like a "benchmark special" to me. If your 
>> driver has some particular behavior where 9K packets take a significant 
>> penalty that 8K do not, then this rightly should be treated in two ways:
>>
>> 1) as a bug -- the hardware/software should have been optimized to make 
>> 9K efficient as it is the most commonly deployed frame size
>> 2) in documentation as a note in the man page.
>>
>> Both of these treatments are appropriate -- this isn't an either/or 
>> situation, I think.
>>     
>
> As a software guy I agree.  However, (1) you don't get to control:
> that's under the control of hardware design people and subject to
> other factors.  And (2), which is the entire point of why I brought
> this up, is unlikely to be seen, and prone to error or staleness.
>   

But again, the "optimum" (or "preferred") value will depend on elements 
that go far beyond the driver. As I said, I believe if you can drive 
9000 bytes packets, then you will generally get better performance. 
(Ignoring other issues like backing store to disk blocks...)

I don't think its generally true that -- even for this hardware -- an 
8150 byte mtu will *always* give better performance than a 9000 byte 
mtu. For some traffic (NFS?) it might... but for other kinds of traffic 
(streaming a big file over a raw socket) I doubt it will. But then this 
value really is a benchmark special, not a general purpose value.

I think exposing this as a single value to customers is a mistake. I 
think knowledge of how to properly configure this value requires much 
more advanced knowledge than we can expose in a single value -- and so 
rightly belongs in documentation. Perhaps a man page. Perhaps a tuning 
guide. I'm not sure, except that I feel pretty strongly that *this* is 
*not* the right place.

And the fact that the tuning is required to get good performance, 
remains, IMO, a bug. We should try hard to get the hardware and driver 
engineers to recognize this problem. I mean, its not like 9K packets are 
suddenly a new thing or such -- that has been the de-facto standard for 
jumbo frames on ethernet pretty much from when they were first proposed.

- Garrett