[brussels-dev] Re: [networking-discuss] Brussels high-level design document

[Sebastien Roy]

Garrett D'Amore wrote:
>> For the mtu case: the MTU definitiion targetted in the doc was intended
>> to address the driver's default_mtu (the tunable typically tweaked
>> by setting various params in the driver.conf to get Jumbo Frames). As 
>> you point out, there is a related concept of the GLDv3 max SDU which
>> can be lowered without involvement from the driver.  We could split the
>> mtu into two items, the "max_mtu" (which would be the classical 
>> ethernet driver's "default_mtu") and the "max_sdu" which would be
>> recognized and handled in the mac layer itself, without attempting to
>> invoke the mc_*prop functions.  Would this make sense?   
> 
> Lowering the MTU is done at the IP layer.

That's a bug IMO.  IP should just use the MTU reported by the layer 
underneath.  Any modification of MTU needs to be done from the bottom of 
the stack, not the top.

> There is little reason to 
> change the SDU size reported by the driver.  dladm shouldn't concern 
> itself with that.

I don't know exactly what you're trying to say.  Are you saying that the 
driver should report different values for its max SDU in mac_register_t 
and its "default MTU" property?  I don't see what the point of that would 
be.  They are effectively the same thing...

> The driver's default_mtu is really the value reported via the maximum 
> SDU in Nemo.   I'd just leave it "default_mtu" for now.

Right...

> Do we have a way for the upper layers of the stack (IP, TCP, VLAN, aggr) 
> to react to a change in the max_sdu?  I don't think so.

Yes.  DL_NOTE_SDU_SIZE.  It works great with IP tunnels, and I have it 
working for Nemo in the Clearview IP tunneling device driver.  It's not 
at all complex as it only required a few lines of code in Nemo to get 
working, and an additional MAC entry point to have the driver tell the 
MAC layer that the max SDU has changed.

I really don't see a reason to restrict changing the MTU to stopped MACs, 
as it's quite trivial to get working.

> Its likely that 
> changing this tunable will require replumbing various layers.   And, 
> quite frankly, that's probably okay, because when switching to a larger 
> MTU it requires other administrative changes as well.  (I.e. you have to 
> make sure all other hosts on the same subnet *also* can cope with large 
> frames.)

Synchronizing the increase of MTU of interfaces of various hosts on the 
network is not related.  I don't see why having the following steps on 8 
different hosts:

ifconfig bge0 unplumb
dladm set-linkprop default_mtu=9000 bge0
ifconfig bge0 plumb dhcp

Is any better than the following step on those same 8 hosts:

dladm set-linkprop default_mtu=9000 bge0

You cannot atomically plumb every interface on the network, so there's 
going to be some period of time when all of the interfaces are not in sync...

> Engineering a solution around this is likely to be "non-trivial".

It is quite trivial.  It took maybe 15 minutes to implement with a few 
dozen lines of code in Nemo.  IP can already handle max SDU changes 
dynamically, so no changes were needed at that layer.

> So it 
> may be simpler to just have a way for brussels to report back to the 
> user that the change won't take effect until the upper layers are 
> replumbed.

I disagree.

-Seb