[Cerowrt-devel] Baby jumbo frames support?

[Cerowrt-devel] Baby jumbo frames support?

[Alexander Hulse]

I've done some basic research and some (unsuccessful) hacking the on the driver source for the ag71xx driver and was wondering about baby jumbo frame support?

I know that the WNDR37/800's built in switch supports passing jumbo frames, but the underlying router interfaces do not support large MTU frames.

However, this:

http://wiki.mikrotik.com/wiki/Manual:Maximum_Transmission_Unit_on_RouterBoards

seems to imply that the built-in interfaces ought to support baby jumbo frames, unless I'm misunderstanding what they are saying.

As to the "Why?", in the UK, the fibre offering by BT using FTTC uses a PPPoE modem that can understand PPPoE packets with an MTU 1508, as described in RFC 4638.

It was possible using a Netgear WNR834T with the application of some patches from the pppd git:

http://git.ozlabs.org/?p=ppp.git;a=commit;h=fd1dcdf758418f040da3ed801ab001b5e46854e7
http://wiki.aa.net.uk/index.php/FTTC_Modem

Would this be a useful feature to add (if possible) to CeroWrt so that full sized frames can be used if the ISP / connection supports it?

Alex

Re: [Cerowrt-devel] Baby jumbo frames support?

[dpreed]

[-- Attachment #1: Type: text/plain, Size: 1831 bytes --]


Digging deeper, there is also a good reason for bigger frames when one is using 600 Mbit 802.11 signalling ( that's 75 bytes per microsecond).  A 1500 byte frame occupies only 50 microseconds of airtime. Not much return for the investment in contention time (5-10 microseconds) by the transmitting station.  9000 byte frames would be good, just as they are for GigE.
 
-----Original Message-----
From: "Alexander Hulse" <ah@amch.net>
Sent: Saturday, June 16, 2012 5:53am
To: cerowrt-devel@lists.bufferbloat.net
Subject: [Cerowrt-devel] Baby jumbo frames support?



I've done some basic research and some (unsuccessful) hacking the on the driver source for the ag71xx driver and was wondering about baby jumbo frame support?

I know that the WNDR37/800's built in switch supports passing jumbo frames, but the underlying router interfaces do not support large MTU frames.

However, this:

http://wiki.mikrotik.com/wiki/Manual:Maximum_Transmission_Unit_on_RouterBoards

seems to imply that the built-in interfaces ought to support baby jumbo frames, unless I'm misunderstanding what they are saying.

As to the "Why?", in the UK, the fibre offering by BT using FTTC uses a PPPoE modem that can understand PPPoE packets with an MTU 1508, as described in RFC 4638.

It was possible using a Netgear WNR834T with the application of some patches from the pppd git:

http://git.ozlabs.org/?p=ppp.git;a=commit;h=fd1dcdf758418f040da3ed801ab001b5e46854e7
http://wiki.aa.net.uk/index.php/FTTC_Modem

Would this be a useful feature to add (if possible) to CeroWrt so that full sized frames can be used if the ISP / connection supports it?

Alex
_______________________________________________
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

[-- Attachment #2: Type: text/html, Size: 2180 bytes --]

Re: [Cerowrt-devel] Baby jumbo frames support?

[Dave Taht]

I'd kind of hoped someone other than me would answer this because the
details are long and intricate.

Another way to put it is that after hearing simon barber, myself,
andrew mcgregor spend 5 hours trying to explain the issues of
wireless-n to kathie nichols and jg and multiple other devs one day, a
prominent networking developer in the room shook his head, and said
"I'm glad I work on 10GigE. It's so much simpler".

Bigger packet sizes are not necessary or useful in the current wifi
standards because of packet aggregation which bundles up to 42 packets
in one burst. There's this HUGE header, followed by X packets in an
aggregated "A-MPDU".

A-MPDU bursts = TSO/GSO on steroids, with similar (but worse)
attendant side effects (big checksum and ack on portions or the whole
bundle!), and dwarf classic (9k) jumbo packets in size - and don't
require reframing when put on the internet.

A-MSDU is a similar concept but proven thus far hard to implement.

On my bad days I think the ratio between ack (68) and max packet
(1500) MTU is too large, already.

As for other wireless standards, say something high bandwidth for tvs,
I could see short haul stuff taking advantage of larger frame sizes,
but as for wifi, gprs, 3g,4g, no.

Felix and I did a pair of podcasts on this stuff, one when we first
met back in the early days of bufferbloat, and the second in august
after we'd been working together for a while. The first half of the
second is largely historical; it got detailed about 40+ minutes in.

I think these are only useful now as historical documents but if enjoy
this sort of background noise, certainly they were stepping stones
towards mutual understanding for felix and I.

http://mirrors.bufferbloat.net/podcasts/BPR-The_Wireless_Stack.mp3
http://huchra.bufferbloat.net/~d/talks/felix/ (includes a transcript)

I ran out of budget for transcripts soon afterward. :(

Anyway...

As the wifi standards are backward compatible, the surrounding frame
is broadcast at a rate that made sense in 1998 in many cases, so the
time spent establishing a frame can far outweigh the actual data
transfered.

This is not a bad description of what happens in wireless-n, but dated.

http://thenetworkguy.typepad.com/nau/2007/12/caveats-of-larg.html

Kind of missed the infinite retry bugs endemic today... among others...

Lastly Andrew McGregor has a wonderful and apparently unpublishable
(far beyond the MPU) paper on how the minstrel rate selection
algorithm works, rate selection and retry being the big thing that
throws the fixed size/time/width calculation you use below overboard,
documenting 1/3 a page of related variables in wireless.

Perhaps he can send that over? I wish I'd recorded the 3 hours he
spent with me explaining in detail how minstrel worked, it was these
three indepth conversations and that paper that were the seed crystals
of converting my obsolete-as-of-2006 understanding of wifi to where I
stand now.

I almost forgot to mention that 802.11e has 4 queues already, each
with different algos for grabbing airtime - and an effectively
invisible set of queues for managing frames and crypto...



On Sat, Jun 16, 2012 at 8:54 AM,  <dpreed@reed.com> wrote:
> Digging deeper, there is also a good reason for bigger frames when one is
> using 600 Mbit 802.11 signalling ( that's 75 bytes per microsecond).  A 1500
> byte frame occupies only 50 microseconds of airtime. Not much return for the
> investment in contention time (5-10 microseconds) by the transmitting
> station.  9000 byte frames would be good, just as they are for GigE.
>
>
>
> -----Original Message-----
> From: "Alexander Hulse" <ah@amch.net>
> Sent: Saturday, June 16, 2012 5:53am
> To: cerowrt-devel@lists.bufferbloat.net
> Subject: [Cerowrt-devel] Baby jumbo frames support?
>
> I've done some basic research and some (unsuccessful) hacking the on the
> driver source for the ag71xx driver and was wondering about baby jumbo frame
> support?
>
> I know that the WNDR37/800's built in switch supports passing jumbo frames,
> but the underlying router interfaces do not support large MTU frames.
>
> However, this:
>
> http://wiki.mikrotik.com/wiki/Manual:Maximum_Transmission_Unit_on_RouterBoards
>
> seems to imply that the built-in interfaces ought to support baby jumbo
> frames, unless I'm misunderstanding what they are saying.
>
> As to the "Why?", in the UK, the fibre offering by BT using FTTC uses a
> PPPoE modem that can understand PPPoE packets with an MTU 1508, as described
> in RFC 4638.
>
> It was possible using a Netgear WNR834T with the application of some patches
> from the pppd git:
>
> http://git.ozlabs.org/?p=ppp.git;a=commit;h=fd1dcdf758418f040da3ed801ab001b5e46854e7
> http://wiki.aa.net.uk/index.php/FTTC_Modem
>
> Would this be a useful feature to add (if possible) to CeroWrt so that full
> sized frames can be used if the ISP / connection supports it?
>
> Alex
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>



-- 
Dave Täht
http://www.bufferbloat.net/projects/cerowrt/wiki - "3.3.8-6 is out
with fq_codel!"

Re: [Cerowrt-devel] Baby jumbo frames support?

[Dave Taht]

On Sat, Jun 16, 2012 at 5:53 AM, Alexander Hulse <ah@amch.net> wrote:
> I've done some basic research and some (unsuccessful) hacking the on the driver source for the ag71xx driver and was wondering about baby jumbo frame support?
>
> I know that the WNDR37/800's built in switch supports passing jumbo frames, but the underlying router interfaces do not support large MTU frames.
>
> However, this:
>
> http://wiki.mikrotik.com/wiki/Manual:Maximum_Transmission_Unit_on_RouterBoards
>
> seems to imply that the built-in interfaces ought to support baby jumbo frames, unless I'm misunderstanding what they are saying.
>
> As to the "Why?", in the UK, the fibre offering by BT using FTTC uses a PPPoE modem that can understand PPPoE packets with an MTU 1508, as described in RFC 4638.
>
> It was possible using a Netgear WNR834T with the application of some patches from the pppd git:
>
> http://git.ozlabs.org/?p=ppp.git;a=commit;h=fd1dcdf758418f040da3ed801ab001b5e46854e7
> http://wiki.aa.net.uk/index.php/FTTC_Modem
>
> Would this be a useful feature to add (if possible) to CeroWrt so that full sized frames can be used if the ISP / connection supports it?

I don't quite understand what you want here.

The internal ethernet connections on the netgear 3800 are both
connected to the switch. The switch is capable of jumbo packets, the
ar71xx driver appears to have vlan support only. I'm told it does not
have jumbo packet capability, but as to what it's upper limit is, is
unknown.

As for PPoE with a size 1508... um... one or the other device is going
to get in your way here. I presume that 1500 works? You would do
better to contact the author of the driver (juhosg) to get your
question answered as I'm under the impression he is under the right
NDAs.

I note that in cerowrt the switch has been configured for jumbo frames
even tho the ethernet chip can't do that - because it cuts latency
down by a lot to pre-allocate for jumbo frames as the on board
buffering on the switch is quite large - in fact, I regard it as too
large, even now...


>
> Alex
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 
Dave Täht
http://www.bufferbloat.net/projects/cerowrt/wiki - "3.3.8-6 is out
with fq_codel!"

Re: [Cerowrt-devel] Baby jumbo frames support?

[Robert Bradley]

On 21 June 2012 01:58, Dave Taht <dave.taht@gmail.com> wrote:
> As for PPoE with a size 1508... um... one or the other device is going
> to get in your way here. I presume that 1500 works? You would do
> better to contact the author of the driver (juhosg) to get your
> question answered as I'm under the impression he is under the right
> NDAs.
>

I think the point here is that MTU=1500 works, but once you add in the
PPPoE header, you end up with an effective MTU of 1492 for outbound
packets:

http://aa.net.uk/kb-broadband-mtu.html
http://tools.ietf.org/html/rfc4638

The short answer is that without baby-jumbo support, you either end up
fragmenting packets or you need to somehow restrict the MTU manually.
You can do that either through MSS clamping or simply configuring each
internal machine to use MTU=1492.  To get around this, the BT ADSL
modems started to support MTU=1508.  This means that the MTU within
the PPPoE tunnel remains at Ethernet-standard 1500, and avoids the
fragmentation or reconfiguration issues.

As for supporting it in CeroWRT ... the ag71xx driver defines
AG71XX_TX_MTU_LEN=1540, so it looks safe enough to use MTU 1508,
especially if you know that no vlans or other additions to the
standard header will be used.  To enable that, you need to reimplement
the eth_change_mtu function for the driver.  The current code uses the
kernel's implementation, which restricts the MTU to 1500.  An initial,
naive patch would look something like:

----
--- C:/Users/robert/AppData/Local/Temp/ag71x-revBASE.svn000.tmp.c	Mon
May 28 03:55:59 2012
+++ C:/Users/robert/Desktop/ag71xx/ag71xx_main.c	Thu Jun 21 13:58:44 2012
@@ -1042,13 +1042,25 @@
 }
 #endif

+/*
+ * Copied from eth_change_mtu and modified so that baby jumbo packets
+ * may be used.  This has not been tested!
+ */
+int ag71xx_change_mtu(struct net_device *dev, int new_mtu)
+{
+        if (new_mtu < 68 || new_mtu > (ETH_DATA_LEN + 8))
+                return -EINVAL;
+        dev->mtu = new_mtu;
+        return 0;
+}
+
 static const struct net_device_ops ag71xx_netdev_ops = {
 	.ndo_open		= ag71xx_open,
 	.ndo_stop		= ag71xx_stop,
 	.ndo_start_xmit		= ag71xx_hard_start_xmit,
 	.ndo_do_ioctl		= ag71xx_do_ioctl,
 	.ndo_tx_timeout		= ag71xx_tx_timeout,
-	.ndo_change_mtu		= eth_change_mtu,
+	.ndo_change_mtu		= ag71xx_change_mtu,
 	.ndo_set_mac_address	= eth_mac_addr,
 	.ndo_validate_addr	= eth_validate_addr,
 #ifdef CONFIG_NET_POLL_CONTROLLER

----

where I've copied the original function and changed the upper limit to
ETH_DATA_LEN+8, then set up the netdev_ops structure to call the new
version.  In reality, you probably want to add some better checks
(testing for MTU+all possible headers<1540?) and remove the magic
constant - in the worst case, something closer to the e1000 driver's
implementation.  I wouldn't recommend using the present version on
anything other than an experimental build, but the default MTU would
be 1500 anyway so should avoid causing too much damage.  Those on BT
ADSL lines can change the MTU on ge00 themselves and see what breaks.
-- 
Robert Bradley

Re: [Cerowrt-devel] Baby jumbo frames support?

[dpreed]

[-- Attachment #1: Type: text/plain, Size: 5885 bytes --]


I understand Dave Taht's long lecture - actually understood it years ago.  But frame aggregation is not the same thing as jumbo frames in a multi-technology Ethernet LAN.   Jumbo frames provide a way to exploit *end-to-end* frame sizes greater than 1500 bytes.  That means the source and destination TCPs get frames that are "whole" (and not random subassemblies of frames that may arrive close together in time).
 
9000 byte frames were invented for 1 GigE transports.   Today's 802.11n and futures approach 1 GigE, and 1 GigE is the standard wiring for most homes, etc.   It does not matter how the underlying radio links chop up the Ethernet frame, retransmit them, ack them etc.   The value I am disucssing is at the *endpoints*.
 
It's tempting for transport link providers to *ignore* TCP and so forth when they design their transports, and focus only on transport-level efficiencies and reliabilities.  This temptation created bufferbloat and also the excessive retry problem.   (and in the past it created the historical predecessor of "bufferbloat" - Frame Relay's "Reliable delivery mode" which would go to extraordinary lengths to never drop a packet, including storing the packets *on disk* in some cases - talk about bloated buffers!)
 
The conversation here (including, but not limited to Taht's comments) shows exactly that *temptation*.
 
Aggregation is NOT the same as large frames.  Not at all.  It achieves internal efficiencies, but not the endpoint efficiencies of receiving a coherent frame, that can be processed immediately and by a single code path.  At 1 Gigabit/sec this was important enough to introduce such frame sizes.
 
The alternative ways to achieve the endpoint goals would be to allow reordering of data delivery to the endpoint app, perhaps by making SCTP work instead of TCP, using a flow/congestion/rate control mechanism other than a window on sequence numbers, etc.  But that would mean changing the entire stack to a new end-to-end theory of operation.
 
There is a real tradeoff space, but unilaterally declaring that packet aggregation is the same as jumbo Ethernet frames is choosing a poor point in the tradeoff space.
 
Regarding "header overhead" - that is minor in the scheme of things.  Obsessing about that indicates a lack of perspective on the systems level issues.
 
 
 
 
-----Original Message-----
From: "Robert Bradley" <robert.bradley1@gmail.com>
Sent: Thursday, June 21, 2012 9:33am
To: cerowrt-devel@lists.bufferbloat.net
Subject: Re: [Cerowrt-devel] Baby jumbo frames support?



On 21 June 2012 01:58, Dave Taht <dave.taht@gmail.com> wrote:
> As for PPoE with a size 1508... um... one or the other device is going
> to get in your way here. I presume that 1500 works? You would do
> better to contact the author of the driver (juhosg) to get your
> question answered as I'm under the impression he is under the right
> NDAs.
>

I think the point here is that MTU=1500 works, but once you add in the
PPPoE header, you end up with an effective MTU of 1492 for outbound
packets:

http://aa.net.uk/kb-broadband-mtu.html
http://tools.ietf.org/html/rfc4638

The short answer is that without baby-jumbo support, you either end up
fragmenting packets or you need to somehow restrict the MTU manually.
You can do that either through MSS clamping or simply configuring each
internal machine to use MTU=1492.  To get around this, the BT ADSL
modems started to support MTU=1508.  This means that the MTU within
the PPPoE tunnel remains at Ethernet-standard 1500, and avoids the
fragmentation or reconfiguration issues.

As for supporting it in CeroWRT ... the ag71xx driver defines
AG71XX_TX_MTU_LEN=1540, so it looks safe enough to use MTU 1508,
especially if you know that no vlans or other additions to the
standard header will be used.  To enable that, you need to reimplement
the eth_change_mtu function for the driver.  The current code uses the
kernel's implementation, which restricts the MTU to 1500.  An initial,
naive patch would look something like:

----
--- C:/Users/robert/AppData/Local/Temp/ag71x-revBASE.svn000.tmp.c	Mon
May 28 03:55:59 2012
+++ C:/Users/robert/Desktop/ag71xx/ag71xx_main.c	Thu Jun 21 13:58:44 2012
@@ -1042,13 +1042,25 @@
 }
 #endif

+/*
+ * Copied from eth_change_mtu and modified so that baby jumbo packets
+ * may be used.  This has not been tested!
+ */
+int ag71xx_change_mtu(struct net_device *dev, int new_mtu)
+{
+        if (new_mtu < 68 || new_mtu > (ETH_DATA_LEN + 8))
+                return -EINVAL;
+        dev->mtu = new_mtu;
+        return 0;
+}
+
 static const struct net_device_ops ag71xx_netdev_ops = {
 .ndo_open		= ag71xx_open,
 .ndo_stop		= ag71xx_stop,
 .ndo_start_xmit		= ag71xx_hard_start_xmit,
 .ndo_do_ioctl		= ag71xx_do_ioctl,
 .ndo_tx_timeout		= ag71xx_tx_timeout,
-	.ndo_change_mtu		= eth_change_mtu,
+	.ndo_change_mtu		= ag71xx_change_mtu,
 .ndo_set_mac_address	= eth_mac_addr,
 .ndo_validate_addr	= eth_validate_addr,
 #ifdef CONFIG_NET_POLL_CONTROLLER

----

where I've copied the original function and changed the upper limit to
ETH_DATA_LEN+8, then set up the netdev_ops structure to call the new
version.  In reality, you probably want to add some better checks
(testing for MTU+all possible headers<1540?) and remove the magic
constant - in the worst case, something closer to the e1000 driver's
implementation.  I wouldn't recommend using the present version on
anything other than an experimental build, but the default MTU would
be 1500 anyway so should avoid causing too much damage.  Those on BT
ADSL lines can change the MTU on ge00 themselves and see what breaks.
-- 
Robert Bradley
_______________________________________________
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

[-- Attachment #2: Type: text/html, Size: 7219 bytes --]

Re: [Cerowrt-devel] Baby jumbo frames support?

[Robert Bradley]

On 21/06/12 15:25, dpreed@reed.com wrote:
> I understand Dave Taht's long lecture - actually understood it years ago.  But frame aggregation is not the same thing as jumbo frames in a multi-technology Ethernet LAN.   Jumbo frames provide a way to exploit *end-to-end* frame sizes greater than 1500 bytes.  That means the source and destination TCPs get frames that are "whole" (and not random subassemblies of frames that may arrive close together in time).
>   

Your post makes more sense as a reply to Dave's post rather than mine, I 
think...

Just in case replying to mine was deliberate, though:

Going back to Alexander's original post, "baby" jumbo packets are quite 
simply a hack to get around the issues of PPPoE and reduced MTU.  For UK 
ADSL customers without loop-unbundling, most of the infrastructure 
between customer and ISP is owned/managed by one provider (BT 
Wholesale), and either PPPoA or PPPoE may be used on this link 
(http://blog.farnz.org.uk/2010/02/on-pppoa-pppoe-atm-and-adsl.html).  On 
the newer network, PPPoE is apparently(?) preferred.  (I am sure that 
there are others reading this that know far more about BT's ADSL setup 
than I do; if any of this is wrong, feel free to correct me here!)

Now, one of the issues of tunnelling is that you have overheads; in this 
case, the PPPoE header is 8 octets, with 1492 octets remaining for the 
tunnelled packet.  In theory, you could simply set the tunnel MTU to 
1492 bytes and have done with it - after all, you can always use path 
MTU discovery!  The problem is that PMTU black holes can and do exist, 
so that is not a viable option.  Ideally, you want your link to be 
1500-byte-safe instead.  One way to achieve this is to increase the 
frame size by just enough to contain a full 1500-octet packet within a 
PPPoE frame (8 octets larger in this case).  Your standard 1500-byte 
packet now becomes a 1508-byte PPPoE packet (8 bytes PPPoE header, and 
1500 bytes of content), which is sent from your router over the BT 
Wholesale network to your ISP's PPPoE server.  The server then extracts 
the embedded Ethernet frame and sticks it on to the ISP's network, with 
no reassembly required.

-- 
Robert Bradley