[Bloat] Remarkably simple bloat managing use by a UK ISP

[Bloat] Remarkably simple bloat managing use by a UK ISP

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 460 bytes --]

Hi Chaps,

I was speaking with Andrews & Arnold (a UK clueful ISP) about their 
experiences with bufferbloat and was sent this graph: 
https://support.aa.net.uk/VMG1312:_QoS

I was surprised at how such a simple 'priority by packet size' scheme 
appears to work.  Clearly there's slightly more going on in that the box 
in question *knows* the outbound bandwidth and doesn't let a standing 
queue form...but priority by packet size?  Hmm.

Kevin


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4791 bytes --]

Re: [Bloat] Remarkably simple bloat managing use by a UK ISP

[Dave Taht]

Yes, making a distinction between large and small packet sizes helps.

There is a clear bifurcation in network traffic at around 300 bytes,
with all kinds of packet sizes below 300, with very few packets sized
in the 300-1100 byte range. webrtc and hangout traffic tends to about
1000 bytes for some reason.

As shipped in openwrt and the sqm-scripts, 300 is the default fq_codel quantum.

Along the way we explored per packet fairness not just with SFQ but
fiddling with the drr (and later, fq_codel) quantum, settling
initially on 500 bytes and after much testing, we cut to 300 bytes.
There was a variant (pfq_codel I think it was in the codebase) that
attempted per packet against all small flows fairness (fiddling with
the new/old queue concept) - but it did badly. I think nfq_codel still
exists in cerowrt as it did quite well also (I can no longer remember
what it did, but it also served more different flows within a
quantum).

It is kind of unclear, scheduling for higher bandwidths (100s of
mbits), what the drr quantum should be. Cake is experimenting with
gradually increasing the quantum as bandwidths grow. I lean towards
sticking with 300 bytes until you crack a gigabit. sch_fq (targetted
at 10gigE+) defaults to 2 packets (3018) which has the nice property
of releasing a delayed-ack on the other side. (I had talked to john
nagle a long time ago and one of the things he sighed about was that
he was deeply unhappy (starting in 198x!)  about the delayed ack
concept, but he never explained why, and I have assumed it was the
loose interaction with fq... and lets not go into stretch acks!)

It had been my hope that the new/old flow distinction in fq_codel at
the default packet quantum (1514) would provide a useful signal to a
tcp type about the actual path RTT, but I think (now, 3 years later),
that gets smoothed away.

Still, the new paced stuff (which can measures the baseline RTT from
the syn/synack pair) interacts nicely with the new/old distinction in
fq_codel at 300 bytes.

To mess up this observable portion of the universe and "easy fix",
there are tcp fast open and quic.


On Fri, Jun 5, 2015 at 7:46 AM, Kevin Darbyshire-Bryant
<kevin@darbyshire-bryant.me.uk> wrote:
> Hi Chaps,
>
> I was speaking with Andrews & Arnold (a UK clueful ISP) about their
> experiences with bufferbloat and was sent this graph:
> https://support.aa.net.uk/VMG1312:_QoS
>
> I was surprised at how such a simple 'priority by packet size' scheme
> appears to work.  Clearly there's slightly more going on in that the box in
> question *knows* the outbound bandwidth and doesn't let a standing queue
> form...but priority by packet size?  Hmm.
>
> Kevin
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>



-- 
Dave Täht
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Bloat] Remarkably simple bloat managing use by a UK ISP

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 2074 bytes --]

Going back to fundamentals, there's a clear distinction between traffic
which is latency sensitive and traffic which is bandwidth sensitive.
Perhaps surprisingly, web traffic tends to fall into the former category,
unless the link bandwidth is very low by current standards (analogue modem
territory).

It sounds like that router's default settings attempt to make that
distinction based on port number, and then perform strict prioritization.
The example of SSH demonstrates why that doesn't work; interactive shell
over SSH is latency sensitive, but SCP and rsync-over-SSH are bandwidth
sensitive, and they all use the same port. The need for explicit
configuration (a database of port numbers) is also a black mark against it,
especially since they managed to leave out such a common protocol as DNS.

Packet size is a better heuristic than port number. Most latency sensitive
protocols do tend to use small packets, and nearly all bandwidth sensitive
traffic uses the largest packets it can. SSH also naturally switches
between small and large packets depending on the type of traffic it's
carrying. If you simply sort your traffic by packet size, you don't even
need to configure it - but otherwise you just have a threshold to set and
forget. Cunningly, this also naturally performs ack and ping promotion.

The downside of packet size as a heuristic is that it's possible to game
it, especially with strict priority in force. All you have to do is send
packets below the threshold if there is one, or slightly smaller than the
competing traffic if there isn't one. The receiver can influence this by
setting the MSS low during TCP handshakes.

That is why fq_codel uses the sparse/saturating flow distinction for
priority. It's a much more robust heuristic than packet size, requires no
configuration at all, and is not so easy to game. The downside? It's only
available if you're already doing flow isolation, which basically solves
the core problem on its own. Still, making that distinction does help with
new flow startup and jitter reduction.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 2224 bytes --]

Re: [Bloat] Remarkably simple bloat managing use by a UK ISP

[Dave Taht]

On Fri, Jun 5, 2015 at 9:35 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
> Going back to fundamentals, there's a clear distinction between traffic
> which is latency sensitive and traffic which is bandwidth sensitive. Perhaps
> surprisingly, web traffic tends to fall into the former category, unless the
> link bandwidth is very low by current standards (analogue modem territory).
>
> It sounds like that router's default settings attempt to make that
> distinction based on port number, and then perform strict prioritization.
> The example of SSH demonstrates why that doesn't work; interactive shell
> over SSH is latency sensitive, but SCP and rsync-over-SSH are bandwidth
> sensitive, and they all use the same port. The need for explicit
> configuration (a database of port numbers) is also a black mark against it,
> especially since they managed to leave out such a common protocol as DNS.
>
> Packet size is a better heuristic than port number. Most latency sensitive
> protocols do tend to use small packets, and nearly all bandwidth sensitive
> traffic uses the largest packets it can. SSH also naturally switches between
> small and large packets depending on the type of traffic it's carrying. If
> you simply sort your traffic by packet size, you don't even need to
> configure it - but otherwise you just have a threshold to set and forget.
> Cunningly, this also naturally performs ack and ping promotion.

I do not regard ping promotion as a "feature". I think ping should be
a measurement of worser-case latency. Openwrt does both synflood and
ping flood protection in their firewall implementation badly with a
fixed limit too low for synflood (it used to be 25/sec), and too high
for ping flood.(1000/sec and extensive filtering)

> The downside of packet size as a heuristic is that it's possible to game it,
> especially with strict priority in force. All you have to do is send packets
> below the threshold if there is one, or slightly smaller than the competing
> traffic if there isn't one. The receiver can influence this by setting the
> MSS low during TCP handshakes.

There is a natural game theory influence pushing packet size larger
for bulkier traffic, and that is header overhead gets pretty
significant as you get below 300 bytes.

>
> That is why fq_codel uses the sparse/saturating flow distinction for
> priority. It's a much more robust heuristic than packet size, requires no
> configuration at all, and is not so easy to game. The downside? It's only
> available if you're already doing flow isolation, which basically solves the
> core problem on its own. Still, making that distinction does help with new
> flow startup and jitter reduction.

Yep. A godawful amount of thought and data went into all this...

... and I stress in
http://www.bufferbloat.net/projects/cerowrt/wiki/Wondershaper_Must_Die

that in 10 years someone might be cursing us for using any of these
techniques so universally because it interacts badly with the latest
brain to network direct connect with high speed voice response

https://www.youtube.com/watch?v=M1ONXea0mXg

(or something like that)

> - Jonathan Morton
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>



-- 
Dave Täht
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Bloat] Remarkably simple bloat managing use by a UK ISP

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 553 bytes --]

> I do not regard ping promotion as a "feature".

Neither do I, but in this case at least ping isn't getting an unfair
advantage over other latency sensitive traffic. Measurements based on ping
therefore correctly show a big improvement over doing nothing.

Fq_codel also naturally promotes pings and acks in the same basic way, and
I'm similarly happy with that, because it's a natural result of a genuine
improvement. The way it works also means that it's not a strict priority,
so it goes away as soon as someone tries to game it.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 658 bytes --]

Re: [Bloat] Remarkably simple bloat managing use by a UK ISP

[Sebastian Moeller]

Hi Dave, hi LIst,

On Jun 5, 2015, at 18:46 , Dave Taht <dave.taht@gmail.com> wrote:

> On Fri, Jun 5, 2015 at 9:35 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>> Going back to fundamentals, there's a clear distinction between traffic
>> which is latency sensitive and traffic which is bandwidth sensitive. Perhaps
>> surprisingly, web traffic tends to fall into the former category, unless the
>> link bandwidth is very low by current standards (analogue modem territory).
>> 
>> It sounds like that router's default settings attempt to make that
>> distinction based on port number, and then perform strict prioritization.
>> The example of SSH demonstrates why that doesn't work; interactive shell
>> over SSH is latency sensitive, but SCP and rsync-over-SSH are bandwidth
>> sensitive, and they all use the same port. The need for explicit
>> configuration (a database of port numbers) is also a black mark against it,
>> especially since they managed to leave out such a common protocol as DNS.
>> 
>> Packet size is a better heuristic than port number. Most latency sensitive
>> protocols do tend to use small packets, and nearly all bandwidth sensitive
>> traffic uses the largest packets it can. SSH also naturally switches between
>> small and large packets depending on the type of traffic it's carrying. If
>> you simply sort your traffic by packet size, you don't even need to
>> configure it - but otherwise you just have a threshold to set and forget.
>> Cunningly, this also naturally performs ack and ping promotion.
> 
> I do not regard ping promotion as a "feature". I think ping should be
> a measurement of worser-case latency.

	I would be most happy with a measurement probe that reflects how the network treats the traffic characteristics currently tested. Now, pings typically are both small and sparse, so both sparse fq-ing and small size priority will treat pings with "loving care”. Changing the packet size with ping is easy if one wants to test how larger packets traverse the network. Making ping non-sparse is a bit more involved as the ping binary somehow thinks that non-root should be throttled to 5 pings per second I believe (which on a a beefy desktop system is easily worked around by starting independent ping instances by hand, but I digress).


Best Regards
        Sebastian


> Openwrt does both synflood and
> ping flood protection in their firewall implementation badly with a
> fixed limit too low for synflood (it used to be 25/sec), and too high
> for ping flood.(1000/sec and extensive filtering)
> 
>> The downside of packet size as a heuristic is that it's possible to game it,
>> especially with strict priority in force. All you have to do is send packets
>> below the threshold if there is one, or slightly smaller than the competing
>> traffic if there isn't one. The receiver can influence this by setting the
>> MSS low during TCP handshakes.
> 
> There is a natural game theory influence pushing packet size larger
> for bulkier traffic, and that is header overhead gets pretty
> significant as you get below 300 bytes.
> 
>> 
>> That is why fq_codel uses the sparse/saturating flow distinction for
>> priority. It's a much more robust heuristic than packet size, requires no
>> configuration at all, and is not so easy to game. The downside? It's only
>> available if you're already doing flow isolation, which basically solves the
>> core problem on its own. Still, making that distinction does help with new
>> flow startup and jitter reduction.
> 
> Yep. A godawful amount of thought and data went into all this...
> 
> ... and I stress in
> http://www.bufferbloat.net/projects/cerowrt/wiki/Wondershaper_Must_Die
> 
> that in 10 years someone might be cursing us for using any of these
> techniques so universally because it interacts badly with the latest
> brain to network direct connect with high speed voice response
> 
> https://www.youtube.com/watch?v=M1ONXea0mXg
> 
> (or something like that)
> 
>> - Jonathan Morton
>> 
>> 
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>> 
> 
> 
> 
> -- 
> Dave Täht
> What will it take to vastly improve wifi for everyone?
> https://plus.google.com/u/0/explore/makewififast
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat