[Cake] lan keyword affects host fairness

[Cake] lan keyword affects host fairness

[Pete Heist]

It seems that the ‘lan’ keyword (and probably other lower rtt settings in general) may adversely impact host fairness in some cases. Is this to be expected? I set up a fairness test with rrul_be_nflows where one client has 2/2 TCP flows and the other has 8/8 TCP flows, then ran five tests:

https://docs.google.com/spreadsheets/d/1SMXWw2fLfmBRU622urfdvA_Ujsuf_KQ4P3uyOH1skOM/edit?usp=sharing

As can be seen, fairness works well with the default rtt, but with the lan keyword, it varies from partially fair when rate limiting is used to as if there’s no fairness at all when unlimited (bql) is used. So either we could find some problem with my test, or this may at least need to be documented somewhere...

Pete

Re: [Cake] lan keyword affects host fairness

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 468 bytes --]

This is most likely an interaction of the AQM with Linux' scheduling
latency.

At the 'lan' setting, the time comstants are similar in magnitude to the
delays induced by Linux itself, so congestion might be signalled
prematurely.  The flows will then become sparse and total throughput
reduced, leaving little or no back-pressure for the fairness logic to work
against.

For this reason, you might have better luck with the next higher RTT
setting.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 557 bytes --]

Re: [Cake] lan keyword affects host fairness

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 1063 bytes --]


> On Nov 23, 2017, at 10:44 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
> This is most likely an interaction of the AQM with Linux' scheduling latency.
> 
> At the 'lan' setting, the time comstants are similar in magnitude to the delays induced by Linux itself, so congestion might be signalled prematurely.  The flows will then become sparse and total throughput reduced, leaving little or no back-pressure for the fairness logic to work against.
> 
> For this reason, you might have better luck with the next higher RTT setting.
> 
Thanks…and using ‘metro’ (rtt 10ms) does improve things (two more tests at the end):

https://docs.google.com/spreadsheets/d/1SMXWw2fLfmBRU622urfdvA_Ujsuf_KQ4P3uyOH1skOM/edit#gid=2072687073

In both cases, soft rate limiting to 950mbit when using lower RTTs works better than relying on bql for the back-pressure (if I’m saying that right).

So it just might be a thing (for the man page?) to avoid confusion. Or a warning emitted in some cases? Maybe there are other opinions on that...

Pete


[-- Attachment #2: Type: text/html, Size: 1848 bytes --]

Re: [Cake] lan keyword affects host fairness

[Dave Taht]

Pete Heist <peteheist@gmail.com> writes:

>     On Nov 23, 2017, at 10:44 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>     
>     This is most likely an interaction of the AQM with Linux' scheduling
>     latency.
>
>     At the 'lan' setting, the time comstants are similar in magnitude to the
>     delays induced by Linux itself, so congestion might be signalled
>     prematurely. The flows will then become sparse and total throughput reduced,
>     leaving little or no back-pressure for the fairness logic to work against.

Agreed. 

man page add:

At the 'lan' setting(1ms), the time constants are similar in magnitude
to the jitter in the Linux kernel itself, so congestion might be
signalled prematurely. The flows will then become sparse and total
throughput reduced, leaving little or no back-pressure for the fairness
logic to work against. Use the "metro" setting for local lans unless you
have a custom kernel.

>
>     For this reason, you might have better luck with the next higher RTT
>     setting.
>
> Thanks…and using ‘metro’ (rtt 10ms) does improve things (two more tests at the
> end):


>
> https://docs.google.com/spreadsheets/d/1SMXWw2fLfmBRU622urfdvA_Ujsuf_KQ4P3uyOH1skOM/edit#gid=2072687073
>
> In both cases, soft rate limiting to 950mbit when using lower RTTs works better
> than relying on bql for the back-pressure (if I’m saying that right).
>
> So it just might be a thing (for the man page?) to avoid confusion. Or a warning
> emitted in some cases? Maybe there are other opinions on that...

Yes, man page.

> Pete
>
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] lan keyword affects host fairness

[Toke Høiland-Jørgensen]

Dave Taht <dave@taht.net> writes:

> Pete Heist <peteheist@gmail.com> writes:
>
>>     On Nov 23, 2017, at 10:44 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>>     
>>     This is most likely an interaction of the AQM with Linux' scheduling
>>     latency.
>>
>>     At the 'lan' setting, the time comstants are similar in magnitude to the
>>     delays induced by Linux itself, so congestion might be signalled
>>     prematurely. The flows will then become sparse and total throughput reduced,
>>     leaving little or no back-pressure for the fairness logic to work against.
>
> Agreed. 
>
> man page add:
>
> At the 'lan' setting(1ms), the time constants are similar in magnitude
> to the jitter in the Linux kernel itself, so congestion might be
> signalled prematurely. The flows will then become sparse and total
> throughput reduced, leaving little or no back-pressure for the fairness
> logic to work against. Use the "metro" setting for local lans unless you
> have a custom kernel.

Erm, doesn't this make the 'lan' keyword pretty much useless? So why not
just remove it? Or redefine it to something that actually works? 3ms?

-Toke

Re: [Cake] lan keyword affects host fairness

[Sebastian Moeller]

> On Nov 24, 2017, at 12:21, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Dave Taht <dave@taht.net> writes:
> 
>> Pete Heist <peteheist@gmail.com> writes:
>> 
>>>    On Nov 23, 2017, at 10:44 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>>> 
>>>    This is most likely an interaction of the AQM with Linux' scheduling
>>>    latency.
>>> 
>>>    At the 'lan' setting, the time comstants are similar in magnitude to the
>>>    delays induced by Linux itself, so congestion might be signalled
>>>    prematurely. The flows will then become sparse and total throughput reduced,
>>>    leaving little or no back-pressure for the fairness logic to work against.
>> 
>> Agreed. 
>> 
>> man page add:
>> 
>> At the 'lan' setting(1ms), the time constants are similar in magnitude
>> to the jitter in the Linux kernel itself, so congestion might be
>> signalled prematurely. The flows will then become sparse and total
>> throughput reduced, leaving little or no back-pressure for the fairness
>> logic to work against. Use the "metro" setting for local lans unless you
>> have a custom kernel.
> 
> Erm, doesn't this make the 'lan' keyword pretty much useless? So why not
> just remove it? Or redefine it to something that actually works? 3ms?

The same applies for datacentre (0.1 ms), no? But I agree, let's not expose these as explicit keywords, one can always use "rtt [100us|1ms]" I assume...


> 
> -Toke
> 
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] lan keyword affects host fairness

[Marcelo Ricardo Leitner]

On Fri, Nov 24, 2017 at 01:06:12PM +0100, Sebastian Moeller wrote:
> 
> > On Nov 24, 2017, at 12:21, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> > 
> > Dave Taht <dave@taht.net> writes:
> > 
> >> Pete Heist <peteheist@gmail.com> writes:
> >> 
> >>>    On Nov 23, 2017, at 10:44 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
> >>> 
> >>>    This is most likely an interaction of the AQM with Linux' scheduling
> >>>    latency.
> >>> 
> >>>    At the 'lan' setting, the time comstants are similar in magnitude to the
> >>>    delays induced by Linux itself, so congestion might be signalled
> >>>    prematurely. The flows will then become sparse and total throughput reduced,
> >>>    leaving little or no back-pressure for the fairness logic to work against.
> >> 
> >> Agreed. 
> >> 
> >> man page add:
> >> 
> >> At the 'lan' setting(1ms), the time constants are similar in magnitude
> >> to the jitter in the Linux kernel itself, so congestion might be
> >> signalled prematurely. The flows will then become sparse and total
> >> throughput reduced, leaving little or no back-pressure for the fairness
> >> logic to work against. Use the "metro" setting for local lans unless you
> >> have a custom kernel.
> > 
> > Erm, doesn't this make the 'lan' keyword pretty much useless? So why not
> > just remove it? Or redefine it to something that actually works? 3ms?
> 
> The same applies for datacentre (0.1 ms), no? But I agree, let's not expose these as explicit keywords, one can always use "rtt [100us|1ms]" I assume...

Which should also contain such disclaimer with it.
"Values smaller than 10ms requires special handling.", for example.

  Marcelo

Re: [Cake] lan keyword affects host fairness

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 2735 bytes --]


> On Nov 24, 2017, at 12:21 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Dave Taht <dave@taht.net> writes:
> 
>> Pete Heist <peteheist@gmail.com> writes:
>> 
>>>    On Nov 23, 2017, at 10:44 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>>> 
>>>    This is most likely an interaction of the AQM with Linux' scheduling
>>>    latency.
>>> 
>>>    At the 'lan' setting, the time comstants are similar in magnitude to the
>>>    delays induced by Linux itself, so congestion might be signalled
>>>    prematurely. The flows will then become sparse and total throughput reduced,
>>>    leaving little or no back-pressure for the fairness logic to work against.
>> 
>> Agreed. 
>> 
>> man page add:
>> 
>> At the 'lan' setting(1ms), the time constants are similar in magnitude
>> to the jitter in the Linux kernel itself, so congestion might be
>> signalled prematurely. The flows will then become sparse and total
>> throughput reduced, leaving little or no back-pressure for the fairness
>> logic to work against. Use the "metro" setting for local lans unless you
>> have a custom kernel.
> 
> Erm, doesn't this make the 'lan' keyword pretty much useless? So why not
> just remove it? Or redefine it to something that actually works? 3ms?

Removing the bandwidth keywords altogether and going back to fq_codel’s specification of target and interval would be my personal preference (unless we can figure out how to make the keywords work well with one another in all cases).

If we want to keep them, I’ll look for more holes where things might not behave as people expect so we can further clear up the docs.

Also, note that even at ‘metro’, fairness is better, but still not fully fair on my hardware:

https://docs.google.com/spreadsheets/d/1SMXWw2fLfmBRU622urfdvA_Ujsuf_KQ4P3uyOH1skOM/edit#gid=2072687073 <https://docs.google.com/spreadsheets/d/1SMXWw2fLfmBRU622urfdvA_Ujsuf_KQ4P3uyOH1skOM/edit#gid=2072687073>

With 950mbit rate limiting, it’s pretty good at 1.16:1. But with bql it’s still ~2:1. At what rtt will fairness actually work? That may depend on hardware, OS version, number of flows, whether bql is being used or not, or other factors. At rtt 100ms fairness worked well for this test with both bql and rate limiting.

Also note that I’m glossing over the fact that on my hardware (Ethernet device with four queues), sometimes there was a bandwidth asymmetry and corresponding fluctuation in fairness when using bql that changed with each flent run, so I had to retry the test once or twice to get a “good” run. I would try to quantify that separately before I get into it further. But that’s something that may also throw folks for a loop.


[-- Attachment #2: Type: text/html, Size: 4058 bytes --]

Re: [Cake] lan keyword affects host fairness

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 991 bytes --]


> On Nov 24, 2017, at 2:49 PM, Pete Heist <peteheist@gmail.com> wrote:
> 
> Removing the bandwidth keywords altogether and going back to fq_codel’s specification of target and interval would be my personal preference (unless we can figure out how to make the keywords work well with one another in all cases).

To add to my comments, this probably came across as too harsh or discontinuous an idea at this stage when we’re in the process of shoring things up- that wasn’t my intent!

There is the other side that these keywords save people from having to know more. Which is better, explaining target and interval to everyone or having them use these? I imagine that was the logic that went into it. Also, if it’s not a good idea to be changing the configuration interface at this point (and it may not be), then there are alternatives, and the man page addition will definitely help people. Maybe I’ll make some runs across a range of rtts to understand this better…


[-- Attachment #2: Type: text/html, Size: 2072 bytes --]

Re: [Cake] lan keyword affects host fairness

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 364 bytes --]

Can I just point out that the four hardware queues will themselves be
interfering with the backpressure on short timescales when Cake is in
unlimited mode, and can easily explain the poorer host-fairness performance.

Conversely, the real performance is seen when the internal shaper is used.
What happens if you say "bandwidth 1Gbit ethernet"?

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 435 bytes --]

Re: [Cake] lan keyword affects host fairness

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 1021 bytes --]


> On Nov 24, 2017, at 8:48 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
> Can I just point out that the four hardware queues will themselves be interfering with the backpressure on short timescales when Cake is in unlimited mode, and can easily explain the poorer host-fairness performance
> 
Ok, just for clarity I have a single cake instance at the root, not four of them under mq.
> Conversely, the real performance is seen when the internal shaper is used.  What happens if you say "bandwidth 1Gbit ethernet"?
> 

Results posted for that with and without ‘lan'. I suppose I’m starting then to lose control of the queue? It's around 1.85:1.

https://docs.google.com/spreadsheets/d/1SMXWw2fLfmBRU622urfdvA_Ujsuf_KQ4P3uyOH1skOM/edit#gid=2072687073 <https://docs.google.com/spreadsheets/d/1SMXWw2fLfmBRU622urfdvA_Ujsuf_KQ4P3uyOH1skOM/edit#gid=2072687073>

Also on the “1Gbit ethernet lan” tab I thought to add the output from “tc -s qdisc” after the test from the server side, for info.


[-- Attachment #2: Type: text/html, Size: 1621 bytes --]

Re: [Cake] lan keyword affects host fairness

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 73 bytes --]

Mmm, that's a lot of drops.  So you're not using ECN?

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 124 bytes --]

Re: [Cake] lan keyword affects host fairness

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 994 bytes --]


> On Nov 24, 2017, at 9:32 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
> Mmm, that's a lot of drops.  So you're not using ECN?
> 
Not using ecn. I did one ecn run but didn’t see much change. I did more runs at 950mbit- 20ms, 50ms 100ms and added a chart at the bottom of rtt vs fairness (same doc, link for reference):

https://docs.google.com/spreadsheets/d/1SMXWw2fLfmBRU622urfdvA_Ujsuf_KQ4P3uyOH1skOM/edit#gid=2072687073 <https://docs.google.com/spreadsheets/d/1SMXWw2fLfmBRU622urfdvA_Ujsuf_KQ4P3uyOH1skOM/edit#gid=2072687073>

There seems to be a weird asymmetry that happens sometimes at rtt 20ms and rtt 50ms. One run at 20ms (right after rebooting) looked good but an earlier one didn’t. Three runs at 50ms all showed some level of asymmetry.

I’ll have to move on to other work, but the results still show fairness working pretty well at 100ms. It sometimes works well at other rtts, but there are variations and sometimes asymmetries. It’s hard to pull apart.

[-- Attachment #2: Type: text/html, Size: 1559 bytes --]

Re: [Cake] lan keyword affects host fairness

[Dave Taht]

I support removing metro and below as keywords.

Note: I am more interested in throughput (w/ecn) at the lower rtt
settings than flow fairness (is the aqm scaling?). If we can have
shorter queues overall while not taking a throughput hit, in the
datacenter, that's a win.

I have a hope, however, that at 10GigE we have sufficient queue size
for both fairness and throughput, with even a 1ms rtt setting (50us
target) but most of us lack the hardware that can test anything at
those rates, and there are other problems in that Linux can't do more
than about 4m PPS, so it is presently impossible, to my knowledge, to
drive 10GigE to saturation with small packets using any qdisc.

And there are many other overheads - notably qdisc locking and, fib
lookup, in the way. It had been my hope that cake could be poured into
ethernet hardware one day, but it has sprouted too many non O(1)
things of late.


On Fri, Nov 24, 2017 at 11:41 AM, Pete Heist <peteheist@gmail.com> wrote:
>
> On Nov 24, 2017, at 2:49 PM, Pete Heist <peteheist@gmail.com> wrote:
>
> Removing the bandwidth keywords altogether and going back to fq_codel’s
> specification of target and interval would be my personal preference (unless
> we can figure out how to make the keywords work well with one another in all
> cases).
>
>
> To add to my comments, this probably came across as too harsh or
> discontinuous an idea at this stage when we’re in the process of shoring
> things up- that wasn’t my intent!
>
> There is the other side that these keywords save people from having to know
> more. Which is better, explaining target and interval to everyone or having
> them use these? I imagine that was the logic that went into it. Also, if
> it’s not a good idea to be changing the configuration interface at this
> point (and it may not be), then there are alternatives, and the man page
> addition will definitely help people. Maybe I’ll make some runs across a
> range of rtts to understand this better…
>
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] lan keyword affects host fairness

[Dave Taht]

Pete Heist <peteheist@gmail.com> writes:

>     On Nov 24, 2017, at 2:49 PM, Pete Heist <peteheist@gmail.com> wrote:
>     
>     
>     Removing the bandwidth keywords altogether and going back to fq_codel’s
>     specification of target and interval would be my personal preference (unless
>     we can figure out how to make the keywords work well with one another in all
>     cases).
>
> To add to my comments, this probably came across as too harsh or discontinuous
> an idea at this stage when we’re in the process of shoring things up- that
> wasn’t my intent!

Well, I expected bumps on trying to get stuff to mainline. This is just
text and documentation. 

The major speedbump would be breaking the xstats API, the testing and
flag day it would entail, etc. I had spare time sufficient to clean
things up, not make major changes like that.

>
> There is the other side that these keywords save people from having to know
> more. Which is better, explaining target and interval to everyone or having them
> use these?

Something that even an ISP could configure.

> I imagine that was the logic that went into it. Also, if it’s not a
> good idea to be changing the configuration interface at this point (and it may
> not be), then there are alternatives, and the man page addition will definitely
> help people. Maybe I’ll make some runs across a range of rtts to understand this
> better…