[Cake] Fwd: net-next is OPEN...

[Cake] Fwd: net-next is OPEN...

[Dave Taht]

do we consider cake ready this time?


---------- Forwarded message ----------
From: David Miller <davem@davemloft.net>
Date: Mon, Apr 16, 2018 at 8:00 AM
Subject: net-next is OPEN...
To: netdev@vger.kernel.org



You all know the drill:

        http://vger.kernel.org/~davem/net-next.html

Now, let's see if you guys can avoid drowning me all at once
this time :-)


-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] Fwd: net-next is OPEN...

[Toke Høiland-Jørgensen]

Dave Taht <dave.taht@gmail.com> writes:

> do we consider cake ready this time?

I'm not aware of anything outstanding, at least...

-Toke

Re: [Cake] net-next is OPEN...

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 975 bytes --]


> On Apr 16, 2018, at 5:12 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Dave Taht <dave.taht@gmail.com> writes:
> 
>> do we consider cake ready this time?
> 
> I'm not aware of anything outstanding, at least...

Do I have the release cycle right that it would have to be upstreamed by “about 10 weeks” from now and would end up in 4.18 (https://www.kernel.org/doc/Documentation/networking/netdev-FAQ.txt)?

I remember that fairness behavior at low RTTs (< 20ms) needed to be either improved or documented, and don’t see anything about that in the man page in the tc-adv repo thus far. Summarizing the host isolation results at http://www.drhleny.cz/bufferbloat/cake/round2/#hostiso_cake_{rtt}_{qos-id} <http://www.drhleny.cz/bufferbloat/cake/round2/#hostiso_cake_%7Brtt%7D_%7Bqos-id%7D>:

RTT: fairness (1.0 == perfect fairness)
---
100us: 2.22
1ms: 1.7
2ms: 1.6
3ms: 1.42
5ms: 1.31
8ms: 1.16
10ms: 1.12
20ms: 1.02
40ms: 1.017




[-- Attachment #2: Type: text/html, Size: 2057 bytes --]

Re: [Cake] net-next is OPEN...

[Toke Høiland-Jørgensen]

Pete Heist <pete@eventide.io> writes:

>> On Apr 16, 2018, at 5:12 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> 
>> Dave Taht <dave.taht@gmail.com> writes:
>> 
>>> do we consider cake ready this time?
>> 
>> I'm not aware of anything outstanding, at least...
>
> Do I have the release cycle right that it would have to be upstreamed
> by “about 10 weeks” from now and would end up in 4.18
> (https://www.kernel.org/doc/Documentation/networking/netdev-FAQ.txt)?

Assuming it is accepted, yes... :)

> I remember that fairness behavior at low RTTs (< 20ms) needed to be
> either improved or documented, and don’t see anything about that in
> the man page in the tc-adv repo thus far. Summarizing the host
> isolation results at
> http://www.drhleny.cz/bufferbloat/cake/round2/#hostiso_cake_{rtt}_{qos-id}
> <http://www.drhleny.cz/bufferbloat/cake/round2/#hostiso_cake_%7Brtt%7D_%7Bqos-id%7D>:
>
> RTT: fairness (1.0 == perfect fairness)
> ---
> 100us: 2.22
> 1ms: 1.7
> 2ms: 1.6
> 3ms: 1.42
> 5ms: 1.31
> 8ms: 1.16
> 10ms: 1.12
> 20ms: 1.02
> 40ms: 1.017

Erm, what's the metric and which data source are you looking at here?

-Toke

Re: [Cake] fairness vs RTT

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 2542 bytes --]


> On Apr 16, 2018, at 11:23 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
>> I remember that fairness behavior at low RTTs (< 20ms) needed to be
>> either improved or documented, and don’t see anything about that in
>> the man page in the tc-adv repo thus far. Summarizing the host
>> isolation results at
>> http://www.drhleny.cz/bufferbloat/cake/round2/#hostiso_cake_{rtt}_{qos-id}
>> <http://www.drhleny.cz/bufferbloat/cake/round2/#hostiso_cake_%7Brtt%7D_%7Bqos-id%7D>:
>> 
>> RTT: fairness (1.0 == perfect fairness)
>> ---
>> 100us: 2.22
>> 1ms: 1.7
>> 2ms: 1.6
>> 3ms: 1.42
>> 5ms: 1.31
>> 8ms: 1.16
>> 10ms: 1.12
>> 20ms: 1.02
>> 40ms: 1.017
> 
> Erm, what's the metric and which data source are you looking at here?


Subject changed...

The clients were as follows:
Client 0- 1 stream up
Client 1- 12 streams up
Client 2- 1 stream down
Client 3- 12 streams down

It looks like what I used before was Client 3’s “TCP Download Sum avg” divided by Client 2’s “TCP Download avg” from the srchost/dsthost tests. The data’s in a table here (see column “Client 3 Mean / Client 2 Mean”):

https://docs.google.com/spreadsheets/d/1e06ZfHSSmecJx9sPU2s2g2GYCS18cMIhBN8PXf1jwaM/edit?usp=sharing <https://docs.google.com/spreadsheets/d/1e06ZfHSSmecJx9sPU2s2g2GYCS18cMIhBN8PXf1jwaM/edit?usp=sharing>

I was calculating by hand before, so the numbers are slightly different, but that’s the idea.

Now, these tests were done at 500Mbit between two cabled APU2s, so we could just be running out of CPU on this hardware at lower RTTs. There are CPU stats included, and a “Flent Data Files” section. Is it possible to tell from this if CPU is the problem? The highest median client load I see looks to be 0.77 for the 40ms tests, for example, with a mean of 0.63.

http://www.drhleny.cz/bufferbloat/cake/round2/hostiso_cake_100us_eg_cakeeth_src_cakeeth_dst_500mbit/index.html <http://www.drhleny.cz/bufferbloat/cake/round2/hostiso_cake_100us_eg_cakeeth_src_cakeeth_dst_500mbit/index.html>

http://www.drhleny.cz/bufferbloat/cake/round2/hostiso_cake_40ms_eg_cakeeth_src_cakeeth_dst_500mbit/index.html <http://www.drhleny.cz/bufferbloat/cake/round2/hostiso_cake_40ms_eg_cakeeth_src_cakeeth_dst_500mbit/index.html>

If we’re not sure, the tests would have to be redone at a lower bitrate. It would probably be best if someone reproduced these results externally anyway, and perhaps it would also be better to not be mixing upload and download tests at the same time… :)


[-- Attachment #2: Type: text/html, Size: 3874 bytes --]

Re: [Cake] fairness vs RTT

[Toke Høiland-Jørgensen]

Pete Heist <pete@eventide.io> writes:

>> On Apr 16, 2018, at 11:23 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> 
>>> I remember that fairness behavior at low RTTs (< 20ms) needed to be
>>> either improved or documented, and don’t see anything about that in
>>> the man page in the tc-adv repo thus far. Summarizing the host
>>> isolation results at
>>> http://www.drhleny.cz/bufferbloat/cake/round2/#hostiso_cake_{rtt}_{qos-id}
>>> <http://www.drhleny.cz/bufferbloat/cake/round2/#hostiso_cake_%7Brtt%7D_%7Bqos-id%7D>:
>>> 
>>> RTT: fairness (1.0 == perfect fairness)
>>> ---
>>> 100us: 2.22
>>> 1ms: 1.7
>>> 2ms: 1.6
>>> 3ms: 1.42
>>> 5ms: 1.31
>>> 8ms: 1.16
>>> 10ms: 1.12
>>> 20ms: 1.02
>>> 40ms: 1.017
>> 
>> Erm, what's the metric and which data source are you looking at here?
>
>
> Subject changed...
>
> The clients were as follows:
> Client 0- 1 stream up
> Client 1- 12 streams up
> Client 2- 1 stream down
> Client 3- 12 streams down
>
> It looks like what I used before was Client 3’s “TCP Download Sum avg” divided by Client 2’s “TCP Download avg” from the srchost/dsthost tests. The data’s in a table here (see column “Client 3 Mean / Client 2 Mean”):
>
> https://docs.google.com/spreadsheets/d/1e06ZfHSSmecJx9sPU2s2g2GYCS18cMIhBN8PXf1jwaM/edit?usp=sharing <https://docs.google.com/spreadsheets/d/1e06ZfHSSmecJx9sPU2s2g2GYCS18cMIhBN8PXf1jwaM/edit?usp=sharing>
>
> I was calculating by hand before, so the numbers are slightly different, but that’s the idea.
>
> Now, these tests were done at 500Mbit between two cabled APU2s, so we
> could just be running out of CPU on this hardware at lower RTTs. There
> are CPU stats included, and a “Flent Data Files” section. Is it
> possible to tell from this if CPU is the problem? The highest median
> client load I see looks to be 0.77 for the 40ms tests, for example,
> with a mean of 0.63.

Well, the CPU usage meter is just summing that first line of /proc/stat;
so yeah, individual CPUs can definitely be 100% loaded. And the fact
that the test only achieves a total of ~200 Mbps rather than the 500
would indicate that this is the case...

-Toke

Re: [Cake] net-next is OPEN...

[Jonathan Morton]

> On 16 Apr, 2018, at 11:55 pm, Pete Heist <pete@eventide.io> wrote:
> 
> I remember that fairness behavior at low RTTs (< 20ms) needed to be either improved or documented

The reason for the behaviour, IIRC, was that throughput dropped below 100% when the latency target was reduced too much.  Since then there has been a small change which might improve it a little, so a retest would be reasonable.

 - Jonathan Morton

Re: [Cake] net-next is OPEN...

[Pete Heist]

I also think I saw this happen at lower bandwidths as well, when the CPU wasn’t loaded. What I’ll do is re-test on the current version I have at, say, 50Mbit (or to where load drops substantially), then update to the head and test again, and let you know...

Pete

> On Apr 17, 2018, at 3:52 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> On 16 Apr, 2018, at 11:55 pm, Pete Heist <pete@eventide.io> wrote:
>> 
>> I remember that fairness behavior at low RTTs (< 20ms) needed to be either improved or documented
> 
> The reason for the behaviour, IIRC, was that throughput dropped below 100% when the latency target was reduced too much.  Since then there has been a small change which might improve it a little, so a retest would be reasonable.
> 
> - Jonathan Morton
> 

Re: [Cake] net-next is OPEN...

[Pete Heist]

> On Apr 18, 2018, at 7:43 AM, Pete Heist <pete@eventide.io> wrote:
> 
> I also think I saw this happen at lower bandwidths as well, when the CPU wasn’t loaded. What I’ll do is re-test on the current version I have at, say, 50Mbit (or to where load drops substantially), then update to the head and test again, and let you know...
> 
>> On Apr 17, 2018, at 3:52 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
>> 
>>> On 16 Apr, 2018, at 11:55 pm, Pete Heist <pete@eventide.io> wrote:
>>> 
>>> I remember that fairness behavior at low RTTs (< 20ms) needed to be either improved or documented
>> 
>> The reason for the behaviour, IIRC, was that throughput dropped below 100% when the latency target was reduced too much.  Since then there has been a small change which might improve it a little, so a retest would be reasonable.

At 50mbit I don’t see nearly as much fairness degradation at low RTTs, although there’s some. Even at 100us, “fairness” is around 1.1 (1.0 being perfectly fair) instead of the 2.x I saw at 500mbit. I do not see much of a difference between the latest code (16d7fed, 2018-04-17) and the previous code I tested (7061401, 2017-12-01), if that info is of use.

RTT: tcp_1up upload Mbps / tcp_12up upload Mbps

7061401 (2017-12-01):

   100us: 23.80 / 25.85
   1ms: 23.89 / 29.46
   10ms: 23.93 / 24.66
   40ms: 23.96 / 24.10
   100ms: 23.97 / 24.12

16d7fed (2018-04-17):

   100us: 23.97 / 26.49
   1ms: 23.89 / 26.27
   10ms: 23.98 / 26.37
   40ms: 23.94 / 24.08
   100ms: 23.97 / 24.12

I can post reports / flent files on request.

So it appears this is CPU related, and not worth exploring further(?) and not worth documenting(?) other than that once things have stabilized, documenting how Cake degrades under various extreme conditions would be informative.

Well, here’s to science and a good walk in the weeds…

Re: [Cake] net-next is OPEN...

[Toke Høiland-Jørgensen]

Pete Heist <pete@eventide.io> writes:

>> On Apr 18, 2018, at 7:43 AM, Pete Heist <pete@eventide.io> wrote:
>> 
>> I also think I saw this happen at lower bandwidths as well, when the CPU wasn’t loaded. What I’ll do is re-test on the current version I have at, say, 50Mbit (or to where load drops substantially), then update to the head and test again, and let you know...
>> 
>>> On Apr 17, 2018, at 3:52 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
>>> 
>>>> On 16 Apr, 2018, at 11:55 pm, Pete Heist <pete@eventide.io> wrote:
>>>> 
>>>> I remember that fairness behavior at low RTTs (< 20ms) needed to be either improved or documented
>>> 
>>> The reason for the behaviour, IIRC, was that throughput dropped below 100% when the latency target was reduced too much.  Since then there has been a small change which might improve it a little, so a retest would be reasonable.
>
> At 50mbit I don’t see nearly as much fairness degradation at low RTTs, although there’s some. Even at 100us, “fairness” is around 1.1 (1.0 being perfectly fair) instead of the 2.x I saw at 500mbit. I do not see much of a difference between the latest code (16d7fed, 2018-04-17) and the previous code I tested (7061401, 2017-12-01), if that info is of use.
>
> RTT: tcp_1up upload Mbps / tcp_12up upload Mbps
>
> 7061401 (2017-12-01):
>
>    100us: 23.80 / 25.85
>    1ms: 23.89 / 29.46
>    10ms: 23.93 / 24.66
>    40ms: 23.96 / 24.10
>    100ms: 23.97 / 24.12
>
> 16d7fed (2018-04-17):
>
>    100us: 23.97 / 26.49
>    1ms: 23.89 / 26.27
>    10ms: 23.98 / 26.37
>    40ms: 23.94 / 24.08
>    100ms: 23.97 / 24.12
>
> I can post reports / flent files on request.
>
> So it appears this is CPU related, and not worth exploring further(?)
> and not worth documenting(?) other than that once things have
> stabilized, documenting how Cake degrades under various extreme
> conditions would be informative.

Awesome, thanks for re-testing! :)

-Toke