[Cake] [Fwd: Re: RHODIUM - nuking blue (for testing)]

[Cake] [Fwd: Re: RHODIUM - nuking blue (for testing)]

[Georgios Amanakis]

[-- Attachment #1: Type: text/plain, Size: 1627 bytes --]

-------- Forwarded Message --------
From: Georgios Amanakis <gamanakis@gmail.com>
To: Dave Taht <dave.taht@gmail.com>
Cc: Cake List <cake@lists.bufferbloat.net>, Jonathan Morton <chromatix9
9@gmail.com>, Pete Heist <peteheist@gmail.com>
Subject: Re: [Cake] RHODIUM - nuking blue (for testing)
Date: Tue, 05 Dec 2017 20:58:39 -0500

I just finished testing Rhodium, and I am supplementing Sunday's
results.

CMTS setup:
server -- delay -- isp -- mbox -- client

ISP limited at 200/10mbit.
Cake/HTB shaping at 180/9mbit.
RTT 20ms with cake's RTT set at 20ms
rrul_be_nflows, 32 download, 8 upload streams.
net.ipv4.tcp_ecn=2 in all nodes (linux default)

Cake tested in all flow-isolation modes with all ack filtering options.
HTB tested with codel,pie,fq,sfq,fq_codel.

Rhodium(cakerhod) doesn't show much difference against Cobalt(cake).

George

On Tue, 2017-12-05 at 11:32 -0800, Dave Taht wrote:
> Given some of the results thus far at various RTTs, and at higher
> loads, like the ginormous one georgios just ran, I thought it might
> be
> useful to run a battery of tests with blue disabled.
> 
> I haven't gone to the trouble of creating a formal branch - tho if I
> did, I'd call it rhodium, as rhodium detectors are used in nuclear
> reactors to measure the neutron flux level.
> 
> untested-but-compiling-patch attached. I'm busy on a few other things
> this week, and it is looking like we'll have to replace the xstats
> structure with something smaller to go mainline.
> 
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

[-- Attachment #2: ping_cdf_cobalt_rhodium_noack.png --]
[-- Type: image/png, Size: 110103 bytes --]

[-- Attachment #3: tcp_rtt_cobalt_rhodium_noack.png --]
[-- Type: image/png, Size: 180626 bytes --]

[-- Attachment #4: ping_cdf_cobalt_rhodium_ackaggr.png --]
[-- Type: image/png, Size: 114693 bytes --]

[-- Attachment #5: tcp_rtt_cobalt_rhodium_ackaggr.png --]
[-- Type: image/png, Size: 183707 bytes --]

Re: [Cake] [Fwd: Re: RHODIUM - nuking blue (for testing)]

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 169 bytes --]

Yup, I wouldn't expect any difference at all in normal loads.  Only when
you get some flows marked as unresponsive does the BLUE portion do anything.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 216 bytes --]

Re: [Cake] [Fwd: Re: RHODIUM - nuking blue (for testing)]

[Dave Taht]

Groovy.

I guess if I had any one goal, was that I wanted to have cake to shape
well on the APU2 at a gigabit. That's the highest end "home router" we
have, although it would be good to get results on things like the turris
omnia, also.

I hope to start a round of profiling as to what can be optimized (out)
this weekend. Last I looked the hashing costs dominated.

Pretty sure we're good at 200Mbit there (aside from the ingress bug).

It would be nice to have a test showing blue being useful.

Did you get cpu stats for the below tests?

Georgios Amanakis <gamanakis@gmail.com> writes:

> -------- Forwarded Message --------
> From: Georgios Amanakis <gamanakis@gmail.com>
> To: Dave Taht <dave.taht@gmail.com>
> Cc: Cake List <cake@lists.bufferbloat.net>, Jonathan Morton <chromatix9
> 9@gmail.com>, Pete Heist <peteheist@gmail.com>
> Subject: Re: [Cake] RHODIUM - nuking blue (for testing)
> Date: Tue, 05 Dec 2017 20:58:39 -0500
>
> I just finished testing Rhodium, and I am supplementing Sunday's
> results.
>
> CMTS setup:
> server -- delay -- isp -- mbox -- client
>
> ISP limited at 200/10mbit.
> Cake/HTB shaping at 180/9mbit.
> RTT 20ms with cake's RTT set at 20ms
> rrul_be_nflows, 32 download, 8 upload streams.
> net.ipv4.tcp_ecn=2 in all nodes (linux default)
>
> Cake tested in all flow-isolation modes with all ack filtering options.
> HTB tested with codel,pie,fq,sfq,fq_codel.
>
> Rhodium(cakerhod) doesn't show much difference against Cobalt(cake).
>
> George
>
> On Tue, 2017-12-05 at 11:32 -0800, Dave Taht wrote:
>> Given some of the results thus far at various RTTs, and at higher
>> loads, like the ginormous one georgios just ran, I thought it might
>> be
>> useful to run a battery of tests with blue disabled.
>> 
>> I haven't gone to the trouble of creating a formal branch - tho if I
>> did, I'd call it rhodium, as rhodium detectors are used in nuclear
>> reactors to measure the neutron flux level.
>> 
>> untested-but-compiling-patch attached. I'm busy on a few other things
>> this week, and it is looking like we'll have to replace the xstats
>> structure with something smaller to go mainline.
>> 
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
>
>
>
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] [Fwd: Re: RHODIUM - nuking blue (for testing)]

[Georgios Amanakis]

[-- Attachment #1.1: Type: text/plain, Size: 215 bytes --]

On Wed, Dec 6, 2017 at 4:08 PM, Dave Taht <dave@taht.net> wrote:

>
> Did you get cpu stats for the below tests?
>
> I did! In the no-ack-filter case CPU usage looks strange though. Seems to
be higher with Rhodium.

[-- Attachment #1.2: Type: text/html, Size: 520 bytes --]

[-- Attachment #2: cpu_cake_cakerhod_ackaggr.png --]
[-- Type: image/png, Size: 539297 bytes --]

[-- Attachment #3: cpu_cake_cakerhod_noack.png --]
[-- Type: image/png, Size: 438269 bytes --]

Re: [Cake] [Fwd: Re: RHODIUM - nuking blue (for testing)]

[Georgios Amanakis]

[-- Attachment #1: Type: text/plain, Size: 349 bytes --]

I will rerun this one, just to be sure.

On Wed, Dec 6, 2017 at 4:17 PM, Georgios Amanakis <gamanakis@gmail.com>
wrote:

>
> On Wed, Dec 6, 2017 at 4:08 PM, Dave Taht <dave@taht.net> wrote:
>
>>
>> Did you get cpu stats for the below tests?
>>
>> I did! In the no-ack-filter case CPU usage looks strange though. Seems to
> be higher with Rhodium.
>

[-- Attachment #2: Type: text/html, Size: 961 bytes --]

Re: [Cake] [Fwd: Re: RHODIUM - nuking blue (for testing)]

[Georgios Amanakis]

[-- Attachment #1.1: Type: text/plain, Size: 611 bytes --]

Redid the triple-isolate/{no-ack-filter, ack-filter-aggressive} with
Cobalt/Rhodium.
ping_cdf, tcp_rtt_cdf and cpu plots.

No apparent difference in CPU usage.

On Wed, Dec 6, 2017 at 4:21 PM, Georgios Amanakis <gamanakis@gmail.com>
wrote:

> I will rerun this one, just to be sure.
>
> On Wed, Dec 6, 2017 at 4:17 PM, Georgios Amanakis <gamanakis@gmail.com>
> wrote:
>
>>
>> On Wed, Dec 6, 2017 at 4:08 PM, Dave Taht <dave@taht.net> wrote:
>>
>>>
>>> Did you get cpu stats for the below tests?
>>>
>>> I did! In the no-ack-filter case CPU usage looks strange though. Seems
>> to be higher with Rhodium.
>>
>
>

[-- Attachment #1.2: Type: text/html, Size: 1570 bytes --]

[-- Attachment #2: cpu_cake_cakerhod.png --]
[-- Type: image/png, Size: 632755 bytes --]

[-- Attachment #3: ping_cdf_cake_cakerhod_ackaggr.png --]
[-- Type: image/png, Size: 386226 bytes --]

[-- Attachment #4: ping_cdf_cake_cakerhod_noack.png --]
[-- Type: image/png, Size: 342039 bytes --]

[-- Attachment #5: rtt_cdf_cake_cakerhod_ackaggr.png --]
[-- Type: image/png, Size: 810020 bytes --]

[-- Attachment #6: rtt_cdf_cake_cakerhod_noack.png --]
[-- Type: image/png, Size: 683106 bytes --]

[-- Attachment #7: rrulbe_cmts_rhodium_cobalt.tgz --]
[-- Type: application/x-gzip, Size: 4139772 bytes --]

Re: [Cake] [Fwd: Re: RHODIUM - nuking blue (for testing)]

[Luis E. Garcia]

[-- Attachment #1: Type: text/plain, Size: 2900 bytes --]

Dave,
What is the highest bandwidth that you have been able to shape with CAKE on
the APU2?

Regards,
Luis Garcia

On Wed, Dec 6, 2017 at 3:08 PM, Dave Taht <dave@taht.net> wrote:

>
> Groovy.
>
> I guess if I had any one goal, was that I wanted to have cake to shape
> well on the APU2 at a gigabit. That's the highest end "home router" we
> have, although it would be good to get results on things like the turris
> omnia, also.
>
> I hope to start a round of profiling as to what can be optimized (out)
> this weekend. Last I looked the hashing costs dominated.
>
> Pretty sure we're good at 200Mbit there (aside from the ingress bug).
>
> It would be nice to have a test showing blue being useful.
>
> Did you get cpu stats for the below tests?
>
> Georgios Amanakis <gamanakis@gmail.com> writes:
>
> > -------- Forwarded Message --------
> > From: Georgios Amanakis <gamanakis@gmail.com>
> > To: Dave Taht <dave.taht@gmail.com>
> > Cc: Cake List <cake@lists.bufferbloat.net>, Jonathan Morton <chromatix9
> > 9@gmail.com>, Pete Heist <peteheist@gmail.com>
> > Subject: Re: [Cake] RHODIUM - nuking blue (for testing)
> > Date: Tue, 05 Dec 2017 20:58:39 -0500
> >
> > I just finished testing Rhodium, and I am supplementing Sunday's
> > results.
> >
> > CMTS setup:
> > server -- delay -- isp -- mbox -- client
> >
> > ISP limited at 200/10mbit.
> > Cake/HTB shaping at 180/9mbit.
> > RTT 20ms with cake's RTT set at 20ms
> > rrul_be_nflows, 32 download, 8 upload streams.
> > net.ipv4.tcp_ecn=2 in all nodes (linux default)
> >
> > Cake tested in all flow-isolation modes with all ack filtering options.
> > HTB tested with codel,pie,fq,sfq,fq_codel.
> >
> > Rhodium(cakerhod) doesn't show much difference against Cobalt(cake).
> >
> > George
> >
> > On Tue, 2017-12-05 at 11:32 -0800, Dave Taht wrote:
> >> Given some of the results thus far at various RTTs, and at higher
> >> loads, like the ginormous one georgios just ran, I thought it might
> >> be
> >> useful to run a battery of tests with blue disabled.
> >>
> >> I haven't gone to the trouble of creating a formal branch - tho if I
> >> did, I'd call it rhodium, as rhodium detectors are used in nuclear
> >> reactors to measure the neutron flux level.
> >>
> >> untested-but-compiling-patch attached. I'm busy on a few other things
> >> this week, and it is looking like we'll have to replace the xstats
> >> structure with something smaller to go mainline.
> >>
> >> _______________________________________________
> >> Cake mailing list
> >> Cake@lists.bufferbloat.net
> >> https://lists.bufferbloat.net/listinfo/cake
> >
> >
> >
> >
> > _______________________________________________
> > Cake mailing list
> > Cake@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cake
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>

[-- Attachment #2: Type: text/html, Size: 4450 bytes --]

Re: [Cake] [Fwd: Re: RHODIUM - nuking blue (for testing)]

[Dave Taht]

On Wed, Dec 6, 2017 at 2:35 PM, Luis E. Garcia <luis@bitamins.net> wrote:
> Dave,
> What is the highest bandwidth that you have been able to shape with CAKE on
> the APU2?

pete ran out of cpu at 900mbit in one of the early test rounds.

https://lists.bufferbloat.net/pipermail/cake/2017-November/003032.html

>
> Regards,
> Luis Garcia
>
> On Wed, Dec 6, 2017 at 3:08 PM, Dave Taht <dave@taht.net> wrote:
>>
>>
>> Groovy.
>>
>> I guess if I had any one goal, was that I wanted to have cake to shape
>> well on the APU2 at a gigabit. That's the highest end "home router" we
>> have, although it would be good to get results on things like the turris
>> omnia, also.
>>
>> I hope to start a round of profiling as to what can be optimized (out)
>> this weekend. Last I looked the hashing costs dominated.
>>
>> Pretty sure we're good at 200Mbit there (aside from the ingress bug).
>>
>> It would be nice to have a test showing blue being useful.
>>
>> Did you get cpu stats for the below tests?
>>
>> Georgios Amanakis <gamanakis@gmail.com> writes:
>>
>> > -------- Forwarded Message --------
>> > From: Georgios Amanakis <gamanakis@gmail.com>
>> > To: Dave Taht <dave.taht@gmail.com>
>> > Cc: Cake List <cake@lists.bufferbloat.net>, Jonathan Morton <chromatix9
>> > 9@gmail.com>, Pete Heist <peteheist@gmail.com>
>> > Subject: Re: [Cake] RHODIUM - nuking blue (for testing)
>> > Date: Tue, 05 Dec 2017 20:58:39 -0500
>> >
>> > I just finished testing Rhodium, and I am supplementing Sunday's
>> > results.
>> >
>> > CMTS setup:
>> > server -- delay -- isp -- mbox -- client
>> >
>> > ISP limited at 200/10mbit.
>> > Cake/HTB shaping at 180/9mbit.
>> > RTT 20ms with cake's RTT set at 20ms
>> > rrul_be_nflows, 32 download, 8 upload streams.
>> > net.ipv4.tcp_ecn=2 in all nodes (linux default)
>> >
>> > Cake tested in all flow-isolation modes with all ack filtering options.
>> > HTB tested with codel,pie,fq,sfq,fq_codel.
>> >
>> > Rhodium(cakerhod) doesn't show much difference against Cobalt(cake).
>> >
>> > George
>> >
>> > On Tue, 2017-12-05 at 11:32 -0800, Dave Taht wrote:
>> >> Given some of the results thus far at various RTTs, and at higher
>> >> loads, like the ginormous one georgios just ran, I thought it might
>> >> be
>> >> useful to run a battery of tests with blue disabled.
>> >>
>> >> I haven't gone to the trouble of creating a formal branch - tho if I
>> >> did, I'd call it rhodium, as rhodium detectors are used in nuclear
>> >> reactors to measure the neutron flux level.
>> >>
>> >> untested-but-compiling-patch attached. I'm busy on a few other things
>> >> this week, and it is looking like we'll have to replace the xstats
>> >> structure with something smaller to go mainline.
>> >>
>> >> _______________________________________________
>> >> Cake mailing list
>> >> Cake@lists.bufferbloat.net
>> >> https://lists.bufferbloat.net/listinfo/cake
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > Cake mailing list
>> > Cake@lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/cake
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
>
>
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] [Fwd: Re: RHODIUM - nuking blue (for testing)]

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 755 bytes --]


> On Dec 7, 2017, at 2:15 AM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> On Wed, Dec 6, 2017 at 2:35 PM, Luis E. Garcia <luis@bitamins.net <mailto:luis@bitamins.net>> wrote:
>> Dave,
>> What is the highest bandwidth that you have been able to shape with CAKE on
>> the APU2?
> 
> pete ran out of cpu at 900mbit in one of the early test rounds.
> 
> https://lists.bufferbloat.net/pipermail/cake/2017-November/003032.html <https://lists.bufferbloat.net/pipermail/cake/2017-November/003032.html>

That may have been because I had only two boxes (both APU2) and was running flent + cake on one box and netserver + cake on the other box. I’m pretty sure the APU2 can handle Gbit if cake is all it’s doing, but I should give that a test…


[-- Attachment #2: Type: text/html, Size: 4882 bytes --]

Re: [Cake] [Fwd: Re: RHODIUM - nuking blue (for testing)]

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 378 bytes --]

> It would be nice to have a test showing blue being useful.

To do that, you would need to have a saturating unresponsive load, such as
a UDP flood, which drives Cake's queue to the memory limit.  Even then, I
suspect you wouldn't see much difference in the traffic stats, but there
might be a CPU advantage since the hard-drop path would be exercised less.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 449 bytes --]

Re: [Cake] [Fwd: Re: RHODIUM - nuking blue (for testing)]

[Kevin Darbyshire-Bryant]

> On 7 Dec 2017, at 02:19, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
> > It would be nice to have a test showing blue being useful.
> 
> To do that, you would need to have a saturating unresponsive load, such as a UDP flood, which drives Cake's queue to the memory limit.  Even then, I suspect you wouldn't see much difference in the traffic stats, but there might be a CPU advantage since the hard-drop path would be exercised less.

This reminds me of a recent discovery/mystery related to ECN, acks, Qnap NAS box and Microsoft onedrive.  I sent a packet dump to Dave for his thoughts.  But in summary, NAS box does file copies to MS Onedrive folder/s across a 19Mbit bit cake controlled egress link (ingress is 70Mbit ingress cake controlled, no large contending bandwidth). Upload uses 4 streams and I saw around 25 packet drops per second/per flow by cake to keep things under control.  What I found curious in combination with the ‘ack drop’ thread is that MS appears to send an ack per packet, not even ack per every 2nd packet.

I wondered if enabling ECN would improve things (ie mark instead of drop),  it didn’t.  Instead cake memory usage goes up to configured limit and it regards the flows as uncontrolled.  My view is that cake handles this particular abuse very well, latency through the link isn’t really affected, though the ‘peak, avg & sparse delay’ figures are remarkably high (peak around 1.6Seconds!  but then getting 4MB of data over 19Mbit is going to take time, unless you drop the packets due to unresponsive)

Curious.

Kevin

Re: [Cake] [Fwd: Re: RHODIUM - nuking blue (for testing)]

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 259 bytes --]

Smells like the NAS and OneDrive are using nonstandard, and at least
partially noncompliant TCP stacks.  Did the ack stream carry ECE flags?

But yes, that is exactly the sort of abuse that the BLUE portion of COBALT
is meant to cope with.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 319 bytes --]