[Cake] Fwd: cake flenter results round 3

[Cake] Fwd: cake flenter results round 3

[Georgios Amanakis]

[-- Attachment #1.1: Type: text/plain, Size: 942 bytes --]

I cannot attach the flent.gz files as they exceed the message size limit of
the list. Hopefully Dave got them.

---------- Forwarded message ----------
From: Georgios Amanakis <gamanakis@gmail.com>
Date: Fri, Dec 1, 2017 at 10:20 PM
Subject: Re: [Cake] cake flenter results round 3
To: Dave Taht <dave.taht@gmail.com>
Cc: Cake List <cake@lists.bufferbloat.net>


Just finished testing. CMTS Setup:
server   --   delay   --   isp   --   mbox   --   client
netserver     100ms      900mbit     810mbit      flent

I am attaching the setup, tried to implement all of Dave's suggestions.
I used the files:
vsetup.sh # topology
vcake.sh{.htb} # qdiscs
sshd.sh # start netserver, irtt etc.
stats.sh # generate the txt files with qdisc stats

Tested pfifo, cake, cake_flowblind, netem_codel, htb_codel, htb_pie,
htb_fq, htb_sfq.

In terms of latency cake_flowblind seems to be as good as
htb_{codel,pie,fq}.
htb_sfq outperforms it slightly.

George

[-- Attachment #1.2: Type: text/html, Size: 1556 bytes --]

[-- Attachment #2: box_totals.png --]
[-- Type: image/png, Size: 109302 bytes --]

[-- Attachment #3: setup.tgz --]
[-- Type: application/x-gzip, Size: 6777 bytes --]

[Cake] cake flenter results round 3

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 1836 bytes --]

http://www.drhleny.cz/bufferbloat/cake/round3/ <http://www.drhleny.cz/bufferbloat/cake/round3/>

Round 3 Tarball: http://www.drhleny.cz/bufferbloat/cake/round3.tgz <http://www.drhleny.cz/bufferbloat/cake/round3.tgz>

*** Notes/Analysis ***

* And another couple hundred tests showing Cake doesn’t crash, at 20/20mbit either!

* I’m now using netns (thanks Dave/Georgios) with a five box setup: client, client router, delay, server router and server. Delay is set up to simulate a 20mbit/20mbit symmetric connection with “bandwidth 20mbit delay 10ms” on each of its two interfaces. That apparently doesn’t hard limit the bandwidth, but no matter, I do get a 20ms RTT as expected, and any tests with rate limiting use a default limit of 18mbit, which seems to control the queue well. The unlimited tests can probably be ignored.

* Now that my CPU can actually run this test, Cake shows good results vs fq_codel on 32/32 nflows:

http://www.drhleny.cz/bufferbloat/cake/round3/nflows_32_32_eg_fq_codel_18.0mbit/index.html
http://www.drhleny.cz/bufferbloat/cake/round3/nflows_32_32_eg_cake_18.0mbit/index.html

* OWD, IPDV (jitter) and lost packets looking good for a VoIP conversation with dual-whatever. The VoIP conversation is simulated along with a normal RRUL test (4/4 flows), a torrent, and tcp_12down simultaneously (family of four?)

http://www.drhleny.cz/bufferbloat/cake/round3/flowiso_mix_eg_cake_dsrc_cake_ddst_18.0mbit/voip-voip_delay_loss.svg

*** Round 4 Plans

* All I can do easily with this setup without more effort is vary rtt, bandwidth, loss or whatever else netem can do. Or I can add other tests with spreads of parameters. Sorry, no asymmetric rates or useful ack filtering tests. Anyone care for high rtt tests, low bandwidth tests like 2/2mbit, or anything else in particular?


[-- Attachment #2: Type: text/html, Size: 2868 bytes --]

Re: [Cake] cake flenter results round 3

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 292 bytes --]

High RTT with relatively high bandwidth is traditionally considered
challenging, and is representative of Australasian and satellite
connections.  Definitely do some of those.

Low bandwidth should also be interesting.  DSL at 4/1 Mbit is typical entry
level package here.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 353 bytes --]

Re: [Cake] cake flenter results round 3

[Georgios Amanakis]

[-- Attachment #1.1: Type: text/plain, Size: 884 bytes --]

I gave high RTT with high bandwidth a try:
server   --   delay  --  mbox  --  client
netserver   300/300ms  45/900mbit  flent

I had to run flent with "-s 0.61" in order to avoid errors with fping dying
prematurely.
Comparing noack, ack, and ack-aggressive, ack seems to give the highest
download rates.
Cake was setup with "rtt 600ms".

George

On Wed, Nov 29, 2017 at 6:25 PM, Jonathan Morton <chromatix99@gmail.com>
wrote:

> High RTT with relatively high bandwidth is traditionally considered
> challenging, and is representative of Australasian and satellite
> connections.  Definitely do some of those.
>
> Low bandwidth should also be interesting.  DSL at 4/1 Mbit is typical
> entry level package here.
>
> - Jonathan Morton
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>
>

[-- Attachment #1.2: Type: text/html, Size: 1620 bytes --]

[-- Attachment #2: rrul_delay600ms_ack_noack.tgz --]
[-- Type: application/x-gzip, Size: 243475 bytes --]

[-- Attachment #3: totals.png --]
[-- Type: image/png, Size: 126979 bytes --]

Re: [Cake] cake flenter results round 3

[Dave Taht]

That's really hard to argue with. An fq_codel same conditions (and
pfifo and pie) might be interesting here.

On Thu, Nov 30, 2017 at 9:13 AM, Georgios Amanakis <gamanakis@gmail.com> wrote:
> I gave high RTT with high bandwidth a try:
> server   --   delay  --  mbox  --  client
> netserver   300/300ms  45/900mbit  flent
>
> I had to run flent with "-s 0.61" in order to avoid errors with fping dying
> prematurely.
> Comparing noack, ack, and ack-aggressive, ack seems to give the highest
> download rates.
> Cake was setup with "rtt 600ms".
>
> George
>
> On Wed, Nov 29, 2017 at 6:25 PM, Jonathan Morton <chromatix99@gmail.com>
> wrote:
>>
>> High RTT with relatively high bandwidth is traditionally considered
>> challenging, and is representative of Australasian and satellite
>> connections.  Definitely do some of those.
>>
>> Low bandwidth should also be interesting.  DSL at 4/1 Mbit is typical
>> entry level package here.
>>
>> - Jonathan Morton
>>
>>
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
>>
>
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] cake flenter results round 3

[Georgios Amanakis]

[-- Attachment #1: Type: text/plain, Size: 2292 bytes --]

cake vs sfq and fq_codel looks great! You already have fq_codel in the
graphs, do you mean codel?

In terms of simulating CMTS I am trying to simulate a cable connection.
Is the following reasonable?
===========8<===========
servers --- delay --- isp --- mbox --- clients

delay:
tc qdisc replace dev delay.r root netem delay ZZms rate XXXmbit limit 100000
tc qdisc replace dev delay.l root netem delay ZZms rate XXXmbit limit 100000

isp:
tc qdisc replace dev isp.l root bfifo limit 1024k
tc qdisc replace dev isp.r root bfifo limit 1024k

mbox:
tc qdisc replace dev mbox.l root cake bandwidth YYYmbit
tc qdisc replace dev mbox.r root cake bandwidth YYYmbit
===========8<===========

George

On Fri, Dec 1, 2017 at 2:23 PM, Dave Taht <dave.taht@gmail.com> wrote:

> That's really hard to argue with. An fq_codel same conditions (and
> pfifo and pie) might be interesting here.
>
> On Thu, Nov 30, 2017 at 9:13 AM, Georgios Amanakis <gamanakis@gmail.com>
> wrote:
> > I gave high RTT with high bandwidth a try:
> > server   --   delay  --  mbox  --  client
> > netserver   300/300ms  45/900mbit  flent
> >
> > I had to run flent with "-s 0.61" in order to avoid errors with fping
> dying
> > prematurely.
> > Comparing noack, ack, and ack-aggressive, ack seems to give the highest
> > download rates.
> > Cake was setup with "rtt 600ms".
> >
> > George
> >
> > On Wed, Nov 29, 2017 at 6:25 PM, Jonathan Morton <chromatix99@gmail.com>
> > wrote:
> >>
> >> High RTT with relatively high bandwidth is traditionally considered
> >> challenging, and is representative of Australasian and satellite
> >> connections.  Definitely do some of those.
> >>
> >> Low bandwidth should also be interesting.  DSL at 4/1 Mbit is typical
> >> entry level package here.
> >>
> >> - Jonathan Morton
> >>
> >>
> >> _______________________________________________
> >> Cake mailing list
> >> Cake@lists.bufferbloat.net
> >> https://lists.bufferbloat.net/listinfo/cake
> >>
> >
> >
> > _______________________________________________
> > Cake mailing list
> > Cake@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cake
> >
>
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619
>

[-- Attachment #2: Type: text/html, Size: 3915 bytes --]

Re: [Cake] cake flenter results round 3

[Dave Taht]

On Fri, Dec 1, 2017 at 11:29 AM, Georgios Amanakis <gamanakis@gmail.com> wrote:
> cake vs sfq and fq_codel looks great! You already have fq_codel in the
> graphs, do you mean codel?
>
> In terms of simulating CMTS I am trying to simulate a cable connection.
> Is the following reasonable?
> ===========8<===========
> servers --- delay --- isp --- mbox --- clients
>
> delay:
> tc qdisc replace dev delay.r root netem delay ZZms rate XXXmbit limit 100000
> tc qdisc replace dev delay.l root netem delay ZZms rate XXXmbit limit 100000

There is not much need to specify the rate here, just the delay. You
could like say 10Gbit, if you wanted.

> isp:
> tc qdisc replace dev isp.l root bfifo limit 1024k
> tc qdisc replace dev isp.r root bfifo limit 1024k

No, because this is not the bottleneck queue. isp.r

    tc qdisc del dev isp.r root # a little uncertain of what happens with handle
    tc qdisc replace dev isp.r root handle 1: netem rate YYYmbit \
             limit 100000
    tc qdisc add dev isp.r parent 1:1 handle 10:1 bfifo limit 1024k

I think.

> mbox:
> tc qdisc replace dev mbox.l root cake bandwidth YYYmbit
> tc qdisc replace dev mbox.r root cake bandwidth YYYmbit

    tc qdisc del dev mbox.r root # to leave it unshaped
    # or put cake back in to shape it after the isp shaper above.



> ===========8<===========
>
> George
>
> On Fri, Dec 1, 2017 at 2:23 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> That's really hard to argue with. An fq_codel same conditions (and
>> pfifo and pie) might be interesting here.
>>
>> On Thu, Nov 30, 2017 at 9:13 AM, Georgios Amanakis <gamanakis@gmail.com>
>> wrote:
>> > I gave high RTT with high bandwidth a try:
>> > server   --   delay  --  mbox  --  client
>> > netserver   300/300ms  45/900mbit  flent
>> >
>> > I had to run flent with "-s 0.61" in order to avoid errors with fping
>> > dying
>> > prematurely.
>> > Comparing noack, ack, and ack-aggressive, ack seems to give the highest
>> > download rates.
>> > Cake was setup with "rtt 600ms".
>> >
>> > George
>> >
>> > On Wed, Nov 29, 2017 at 6:25 PM, Jonathan Morton <chromatix99@gmail.com>
>> > wrote:
>> >>
>> >> High RTT with relatively high bandwidth is traditionally considered
>> >> challenging, and is representative of Australasian and satellite
>> >> connections.  Definitely do some of those.
>> >>
>> >> Low bandwidth should also be interesting.  DSL at 4/1 Mbit is typical
>> >> entry level package here.
>> >>
>> >> - Jonathan Morton
>> >>
>> >>
>> >> _______________________________________________
>> >> Cake mailing list
>> >> Cake@lists.bufferbloat.net
>> >> https://lists.bufferbloat.net/listinfo/cake
>> >>
>> >
>> >
>> > _______________________________________________
>> > Cake mailing list
>> > Cake@lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/cake
>> >
>>
>>
>>
>> --
>>
>> Dave Täht
>> CEO, TekLibre, LLC
>> http://www.teklibre.com
>> Tel: 1-669-226-2619
>
>



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] cake flenter results round 3

[Dave Taht]

On Fri, Dec 1, 2017 at 11:45 AM, Dave Taht <dave.taht@gmail.com> wrote:
> On Fri, Dec 1, 2017 at 11:29 AM, Georgios Amanakis <gamanakis@gmail.com> wrote:
>> cake vs sfq and fq_codel looks great! You already have fq_codel in the
>> graphs, do you mean codel?

Heh. I meant codel, yes.

>> In terms of simulating CMTS I am trying to simulate a cable connection.
>> Is the following reasonable?
>> ===========8<===========
>> servers --- delay --- isp --- mbox --- clients
>>
>> delay:
>> tc qdisc replace dev delay.r root netem delay ZZms rate XXXmbit limit 100000
>> tc qdisc replace dev delay.l root netem delay ZZms rate XXXmbit limit 100000
>
> There is not much need to specify the rate here, just the delay. You
> could like say 10Gbit, if you wanted.
>
>> isp:
>> tc qdisc replace dev isp.l root bfifo limit 1024k
>> tc qdisc replace dev isp.r root bfifo limit 1024k
>
> No, because this is not the bottleneck queue. isp.r
>
>     tc qdisc del dev isp.r root # a little uncertain of what happens with handle
>     tc qdisc replace dev isp.r root handle 1: netem rate YYYmbit \
>              limit 100000
>     tc qdisc add dev isp.r parent 1:1 handle 10:1 bfifo limit 1024k
>
> I think.
>
>> mbox:
>> tc qdisc replace dev mbox.l root cake bandwidth YYYmbit
>> tc qdisc replace dev mbox.r root cake bandwidth YYYmbit
>
>     tc qdisc del dev mbox.r root # to leave it unshaped
>     # or put cake back in to shape it after the isp shaper above.
>
>
>
>> ===========8<===========
>>
>> George
>>
>> On Fri, Dec 1, 2017 at 2:23 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>>
>>> That's really hard to argue with. An fq_codel same conditions (and
>>> pfifo and pie) might be interesting here.
>>>
>>> On Thu, Nov 30, 2017 at 9:13 AM, Georgios Amanakis <gamanakis@gmail.com>
>>> wrote:
>>> > I gave high RTT with high bandwidth a try:
>>> > server   --   delay  --  mbox  --  client
>>> > netserver   300/300ms  45/900mbit  flent
>>> >
>>> > I had to run flent with "-s 0.61" in order to avoid errors with fping
>>> > dying
>>> > prematurely.
>>> > Comparing noack, ack, and ack-aggressive, ack seems to give the highest
>>> > download rates.
>>> > Cake was setup with "rtt 600ms".
>>> >
>>> > George
>>> >
>>> > On Wed, Nov 29, 2017 at 6:25 PM, Jonathan Morton <chromatix99@gmail.com>
>>> > wrote:
>>> >>
>>> >> High RTT with relatively high bandwidth is traditionally considered
>>> >> challenging, and is representative of Australasian and satellite
>>> >> connections.  Definitely do some of those.
>>> >>
>>> >> Low bandwidth should also be interesting.  DSL at 4/1 Mbit is typical
>>> >> entry level package here.
>>> >>
>>> >> - Jonathan Morton
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> Cake mailing list
>>> >> Cake@lists.bufferbloat.net
>>> >> https://lists.bufferbloat.net/listinfo/cake
>>> >>
>>> >
>>> >
>>> > _______________________________________________
>>> > Cake mailing list
>>> > Cake@lists.bufferbloat.net
>>> > https://lists.bufferbloat.net/listinfo/cake
>>> >
>>>
>>>
>>>
>>> --
>>>
>>> Dave Täht
>>> CEO, TekLibre, LLC
>>> http://www.teklibre.com
>>> Tel: 1-669-226-2619
>>
>>
>
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] cake flenter results round 3

[Georgios Amanakis]

[-- Attachment #1.1: Type: text/plain, Size: 929 bytes --]

Thank you Dave!
I am playing around with this setup:
server -- delay -- isp -- mbox -- client

delay:
tc qdisc replace dev ${host}.l root netem delay 50ms limit 100000
tc qdisc replace dev ${host}.r root netem delay 50ms limit 100000

isp:
tc qdisc replace dev ${host}.l root handle 1:0 netem rate 900mbit limit
100000
tc qdisc replace dev ${host}.r root handle 1:0 netem rate 900mbit limit
100000
tc qdisc replace dev ${host}.r parent 1:1 handle 10:1 bfifo limit 1024k


I played with mbox's qdiscs (3 variations).
mbox:
tc qdisc replace dev ${host}.l root cake bandwidth 890mbit
tc qdisc replace dev ${host}.r root cake bandwidth 890mbit


mbox:
tc qdisc replace dev ${host}.l root codel
tc qdisc replace dev ${host}.r root codel


mbox:
tc qdisc replace dev ${host}.l root fq maxrate 890mbit
tc qdisc replace dev ${host}.r root fq maxrate 890mbit


rrul_be_nflows, 32 upstream, 32 downstream. In every case cake wins.

George

[-- Attachment #1.2: Type: text/html, Size: 1214 bytes --]

[-- Attachment #2: box_totals_cake_codel.png --]
[-- Type: image/png, Size: 66959 bytes --]

[-- Attachment #3: box_totals_cake_fq.png --]
[-- Type: image/png, Size: 60263 bytes --]

[-- Attachment #4: rrul_be_nflows-2017-12-01T160646.489200.nflows_vs_rate_32_32_cake_890mbit.flent.gz --]
[-- Type: application/x-gzip, Size: 976251 bytes --]

[-- Attachment #5: rrul_be_nflows-2017-12-01T160849.976928.nflows_vs_rate_32_32_codel_900mbit.flent.gz --]
[-- Type: application/x-gzip, Size: 960411 bytes --]

[-- Attachment #6: rrul_be_nflows-2017-12-01T161606.795041.nflows_vs_rate_32_32_fq_890mbit.flent.gz --]
[-- Type: application/x-gzip, Size: 850898 bytes --]

[-- Attachment #7: tcp_rtt_cake_codel.png --]
[-- Type: image/png, Size: 182506 bytes --]

[-- Attachment #8: tcp_rtt_cake_fq.png --]
[-- Type: image/png, Size: 165338 bytes --]

[Cake] Fwd: cake flenter results round 3

[Georgios Amanakis]

[-- Attachment #1.1: Type: text/plain, Size: 1680 bytes --]

---------- Forwarded message ----------
From: Georgios Amanakis <gamanakis@gmail.com>
Date: Fri, Dec 1, 2017 at 8:25 PM
Subject: Re: [Cake] cake flenter results round 3
To: Dave Taht <dave.taht@gmail.com>


The only thing in the CMTS setup 900mbit that appears to be close to cake
is rate limiting with netem and applying codel
 as sub-qdisc.

host=mbox
tc qdisc replace dev ${host}.l root handle 1:0 netem rate 890mbit
tc qdisc replace dev ${host}.r root handle 1:0 netem rate 890mbit
tc qdisc replace dev ${host}.l parent 1:1 handle 10:1 codel
tc qdisc replace dev ${host}.r parent 1:1 handle 10:1 codel


On Fri, Dec 1, 2017 at 4:41 PM, Georgios Amanakis <gamanakis@gmail.com>
wrote:

> Thank you Dave!
> I am playing around with this setup:
> server -- delay -- isp -- mbox -- client
>
> delay:
> tc qdisc replace dev ${host}.l root netem delay 50ms limit 100000
> tc qdisc replace dev ${host}.r root netem delay 50ms limit 100000
>
> isp:
> tc qdisc replace dev ${host}.l root handle 1:0 netem rate 900mbit limit
> 100000
> tc qdisc replace dev ${host}.r root handle 1:0 netem rate 900mbit limit
> 100000
> tc qdisc replace dev ${host}.r parent 1:1 handle 10:1 bfifo limit 1024k
>
>
> I played with mbox's qdiscs (3 variations).
> mbox:
> tc qdisc replace dev ${host}.l root cake bandwidth 890mbit
> tc qdisc replace dev ${host}.r root cake bandwidth 890mbit
>
>
> mbox:
> tc qdisc replace dev ${host}.l root codel
> tc qdisc replace dev ${host}.r root codel
>
>
> mbox:
> tc qdisc replace dev ${host}.l root fq maxrate 890mbit
> tc qdisc replace dev ${host}.r root fq maxrate 890mbit
>
>
> rrul_be_nflows, 32 upstream, 32 downstream. In every case cake wins.
>
> George
>
>

[-- Attachment #1.2: Type: text/html, Size: 2518 bytes --]

[-- Attachment #2: tcp_rtt_cake_netem_codel.png --]
[-- Type: image/png, Size: 172662 bytes --]

[-- Attachment #3: box_totals_cake_netem_codel.png --]
[-- Type: image/png, Size: 58567 bytes --]

[-- Attachment #4: rrul_be_nflows-2017-12-01T201958.740500.nflows_vs_rate_32_32_netem_codel_900mbit.flent.gz --]
[-- Type: application/x-gzip, Size: 968265 bytes --]

[-- Attachment #5: rrul_be_nflows-2017-12-01T160646.489200.nflows_vs_rate_32_32_cake_890mbit.flent.gz --]
[-- Type: application/x-gzip, Size: 976251 bytes --]