[Cake] Testing variants of the MTU latency scaling

[Cake] Testing variants of the MTU latency scaling

[Toke Høiland-Jørgensen]

[-- Attachment #1: Type: text/plain, Size: 1872 bytes --]

So as promised I ran a bunch of tests with different variants of the MTU
minimum queue occupancy scaling feature that we were discussing.

I added instrumentation so the scaling can be turned on and off and set
at various levels (see the test-instrumentation branch). I then ran
tests with the minimum queue level (i.e., where the AQM turns off) set
to, respectively:

- 1 MTU
- 2 MTU
- 2 MTU x number of flows
- 4 MTU
- 4 MTU x number of flows

The "x number of flows" tests are referred to as "scaling" in the
dataset.

I ran the RRUL test and 32-flow TCP download and upload tests on a setup
where I installed Cake before the bottleneck link on upstream and on an
IFB in 'ingress' mode. The other side of the bottleneck link was a TBF
1000-packet FIFO with the same bandwidth setting. Cake had Ethernet
overhead compensation turned on, which kept the bottleneck under control
(as you can see by the latency results).

Full dataset is here:
https://kau.toke.dk/experiments/cake/cake-mtuscale.tar.gz



Takeaways (see attached plots):

- The MTU scaling does indeed give a nice benefit in egress mode
  "tcp-download-totals" plot. From just over 6 Mbps to just over 8 Mbps
  of goodput on the 10 Mbit link. There is not a large difference
  between 2MTU and 4MTU, except that 4MTU hurts inter-flow latency
  somewhat.

- The effect for upload flows (where Cake is before the bottleneck;
  10mbit-upload.png) is negligible.

- The MTU scaling really hurts TCP RTT (intra-flow latency;
  tcp-upload-tcprtt-10mbit.png and rrul-tcprtt.png).

- For bidirectional traffic the combined effect is also negligible.


Based on all this, I propose we change the scaling mechanism so that it
is only active in egress mode, and change it from 4 MTUs to 2. I'll
merge Kevin's patch to do this unless someone complains loudly :)

If you want me to run other tests, let me know.

-Toke


[-- Attachment #2: 10mbit-upload.png --]
[-- Type: image/png, Size: 135745 bytes --]

[-- Attachment #3: rrul-box.png --]
[-- Type: image/png, Size: 62707 bytes --]

[-- Attachment #4: rrul-icmp-cdf.png --]
[-- Type: image/png, Size: 55056 bytes --]

[-- Attachment #5: rrul-tcprtt.png --]
[-- Type: image/png, Size: 44198 bytes --]

[-- Attachment #6: tcp-download-totals.png --]
[-- Type: image/png, Size: 129915 bytes --]

[-- Attachment #7: tcp-upload-tcprtt-10mbit.png --]
[-- Type: image/png, Size: 386028 bytes --]

Re: [Cake] Testing variants of the MTU latency scaling

[Jonathan Morton]

> Takeaways (see attached plots):
> 
> - The MTU scaling does indeed give a nice benefit in egress mode
>  "tcp-download-totals" plot. From just over 6 Mbps to just over 8 Mbps
>  of goodput on the 10 Mbit link. There is not a large difference
>  between 2MTU and 4MTU, except that 4MTU hurts inter-flow latency
>  somewhat.
> 
> - The effect for upload flows (where Cake is before the bottleneck;
>  10mbit-upload.png) is negligible.
> 
> - The MTU scaling really hurts TCP RTT (intra-flow latency;
>  tcp-upload-tcprtt-10mbit.png and rrul-tcprtt.png).
> 
> - For bidirectional traffic the combined effect is also negligible.
> 
> 
> Based on all this, I propose we change the scaling mechanism so that it
> is only active in egress mode, and change it from 4 MTUs to 2. I'll
> merge Kevin's patch to do this unless someone complains loudly :)
> 
> If you want me to run other tests, let me know.

I'm not actually sure what you've measured here - unless you've somehow managed to swap "ingress" with "egress" mode in a strange manner.  I don't see any systematic measurement of the different MTU scales in ingress mode in your results, which makes your assertion that it should only be active in egress mode rather odd.

 - Jonathan Morton

Re: [Cake] Testing variants of the MTU latency scaling

[Toke Høiland-Jørgensen]

On 22 April 2018 23:09:54 CEST, Jonathan Morton <chromatix99@gmail.com> wrote:
>> Takeaways (see attached plots):
>> 
>> - The MTU scaling does indeed give a nice benefit in egress mode
>>  "tcp-download-totals" plot. From just over 6 Mbps to just over 8
>Mbps
>>  of goodput on the 10 Mbit link. There is not a large difference
>>  between 2MTU and 4MTU, except that 4MTU hurts inter-flow latency
>>  somewhat.
>> 
>> - The effect for upload flows (where Cake is before the bottleneck;
>>  10mbit-upload.png) is negligible.
>> 
>> - The MTU scaling really hurts TCP RTT (intra-flow latency;
>>  tcp-upload-tcprtt-10mbit.png and rrul-tcprtt.png).
>> 
>> - For bidirectional traffic the combined effect is also negligible.
>> 
>> 
>> Based on all this, I propose we change the scaling mechanism so that
>it
>> is only active in egress mode, and change it from 4 MTUs to 2. I'll
>> merge Kevin's patch to do this unless someone complains loudly :)
>> 
>> If you want me to run other tests, let me know.
>
>I'm not actually sure what you've measured here - unless you've somehow
>managed to swap "ingress" with "egress" mode in a strange manner.  I
>don't see any systematic measurement of the different MTU scales in
>ingress mode in your results, which makes your assertion that it should
>only be active in egress mode rather odd.

Ah, that was a typo. I meant that it should only be active in *ingress* mode. 

As for the results, as I wrote in my original email, all download flows go through cake in ingress mode (the device running cake is between the client and the bottleneck).

-Toke

Re: [Cake] Testing variants of the MTU latency scaling

[Jonathan Morton]

> Based on all this, I propose we change the scaling mechanism so that it
> is only active in egress mode, and change it from 4 MTUs to 2. I'll
> merge Kevin's patch to do this unless someone complains loudly :)

I suppose that makes enough sense.  But let me look over the patch again - there's probably a better way to do it.

 - Jonathan Morton

Re: [Cake] Testing variants of the MTU latency scaling

[Toke Høiland-Jørgensen]

Jonathan Morton <chromatix99@gmail.com> writes:

>> Based on all this, I propose we change the scaling mechanism so that it
>> is only active in egress mode, and change it from 4 MTUs to 2. I'll
>> merge Kevin's patch to do this unless someone complains loudly :)
>
> I suppose that makes enough sense.  But let me look over the patch
> again - there's probably a better way to do it.

Right. Pushed the patch to the cobalt branch, feel free to fix it up :)

-Toke

Re: [Cake] Testing variants of the MTU latency scaling

[Jonathan Morton]

> Right. Pushed the patch to the cobalt branch, feel free to fix it up :)

Given that I basically had to revert 80% of it and start again, perhaps not the best policy.

 - Jonathan Morton

Re: [Cake] Testing variants of the MTU latency scaling

[Jonas Mårtensson]

[-- Attachment #1: Type: text/plain, Size: 1724 bytes --]

One thing that is still not clear to me from these results: if I run cake
on an IFB without ingress mode (i.e. the default?), does the MTU scaling
have any impact on TCP download throughput?

/Jonas

On Sun, Apr 22, 2018 at 11:09 PM, Jonathan Morton <chromatix99@gmail.com>
wrote:

> > Takeaways (see attached plots):
> >
> > - The MTU scaling does indeed give a nice benefit in egress mode
> >  "tcp-download-totals" plot. From just over 6 Mbps to just over 8 Mbps
> >  of goodput on the 10 Mbit link. There is not a large difference
> >  between 2MTU and 4MTU, except that 4MTU hurts inter-flow latency
> >  somewhat.
> >
> > - The effect for upload flows (where Cake is before the bottleneck;
> >  10mbit-upload.png) is negligible.
> >
> > - The MTU scaling really hurts TCP RTT (intra-flow latency;
> >  tcp-upload-tcprtt-10mbit.png and rrul-tcprtt.png).
> >
> > - For bidirectional traffic the combined effect is also negligible.
> >
> >
> > Based on all this, I propose we change the scaling mechanism so that it
> > is only active in egress mode, and change it from 4 MTUs to 2. I'll
> > merge Kevin's patch to do this unless someone complains loudly :)
> >
> > If you want me to run other tests, let me know.
>
> I'm not actually sure what you've measured here - unless you've somehow
> managed to swap "ingress" with "egress" mode in a strange manner.  I don't
> see any systematic measurement of the different MTU scales in ingress mode
> in your results, which makes your assertion that it should only be active
> in egress mode rather odd.
>
>  - Jonathan Morton
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>

[-- Attachment #2: Type: text/html, Size: 2436 bytes --]

Re: [Cake] Testing variants of the MTU latency scaling

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 1900 bytes --]



> On 23 Apr 2018, at 11:50, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> Right. Pushed the patch to the cobalt branch, feel free to fix it up :)
> 
> Given that I basically had to revert 80% of it and start again, perhaps not the best policy.

Ha :-)

cobalt_should_drop(&flow->cvars, &b->cparams, now, skb,
                        b->bulk_flow_count * !!(q->rate_flags & CAKE_FLAG_INGRESS)

I like the conditional multiply (by 1) based on ingress mode, so we only pass a non zero flow count in ingress mode only.  Saves passing an additional parameter which I disliked but could see no way around.


I’m not sure that the replacement code in cobalt_should_drop itself is an equivalent of what Toke tested in his many combinations:

        bool over_target = sojourn > p->target && (
                           sojourn > p->mtu_time * bulk_flows * 2 ||
                           sojourn > p->mtu_time * 4 );

So in egress mode the first part of the OR statement will always be true (p->mtu_time * bulk_flows (0) * 2) is always 0, I’d like to think sojourn will always be bigger than 0.  So at run time it simplifies to

bool over_target = sojourn > p->target


ingress mode is more interesting since bulk_flows is now non zero.

sojourn > p->mtu_time * bulk_flows * 2 (I think in essence this is ‘permit up to 2 outstanding MTU packets(time) per flow overall) - this wasn’t tested
OR
sojourn > p->mtu_time * 4 ); (up to 4 MTU packets(time) on this flow only)   - this I believe was tested.

and this particular combination of the two wasn’t tested.   I wonder if I could politely ask Toke to run his test on this version.

I’m arguing/commenting from a position of ignorance and a double migraine yesterday, so to say I’m foggy today is an understatement!


Cheers,

Kevin D-B

012C ACB2 28C6 C53E 9775  9123 B3A2 389B 9DE2 334A


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Cake] Testing variants of the MTU latency scaling

[Jonathan Morton]

> So in egress mode the first part of the OR statement will always be true (p->mtu_time * bulk_flows (0) * 2) is always 0,

Yeah, that's what happens when I get rushed.  Let me see if I can fix the logic.

 - Jonathan Morton

Re: [Cake] Testing variants of the MTU latency scaling

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 584 bytes --]



> On 24 Apr 2018, at 09:14, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> So in egress mode the first part of the OR statement will always be true (p->mtu_time * bulk_flows (0) * 2) is always 0,
> 
> Yeah, that's what happens when I get rushed.  Let me see if I can fix the logic.

Actually I wonder if that’s a happy accident :-)   It drops any mtu/bulk flow related cleverness in egress mode completely, reserving the clever stuff for ingress only.

> 
> - Jonathan Morton
> 


Cheers,

Kevin D-B

012C ACB2 28C6 C53E 9775  9123 B3A2 389B 9DE2 334A


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Cake] Testing variants of the MTU latency scaling

[Toke Høiland-Jørgensen]

Jonas Mårtensson <martensson.jonas@gmail.com> writes:

> One thing that is still not clear to me from these results: if I run
> cake on an IFB without ingress mode (i.e. the default?), does the MTU
> scaling have any impact on TCP download throughput?

Odds are that not using ingress mode will make Cake lose control of the
bottleneck (that is what happened when I tried running a quick test),
and so will mess up both latency and throughput as you hit the bloated
upstream link buffer...

-Toke

Re: [Cake] Testing variants of the MTU latency scaling

[Jonas Mårtensson]

[-- Attachment #1: Type: text/plain, Size: 818 bytes --]

On Tue, Apr 24, 2018 at 1:45 PM, Toke Høiland-Jørgensen <toke@toke.dk>
wrote:

> Jonas Mårtensson <martensson.jonas@gmail.com> writes:
>
> > One thing that is still not clear to me from these results: if I run
> > cake on an IFB without ingress mode (i.e. the default?), does the MTU
> > scaling have any impact on TCP download throughput?
>
> Odds are that not using ingress mode will make Cake lose control of the
> bottleneck (that is what happened when I tried running a quick test),
> and so will mess up both latency and throughput as you hit the bloated
> upstream link buffer...


So using cake through sqm-scripts in OpenWRT/LEDE for ingress shaping does
not currently work very well then? I guess the sqm-scripts should be
updated to actually use ingress mode at some point...

/Jonas

[-- Attachment #2: Type: text/html, Size: 1245 bytes --]

Re: [Cake] Testing variants of the MTU latency scaling

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 1427 bytes --]

Assuming you’re using luci to configure then enabling both show and use advanced configuration & show and use dangerous configurations… then enter ‘ingress’ in the ‘advanced option string to pass to ingress queuing’ will enable ingress mode.

Maybe that helps?

> On 24 Apr 2018, at 20:04, Jonas Mårtensson <martensson.jonas@gmail.com> wrote:
> 
> 
> 
> On Tue, Apr 24, 2018 at 1:45 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> Jonas Mårtensson <martensson.jonas@gmail.com> writes:
> 
> > One thing that is still not clear to me from these results: if I run
> > cake on an IFB without ingress mode (i.e. the default?), does the MTU
> > scaling have any impact on TCP download throughput?
> 
> Odds are that not using ingress mode will make Cake lose control of the
> bottleneck (that is what happened when I tried running a quick test),
> and so will mess up both latency and throughput as you hit the bloated
> upstream link buffer...
> 
> So using cake through sqm-scripts in OpenWRT/LEDE for ingress shaping does not currently work very well then? I guess the sqm-scripts should be updated to actually use ingress mode at some point...
> 
> /Jonas
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake


Cheers,

Kevin D-B

012C ACB2 28C6 C53E 9775  9123 B3A2 389B 9DE2 334A


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Cake] Testing variants of the MTU latency scaling

[Jonas Mårtensson]

[-- Attachment #1: Type: text/plain, Size: 513 bytes --]

On Tue, Apr 24, 2018 at 9:22 PM, Kevin Darbyshire-Bryant <
kevin@darbyshire-bryant.me.uk> wrote:

> Assuming you’re using luci to configure then enabling both show and use
> advanced configuration & show and use dangerous configurations… then enter
> ‘ingress’ in the ‘advanced option string to pass to ingress queuing’ will
> enable ingress mode.
>
> Maybe that helps?


Just typing ingress in that box seems to disable cake and instead sets
default fq_codel as ingress qdisc.

/Jonas

[-- Attachment #2: Type: text/html, Size: 858 bytes --]

Re: [Cake] Testing variants of the MTU latency scaling

[Sebastian Moeller]

Hi Jonas,


> On Apr 24, 2018, at 22:27, Jonas Mårtensson <martensson.jonas@gmail.com> wrote:
> 
> 
> 
> On Tue, Apr 24, 2018 at 9:22 PM, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
> Assuming you’re using luci to configure then enabling both show and use advanced configuration & show and use dangerous configurations… then enter ‘ingress’ in the ‘advanced option string to pass to ingress queuing’ will enable ingress mode.

That is odd; could you post the output of:

cat /etc/config/sqm
tc -s qdisc

I am assuming openwrt/lede here otherwise, please let me know what distribution you use...

Best Regards
	Sebastian




> 
> Maybe that helps?
> 
> Just typing ingress in that box seems to disable cake and instead sets default fq_codel as ingress qdisc.
> 
> /Jonas 
> 
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake