[Cake] cake target corner cases?

[Cake] cake target corner cases?

[Sebastian Moeller]

Dear cake committee,

I just played around with the most recent sch_cake and noticed:

user@computer:~/CODE/tc-adv/tc> sudo tc-adv qdisc del dev eth0 root
user@computer:~/CODE/tc-adv/tc> sudo tc-adv qdisc replace dev eth0 root cake bandwidth 1Mbit ; sudo tc-adv -s qdisc
qdisc cake 8005: dev eth0 root refcnt 6 bandwidth 1Mbit diffserv4 flows rtt 100.0ms raw 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
capacity estimate: 1Mbit
             Tin 0       Tin 1       Tin 2       Tin 3  
  thresh       1Mbit   937504bit     750Kbit     250Kbit
  target      18.2ms      19.4ms      24.2ms      72.7ms
interval     145.3ms     155.0ms     193.8ms     581.4ms
Pk-delay         0us         0us         0us         0us
Av-delay         0us         0us         0us         0us
Sp-delay         0us         0us         0us         0us
  pkts             0           0           0           0
  bytes            0           0           0           0
way-inds           0           0           0           0
way-miss           0           0           0           0
way-cols           0           0           0           0
  drops            0           0           0           0
  marks            0           0           0           0
Sp-flows           0           0           0           0
Bk-flows           0           0           0           0
last-len           0           0           0           0
max-len            0           0           0           0

Here target is always 12.5% of interval instead of the expected 6.25%
1/16 = 0.0625
72.7/581.4 = 0.125042999656
24.2/193.8 = 0.124871001032
19.4/155.0 = 0.125161290323
18.2/145.3 = 0.125258086717
But the bandwidth is really low, so pushing target closer to the bandwidth conserving side of the codel rationale might be fine, since latency is bad to begin with and bandwidth also pretty scarce. But it might be interesting to do a few more measurements at low bandwidths to confirm that the 12.5% of interval logic holds water; one could also argue that people with such links (a lot of DSL lines have even less upload, so this certainly is not extreme) might think that any added ms of delay matters (more than bandwidth); currently we leave the user no remedy...





user@computer:~/CODE/tc-adv/tc> sudo tc-adv qdisc del dev eth0 root
user@computer:~/CODE/tc-adv/tc> sudo tc-adv qdisc replace dev eth0 root cake bandwidth 10Mbit ; sudo tc-adv -s qdisc
qdisc cake 8006: dev eth0 root refcnt 6 bandwidth 10Mbit diffserv4 flows rtt 100.0ms raw 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
capacity estimate: 10Mbit
             Tin 0       Tin 1       Tin 2       Tin 3  
  thresh      10Mbit    9375Kbit    7500Kbit    2500Kbit
  target       6.2ms       6.2ms       6.2ms       7.3ms
interval     100.0ms     100.0ms     100.0ms     101.0ms
Pk-delay         0us         0us         0us         0us
Av-delay         0us         0us         0us         0us
Sp-delay         0us         0us         0us         0us
  pkts             0           0           0           0
  bytes            0           0           0           0
way-inds           0           0           0           0
way-miss           0           0           0           0
way-cols           0           0           0           0
  drops            0           0           0           0
  marks            0           0           0           0
Sp-flows           0           0           0           0
Bk-flows           0           0           0           0
last-len           0           0           0           0
max-len            0           0           0           0

This looks okay, except Tin3 has target at 7.3/101.0 = 0.0722772277228 7% of interval.



Both observations might actually be on purpose, but if so we should document that behavior as expected, for example in the man page…

Best Regards
	Sebastian

Re: [Cake] cake target corner cases?

[Alan Jenkins]

On 01/11/2015, Sebastian Moeller <moeller0@gmx.de> wrote:
> Dear cake committee,
>
> I just played around with the most recent sch_cake and noticed:
>
> user@computer:~/CODE/tc-adv/tc> sudo tc-adv qdisc del dev eth0 root
> user@computer:~/CODE/tc-adv/tc> sudo tc-adv qdisc replace dev eth0 root cake
> bandwidth 1Mbit ; sudo tc-adv -s qdisc
> qdisc cake 8005: dev eth0 root refcnt 6 bandwidth 1Mbit diffserv4 flows rtt
> 100.0ms raw
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
> capacity estimate: 1Mbit
>              Tin 0       Tin 1       Tin 2       Tin 3
>   thresh       1Mbit   937504bit     750Kbit     250Kbit
>   target      18.2ms      19.4ms      24.2ms      72.7ms
> interval     145.3ms     155.0ms     193.8ms     581.4ms


> Here target is always 12.5% of interval instead of the expected 6.25%
> 1/16 = 0.0625
> 72.7/581.4 = 0.125042999656
> 24.2/193.8 = 0.124871001032
> 19.4/155.0 = 0.125161290323
> 18.2/145.3 = 0.125258086717
> But the bandwidth is really low, so pushing target closer to the bandwidth
> conserving side of the codel rationale might be fine,

Pretty sure it's a minimum derived from the MTU

((mtu=1.5kbyte) * 8 bits/byte) / 1000 Mbit/s = 0.012s

except I don't know where the .5 comes from, that's incredibly
suspicious to have a round 1/8th :).

The point is that if buffering falls below the MTU, the connection
will be completely clobbered.

In a way it's nice cake reports this in the target.  Otherwise cake
would claim the target is 5ms, but measurements would show the
effective target is more than twice as high.

> since latency is bad
> to begin with and bandwidth also pretty scarce. But it might be interesting
> to do a few more measurements at low bandwidths to confirm that the 12.5% of
> interval logic holds water; one could also argue that people with such links
> (a lot of DSL lines have even less upload, so this certainly is not extreme)
> might think that any added ms of delay matters (more than bandwidth);
> currently we leave the user no remedy...
>
>

<snip>

> This looks okay, except Tin3 has target at 7.3/101.0 = 0.0722772277228 7% of
> interval.

Looks like the same thing.


> Both observations might actually be on purpose, but if so we should document
> that behavior as expected, for example in the man page…
>
> Best Regards
> 	Sebastian


I'm afraid I can't help mention my old niggle :).  _If_ you mention
this alongside instructions for RRUL, I think you'd also want to
explain^W mention the measurement increase for diffserv4 v.s.
besteffort.

I think the ICMP ping measurement increases by another 10ms on my
connection (11500k down /  850k up, so an mtu is ~15ms).  I concluded
it was inherent in prioritization.  Now I guess it's equal to the sum
of target * bandwidth_fraction for each class "above" icmp ping (and
could be tested).

I have graphs from sqm with and without classification.  I did test
cake once and I think it's the same (otherwise would be a bug).

https://dl.dropboxusercontent.com/u/49925445/bufferbloat.net/220-cdf-531414.sqm_simplest_11500_850_atm40_udppingfix.svg

https://dl.dropboxusercontent.com/u/49925445/bufferbloat.net/221-cdf-360505.sqm_simple_11500_850_atm40_udppingfix.svg

Warm regards
Alan

Re: [Cake] cake target corner cases?

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 3904 bytes --]


Can't give full explanation now but the bytes per ns calculation which is used as a basis for target on 'slow' links uses mtu*1.5

--
Cheers,

Kevin@Darbyshire-Bryant.me.uk

> On 1 Nov 2015, at 20:58, Alan Jenkins <alan.christopher.jenkins@gmail.com> wrote:
> 
>> On 01/11/2015, Sebastian Moeller <moeller0@gmx.de> wrote:
>> Dear cake committee,
>> 
>> I just played around with the most recent sch_cake and noticed us:
>> 
>> user@computer:~/CODE/tc-adv/tc> sudo tc-adv qdisc del dev eth0 root
>> user@computer:~/CODE/tc-adv/tc> sudo tc-adv qdisc replace dev eth0 root cake
>> bandwidth 1Mbit ; sudo tc-adv -s qdisc
>> qdisc cake 8005: dev eth0 root refcnt 6 bandwidth 1Mbit diffserv4 flows rtt
>> 100.0ms raw
>> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>> backlog 0b 0p requeues 0
>> capacity estimate: 1Mbit
>>             Tin 0       Tin 1       Tin 2       Tin 3
>>  thresh       1Mbit   937504bit     750Kbit     250Kbit
>>  target      18.2ms      19.4ms      24.2ms      72.7ms
>> interval     145.3ms     155.0ms     193.8ms     581.4ms
> 
> 
>> Here target is always 12.5% of interval instead of the expected 6.25%
>> 1/16 = 0.0625
>> 72.7/581.4 = 0.125042999656
>> 24.2/193.8 = 0.124871001032
>> 19.4/155.0 = 0.125161290323
>> 18.2/145.3 = 0.125258086717
>> But the bandwidth is really low, so pushing target closer to the bandwidth
>> conserving side of the codel rationale might be fine,
> 
> Pretty sure it's a minimum derived from the MTU
> 
> ((mtu=1.5kbyte) * 8 bits/byte) / 1000 Mbit/s = 0.012s
> 
> except I don't know where the .5 comes from, that's incredibly
> suspicious to have a round 1/8th :).
> 
> The point is that if buffering falls below the MTU, the connection
> will be completely clobbered.
> 
> In a way it's nice cake reports this in the target.  Otherwise cake
> would claim the target is 5ms, but measurements would show the
> effective target is more than twice as high.
> 
>> since latency is bad
>> to begin with and bandwidth also pretty scarce. But it might be interesting
>> to do a few more measurements at low bandwidths to confirm that the 12.5% of
>> interval logic holds water; one could also argue that people with such links
>> (a lot of DSL lines have even less upload, so this certainly is not extreme)
>> might think that any added ms of delay matters (more than bandwidth);
>> currently we leave the user no remedy...
>> 
>> 
> 
> <snip>
> 
>> This looks okay, except Tin3 has target at 7.3/101.0 = 0.0722772277228 7% of
>> interval.
> 
> Looks like the same thing.
> 
> 
>> Both observations might actually be on purpose, but if so we should document
>> that behavior as expected, for example in the man page…
>> 
>> Best Regards
>>    Sebastian
> 
> 
> I'm afraid I can't help mention my old niggle :).  _If_ you mention
> this alongside instructions for RRUL, I think you'd also want to
> explain^W mention the measurement increase for diffserv4 v.s.
> besteffort.
> 
> I think the ICMP ping measurement increases by another 10ms on my
> connection (11500k down /  850k up, so an mtu is ~15ms).  I concluded
> it was inherent in prioritization.  Now I guess it's equal to the sum
> of target * bandwidth_fraction for each class "above" icmp ping (and
> could be tested).
> 
> I have graphs from sqm with and without classification.  I did test
> cake once and I think it's the same (otherwise would be a bug).
> 
> https://dl.dropboxusercontent.com/u/49925445/bufferbloat.net/220-cdf-531414.sqm_simplest_11500_850_atm40_udppingfix.svg
> 
> https://dl.dropboxusercontent.com/u/49925445/bufferbloat.net/221-cdf-360505.sqm_simple_11500_850_atm40_udppingfix.svg
> 
> Warm regards
> Alan
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 3062 bytes --]

Re: [Cake] cake target corner cases?

[Alan Jenkins]

On 01/11/2015, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
>
> Can't give full explanation now but the bytes per ns calculation which is
> used as a basis for target on 'slow' links uses mtu*1.5
>
> --
> Cheers,

Right.  And mtu*1.5 matches better than my math, if I look at Sebs
figures again.  Oops :).

I thought SQM works fine with mtu * 1.0.  But cake has been like this
since at least April, and I didn't notice an extra 7ms when I tried
it.  Confusing.

Sebs 12.5% is indeed 1/8th.  (See last line).


	u32 byte_target = mtu + (mtu >> 1);	/* MTU * 1.5 */
...
	byte_target_ns = (byte_target * rate_ns) >> rate_shft;

	b->cparams.target = max(byte_target_ns, ns_target);
	b->cparams.interval = max(rtt_est_ns +
				     b->cparams.target - ns_target,
				     b->cparams.target * 8);

Maybe this keeps the target/interval ratio from going too far over the
recommended 10%.  Though that's not what CoDel says to do.

In the CoDel draft you just don't drop below 1 MTU.  You don't adjust
target for the rate.  There's no suggestion to increase interval
either.  It makes sense that long transmission times add to the
expected rtt (SQM does this too).  I can't see what the justification
is for `target * 8` though.  If there's a good reason maybe it could
use a comment.

Thanks
Alan

Re: [Cake] cake target corner cases?

[Sebastian Moeller]

Hi Alan,

On Nov 2, 2015, at 01:20 , Alan Jenkins <alan.christopher.jenkins@gmail.com> wrote:

> On 01/11/2015, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
>> 
>> Can't give full explanation now but the bytes per ns calculation which is
>> used as a basis for target on 'slow' links uses mtu*1.5
>> 
>> --
>> Cheers,
> 
> Right.  And mtu*1.5 matches better than my math, if I look at Sebs
> figures again.  Oops :).
> 
> I thought SQM works fine with mtu * 1.0.  But cake has been like this
> since at least April, and I didn't notice an extra 7ms when I tried
> it.  Confusing.

	Well, sqm-scripts does take an MTU of 1540 bytes, which is less than ideal because for ATM encapsulation the on wire size is actually 32 or 32 * 53, so the worst case should be 33*53 = 1749 bytes, but in reality the actual limit itself only needs to be approximately right.. That said, I guess I fixed up sqm-scripts right now (by making target large enough for one 1749 byte packet, I would be delighted if you could go measure whether this changes things, before increasing auto-target to 1.5 worst case on-wire-packets, and also scaling interval so that target <= 0.1*interval stays true)...

> 
> Sebs 12.5% is indeed 1/8th.  (See last line).
> 
> 
> 	u32 byte_target = mtu + (mtu >> 1);	/* MTU * 1.5 */
> ...
> 	byte_target_ns = (byte_target * rate_ns) >> rate_shft;
> 
> 	b->cparams.target = max(byte_target_ns, ns_target);
> 	b->cparams.interval = max(rtt_est_ns +
> 				     b->cparams.target - ns_target,
> 				     b->cparams.target * 8);

	I sort of cheated, I knew this was in the code, but I wanted to highlight that we need a better description of what cake does to not confuse future users, sorry for that.

> 
> Maybe this keeps the target/interval ratio from going too far over the
> recommended 10%.  Though that's not what CoDel says to do.
> 
> In the CoDel draft you just don't drop below 1 MTU.  You don't adjust
> target for the rate.  There's no suggestion to increase interval
> either.  It makes sense that long transmission times add to the
> expected rtt (SQM does this too).  

	I know, I believe that K. Nichols recommended to increase interval, and I opted for the simplest additive increase (as effectively the expected RTTs increase by the amount of transfer time, and for asymmetric links we can ignore the faster direction mostly). 
	Looking at the rationale in the codel RFC I now believe that scaling would be the right thing… in theory smaller target better preserves responsiveness at a slightly higher bandwidth sacrifice, and with slow links I believe responsiveness is more important; but then I wonder what is more responsive, a higher target ratio with a (smaller) estimated reaction time (in first approximation interval gives the time we allow sender and receiver to react to our signaling) or the theoretical better target ratio with an artificially increased interval. I guess that needs data...

> I can't see what the justification
> is for `target * 8` though.  If there's a good reason maybe it could
> use a comment.

	I fully agree, and hopefully not only in the code but also in the man page as that has potential to confuse users if not documented (and even if in the man page it will still cause confusion at least we can say RTFM ;))

Best Regards
	Sebastian

> 
> Thanks
> Alan

Re: [Cake] cake target corner cases?

[Alan Jenkins]

On 02/11/2015, Sebastian Moeller <moeller0@gmx.de> wrote:
> Hi Alan,
>
> On Nov 2, 2015, at 01:20 , Alan Jenkins <alan.christopher.jenkins@gmail.com>
> wrote:
>
>> I thought SQM works fine with mtu * 1.0.  But cake has been like this
>> since at least April, and I didn't notice an extra 7ms when I tried
>> it.  Confusing.
>
> 	Well, sqm-scripts does take an MTU of 1540 bytes, which is less than ideal
> because for ATM encapsulation the on wire size is actually 32 or 32 * 53, so
> the worst case should be 33*53 = 1749 bytes, but in reality the actual limit
> itself only needs to be approximately right.. That said, I guess I fixed up
> sqm-scripts right now (by making target large enough for one 1749 byte
> packet, I would be delighted if you could go measure whether this changes
> things, before increasing auto-target to 1.5 worst case on-wire-packets, and
> also scaling interval so that target <= 0.1*interval stays true)...

Ok.  Actually I wouldn't have measured +7ms because I wasn't testing
cake flowblind, aka codel.  (And I was looking at ping under load, not
tc -s qdisc, or the tcp rtt's).

The increased target doesn't increase my bandwidth.  Actually I'm
failing to measure any changes at all, even for mtu*1.5.  I'm sure
there's a real effect on buffering, if I had a less noisy way to
measure it :(.  I couldn't work out exactly what you wanted testing;
if there's something more specific I could probably spend more time on
it.

I agree with the reasoning for 1749 bytes.  If you think mtu*1.5 is a
good idea for SQM, I can't follow the logic, but it wouldn't bother me
for fq_codel.  I'm not really worried about cake, because I don't
understand it.  cake flowblind with mtu*1.5 hard-coded sounds less
good, but I don't have any reason to use it in that mode :).


>> 	b->cparams.interval = max(rtt_est_ns +
>> 				     b->cparams.target - ns_target,
>> 				     b->cparams.target * 8);
>
> 	I sort of cheated, I knew this was in the code, but I wanted to highlight
> that we need a better description of what cake does to not confuse future
> users, sorry for that.

I won't complain, I'm probably the one being obstructive here.


>> Maybe this keeps the target/interval ratio from going too far over the
>> recommended 10%.  Though that's not what CoDel says to do.
>>
>> In the CoDel draft you just don't drop below 1 MTU.  You don't adjust
>> target for the rate.  There's no suggestion to increase interval
>> either.  It makes sense that long transmission times add to the
>> expected rtt (SQM does this too).
>
> 	I know, I believe that K. Nichols recommended to increase interval, and I
> opted for the simplest additive increase (as effectively the expected RTTs
> increase by the amount of transfer time, and for asymmetric links we can
> ignore the faster direction mostly).
> 	Looking at the rationale in the codel RFC I now believe that scaling would
> be the right thing… in theory smaller target better preserves responsiveness
> at a slightly higher bandwidth sacrifice, and with slow links I believe
> responsiveness is more important; but then I wonder what is more responsive,
> a higher target ratio with a (smaller) estimated reaction time (in first
> approximation interval gives the time we allow sender and receiver to react
> to our signaling) or the theoretical better target ratio with an
> artificially increased interval. I guess that needs data...

Data always good.  AIUI the ratio is about how low you can set target,
before the bandwidth reduction becomes painful.

I can imagine expected rtt increases e.g. in Tin 4, when it's
de-prioritised because of exceeding the bandwidth threshold.  I don't
see why it would increase by over 400ms, in the 1Mbit/s case you
showed.

Regards
Alan

Re: [Cake] cake target corner cases?

[Sebastian Moeller]

Hi Alan,

On Nov 2, 2015, at 17:20 , Alan Jenkins <alan.christopher.jenkins@gmail.com> wrote:

> On 02/11/2015, Sebastian Moeller <moeller0@gmx.de> wrote:
>> Hi Alan,
>> 
>> On Nov 2, 2015, at 01:20 , Alan Jenkins <alan.christopher.jenkins@gmail.com>
>> wrote:
>> 
>>> I thought SQM works fine with mtu * 1.0.  But cake has been like this
>>> since at least April, and I didn't notice an extra 7ms when I tried
>>> it.  Confusing.
>> 
>> 	Well, sqm-scripts does take an MTU of 1540 bytes, which is less than ideal
>> because for ATM encapsulation the on wire size is actually 32 or 32 * 53, so
>> the worst case should be 33*53 = 1749 bytes, but in reality the actual limit
>> itself only needs to be approximately right.. That said, I guess I fixed up
>> sqm-scripts right now (by making target large enough for one 1749 byte
>> packet, I would be delighted if you could go measure whether this changes
>> things, before increasing auto-target to 1.5 worst case on-wire-packets, and
>> also scaling interval so that target <= 0.1*interval stays true)...
> 
> Ok.  Actually I wouldn't have measured +7ms because I wasn't testing
> cake flowblind, aka codel.  (And I was looking at ping under load, not
> tc -s qdisc, or the tcp rtf’s).

	Yes, you looked at the real data, the set-points is what I hoped to use to affect the real latency under load.

> 
> The increased target doesn't increase my bandwidth.  

	I believe we are talking changes so small it should not matter one way or the other, but then it is nice to be at least consistent with one’s own theory ;)

> Actually I'm
> failing to measure any changes at all, even for mtu*1.5.  I'm sure
> there's a real effect on buffering, if I had a less noisy way to
> measure it :(.  I couldn't work out exactly what you wanted testing;
> if there's something more specific I could probably spend more time on
> it.

	This is pretty much what I expected; the only interesting measurement left then is to disable this completely and see how a to small target affects latency under load… (I just wonder if this might be better measured with larger probe packets, say full MTU ICMP packets instead of small ones?)

> 
> I agree with the reasoning for 1749 bytes.  If you think mtu*1.5 is a
> good idea for SQM, I can't follow the logic, but it wouldn't bother me
> for fq_codel.  

	Oh, the 1.5 * 1749 is really just to be on the safe side, but until this setting makes a dent in at least one measurement, I guess I am not going to bother increasing this further; I will try to play with keeping interval at 20 to 10 times target though...

> I'm not really worried about cake, because I don't
> understand it.  cake flowblind with mtu*1.5 hard-coded sounds less
> good, but I don't have any reason to use it in that mode :).
> 
> 
>>> 	b->cparams.interval = max(rtt_est_ns +
>>> 				     b->cparams.target - ns_target,
>>> 				     b->cparams.target * 8);
>> 
>> 	I sort of cheated, I knew this was in the code, but I wanted to highlight
>> that we need a better description of what cake does to not confuse future
>> users, sorry for that.
> 
> I won't complain, I'm probably the one being obstructive here.

	Not at all; you are right with your analysis and I should have been less circumvent...

> 
> 
>>> Maybe this keeps the target/interval ratio from going too far over the
>>> recommended 10%.  Though that's not what CoDel says to do.
>>> 
>>> In the CoDel draft you just don't drop below 1 MTU.  You don't adjust
>>> target for the rate.  There's no suggestion to increase interval
>>> either.  It makes sense that long transmission times add to the
>>> expected rtt (SQM does this too).
>> 
>> 	I know, I believe that K. Nichols recommended to increase interval, and I
>> opted for the simplest additive increase (as effectively the expected RTTs
>> increase by the amount of transfer time, and for asymmetric links we can
>> ignore the faster direction mostly).
>> 	Looking at the rationale in the codel RFC I now believe that scaling would
>> be the right thing… in theory smaller target better preserves responsiveness
>> at a slightly higher bandwidth sacrifice, and with slow links I believe
>> responsiveness is more important; but then I wonder what is more responsive,
>> a higher target ratio with a (smaller) estimated reaction time (in first
>> approximation interval gives the time we allow sender and receiver to react
>> to our signaling) or the theoretical better target ratio with an
>> artificially increased interval. I guess that needs data...
> 
> Data always good.  AIUI the ratio is about how low you can set target,
> before the bandwidth reduction becomes painful.

	Yes, but also target sort of correlates with the median expected queuing delay under saturating load, so if target is too high latency suffers, not exactly what codel aims to fix...


> 
> I can imagine expected rtt increases e.g. in Tin 4, when it's
> de-prioritised because of exceeding the bandwidth threshold.  I don't
> see why it would increase by over 400ms, in the 1Mbit/s case you
> showed.

	Well, I am not sure that I fully understand whether cake is doing what it wants to do there in the first place, and I am also unsure if cake achieves its goal why it does that. I would be delighted if someone in the know could elucidate that for me…

Best Regards
	Sebastian


> 
> Regards
> Alan

Re: [Cake] cake target corner cases?

[Alan Jenkins]

On 02/11/15 18:49, Sebastian Moeller wrote:
> Hi Alan,
Hi again Sebastian

> On Nov 2, 2015, at 17:20 , Alan Jenkins <alan.christopher.jenkins@gmail.com> wrote:
>
>> Actually I'm
>> failing to measure any changes at all, even for mtu*1.5.  I'm sure
>> there's a real effect on buffering, if I had a less noisy way to
>> measure it :(.  I couldn't work out exactly what you wanted testing;
>> if there's something more specific I could probably spend more time on
>> it.
> 	This is pretty much what I expected; the only interesting measurement left then is to disable this completely and see how a to small target affects latency under load… (I just wonder if this might be better measured with larger probe packets, say full MTU ICMP packets instead of small ones?)

Good point!  I should test with the "adaptation" feature entirely 
disabled, i.e. target=5ms.  There should be a case where it decreases 
bandwidth.  Setting the rate artificially low might be useful to make it 
easier to measure.

Ah, the worst case for bandwidth should be a single stream, with tcp 
congestion control set to Reno (like non-server versions of Windows).  
Whereas my basic tests are multi-stream and use Linux' default Cubic.  
So that's probably what I was doing wrong.

>> I agree with the reasoning for 1749 bytes.  If you think mtu*1.5 is a
>> good idea for SQM, I can't follow the logic, but it wouldn't bother me
>> for fq_codel.
> 	Oh, the 1.5 * 1749 is really just to be on the safe side, but until this setting makes a dent in at least one measurement, I guess I am not going to bother increasing this further; I will try to play with keeping interval at 20 to 10 times target though...

I guess you want a similar setup to the 1s rtt test.  Simulate a path 
delay of exactly 100ms, set a low rate, then look at the effect interval 
has on bandwidth.

Regards
Alan

Re: [Cake] cake target corner cases?

[Sebastian Moeller]

Hi Kevin,

On Nov 1, 2015, at 22:52 , Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:

> 
> Can't give full explanation now but the bytes per ns calculation which is used as a basis for target on 'slow' links uses mtu*1.5

	Which has the advantage of being larger than ATM encapsulation expansion would account for so this is on the safe side…

Best Regards
	Sebastian

> 
> --
> Cheers,
> 
> Kevin@Darbyshire-Bryant.me.uk
> 
>> On 1 Nov 2015, at 20:58, Alan Jenkins <alan.christopher.jenkins@gmail.com> wrote:
>> 
>>> On 01/11/2015, Sebastian Moeller <moeller0@gmx.de> wrote:
>>> Dear cake committee,
>>> 
>>> I just played around with the most recent sch_cake and noticed us:
>>> 
>>> user@computer:~/CODE/tc-adv/tc> sudo tc-adv qdisc del dev eth0 root
>>> user@computer:~/CODE/tc-adv/tc> sudo tc-adv qdisc replace dev eth0 root cake
>>> bandwidth 1Mbit ; sudo tc-adv -s qdisc
>>> qdisc cake 8005: dev eth0 root refcnt 6 bandwidth 1Mbit diffserv4 flows rtt
>>> 100.0ms raw
>>> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>> backlog 0b 0p requeues 0
>>> capacity estimate: 1Mbit
>>>            Tin 0       Tin 1       Tin 2       Tin 3
>>> thresh       1Mbit   937504bit     750Kbit     250Kbit
>>> target      18.2ms      19.4ms      24.2ms      72.7ms
>>> interval     145.3ms     155.0ms     193.8ms     581.4ms
>> 
>> 
>>> Here target is always 12.5% of interval instead of the expected 6.25%
>>> 1/16 = 0.0625
>>> 72.7/581.4 = 0.125042999656
>>> 24.2/193.8 = 0.124871001032
>>> 19.4/155.0 = 0.125161290323
>>> 18.2/145.3 = 0.125258086717
>>> But the bandwidth is really low, so pushing target closer to the bandwidth
>>> conserving side of the codel rationale might be fine,
>> 
>> Pretty sure it's a minimum derived from the MTU
>> 
>> ((mtu=1.5kbyte) * 8 bits/byte) / 1000 Mbit/s = 0.012s
>> 
>> except I don't know where the .5 comes from, that's incredibly
>> suspicious to have a round 1/8th :).
>> 
>> The point is that if buffering falls below the MTU, the connection
>> will be completely clobbered.
>> 
>> In a way it's nice cake reports this in the target.  Otherwise cake
>> would claim the target is 5ms, but measurements would show the
>> effective target is more than twice as high.
>> 
>>> since latency is bad
>>> to begin with and bandwidth also pretty scarce. But it might be interesting
>>> to do a few more measurements at low bandwidths to confirm that the 12.5% of
>>> interval logic holds water; one could also argue that people with such links
>>> (a lot of DSL lines have even less upload, so this certainly is not extreme)
>>> might think that any added ms of delay matters (more than bandwidth);
>>> currently we leave the user no remedy...
>>> 
>>> 
>> 
>> <snip>
>> 
>>> This looks okay, except Tin3 has target at 7.3/101.0 = 0.0722772277228 7% of
>>> interval.
>> 
>> Looks like the same thing.
>> 
>> 
>>> Both observations might actually be on purpose, but if so we should document
>>> that behavior as expected, for example in the man page…
>>> 
>>> Best Regards
>>>   Sebastian
>> 
>> 
>> I'm afraid I can't help mention my old niggle :).  _If_ you mention
>> this alongside instructions for RRUL, I think you'd also want to
>> explain^W mention the measurement increase for diffserv4 v.s.
>> besteffort.
>> 
>> I think the ICMP ping measurement increases by another 10ms on my
>> connection (11500k down /  850k up, so an mtu is ~15ms).  I concluded
>> it was inherent in prioritization.  Now I guess it's equal to the sum
>> of target * bandwidth_fraction for each class "above" icmp ping (and
>> could be tested).
>> 
>> I have graphs from sqm with and without classification.  I did test
>> cake once and I think it's the same (otherwise would be a bug).
>> 
>> https://dl.dropboxusercontent.com/u/49925445/bufferbloat.net/220-cdf-531414.sqm_simplest_11500_850_atm40_udppingfix.svg
>> 
>> https://dl.dropboxusercontent.com/u/49925445/bufferbloat.net/221-cdf-360505.sqm_simple_11500_850_atm40_udppingfix.svg
>> 
>> Warm regards
>> Alan
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] cake target corner cases?

[Sebastian Moeller]

Hi Alan,

On Nov 1, 2015, at 21:58 , Alan Jenkins <alan.christopher.jenkins@gmail.com> wrote:

> On 01/11/2015, Sebastian Moeller <moeller0@gmx.de> wrote:
>> Dear cake committee,
>> 
>> I just played around with the most recent sch_cake and noticed:
>> 
>> user@computer:~/CODE/tc-adv/tc> sudo tc-adv qdisc del dev eth0 root
>> user@computer:~/CODE/tc-adv/tc> sudo tc-adv qdisc replace dev eth0 root cake
>> bandwidth 1Mbit ; sudo tc-adv -s qdisc
>> qdisc cake 8005: dev eth0 root refcnt 6 bandwidth 1Mbit diffserv4 flows rtt
>> 100.0ms raw
>> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>> backlog 0b 0p requeues 0
>> capacity estimate: 1Mbit
>>             Tin 0       Tin 1       Tin 2       Tin 3
>>  thresh       1Mbit   937504bit     750Kbit     250Kbit
>>  target      18.2ms      19.4ms      24.2ms      72.7ms
>> interval     145.3ms     155.0ms     193.8ms     581.4ms
> 
> 
>> Here target is always 12.5% of interval instead of the expected 6.25%
>> 1/16 = 0.0625
>> 72.7/581.4 = 0.125042999656
>> 24.2/193.8 = 0.124871001032
>> 19.4/155.0 = 0.125161290323
>> 18.2/145.3 = 0.125258086717
>> But the bandwidth is really low, so pushing target closer to the bandwidth
>> conserving side of the codel rationale might be fine,
> 
> Pretty sure it's a minimum derived from the MTU
> 
> ((mtu=1.5kbyte) * 8 bits/byte) / 1000 Mbit/s = 0.012s
> 
> except I don't know where the .5 comes from, that's incredibly
> suspicious to have a round 1/8th :).

	As we agree later/earlier 1/8th it is ;)

> 
> The point is that if buffering falls below the MTU, the connection
> will be completely clobbered.
> 
> In a way it's nice cake reports this in the target.  Otherwise cake
> would claim the target is 5ms, but measurements would show the
> effective target is more than twice as high.
> 
>> since latency is bad
>> to begin with and bandwidth also pretty scarce. But it might be interesting
>> to do a few more measurements at low bandwidths to confirm that the 12.5% of
>> interval logic holds water; one could also argue that people with such links
>> (a lot of DSL lines have even less upload, so this certainly is not extreme)
>> might think that any added ms of delay matters (more than bandwidth);
>> currently we leave the user no remedy...
>> 
>> 
> 
> <snip>
> 
>> This looks okay, except Tin3 has target at 7.3/101.0 = 0.0722772277228 7% of
>> interval.
> 
> Looks like the same thing.

	Well, I believe there is a gradual shift from the default 5ms/6.2ms target to the 1/8th target and this is just somewhere in between, by virtue of the smallest bandwidth or so...

> 
> 
>> Both observations might actually be on purpose, but if so we should document
>> that behavior as expected, for example in the man page…
>> 
>> Best Regards
>> 	Sebastian
> 
> 
> I'm afraid I can't help mention my old niggle :).  _If_ you mention
> this alongside instructions for RRUL, I think you'd also want to
> explain^W mention the measurement increase for diffserv4 v.s.
> besteffort.
> 
> I think the ICMP ping measurement increases by another 10ms on my
> connection (11500k down /  850k up, so an mtu is ~15ms).  I concluded
> it was inherent in prioritization.  Now I guess it's equal to the sum
> of target * bandwidth_fraction for each class "above" icmp ping (and
> could be tested).

	Well, with standard sqm-scripts (htb and fq_codel) we leverage iptables to do the filtering, which comes with its own computation cost…  then again with your bandwidths there should be enough cycles left to do this...

> 
> I have graphs from sqm with and without classification.  I did test
> cake once and I think it's the same (otherwise would be a bug).
> 
> https://dl.dropboxusercontent.com/u/49925445/bufferbloat.net/220-cdf-531414.sqm_simplest_11500_850_atm40_udppingfix.svg
> 
> https://dl.dropboxusercontent.com/u/49925445/bufferbloat.net/221-cdf-360505.sqm_simple_11500_850_atm40_udppingfix.svg

	Yes, that is the picture I know, I believe cake looks a bit different, by virtue of doing a few things in a more integrated, clever way; not sure how it looks at low bandwidth though, I rarely test at low bandwidth nowadays. Even though I should…

Best Regards
	Sebastian

> 
> Warm regards
> Alan