[Cake] Running Cake at long RTTs

[Cake] Running Cake at long RTTs

[Toke Høiland-Jørgensen]

[-- Attachment #1: Type: text/plain, Size: 737 bytes --]

So Dave and I did a bit of testing of Cake at a 1-second base RTT. That
showed that, compared with a straight FIFO queue (with a sufficiently
large buffer), throughput was suffering quite a bit under Cake,
especially at large bandwidths. We did two changes to fix this:

- Turn the hard packet queue size into a lower bound rather than an
  upper bound.

- Scale the target to be 1/16th of the interval.

The first change allows Cake to actually reach the target throughput,
but it still takes a long while to get there. With the second change, we
actually get the desired behaviour. The attached plot shows the
difference, with the solid line being before the change and the dashed
line being after the change.

Patch attached.

-Toke


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: cake-longrtt.patch --]
[-- Type: text/x-diff, Size: 859 bytes --]

diff --git a/sch_cake.c b/sch_cake.c
index c97a212..91e556c 100644
--- a/sch_cake.c
+++ b/sch_cake.c
@@ -854,10 +854,11 @@ static void cake_set_rate(struct cake_tin_data *b, u64 rate, u32 mtu,
 
 	byte_target_ns = (byte_target * rate_ns) >> rate_shft;
 
-	b->cparams.target = max(byte_target_ns, ns_target);
 	b->cparams.interval = max(rtt_est_ns +
 				     b->cparams.target - ns_target,
 				     b->cparams.target * 8);
+	b->cparams.target = max(max(byte_target_ns, ns_target),
+				b->cparams.interval >> 4);
 	b->cparams.threshold = (b->cparams.target >> 15) *
 		(b->cparams.interval >> 15) * 2;
 
@@ -1144,7 +1145,7 @@ static void cake_reconfigure(struct Qdisc *sch)
 		q->peel_threshold = 0;
 	}
 
-	q->buffer_limit = min(q->buffer_limit, sch->limit *
+	q->buffer_limit = max(q->buffer_limit, sch->limit *
 			      psched_mtu(qdisc_dev(sch)));
 }
 

[-- Attachment #3: cake-longrtt.pdf --]
[-- Type: application/pdf, Size: 149611 bytes --]

Re: [Cake] Running Cake at long RTTs

[Loganaden Velvindron]

[-- Attachment #1: Type: text/plain, Size: 1151 bytes --]

On Tue, Oct 27, 2015 at 3:14 PM, Toke Høiland-Jørgensen <toke@toke.dk>
wrote:

> So Dave and I did a bit of testing of Cake at a 1-second base RTT. That
> showed that, compared with a straight FIFO queue (with a sufficiently
> large buffer), throughput was suffering quite a bit under Cake,
> especially at large bandwidths. We did two changes to fix this:
>
> - Turn the hard packet queue size into a lower bound rather than an
>   upper bound.
>
> - Scale the target to be 1/16th of the interval.
>
> The first change allows Cake to actually reach the target throughput,
> but it still takes a long while to get there. With the second change, we
> actually get the desired behaviour. The attached plot shows the
> difference, with the solid line being before the change and the dashed
> line being after the change.
>
> Patch attached.
>
> Fantastic !

I'm going to test it and report back. I'm interested in how this will
perform for 350ms RTT.



> -Toke
>
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>
>

[-- Attachment #2: Type: text/html, Size: 1900 bytes --]

Re: [Cake] Running Cake at long RTTs

[Toke Høiland-Jørgensen]

Sebastian Moeller <moeller0@gmx.de> writes:

>> - Turn the hard packet queue size into a lower bound rather than an
>>  upper bound.
>
> 	This seems dangerous for low memory configurations?

Well, yeah, the max packet count allowed can be higher this way, but I
guess the only sane answer to that is "don't configure cake for 100Mbit
and 1s+ of latency if you don't have the memory to spare".

Having the (now lower) bound be a bit lower than it is currently may be
a good idea, but not sure what it should be... The current 10240 matches
fq_codel.

>> - Scale the target to be 1/16th of the interval.
>
> 	Codel theory claims 5-10%, so 1/20 to 1/10, so it seems you
> 	confirm the theory, not sure why cake did not do this as a
> 	default… rem, what actually was cake setting target to? Asked
> 	differently how sensitive to target being exactly 1/16th was the
> 	throughput?

Yup. 1/16th has the benefit of being implementable as a 4bit shift :)

Before, the target was hard-coded as 5 ms; so this was definitely an
oversight.

-Toke

Re: [Cake] Running Cake at long RTTs

[Sebastian Moeller]

Hi Toke,

On Oct 27, 2015, at 17:50 , Toke Høiland-Jørgensen <toke@toke.dk> wrote:

> Sebastian Moeller <moeller0@gmx.de> writes:
> 
>>> - Turn the hard packet queue size into a lower bound rather than an
>>> upper bound.
>> 
>> 	This seems dangerous for low memory configurations?
> 
> Well, yeah, the max packet count allowed can be higher this way, but I
> guess the only sane answer to that is "don't configure cake for 100Mbit
> and 1s+ of latency if you don't have the memory to spare”.

	What about cake honoring it as a hard limit but complaining to syslog, if cake deems the limit too low? If I recall correctly, having a qdisc OOM a router basically functionally bricks the router (unit the next reboot) or forces a reboot, both not ideal.

> 
> Having the (now lower) bound be a bit lower than it is currently may be
> a good idea, but not sure what it should be... The current 10240 matches
> fq_codel.

	Oh, I guess that is fine, it just means that for low memory devices we need to teach sqm-scripts to reign in a bit...

> 
>>> - Scale the target to be 1/16th of the interval.
>> 
>> 	Codel theory claims 5-10%, so 1/20 to 1/10, so it seems you
>> 	confirm the theory, not sure why cake did not do this as a
>> 	default… rem, what actually was cake setting target to? Asked
>> 	differently how sensitive to target being exactly 1/16th was the
>> 	throughput?
> 
> Yup. 1/16th has the benefit of being implementable as a 4bit shift :)

	So is 1/8th ;) just curious, which values did you try? Codel RFC argues for 5-10% with 10% giving ever so slightly more bandwidth, while 5% keeps latency under load increase a bit smaller. But I really like that reality fits theory here, chalk this one up for code;l’s designers ;)

> 
> Before, the target was hard-coded as 5 ms; so this was definitely an
> oversight.

	Not sure, we had discussed this and Jonathan argued that it requires empiric testing if I recall correctly, exactly what you have done ;)
I still believe that for any automatic choice cake does there should be a manual override option for experimentation, most urgently a manual target specification that will override all automatic optimizations. Well, just as for the queue max size I believe cake should complain if it thinks things are unreasonable but still do follow the users request if non impossible…


Best Regards
	Sebastian

> 
> -Toke

Re: [Cake] Running Cake at long RTTs

[Jonathan Morton]

> On 27 Oct, 2015, at 17:14, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> - Turn the hard packet queue size into a lower bound rather than an
>  upper bound.
> 
> - Scale the target to be 1/16th of the interval.
> 
> The first change allows Cake to actually reach the target throughput,
> but it still takes a long while to get there. With the second change, we
> actually get the desired behaviour.

The first change implies that the 10K packets * MTU default (ie. 15MB) isn’t big enough, but the 50MB calculated by 4 * RTT * rate is.  That’s a fairly narrow range, which suggests that the latter calculation is correct.

However, have you tried it with just the second change and not the first?  Conventional wisdom suggests that 15MB (which is slightly more than one BDP) should be about the right size for a buffer under these conditions.  I don’t want to gain a false impression of what the minimum usable buffer size is.

Limiting it to 10K * MTU was a safety valve to avoid the buffer size exploding.  With the first change in place, the “interplanetary” setting effectively removes any limit on buffer size as well.  This makes me deeply uncomfortable.  Instead, we should arrange to configure the safety-valve limit when accommodating LFNs.

The result from the second change is more clearly beneficial, but I’ll put it in userspace (ie. tc) rather than in the kernel.  I think the patch as given has some unintended consequences.  Scaling target with interval also means I have to take more care with the threshold calculation, since the threshold scales with the product of the two.

 - Jonathan Morton

Re: [Cake] Running Cake at long RTTs

[Toke Høiland-Jørgensen]

Jonathan Morton <chromatix99@gmail.com> writes:

> The first change implies that the 10K packets * MTU default (ie. 15MB)
> isn’t big enough, but the 50MB calculated by 4 * RTT * rate is. That’s
> a fairly narrow range, which suggests that the latter calculation is
> correct.
>
> However, have you tried it with just the second change and not the first?
> Conventional wisdom suggests that 15MB (which is slightly more than one BDP)
> should be about the right size for a buffer under these conditions. I don’t want
> to gain a false impression of what the minimum usable buffer size is.

Well, indirectly: The first change was actually to address the problem
that Cake with CoDel turned off (i.e. interval 300s) did not achieve
full throughput. Changing the buffer size fixed that. Guess I can go
back and revert that and try again; will do so tomorrow.

> Limiting it to 10K * MTU was a safety valve to avoid the buffer size
> exploding. With the first change in place, the “interplanetary”
> setting effectively removes any limit on buffer size as well. This
> makes me deeply uncomfortable. Instead, we should arrange to configure
> the safety-valve limit when accommodating LFNs.

I agree that some sort of safety valve is probably needed (as is a
minimum; but that's another matter). Perhaps just setting the limit
higher could be a pragmatic solution?

> The result from the second change is more clearly beneficial, but I’ll
> put it in userspace (ie. tc) rather than in the kernel.

I'm not sure I agree with that. Putting it in userspace means basically
exposing the target. Even if it's not recognised in iproute, it will be
exposed in the netlink API, which sooner or later some other tool is
going to speak. We will then risk ending up in a situation where the
target is set arbitrarily, leading to a need to print out what it
actually is; which in turn brings us back to having exposed target
completely.

Now, I'm not ruling out that exposing target may eventually be the right
thing to do. However, I do like the simplicity of having only the one
parameter; so if we can avoid it I think we should. I guess more
experimentation is needed to answer this.

> I think the patch as given has some unintended consequences. Scaling
> target with interval also means I have to take more care with the
> threshold calculation, since the threshold scales with the product of
> the two.

That is most probably true. :)

-Toke

Re: [Cake] Running Cake at long RTTs

[Sebastian Moeller]

Hi Toke, hi Jonathan,

On Oct 28, 2015, at 00:57 , Toke Høiland-Jørgensen <toke@toke.dk> wrote:

> Jonathan Morton <chromatix99@gmail.com> writes:
> 
>> The first change implies that the 10K packets * MTU default (ie. 15MB)
>> isn’t big enough, but the 50MB calculated by 4 * RTT * rate is. That’s
>> a fairly narrow range, which suggests that the latter calculation is
>> correct.
>> 
>> However, have you tried it with just the second change and not the first?
>> Conventional wisdom suggests that 15MB (which is slightly more than one BDP)
>> should be about the right size for a buffer under these conditions. I don’t want
>> to gain a false impression of what the minimum usable buffer size is.
> 
> Well, indirectly: The first change was actually to address the problem
> that Cake with CoDel turned off (i.e. interval 300s) did not achieve
> full throughput. Changing the buffer size fixed that. Guess I can go
> back and revert that and try again; will do so tomorrow.
> 
>> Limiting it to 10K * MTU was a safety valve to avoid the buffer size
>> exploding. With the first change in place, the “interplanetary”
>> setting effectively removes any limit on buffer size as well. This
>> makes me deeply uncomfortable. Instead, we should arrange to configure
>> the safety-valve limit when accommodating LFNs.
> 
> I agree that some sort of safety valve is probably needed (as is a
> minimum; but that's another matter). Perhaps just setting the limit
> higher could be a pragmatic solution?

	Might I propose again to actually expose the hard maximal limit somehow to user space? Both 15MB as well as well as 50MB will wreck havoc on my poor 64MB wndr3700v2. I believe other qdiscs allow to control the maximum buffering (at least approximately, since a buffer limit in packets will ignore all the sib overhead that seems relevant; ideally I would like cake to honor what ever hard limit the user set).

> 
>> The result from the second change is more clearly beneficial, but I’ll
>> put it in userspace (ie. tc) rather than in the kernel.
> 
> I'm not sure I agree with that. Putting it in userspace means basically
> exposing the target.

	I believe it already is exposed, so if this is unwanted we should change this back before upstreaming?

> Even if it's not recognised in iproute, it will be
> exposed in the netlink API, which sooner or later some other tool is
> going to speak. We will then risk ending up in a situation where the
> target is set arbitrarily, leading to a need to print out what it
> actually is; which in turn brings us back to having exposed target
> completely.

	If I might repeat my opinion here, let’s expose it, it is one thing to be pro-active and have well working defaults that do not need manual intervention, but it is a totally different thing saying “trust me I know what is best for you”.

> 
> Now, I'm not ruling out that exposing target may eventually be the right
> thing to do. However, I do like the simplicity of having only the one
> parameter; so if we can avoid it I think we should. I guess more
> experimentation is needed to answer this.

	I would argue that cake should do what we consider to be ideal by default without user intervention, and if the user starts specifying things it should honor them (still using its heuristics for the parameters not explicitly specified), though cake might complain to syslog that it thinks the user requested something silly. It is fine to reject the impossible ;). The advantage of exposed knobs is that it allows experimentation without recompilation from source...

Best Regards
	Sebastian

> 
>> I think the patch as given has some unintended consequences. Scaling
>> target with interval also means I have to take more care with the
>> threshold calculation, since the threshold scales with the product of
>> the two.
> 
> That is most probably true. :)
> 
> -Toke
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Running Cake at long RTTs

[Sebastian Moeller]

[-- Attachment #1: Type: text/plain, Size: 5178 bytes --]

Hi Toke,

so I just pulled both Dave’s ach_cake and your iproute2 repositories (edited codel5.h to exclude kvfree which is not needed on my opensuse patched? 3.16 kernel) and build the new stuff. After rmmod’ing the old module before insmod’ing the new one things started to work:

moeller@happy-horse:~/CODE/sch_cake> sudo tc-toke qdisc replace dev eth0 root cake rtt 120ms
moeller@happy-horse:~/CODE/sch_cake> sudo tc-toke -s qdisc
qdisc cake 8004: dev eth0 root refcnt 6 unlimited besteffort flows rtt 120.0ms raw 
 Sent 42990 bytes 261 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
capacity estimate: 0bit
             Tin 0  
  thresh        0bit
  target       7.7ms
interval     122.7ms
Pk-delay         0us
Av-delay         0us
Sp-delay         0us
  pkts             0
  bytes            0
way-inds           0
way-miss           0
way-cols           0
  drops            0
  marks            0
Sp-flows           0
Bk-flows           0
last-len           0
max-len            0


Except I requested rtf 120 ms and got 122.7, which admittedly is close. I know I repeat myself, but on of the is one things that irritate me in software is if software silently pretends to know better… 
Now 122.7 versus 120 might be in the noise, but look at that:

moeller@happy-horse:~/CODE/sch_cake> sudo tc-toke qdisc replace dev eth0 root cake bandwidth 1Mbit besteffort rtt 100ms
moeller@happy-horse:~/CODE/sch_cake> sudo tc-toke -s qdisc
qdisc cake 8008: dev eth0 root refcnt 6 bandwidth 1Mbit besteffort flows rtt 100.0ms raw 
 Sent 27367 bytes 188 pkt (dropped 0, overlimits 26 requeues 0) 
 backlog 0b 0p requeues 0 
capacity estimate: 981464bit
             Tin 0  
  thresh       1Mbit
  target      18.2ms
interval     145.3ms
Pk-delay       1.1ms
Av-delay        48us
Sp-delay         0us
  pkts           130
  bytes        18905
way-inds           0
way-miss          22
way-cols           0
  drops            0
  marks            0
Sp-flows           0
Bk-flows           1
last-len         159
max-len          584

	145.3 versus 100 that is way off (I know 145.2 = 8 * target, or target is 12.5% of interval), Not sure where the 18.2ms target comes from, the worst case single packet wire size is 33 ATM cells of 53 bytes which at the set rate of 1Mbps results in ((33*53*8)/1000^3) * 1000 = 0.013992 seconds = 14.0 ms seconds and 14*8 = 112 ms. So color me confused in what cake does under the hood. I would have preferred to find at least a “I'm sorry, Dave, I'm afraid I can't do that.” in the syslog regarding my request for an rtt of 100ms...



	Also I notice the disparity between parameter name “rtt” and the name of the reported statistic “interval”. Jonathan, Dave, what is your preference change the reported stats name or the parameter name (I believe it should be that statistic that needs renaming).
Next question what is the opinion on exposing target in addition to interval? My point, which I might have over-repeated already, is we should expose and honor it if possible. (Any with honor I mean replace the 5ms default but do all calculations as cake usually does for different priority bands; that also means that if only one of interval or target where specified cake is free to scale the other one to what it considers reasonable).

Best Regards
	Sebastian



On Oct 27, 2015, at 16:14 , Toke Høiland-Jørgensen <toke@toke.dk> wrote:

> So Dave and I did a bit of testing of Cake at a 1-second base RTT. That
> showed that, compared with a straight FIFO queue (with a sufficiently
> large buffer), throughput was suffering quite a bit under Cake,
> especially at large bandwidths. We did two changes to fix this:
> 
> - Turn the hard packet queue size into a lower bound rather than an
>  upper bound.
> 
> - Scale the target to be 1/16th of the interval.
> 
> The first change allows Cake to actually reach the target throughput,
> but it still takes a long while to get there. With the second change, we
> actually get the desired behaviour. The attached plot shows the
> difference, with the solid line being before the change and the dashed
> line being after the change.
> 
> Patch attached.
> 
> -Toke
> 
> diff --git a/sch_cake.c b/sch_cake.c
> index c97a212..91e556c 100644
> --- a/sch_cake.c
> +++ b/sch_cake.c
> @@ -854,10 +854,11 @@ static void cake_set_rate(struct cake_tin_data *b, u64 rate, u32 mtu,
> 
> 	byte_target_ns = (byte_target * rate_ns) >> rate_shft;
> 
> -	b->cparams.target = max(byte_target_ns, ns_target);
> 	b->cparams.interval = max(rtt_est_ns +
> 				     b->cparams.target - ns_target,
> 				     b->cparams.target * 8);
> +	b->cparams.target = max(max(byte_target_ns, ns_target),
> +				b->cparams.interval >> 4);
> 	b->cparams.threshold = (b->cparams.target >> 15) *
> 		(b->cparams.interval >> 15) * 2;
> 
> @@ -1144,7 +1145,7 @@ static void cake_reconfigure(struct Qdisc *sch)
> 		q->peel_threshold = 0;
> 	}
> 
> -	q->buffer_limit = min(q->buffer_limit, sch->limit *
> +	q->buffer_limit = max(q->buffer_limit, sch->limit *
> 			      psched_mtu(qdisc_dev(sch)));
> }
> 

[-- Attachment #2: cake-longrtt.pdf --]
[-- Type: application/pdf, Size: 149611 bytes --]

[-- Attachment #3: Type: text/plain, Size: 146 bytes --]

> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Running Cake at long RTTs

[Toke Høiland-Jørgensen]

Sebastian Moeller <moeller0@gmx.de> writes:

> Except I requested rtf 120 ms and got 122.7, which admittedly is
> close. I know I repeat myself, but on of the is one things that
> irritate me in software is if software silently pretends to know
> better… Now 122.7 versus 120 might be in the noise, but look at that:

I don't remember from where, but I suspect there's something wrong with
the 'change' logic in Cake. Can you try removing the qdisc completely,
then re-adding it, rather than doing a straight replace?

Might not be that, but do try it out just to be sure. I have not looked
at the scaling of target/interval that goes on in the different bands;
but there is one? Or is there? If there is, why? It's not immediately
obvious that different diffserv markings needs different settings for
the AQM...

> Also I notice the disparity between parameter name “rtt” and the name
> of the reported statistic “interval”. Jonathan, Dave, what is your
> preference change the reported stats name or the parameter name (I
> believe it should be that statistic that needs renaming).

Yes, if we're calling it RTT it should be RTT throughout. I do think
that makes more sense than interval.

> Next question what is the opinion on exposing target in addition to
> interval? My point, which I might have over-repeated already, is we
> should expose and honor it if possible. (Any with honor I mean replace
> the 5ms default but do all calculations as cake usually does for
> different priority bands; that also means that if only one of interval
> or target where specified cake is free to scale the other one to what
> it considers reasonable).

I seem to recall that at some point we has "easy to configure" as a goal
of Cake.

-Toke

Re: [Cake] Running Cake at long RTTs

[Sebastian Moeller]

Hi Toke,

On Oct 28, 2015, at 16:36 , Toke Høiland-Jørgensen <toke@toke.dk> wrote:

> Sebastian Moeller <moeller0@gmx.de> writes:
> 
>> Except I requested rtf 120 ms and got 122.7, which admittedly is
>> close. I know I repeat myself, but on of the is one things that
>> irritate me in software is if software silently pretends to know
>> better… Now 122.7 versus 120 might be in the noise, but look at that:
> 
> I don't remember from where, but I suspect there's something wrong with
> the 'change' logic in Cake. Can you try removing the qdisc completely,
> then re-adding it, rather than doing a straight replace?

Good point:
moeller@happy-horse:~/CODE/sch_cake> sudo tc-toke qdisc replace dev eth0 root pfifo_fast
moeller@happy-horse:~/CODE/sch_cake> sudo tc-toke qdisc replace dev eth0 root cake bandwidth 1Mbit besteffort rtt 100ms ; sudo tc-toke -s qdisc
qdisc cake 800c: dev eth0 root refcnt 6 bandwidth 1Mbit besteffort flows rtt 100.0ms raw 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
capacity estimate: 1Mbit
             Tin 0  
  thresh       1Mbit
  target      18.2ms
interval      95.0ms
Pk-delay         0us
Av-delay         0us
Sp-delay         0us
  pkts             0
  bytes            0
way-inds           0
way-miss           0
way-cols           0
  drops            0
  marks            0
Sp-flows           0
Bk-flows           0
last-len           0
max-len            0

moeller@happy-horse:~/CODE/sch_cake> 

Just 95 is not equal to 100 either…





> 
> Might not be that, but do try it out just to be sure. I have not looked
> at the scaling of target/interval that goes on in the different bands;
> but there is one? Or is there? If there is, why? It's not immediately
> obvious that different diffserv markings needs different settings for
> the AQM…

Well:
moeller@happy-horse:~/CODE/sch_cake> sudo tc-toke qdisc replace dev eth0 root cake bandwidth 1Mbit diffserv4 rtt 100ms ; sudo tc-toke -s qdisc
qdisc cake 800e: dev eth0 root refcnt 6 bandwidth 1Mbit diffserv4 flows rtt 100.0ms raw 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
capacity estimate: 1Mbit
             Tin 0       Tin 1       Tin 2       Tin 3  
  thresh       1Mbit   937504bit     750Kbit     250Kbit
  target      18.2ms      19.4ms      24.2ms      72.7ms
interval      95.0ms      95.0ms      95.0ms      95.0ms
Pk-delay         0us         0us         0us         0us
Av-delay         0us         0us         0us         0us
Sp-delay         0us         0us         0us         0us
  pkts             0           0           0           0
  bytes            0           0           0           0
way-inds           0           0           0           0
way-miss           0           0           0           0
way-cols           0           0           0           0
  drops            0           0           0           0
  marks            0           0           0           0
Sp-flows           0           0           0           0
Bk-flows           0           0           0           0
last-len           0           0           0           0
max-len            0           0           0           0

moeller@happy-horse:~/CODE/sch_cake> 

The different tins have different bandwidths and so might need to scale target differentially as the packet transfer time is different for the different tins (I note in my contrived example I have set the bandwidth to a measly 1Mbps). Interestingly rtt is auto-adjusted to 95.0 ms even if I do not request rtt 100:

moeller@happy-horse:~/CODE/sch_cake> sudo tc-toke qdisc del dev eth0 root
RTNETLINK answers: No such file or directory
moeller@happy-horse:~/CODE/sch_cake> sudo tc-toke qdisc replace dev eth0 root cake bandwidth 1Mbit diffserv4 rtt 100ms ; sudo tc-toke -s qdisc
qdisc cake 8016: dev eth0 root refcnt 6 bandwidth 1Mbit diffserv4 flows rtt 100.0ms raw 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
capacity estimate: 1Mbit
             Tin 0       Tin 1       Tin 2       Tin 3  
  thresh       1Mbit   937504bit     750Kbit     250Kbit
  target      18.2ms      19.4ms      24.2ms      72.7ms
interval      95.0ms      95.0ms      95.0ms      95.0ms
Pk-delay         0us         0us         0us         0us
Av-delay         0us         0us         0us         0us
Sp-delay         0us         0us         0us         0us
  pkts             0           0           0           0
  bytes            0           0           0           0
way-inds           0           0           0           0
way-miss           0           0           0           0
way-cols           0           0           0           0
  drops            0           0           0           0
  marks            0           0           0           0
Sp-flows           0           0           0           0
Bk-flows           0           0           0           0
last-len           0           0           0           0
max-len            0           0           0           0

moeller@happy-horse:~/CODE/sch_cake> sudo tc-toke qdisc del dev eth0 root
moeller@happy-horse:~/CODE/sch_cake> sudo tc-toke qdisc replace dev eth0 root cake bandwidth 1Mbit diffserv4 ; sudo tc-toke -s qdisc
qdisc cake 8017: dev eth0 root refcnt 6 bandwidth 1Mbit diffserv4 flows rtt 100.0ms raw 
 Sent 146 bytes 1 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
capacity estimate: 996096bit
             Tin 0       Tin 1       Tin 2       Tin 3  
  thresh       1Mbit   937504bit     750Kbit     250Kbit
  target      18.2ms      19.4ms      24.2ms      72.7ms
interval      95.0ms      95.0ms      95.0ms      95.0ms
Pk-delay         0us         0us         0us         0us
Av-delay         0us         0us         0us         0us
Sp-delay         0us         0us         0us         0us
  pkts             0           1           0           0
  bytes            0         146           0           0
way-inds           0           0           0           0
way-miss           0           1           0           0
way-cols           0           0           0           0
  drops            0           0           0           0
  marks            0           0           0           0
Sp-flows           0           1           0           0
Bk-flows           0           0           0           0
last-len           0         146           0           0
max-len            0         146           0           0

moeller@happy-horse:~/CODE/sch_cake> 

At least in full automatic mode I would have assumed cake would also increase the interval according to the available bandwidth in the different Tins...



> 
>> Also I notice the disparity between parameter name “rtt” and the name
>> of the reported statistic “interval”. Jonathan, Dave, what is your
>> preference change the reported stats name or the parameter name (I
>> believe it should be that statistic that needs renaming).
> 
> Yes, if we're calling it RTT it should be RTT throughout. I do think
> that makes more sense than interval.

	Ah, thanks. (Personaly I would prefer codel_interval ;) )

> 
>> Next question what is the opinion on exposing target in addition to
>> interval? My point, which I might have over-repeated already, is we
>> should expose and honor it if possible. (Any with honor I mean replace
>> the 5ms default but do all calculations as cake usually does for
>> different priority bands; that also means that if only one of interval
>> or target where specified cake is free to scale the other one to what
>> it considers reasonable).
> 
> I seem to recall that at some point we has "easy to configure" as a goal
> of Cake.

	Well, in the easy configuration it lis full automatic and does the right thing; it does not get any easier to configure than:
tc qdisc replace dev eth0 root cake

All the additional parameters are there to tweak and experiment. So I argue exposing knobs is not making it difficult to configure as long as nobody needs to touch these (I also volunteer to make the tc help more specific in that regard…)

Best Regards
	Sebastian


> 
> -Toke

Re: [Cake] Running Cake at long RTTs

[Toke Høiland-Jørgensen]

Sebastian Moeller <moeller0@gmx.de> writes:

> Hi Toke,
>
> On Oct 28, 2015, at 16:36 , Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
>> Sebastian Moeller <moeller0@gmx.de> writes:
>> 
>>> Except I requested rtf 120 ms and got 122.7, which admittedly is
>>> close. I know I repeat myself, but on of the is one things that
>>> irritate me in software is if software silently pretends to know
>>> better… Now 122.7 versus 120 might be in the noise, but look at that:
>> 
>> I don't remember from where, but I suspect there's something wrong with
>> the 'change' logic in Cake. Can you try removing the qdisc completely,
>> then re-adding it, rather than doing a straight replace?
>
> Good point:
> moeller@happy-horse:~/CODE/sch_cake> sudo tc-toke qdisc replace dev eth0 root pfifo_fast
> moeller@happy-horse:~/CODE/sch_cake> sudo tc-toke qdisc replace dev eth0 root cake bandwidth 1Mbit besteffort rtt 100ms ; sudo tc-toke -s qdisc

Right, so there's definitely a bug in the 'change' logic.

> Just 95 is not equal to 100 either…

That's also a bug as far as I'm concerned.

> At least in full automatic mode I would have assumed cake would also
> increase the interval according to the available bandwidth in the
> different Tins...

What has the interval got to do with the bandwidth? There's definitely
some value in capping the target to be at least the time it takes to
transmit one MTU's worth of data.

> All the additional parameters are there to tweak and experiment. So I
> argue exposing knobs is not making it difficult to configure as long
> as nobody needs to touch these (I also volunteer to make the tc help
> more specific in that regard…)

If no one needs to touch them, why are they there? At best that will
just make things break when they bitrot.

I can see how getting around the need for the encapsulation variables is
going to be hard; but for the target, I am not sure I see the value in
exposing this for "experimentation". If we are sure that the relation
between target and interval should be fixed (and I'm not entirely
convinced of that yet, but will think about it some more), then exposing
target is just going to enable invalid configurations.

-Toke

Re: [Cake] Running Cake at long RTTs

[Sebastian Moeller]

Hi Toke,

On Oct 28, 2015, at 17:34 , Toke Høiland-Jørgensen <toke@toke.dk> wrote:

> Sebastian Moeller <moeller0@gmx.de> writes:
> 
>> Hi Toke,
>> 
>> On Oct 28, 2015, at 16:36 , Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> 
>>> Sebastian Moeller <moeller0@gmx.de> writes:
>>> 
>>>> Except I requested rtf 120 ms and got 122.7, which admittedly is
>>>> close. I know I repeat myself, but on of the is one things that
>>>> irritate me in software is if software silently pretends to know
>>>> better… Now 122.7 versus 120 might be in the noise, but look at that:
>>> 
>>> I don't remember from where, but I suspect there's something wrong with
>>> the 'change' logic in Cake. Can you try removing the qdisc completely,
>>> then re-adding it, rather than doing a straight replace?
>> 
>> Good point:
>> moeller@happy-horse:~/CODE/sch_cake> sudo tc-toke qdisc replace dev eth0 root pfifo_fast
>> moeller@happy-horse:~/CODE/sch_cake> sudo tc-toke qdisc replace dev eth0 root cake bandwidth 1Mbit besteffort rtt 100ms ; sudo tc-toke -s qdisc
> 
> Right, so there's definitely a bug in the 'change' logic.
> 
>> Just 95 is not equal to 100 either…
> 
> That's also a bug as far as I'm concerned.
> 
>> At least in full automatic mode I would have assumed cake would also
>> increase the interval according to the available bandwidth in the
>> different Tins...
> 
> What has the interval got to do with the bandwidth? There's definitely
> some value in capping the target to be at least the time it takes to
> transmit one MTU's worth of data.

	Well according to codel’s theory target shall be between 5 to 10% of interval (I note this range is not deduced mathematically and so 1/8 probably is well enough as is 1/16, heck even 1/32th might still be good enough). So if we increase target we also should accordingly increase interval to keep the relation correct. I believe your tests at long RTTs have just shown that the theory is not completely off. So I am willing to entertain the notion that adapting to slow bandwidths should have an effect on target and interval… 


> 
>> All the additional parameters are there to tweak and experiment. So I
>> argue exposing knobs is not making it difficult to configure as long
>> as nobody needs to touch these (I also volunteer to make the tc help
>> more specific in that regard…)
> 
> If no one needs to touch them, why are they there? At best that will
> just make things break when they bitrot.

	Do you really envision lots of changes to cake after upstreaming, as otherwise things should pretty much stay as well debugged as when upstreamed: "no change no bitrot” or so. But the point for exposing it that your experiments have shown that the automatic defaults did not do the right thing, are we really sure that we tested all relevant combinations of the parameter space to say, we are done, the target interval relation is solved?
	Note I am fine with exposing some parameters only if the user also adds a silly option like “willing_to_keep_the_pieces_if_cake_breaks” or is it crumps...

> 
> I can see how getting around the need for the encapsulation variables is
> going to be hard;

	Numeric overhead is all that is needed ;)

> but for the target, I am not sure I see the value in
> exposing this for "experimentation”.

	But so far we have not gotten it right, so experimentation was/is still required.

> If we are sure that the relation
> between target and interval should be fixed (and I'm not entirely
> convinced of that yet, but will think about it some more), then exposing
> target is just going to enable invalid configurations.

	No, unreasonable maybe but not invalid; I am fine with cake not accepting target > interval which I admit is invalid, but target = interval/2 is valid no matter whether I think it is reasonable. If we go the “I know what is best for you route” we should be really really certain that we actually do. But do we? The fact that we are having this discussion makes me wonder a bit.
	The codel RFC give some guidance on the relation of interval and target for TCPs (https://datatracker.ietf.org/doc/draft-ietf-aqm-codel/?include_text=1 Section 3.2) but we so far  cake does/did not follow this guidance, arguably recent cake was doing the wrong thing, exposed interval and target would have made it possible for people using deployed sch_cake’s to fix the behavior. Currently the only redress is to edit the source and build the module afresh, certainly outside the scope of cake’s “make common things simple” motto, no?


Best Regards
	Sebastian

> 
> -Toke

Re: [Cake] Running Cake at long RTTs

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 1553 bytes --]



On 27/10/15 15:14, Toke Høiland-Jørgensen wrote:
> So Dave and I did a bit of testing of Cake at a 1-second base RTT. That
> showed that, compared with a straight FIFO queue (with a sufficiently
> large buffer), throughput was suffering quite a bit under Cake,
> especially at large bandwidths. We did two changes to fix this:
>
> - Turn the hard packet queue size into a lower bound rather than an
>   upper bound.
>
> - Scale the target to be 1/16th of the interval.
>
> The first change allows Cake to actually reach the target throughput,
> but it still takes a long while to get there. With the second change, we
> actually get the desired behaviour. The attached plot shows the
> difference, with the solid line being before the change and the dashed
> line being after the change.
>
> Patch attached.
>
> -Toke

That patch is really going to behave strangely indeed.  You end up with
    b->cparams.interval = max(rtt_est_ns +  b->cparams.target -
ns_target,b->cparams.target * 8);
    b->cparams.target = max(max(byte_target_ns, ns_target),
b->cparams.interval >> 4);

So interval no matter what happens takes note of 'cparams.target' which
hasn't yet been set (on first run is 0)
And target wants to take interval/16, which may be based on the previous
calculation.  Erm.  Needs more thought.

My brain isn't working at the moment due to illness, but recursively
playing with bandwidth settings on cake is most entertaining at the
moment ;-)

Kevin (dead with cold/high as a kite on cold drugs)





[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4816 bytes --]

Re: [Cake] Running Cake at long RTTs

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 1793 bytes --]



On 29/10/15 17:36, Kevin Darbyshire-Bryant wrote:
>
> On 27/10/15 15:14, Toke Høiland-Jørgensen wrote:
>> So Dave and I did a bit of testing of Cake at a 1-second base RTT. That
>> showed that, compared with a straight FIFO queue (with a sufficiently
>> large buffer), throughput was suffering quite a bit under Cake,
>> especially at large bandwidths. We did two changes to fix this:
>>
>> - Turn the hard packet queue size into a lower bound rather than an
>>   upper bound.
>>
>> - Scale the target to be 1/16th of the interval.
>>
>> The first change allows Cake to actually reach the target throughput,
>> but it still takes a long while to get there. With the second change, we
>> actually get the desired behaviour. The attached plot shows the
>> difference, with the solid line being before the change and the dashed
>> line being after the change.
>>
>> Patch attached.
>>
>> -Toke
> That patch is really going to behave strangely indeed.  You end up with
>     b->cparams.interval = max(rtt_est_ns +  b->cparams.target -
> ns_target,b->cparams.target * 8);
>     b->cparams.target = max(max(byte_target_ns, ns_target),
> b->cparams.interval >> 4);
>
> So interval no matter what happens takes note of 'cparams.target' which
> hasn't yet been set (on first run is 0)
> And target wants to take interval/16, which may be based on the previous
> calculation.  Erm.  Needs more thought.
>
> My brain isn't working at the moment due to illness, but recursively
> playing with bandwidth settings on cake is most entertaining at the
> moment ;-)
>
> Kevin (dead with cold/high as a kite on cold drugs)
Right,  I've just pushed a slightly uncomfortable fix for this but it
does at least make cake behave in a deterministic manner again.
>
>



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4816 bytes --]

Re: [Cake] Running Cake at long RTTs

[Dave Taht]

Good spot, thx!
Dave Täht
I just invested five years of my life to making wifi better. And,
now... the FCC wants to make my work, illegal for people to install.
https://www.gofundme.com/savewifi


On Thu, Oct 29, 2015 at 1:36 PM, Kevin Darbyshire-Bryant
<kevin@darbyshire-bryant.me.uk> wrote:
>
>
> On 27/10/15 15:14, Toke Høiland-Jørgensen wrote:
>> So Dave and I did a bit of testing of Cake at a 1-second base RTT. That
>> showed that, compared with a straight FIFO queue (with a sufficiently
>> large buffer), throughput was suffering quite a bit under Cake,
>> especially at large bandwidths. We did two changes to fix this:
>>
>> - Turn the hard packet queue size into a lower bound rather than an
>>   upper bound.
>>
>> - Scale the target to be 1/16th of the interval.
>>
>> The first change allows Cake to actually reach the target throughput,
>> but it still takes a long while to get there. With the second change, we
>> actually get the desired behaviour. The attached plot shows the
>> difference, with the solid line being before the change and the dashed
>> line being after the change.
>>
>> Patch attached.
>>
>> -Toke
>
> That patch is really going to behave strangely indeed.  You end up with
>     b->cparams.interval = max(rtt_est_ns +  b->cparams.target -
> ns_target,b->cparams.target * 8);
>     b->cparams.target = max(max(byte_target_ns, ns_target),
> b->cparams.interval >> 4);
>
> So interval no matter what happens takes note of 'cparams.target' which
> hasn't yet been set (on first run is 0)
> And target wants to take interval/16, which may be based on the previous
> calculation.  Erm.  Needs more thought.
>
> My brain isn't working at the moment due to illness, but recursively
> playing with bandwidth settings on cake is most entertaining at the
> moment ;-)
>
> Kevin (dead with cold/high as a kite on cold drugs)
>
>
>
>
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>