[Cerowrt-devel] openwrt build available with latest cake and fq_pie

[Cerowrt-devel] openwrt build available with latest cake and fq_pie

[Dave Taht]

Hopefully, by creating a "tc-adv" package (now in ceropackages) we are
nearly at the last step for being able to do builds out of the main
openwrt tree. I am puzzled as to how to correctly override the default
"tc" package, but at least this built and worked for me the first
time.

so you can kill any local mods to the iproute2 package in your own
openwrt builds, and merely add tc-adv to your own build instead, and
build kmod-sched-fq_pie and kmod-sched-cake, and walla!

assuming this is now correct, the next step would be to push tc-adv
into some mainline openwrt repo (routing?) and get it and the kmod-*
stuff built regularly out of their build system. (and then! yea! try
some faster boxes like the linksys ac1900 and see what new breaks!)

Anyway, my barely tested latest build (cake works, at least) is at here:

http://snapon.lab.bufferbloat.net/~cero3/lupin/ar71xx/

This also includes the latest cake, although I disagree with jonathon
about the count/2 mod, might as well test.

-- 
Dave Täht
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] [Cake] openwrt build available with latest cake and fq_pie

[Alan Jenkins]

On 13/06/2015, Dave Taht <dave.taht@gmail.com> wrote:
> Hopefully, by creating a "tc-adv" package (now in ceropackages) we are
> nearly at the last step for being able to do builds out of the main
> openwrt tree. I am puzzled as to how to correctly override the default
> "tc" package, but at least this built and worked for me the first
> time.
>
> so you can kill any local mods to the iproute2 package in your own
> openwrt builds, and merely add tc-adv to your own build instead, and
> build kmod-sched-fq_pie and kmod-sched-cake, and walla!
>
> assuming this is now correct, the next step would be to push tc-adv
> into some mainline openwrt repo (routing?) and get it and the kmod-*
> stuff built regularly out of their build system. (and then! yea! try
> some faster boxes like the linksys ac1900 and see what new breaks!)
>
> Anyway, my barely tested latest build (cake works, at least) is at here:
>
> http://snapon.lab.bufferbloat.net/~cero3/lupin/ar71xx/
>
> This also includes the latest cake, although I disagree with jonathon
> about the count/2 mod, might as well test.

New build still works for my link :).  (15/1M dsl in the UK).

If I want to test cake continuously, I'll need to fix sqm-scripts to
pass the cake ATM options.  I switched back to fq_codel for now (that
worked as well).

Alan

Re: [Cerowrt-devel] [Cake] openwrt build available with latest cake and fq_pie

[Dave Taht]

On Sun, Jun 14, 2015 at 8:53 AM, Alan Jenkins
<alan.christopher.jenkins@gmail.com> wrote:
> On 13/06/2015, Dave Taht <dave.taht@gmail.com> wrote:
>> Hopefully, by creating a "tc-adv" package (now in ceropackages) we are
>> nearly at the last step for being able to do builds out of the main
>> openwrt tree. I am puzzled as to how to correctly override the default
>> "tc" package, but at least this built and worked for me the first
>> time.
>>
>> so you can kill any local mods to the iproute2 package in your own
>> openwrt builds, and merely add tc-adv to your own build instead, and
>> build kmod-sched-fq_pie and kmod-sched-cake, and walla!
>>
>> assuming this is now correct, the next step would be to push tc-adv
>> into some mainline openwrt repo (routing?) and get it and the kmod-*
>> stuff built regularly out of their build system. (and then! yea! try
>> some faster boxes like the linksys ac1900 and see what new breaks!)
>>
>> Anyway, my barely tested latest build (cake works, at least) is at here:
>>
>> http://snapon.lab.bufferbloat.net/~cero3/lupin/ar71xx/
>>
>> This also includes the latest cake, although I disagree with jonathon
>> about the count/2 mod, might as well test.

I do pretty strongly think count - 1 is the rightest thing still. I
did come up with two
other ideas for the resumption phase that might work.

1) schedule the next drop based on count, THEN do a count - 2 or
count/2 to be scheduled for the drop phase afterwards

2) measure the amount of time since the last entrance to the
resumption phase and resume with count at some level slightly above
that.

probably even more useful than just hacking around:

A) measure all the time spent in all the phases relative to the
induced workloads.

we had code like that in the original versions of codel.

> New build still works for my link :).  (15/1M dsl in the UK).
>
> If I want to test cake continuously, I'll need to fix sqm-scripts to
> pass the cake ATM options.  I switched back to fq_codel for now (that
> worked as well).

Patches gladly accepted (tc-adv now does parse the new keywords I
think), and we really, really, really do need to confirm that the atm
code works in every circumstance.

> Alan



-- 
Dave Täht
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] [Cake] openwrt build available with latest cake and fq_pie

[Jonathan Morton]

> On 14 Jun, 2015, at 19:09, Dave Taht <dave.taht@gmail.com> wrote:
> 
> I do pretty strongly think count - 1 is the rightest thing still.

I really don’t.  Here’s why:

Every time Codel triggers the dropping state, it will mark or drop at least one packet, and increment count by that number.  With count decremented only by 1 on recovery, it will effectively remain constant *if*, by some miracle, the queue empties before the second signal was sent; it cannot decrease between episodes unless it resets or wraps.

With count decremented by 2 on recovery, it is possible for count to decrease slowly in that ideal case, but it’ll remain constant if two signals were sent before the queue cleared, and - this is important - it will always continue to increase if three or more signals are sent before the queue empties.

If one signal did suffice to clear the queue, then logically the value of count was irrelevant to that congestion episode and shouldn’t be preserved.  This is true regardless of the actual reason the queue emptied.

The problem arises when more than one signal is sent before the queue is observed to clear.  This could be a sign of several distinct network conditions:

- The RTT is longer than interval / sqrt(count), in which case one signal would still have been sufficient, and the ideal value of count is less than its current value.  On non-ECN TCP flows, this results in more retransmissions than necessary.

- The RTT is much shorter than interval / sqrt(count), so the congestion window is recovering faster than the signalling rate, and count needs to increase to compensate for that.

- There is more than one flow sharing the queue, and it was necessary to signal to all of them, in which case count should reflect the flow count and be capable of adjusting both up and down.

- The flow is unresponsive, so count should adjust to provide the correct dropping rate, and RTT is irrelevant.  With default parameters, the maximum drop rate is presently 25600 pps (which would cause count to wrap after a few seconds, until I put in the saturating arithmetic).

How does Codel distinguish between those cases?  It can’t - at least, not reliably.  So it must allow count to increase until the queue is observed to be controlled, and then decrease count by some other means to cover the case where it was overestimated.  For this latter phase, count-2 is obviously insufficient to cope with the case where count is actually correct, but more than one signal per episode is required.

*That* is why I put in count/2.  A multiplicative decrease allows count to stabilise at some value which adequately controls the queue, rather than continuously increasing past it.  For the typical cake case where there is one flow per Codel instance and the RTT is of Internet scale, this should work at least as well as an additive decrease; in particular, the behaviour is identical where count ended at 2, 3 or 4 (it can’t end at 1).

Of course, hard data would help to evaluate it, but I do think it’s theoretically sound.

 - Jonathan Morton

Re: [Cerowrt-devel] [Cake] openwrt build available with latest cake and fq_pie

[Toke Høiland-Jørgensen]

Jonathan Morton <chromatix99@gmail.com> writes:

> *That* is why I put in count/2. A multiplicative decrease allows count
> to stabilise at some value which adequately controls the queue, rather
> than continuously increasing past it. For the typical cake case where
> there is one flow per Codel instance and the RTT is of Internet scale,
> this should work at least as well as an additive decrease; in
> particular, the behaviour is identical where count ended at 2, 3 or 4
> (it can’t end at 1).

Is there any reason why the decrease couldn't be some sort of decay?
I.e. a function of how long ago the drop state was exited?

-Toke

Re: [Cerowrt-devel] [Cake] openwrt build available with latest cake and fq_pie

[Dave Taht]

adding in codel list. I do think we need a better understanding from
observing flow behavior in the fq case than we have.

certainly we had a case in the early days where count increased without bound.

On Sun, Jun 14, 2015 at 10:19 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>
>> On 14 Jun, 2015, at 19:09, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> I do pretty strongly think count - 1 is the rightest thing still.
>
> I really don’t.  Here’s why:
>
> Every time Codel triggers the dropping state, it will mark or drop at least one packet, and increment count by that number.  With count decremented only by 1 on recovery, it will effectively remain constant *if*, by some miracle, the queue empties before the second signal was sent; it cannot decrease between episodes unless it resets or wraps.

It aint a miracle, it is hopefully within an rtt.

> With count decremented by 2 on recovery, it is possible for count to decrease slowly in that ideal case, but it’ll remain constant if two signals were sent before the queue cleared, and - this is important - it will always continue to increase if three or more signals are sent before the queue empties.

quibble: queue's latency falls below the target delay.

>
> If one signal did suffice to clear the queue, then logically the value of count was irrelevant to that congestion episode and shouldn’t be preserved.  This is true regardless of the actual reason the queue emptied.
>
> The problem arises when more than one signal is sent before the queue is observed to clear.  This could be a sign of several distinct network conditions:
>
> - The RTT is longer than interval / sqrt(count), in which case one signal would still have been sufficient, and the ideal value of count is less than its current value.  On non-ECN TCP flows, this results in more retransmissions than necessary.
>
> - The RTT is much shorter than interval / sqrt(count), so the congestion window is recovering faster than the signalling rate, and count needs to increase to compensate for that.
>
> - There is more than one flow sharing the queue, and it was necessary to signal to all of them, in which case count should reflect the flow count and be capable of adjusting both up and down.
>
> - The flow is unresponsive, so count should adjust to provide the correct dropping rate, and RTT is irrelevant.  With default parameters, the maximum drop rate is presently 25600 pps (which would cause count to wrap after a few seconds, until I put in the saturating arithmetic).

There was some good discussion of things on another list recently
(can't remember which) at 100Gig rates.

>
> How does Codel distinguish between those cases?  It can’t - at least, not reliably.  So it must allow count to increase until the queue is observed to be controlled, and then decrease count by some other means to cover the case where it was overestimated.  For this latter phase, count-2 is obviously insufficient to cope with the case where count is actually correct, but more than one signal per episode is required.
>
> *That* is why I put in count/2.

When resuming the drop phase of codel, it is almost *already* too late
to catch that burst incurring the latency.

Sometimes I think we need to do away with the count idea and measure
slopes of curves instead, and "harmonics".

>A multiplicative decrease allows count to stabilise at some value which adequately controls the queue, rather than continuously increasing past it.  For the typical cake case where there is one flow per Codel instance and the RTT is of Internet scale, this should work at least as well as an additive decrease; in particular, the behaviour is identical where count ended at 2, 3 or 4 (it can’t end at 1).

> Of course, hard data would help to evaluate it, but I do think it’s theoretically sound.

adding in codel list.

The brief bits of data I got so far show it failing to control queue,
building over time.

>  - Jonathan Morton
>



-- 
Dave Täht
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] [Cake] openwrt build available with latest cake and fq_pie

[Jonathan Morton]

> On 14 Jun, 2015, at 20:38, Dave Taht <dave.taht@gmail.com> wrote:
> 
>> Every time Codel triggers the dropping state, it will mark or drop at least one packet, and increment count by that number.  With count decremented only by 1 on recovery, it will effectively remain constant *if*, by some miracle, the queue empties before the second signal was sent; it cannot decrease between episodes unless it resets or wraps.
> 
> It aint a miracle, it is hopefully within an rtt.

No, it is *at minimum* one RTT.  It takes that long for the congestion signal to reach the receiver, be reflected back to the sender, the sender’s reaction to *begin to* appear at the queue.  Then the queue *starts* to empty, if the signal is what’s required to make it do so.

> When resuming the drop phase of codel, it is almost *already* too late
> to catch that burst incurring the latency.

Yes, but that’s what FQ is for.  And ELR, if we ever get that properly started.

> Sometimes I think we need to do away with the count idea and measure
> slopes of curves instead, and "harmonics”.


> Is there any reason why the decrease couldn't be some sort of decay?
> I.e. a function of how long ago the drop state was exited?

Such things are theoretically possible, but require further thought to determine how best to do them.

 - Jonathan Morton

Re: [Cerowrt-devel] [Cake] openwrt build available with latest cake and fq_pie

[Dave Taht]

On Sun, Jun 14, 2015 at 11:07 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>
>> On 14 Jun, 2015, at 20:38, Dave Taht <dave.taht@gmail.com> wrote:
>>
>>> Every time Codel triggers the dropping state, it will mark or drop at least one packet, and increment count by that number.  With count decremented only by 1 on recovery, it will effectively remain constant *if*, by some miracle, the queue empties before the second signal was sent; it cannot decrease between episodes unless it resets or wraps.
>>
>> It aint a miracle, it is hopefully within an rtt.
>
> No, it is *at minimum* one RTT.  It takes that long for the congestion signal to reach the receiver, be reflected back to the sender, the sender’s reaction to *begin to* appear at the queue.  Then the queue *starts* to empty, if the signal is what’s required to make it do so.

Flows, btw, do end quite rapidly in the real world. What was it, 95%
of all web flows ended inside of IW10? Dropping a packet at the tail
end of an ending flow forces a retransmit. Far too much of our testing
and thinking is oriented to full rate flows.

That is what I meant by "hopefully". I would like to build a test that
consisted primarily of short flows, one that hopefully looked more
like normal traffic.

>> When resuming the drop phase of codel, it is almost *already* too late
>> to catch that burst incurring the latency.
>
> Yes, but that’s what FQ is for.  And ELR, if we ever get that properly started.
>
>> Sometimes I think we need to do away with the count idea and measure
>> slopes of curves instead, and "harmonics”.
>
>
>> Is there any reason why the decrease couldn't be some sort of decay?
>> I.e. a function of how long ago the drop state was exited?
>
> Such things are theoretically possible, but require further thought to determine how best to do them.
>
>  - Jonathan Morton
>



-- 
Dave Täht
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] [Cake] openwrt build available with latest cake and fq_pie

[Jonathan Morton]

> On 14 Jun, 2015, at 21:24, Dave Taht <dave.taht@gmail.com> wrote:
> 
> Flows, btw, do end quite rapidly in the real world. What was it, 95%
> of all web flows ended inside of IW10?

It might be worth thinking about how heavily loaded a network needs to be for Codel to trigger on such a flow.  However, with perfect flow isolation, count will start at 1, making it less relevant to the present thread.

Cake will start triggering on a instantly-arrived burst (call it a packet salvo) after 35ms.  Thus, a ten-packet burst will not trigger provided at least 4.5 Mbps is available to that flow.  However, on many links that is still a tall order, since if the flows really are that short, there are probably lots of them in parallel.

A paced burst is much more friendly.  At a 100ms RTT, IW10 could be delivered at 100pps (1.5 Mbps), extending the range of link speeds on which congestion signalling will not occur by at least 3x (and generally more).  Since this greatly reduces the risk of packet loss, it might actually reduce the average time that the sender needs to maintain the connection’s buffers, despite the deliberate 1-RTT delay introduced.

 - Jonathan Morton

Re: [Cerowrt-devel] [Cake] openwrt build available with latest cake and fq_pie

[Dave Taht]

On Sun, Jun 14, 2015 at 12:35 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
>
>> On 14 Jun, 2015, at 21:24, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> Flows, btw, do end quite rapidly in the real world. What was it, 95%
>> of all web flows ended inside of IW10?
>
> It might be worth thinking about how heavily loaded a network needs to be for Codel to trigger on such a flow.  However, with perfect flow isolation, count will start at 1, making it less relevant to the present thread.
>
> Cake will start triggering on a instantly-arrived burst (call it a packet salvo) after 35ms.  Thus, a ten-packet burst will not trigger provided at least 4.5 Mbps is available to that flow.  However, on many links that is still a tall order, since if the flows really are that short, there are probably lots of them in parallel.
>
> A paced burst is much more friendly.  At a 100ms RTT, IW10 could be delivered at 100pps (1.5 Mbps), extending the range of link speeds on which congestion signalling will not occur by at least 3x (and generally more).  Since this greatly reduces the risk of packet loss, it might actually reduce the average time that the sender needs to maintain the connection’s buffers, despite the deliberate 1-RTT delay introduced.

Data point: Quic is presently 10 packet burst, 22 packets paced. Not
that I like it.

>  - Jonathan Morton
>



-- 
Dave Täht
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] [Cake] openwrt build available with latest cake and fq_pie

[Alan Jenkins]

On 14/06/15 17:09, Dave Taht wrote:
> On Sun, Jun 14, 2015 at 8:53 AM, Alan Jenkins
> <alan.christopher.jenkins@gmail.com> wrote:
>> On 13/06/2015, Dave Taht <dave.taht@gmail.com> wrote:
>>> Hopefully, by creating a "tc-adv" package (now in ceropackages) we are
>>> nearly at the last step for being able to do builds out of the main
>>> openwrt tree. I am puzzled as to how to correctly override the default
>>> "tc" package, but at least this built and worked for me the first
>>> time.
>>>
>>> so you can kill any local mods to the iproute2 package in your own
>>> openwrt builds, and merely add tc-adv to your own build instead, and
>>> build kmod-sched-fq_pie and kmod-sched-cake, and walla!
>>>
>>> assuming this is now correct, the next step would be to push tc-adv
>>> into some mainline openwrt repo (routing?) and get it and the kmod-*
>>> stuff built regularly out of their build system. (and then! yea! try
>>> some faster boxes like the linksys ac1900 and see what new breaks!)
>>>
>>> Anyway, my barely tested latest build (cake works, at least) is at here:
>>>
>>> http://snapon.lab.bufferbloat.net/~cero3/lupin/ar71xx/
>>>
>>> This also includes the latest cake, although I disagree with jonathon
>>> about the count/2 mod, might as well test.

>> New build still works for my link :).  (15/1M dsl in the UK).
>>
>> If I want to test cake continuously, I'll need to fix sqm-scripts to
>> pass the cake ATM options.  I switched back to fq_codel for now (that
>> worked as well).
> Patches gladly accepted (tc-adv now does parse the new keywords I
> think)

Yes to both.  I'd already tested "cake atm" + "stab overhead".  This 
time I was dropping "stab" and using "cake atm overhead 44", which tc 
accepted...

Sigh, I forgot the main reason I watched for a second build.  To be sure 
of "cake overhead" I really need to retest closer to the link speed.  I 
have a specific method for it.

The test is whether it matches "tc stab overhead" in allowing higher 
rates/lower latency on RRUL.  As RRUL saturates the uplink, you have to 
account for high ATM overhead on the TCP ACK packets there.  And the 
bandwidth consumed by ACKs (and their overhead) is significant on the 
uplink because of the asymmetric link rate.  My pleasure at 
understanding this is mitigated by how long it took for the light to 
dawn :).

> , and we really, really, really do need to confirm that the atm
> code works in every circumstance.

I'm still with you :), I'll have another go in a few days.  I've got 
some pretty monitoring (smokeping) now, for if I get cake running 
permanently.  It doesn't seem particularly sensitive to this stuff[1] 
but it should show any massive screwup in the rate-limiter :).

Alan

[1] it seems my link is already relatively good & my usage is relatively 
undemanding.

Re: [Cerowrt-devel] [Cake] openwrt build available with latest cake and fq_pie

[Sebastian Moeller]

On Jun 14, 2015, at 21:32 , Alan Jenkins <alan.christopher.jenkins@gmail.com> wrote:

> On 14/06/15 17:09, Dave Taht wrote:
>>> [...]
>> Patches gladly accepted (tc-adv now does parse the new keywords I
>> think)
> 
> Yes to both.  I'd already tested "cake atm" + "stab overhead".  This time I was dropping "stab" and using "cake atm overhead 44", which tc accepted…

	Just to be explicit: this combination worked correctly? If so, changing sqm-scripts to allow the cake way of specifying overheads gets less urgent… Did you by any chance also try to use tic’s stab method to specify overhead and link layer ATM? If that does not work sqm-scripts need a fix quickly anyways (otherwise people will get tangled in our crafty net…)
	I am currently mulling over how to include cake’s more user-friendly way to specify overheads into sqm-scripts, and have no good solution yet. One hack I would like to propose is to attach the “Advanced option string to pass to the ingress queueing disciplines; no error checking, use very carefully.” to the cake invocation to allow early adopters to pass arbitrary strings to cake, so they can keep the guy. The main issue is that this comes with no safety checks at all, and I have no idea how cake deals with wrong inputs (as I have not been able to build it under opens use 13.2 yet).

> 
> Sigh, I forgot the main reason I watched for a second build.  To be sure of "cake overhead" I really need to retest closer to the link speed.  I have a specific method for it.

	So for me the following worked well enough: set the shaper to the exact uplink sync rate as specified in the modem, run RRUL against the nearest server for 300 seconds or so, with the correct overhead and link layer options. On my link latency under load started to increase sharply once the overhead was reduced by a single byte. 
	This sensitivity actually allowed me to catch an episode when my ISP had increased the overhead by 4 bytes temporarily (after the RRUL results I re-ran my arm-overhead detector which confirmed the increase by 4 bytes).

> 
> The test is whether it matches "tc stab overhead" in allowing higher rates/lower latency on RRUL.  As RRUL saturates the uplink, you have to account for high ATM overhead on the TCP ACK packets there.  And the bandwidth consumed by ACKs (and their overhead) is significant on the uplink because of the asymmetric link rate.  

	Back of envelop calculation gives an estimate of 2% of downlink-bandwidth as reverse traffic as ACK (before ATM does its dirty work and quantisation).

> My pleasure at understanding this is mitigated by how long it took for the light to dawn :).

	Yeah, I guess you will not shed a tear once ATM goes the way of the dodo?


Best Regards
	Sebastian

> 
>> , and we really, really, really do need to confirm that the atm
>> code works in every circumstance.
> 
> I'm still with you :), I'll have another go in a few days.  I've got some pretty monitoring (smokeping) now, for if I get cake running permanently.  It doesn't seem particularly sensitive to this stuff[1] but it should show any massive screwup in the rate-limiter :).
> 
> Alan
> 
> [1] it seems my link is already relatively good & my usage is relatively undemanding.
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Cake] openwrt build available with latest cake and fq_pie

[Alan Jenkins]

On 14/06/15 20:47, Sebastian Moeller wrote:
> On Jun 14, 2015, at 21:32 , Alan Jenkins <alan.christopher.jenkins@gmail.com> wrote:
>
>> On 14/06/15 17:09, Dave Taht wrote:
>>>> [...]
>>> Patches gladly accepted (tc-adv now does parse the new keywords I
>>> think)
>> Yes to both.  I'd already tested "cake atm" + "stab overhead".  This time I was dropping "stab" and using "cake atm overhead 44", which tc accepted…
> 	Just to be explicit: this combination worked correctly?

Yes.  I found "cake atm" + "stab overhead" worked fine (and reported it 
on the cake list).

> If so, changing sqm-scripts to allow the cake way of specifying overheads gets less urgent…

If I'm reading the code correctly, sqm-scripts doesn't use _any_ way of 
setting overhead when qdisc is set to cake.  Maybe I got it wrong though.

(other todo would be to add cake to luci-app-sqm, doesn't bother me 
either way though)

> Did you by any chance also try to use tic’s stab method to specify overhead and link layer ATM? If that does not work sqm-scripts need a fix quickly anyways (otherwise people will get tangled in our crafty net…)

Hmm, looks like I didn't record that.  I thought it works but I'm not 
sure now.

> 	I am currently mulling over how to include cake’s more user-friendly way to specify overheads into sqm-scripts, and have no good solution yet. One hack I would like to propose is to attach the “Advanced option string to pass to the ingress queueing disciplines; no error checking, use very carefully.” to the cake invocation to allow early adopters to pass arbitrary strings to cake, so they can keep the guy. The main issue is that this comes with no safety checks at all, and I have no idea how cake deals with wrong inputs (as I have not been able to build it under opens use 13.2 yet).
>
>> Sigh, I forgot the main reason I watched for a second build.  To be sure of "cake overhead" I really need to retest closer to the link speed.  I have a specific method for it.
> 	So for me the following worked well enough: set the shaper to the exact uplink sync rate as specified in the modem, run RRUL against the nearest server for 300 seconds or so, with the correct overhead and link layer options. On my link latency under load started to increase sharply once the overhead was reduced by a single byte.
> 	This sensitivity actually allowed me to catch an episode when my ISP had increased the overhead by 4 bytes temporarily (after the RRUL results I re-ran my arm-overhead detector which confirmed the increase by 4 bytes).

Pretty sure I have to set the rate lower :(.  Maybe an ISP shaper or 
policer like you mentioned noticing.

>
>> The test is whether it matches "tc stab overhead" in allowing higher rates/lower latency on RRUL.  As RRUL saturates the uplink, you have to account for high ATM overhead on the TCP ACK packets there.  And the bandwidth consumed by ACKs (and their overhead) is significant on the uplink because of the asymmetric link rate.
> 	Back of envelop calculation gives an estimate of 2% of downlink-bandwidth as reverse traffic as ACK (before ATM does its dirty work and quantisation).

It's getting late now :).  But my theoretical calculation was much 
higher, and it matched the gap in measured bandwidth (tcp goodput) 
between a 1-way and 2-way test (rrul).  740 and 390 K respectively.

http://www.dslreports.com/forum/r30046195-FYI-for-general-feedback-on-the-new-speedtest

that said my calculation assumed 1 ack for each tcp segment (packet).  
Apparently 1 ack per 2 segments is also common ("delayed ack") and I 
didn't look at whether that happened in my test.

>> My pleasure at understanding this is mitigated by how long it took for the light to dawn :).
> 	Yeah, I guess you will not shed a tear once ATM goes the way of the dodo?

I like your jokes.  Don't you know, old technology never dies :-D

> Best Regards
> 	Sebastian
>
>>> , and we really, really, really do need to confirm that the atm
>>> code works in every circumstance.
>> I'm still with you :), I'll have another go in a few days.  I've got some pretty monitoring (smokeping) now, for if I get cake running permanently.  It doesn't seem particularly sensitive to this stuff[1] but it should show any massive screwup in the rate-limiter :).
>>
>> Alan
>>
>> [1] it seems my link is already relatively good & my usage is relatively undemanding.
>>

Re: [Cerowrt-devel] [Cake] openwrt build available with latest cake and fq_pie

[Sebastian Moeller]

Hi Alan,

On Jun 14, 2015, at 22:43 , Alan Jenkins <alan.christopher.jenkins@gmail.com> wrote:

> On 14/06/15 20:47, Sebastian Moeller wrote:
>> On Jun 14, 2015, at 21:32 , Alan Jenkins <alan.christopher.jenkins@gmail.com> wrote:
>> 
>>> On 14/06/15 17:09, Dave Taht wrote:
>>>>> [...]
>>>> Patches gladly accepted (tc-adv now does parse the new keywords I
>>>> think)
>>> Yes to both.  I'd already tested "cake atm" + "stab overhead".  This time I was dropping "stab" and using "cake atm overhead 44", which tc accepted…
>> 	Just to be explicit: this combination worked correctly?
> 
> Yes.  I found "cake atm" + "stab overhead" worked fine (and reported it on the cake list).
> 
>> If so, changing sqm-scripts to allow the cake way of specifying overheads gets less urgent…
> 
> If I'm reading the code correctly, sqm-scripts doesn't use _any_ way of setting overhead when qdisc is set to cake.  Maybe I got it wrong though.

	No, you are right, it is not hooked up at all, my bad. Since I still fail t building cake I can not test it, and my track record is bad enough with changes I could have tested, that I want to avoid checking stuff that should work in theory, without any testing ;)

> (other todo would be to add cake to luci-app-sqm, doesn't bother me either way though)
> 
>> Did you by any chance also try to use tic’s stab method to specify overhead and link layer ATM? If that does not work sqm-scripts need a fix quickly anyways (otherwise people will get tangled in our crafty net…)
> 
> Hmm, looks like I didn't record that.  I thought it works but I'm not sure now.

	If you could retest that some of these days I would be delighted, but then cake’s and ht.’s ways of dealing with ATM are more generic than tc stab’s so maybe I can not avoid doing the cake changes anyway...

> 
>> 	I am currently mulling over how to include cake’s more user-friendly way to specify overheads into sqm-scripts, and have no good solution yet. One hack I would like to propose is to attach the “Advanced option string to pass to the ingress queueing disciplines; no error checking, use very carefully.” to the cake invocation to allow early adopters to pass arbitrary strings to cake, so they can keep the guy. The main issue is that this comes with no safety checks at all, and I have no idea how cake deals with wrong inputs (as I have not been able to build it under opens use 13.2 yet).
>> 
>>> Sigh, I forgot the main reason I watched for a second build.  To be sure of "cake overhead" I really need to retest closer to the link speed.  I have a specific method for it.
>> 	So for me the following worked well enough: set the shaper to the exact uplink sync rate as specified in the modem, run RRUL against the nearest server for 300 seconds or so, with the correct overhead and link layer options. On my link latency under load started to increase sharply once the overhead was reduced by a single byte.
>> 	This sensitivity actually allowed me to catch an episode when my ISP had increased the overhead by 4 bytes temporarily (after the RRUL results I re-ran my arm-overhead detector which confirmed the increase by 4 bytes).
> 
> Pretty sure I have to set the rate lower :(.  Maybe an ISP shaper or policer like you mentioned noticing.

	Isn’t it great, every ISP a snow flake ;)

> 
>> 
>>> The test is whether it matches "tc stab overhead" in allowing higher rates/lower latency on RRUL.  As RRUL saturates the uplink, you have to account for high ATM overhead on the TCP ACK packets there.  And the bandwidth consumed by ACKs (and their overhead) is significant on the uplink because of the asymmetric link rate.
>> 	Back of envelop calculation gives an estimate of 2% of downlink-bandwidth as reverse traffic as ACK (before ATM does its dirty work and quantisation).
> 
> It's getting late now :).  But my theoretical calculation was much higher, and it matched the gap in measured bandwidth (tcp goodput) between a 1-way and 2-way test (rrul).  740 and 390 K respectively.
> 
> http://www.dslreports.com/forum/r30046195-FYI-for-general-feedback-on-the-new-speedtest
> 
> that said my calculation assumed 1 ack for each tcp segment (packet).  Apparently 1 ack per 2 segments is also common ("delayed ack") and I didn't look at whether that happened in my test.

	I believe the RFC’s state at least one ACK every two MSSs, and if I look at iftop while running rrul tests (under macosx 10.9.5) i get something close to 2%, but I tried to hedge with the “back of envelop” ;)

> 
>>> My pleasure at understanding this is mitigated by how long it took for the light to dawn :).
>> 	Yeah, I guess you will not shed a tear once ATM goes the way of the dodo?
> 
> I like your jokes.  Don't you know, old technology never dies :-D

	Right you are, this is partly th reason why I think that cake should be able to solve this for unsuspecting users that might be trapped on planet ATM for years to come. (I always wonder whether ATM gear really is nearing EOL or whether it will get a second wind in a less developed market; I certainly hope that it truly dies one of the days, like vampires ATM has its charms and yet the world would be a better place without ;) )

Best Regards
        Sebastian

> 
>> Best Regards
>> 	Sebastian
>> 
>>>> , and we really, really, really do need to confirm that the atm
>>>> code works in every circumstance.
>>> I'm still with you :), I'll have another go in a few days.  I've got some pretty monitoring (smokeping) now, for if I get cake running permanently.  It doesn't seem particularly sensitive to this stuff[1] but it should show any massive screwup in the rate-limiter :).
>>> 
>>> Alan
>>> 
>>> [1] it seems my link is already relatively good & my usage is relatively undemanding.
>>> 
>