[Cake] memory

[Cake] memory

[Dave Taht]

There has been so much traffic here that I can't summarize.

A) But a bit on memory limits - the memory limit enforced in cake is a
sanity check sort of limit. There are *no* allocations of memory in
it. It will not fail itself, due to running out of memory except at
init time.

So if you run out of memory elsewhere in the system, the normal set of
bad things happen. cake's "sane - and, yes, could use more smarts"
limits can reduce memory pressure elsewhere in the system by
discarding things when it gets irrational, but packet memory tends to
be fragmented and hard to recover cleanly in the first place.

B) Similarly a HUGE waster of memory is small packets, which usually
get a full slab of bytes (2k) to play with on the rx side. This
problem got so bad in some testing that openwrt contains a clever (I
would say that because I wrote it), patch that once we get to tons of
packets more than we think is sane and we get close to running out of
2k slabs that we start reallocating packets to fit into much smaller
slabs (like 512 bytes) and copying them into those.

so, briefly, memory allocation and release patterns are more complex
than the discussion I sort of saw go by a over the last few days.



Dave Täht
I just invested five years of my life to making wifi better. And,
now... the FCC wants to make my work, illegal for people to install.
https://www.gofundme.com/savewifi

Re: [Cake] memory

[Sebastian Moeller]

Hi Dave,

On Oct 29, 2015, at 10:01 , Dave Taht <dave.taht@gmail.com> wrote:

> There has been so much traffic here that I can't summarize.
> 
> A) But a bit on memory limits - the memory limit enforced in cake is a
> sanity check sort of limit. There are *no* allocations of memory in
> it. It will not fail itself, due to running out of memory except at
> init time.

	This is all quite interesting and correct; the point I failed to make clearly it seems, is that a 50MB limit on a 64MB router can not be called sane or safe, no matter how you slice or dice it… It is one thing having cake tell me that it thinks the limit I gave is to low to reach full throughput as long as it accepts the given limit.

> 
> So if you run out of memory elsewhere in the system, the normal set of
> bad things happen. cake's "sane - and, yes, could use more smarts"
> limits can reduce memory pressure elsewhere in the system by
> discarding things when it gets irrational, but packet memory tends to
> be fragmented and hard to recover cleanly in the first place.

	To come back to my example, the big issue is that unlike a lot of other allocations the queue is filled by tpackets generated by connected machines, it would be sweet if it would be harder for a mischievous user in my internal network to OOM my router… I guess what I want to say is there is not one size fits all size for a safe limit, it really depends on the machines memory (on the lower end) and on expected traffic at the upper end.


> 
> B) Similarly a HUGE waster of memory is small packets, which usually
> get a full slab of bytes (2k) to play with on the rx side. This
> problem got so bad in some testing that openwrt contains a clever (I
> would say that because I wrote it), patch that once we get to tons of
> packets more than we think is sane and we get close to running out of
> 2k slabs that we start reallocating packets to fit into much smaller
> slabs (like 512 bytes) and copying them into those.

	Clever! This argues for a limit on the sum of skb->true size of all queued packets instead of just the number of packets. On the other hand number of packets will not treat small and large packets differently and hence the number of packets limit should correlate pretty nicely with the consumed memory. But for this to work cake needs to interpret the limit as a hard maximum limit.

> 
> so, briefly, memory allocation and release patterns are more complex
> than the discussion I sort of saw go by a over the last few days.

	I know that I am just a layman, but the limit thing does not seem to be too complex, we are really just talking about a safety valve, that does not need to be hyper precise, but approximately correlated with the consumed memory. Currently, setting oldish cake up on in- and egress would have the worst case queue memory somewhere at (2 * 10240 * 2) = 40960 KB which on a 64MB is already larger than I would prefer and totally ineffective on a 32MB router ;). I am by the war fine with cake defaulting to what ever number is deemed good, all I want is to be able to tell cake something smaller if need be. 
	But than cake is not my baby, and all I can do is try to convince Jonathan, Toke and you that I believe that while cake’s principle should be simplicity in use that does not need to mean no knobs as long as everything works fine in most cases without the need to touch those knobs ;)


Best Regards
	Sebastian


> 
> Dave Täht
> I just invested five years of my life to making wifi better. And,
> now... the FCC wants to make my work, illegal for people to install.
> https://www.gofundme.com/savewifi
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] memory

[Toke Høiland-Jørgensen]

Sebastian Moeller <moeller0@gmx.de> writes:

> 	This is all quite interesting and correct; the point I failed to make
> clearly it seems, is that a 50MB limit on a 64MB router can not be called sane
> or safe, no matter how you slice or dice it… It is one thing having cake tell me
> that it thinks the limit I gave is to low to reach full throughput as long as it
> accepts the given limit.

Well, a solution would be to not configure your 64-MB router with a
100Mbit / 1000ms connection :)

-Toke

Re: [Cake] memory

[Sebastian Moeller]

Hi Toke,

On Oct 29, 2015, at 11:04 , Toke Høiland-Jørgensen <toke@toke.dk> wrote:

> Sebastian Moeller <moeller0@gmx.de> writes:
> 
>> 	This is all quite interesting and correct; the point I failed to make
>> clearly it seems, is that a 50MB limit on a 64MB router can not be called sane
>> or safe, no matter how you slice or dice it… It is one thing having cake tell me
>> that it thinks the limit I gave is to low to reach full throughput as long as it
>> accepts the given limit.
> 
> Well, a solution would be to not configure your 64-MB router with a
> 100Mbit / 1000ms connection :)

	Fair enough ;) Let’s get the soldering iron out and see whether I can replace the phi with a 10Mbit one ;), mmmh, maybe I should run 10BASE5 and do away with all that modernist “hub” and “switch” witch craft while I am at it
Joking aside, I believe the issue only occurred on loooong RTTs so it will only be a minor issue in real life, but still it would be nice if cake would not introduce the possibility for remote parties to make my router OOM/reboot ;). I might be odd, but I prefer a hit in throughput to a forced reboot any day ;)

Best Regards
	Sebastian

> 
> -Toke

Re: [Cake] memory

[Toke Høiland-Jørgensen]

Sebastian Moeller <moeller0@gmx.de> writes:

> Fair enough ;) Let’s get the soldering iron out and see whether I can
> replace the phi with a 10Mbit one ;), mmmh, maybe I should run 10BASE5
> and do away with all that modernist “hub” and “switch” witch craft
> while I am at it Joking aside, I believe the issue only occurred on
> loooong RTTs so it will only be a minor issue in real life, but still
> it would be nice if cake would not introduce the possibility for
> remote parties to make my router OOM/reboot ;). I might be odd, but I
> prefer a hit in throughput to a forced reboot any day ;)

The "1 second RTT" part was a key part in that... If the RTT is set to
100ms (the default), you'll only get a limit of 12.5 MB...

Now, there is still an issue to be resolved in "unlimited" mode. But
that's a separate issue.

-Toke

Re: [Cake] memory

[Sebastian Moeller]

Hi Toke,


On Oct 29, 2015, at 11:14 , Toke Høiland-Jørgensen <toke@toke.dk> wrote:

> Sebastian Moeller <moeller0@gmx.de> writes:
> 
>> Fair enough ;) Let’s get the soldering iron out and see whether I can
>> replace the phi with a 10Mbit one ;), mmmh, maybe I should run 10BASE5
>> and do away with all that modernist “hub” and “switch” witch craft
>> while I am at it Joking aside, I believe the issue only occurred on
>> loooong RTTs so it will only be a minor issue in real life, but still
>> it would be nice if cake would not introduce the possibility for
>> remote parties to make my router OOM/reboot ;). I might be odd, but I
>> prefer a hit in throughput to a forced reboot any day ;)
> 
> The "1 second RTT" part was a key part in that... If the RTT is set to
> 100ms (the default), you'll only get a limit of 12.5 MB…

	Yes, while trying to be funny I glossed over that part a bit (the loooong RTTs part was aiming at that), sorry. But really I am wonder what it is that makes my position that cake should expose and honor a packet limit for its queueing so controversial; I am totally happy with cake automatically selecting a good default automatically using all information it has available, like configured bandwidth and RTT, all I want to avoid the situation where cake divined something that is totally consistent within the information it has available, but is obviously wrong given additional information out of cake’s control (like total memory size or even maximum memory size the user is willing to put aside for cake). I think cake’s goal of being simple to use is commendable, but we should make sure we do not accidentally end up with cake being simplistic… (rather make the normal things easy/simple and complex/crazy things possible ;) )

Best Regards
	Sebastian


> 
> Now, there is still an issue to be resolved in "unlimited" mode. But
> that's a separate issue.
> 
> -Toke

Re: [Cake] memory

[Toke Høiland-Jørgensen]

Sebastian Moeller <moeller0@gmx.de> writes:

> Yes, while trying to be funny I glossed over that part a bit (the
> loooong RTTs part was aiming at that), sorry. But really I am wonder
> what it is that makes my position that cake should expose and honor a
> packet limit for its queueing so controversial;

The suggestion in itself is not necessarily controversial. However, the
road to bloated interfaces is covered in good intentions. Each addition
may be worthwhile in itself, but the sum of them becomes a horrible mess
to configure (I'm being a bit hyperbolic here, but I'm sure you get my
gist). So I'd rather say that for each possible configuration variable,
we should figure out if there is a way to avoid exposing it, and only
exposing it if we can't find a way to do that. Rather than going with
the "easy" solution of just making it configurable and leaving the user
to figure it out.

-Toke

Re: [Cake] memory

[Toke Høiland-Jørgensen]

Sebastian Moeller <moeller0@gmx.de> writes:

> Hi Toke,
>
> On Oct 29, 2015, at 12:02 , Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
>> Sebastian Moeller <moeller0@gmx.de> writes:
>> 
>>> Yes, while trying to be funny I glossed over that part a bit (the
>>> loooong RTTs part was aiming at that), sorry. But really I am wonder
>>> what it is that makes my position that cake should expose and honor a
>>> packet limit for its queueing so controversial;
>> 
>> The suggestion in itself is not necessarily controversial. However, the
>> road to bloated interfaces is covered in good intentions. Each addition
>> may be worthwhile in itself, but the sum of them becomes a horrible mess
>> to configure (I'm being a bit hyperbolic here, but I'm sure you get my
>> gist). So I'd rather say that for each possible configuration variable,
>> we should figure out if there is a way to avoid exposing it, and only
>> exposing it if we can't find a way to do that. Rather than going with
>> the "easy" solution of just making it configurable and leaving the user
>> to figure it out.
>
> 	I am all for empowering the users to change the settings (well, the few
> settings we talk about here, target, interval, limit, encapsulation and
> overhead, I believe bandwidth and diffserv-scheme not contentious ;) ) in
> COMBINATION with having sane defaults that make twiddling not necessary AND
> potentially scary warnings if users configure something cake considers insane.
> Exposing parameters that by all probability will not go away does not seem
> dangerous to me, but the right thing to do. I venture the guess that if cake
> should give up using codel underneath it is time for a rename, and cake2 or
> cookie can have different exposed parameters than cake, no?

> 	We avoid the “horrible mess” part by basically requiring none of
> the parameters (or just few if none is impossible) to be specified and
> fill in all not specified parameters using the current auto-setting
> methods. I am fine with not exposing implementation details, but the
> parameters that will stay with us until we change the algorithms like
> target and interval should be exposed (unless we are dead certain that
> our automatic methods will configure the objectively best parameters
> for all conditions).

Well, that's the thing: It is not clear to me that limit and target are
not exactly that: implementation details.

Note that I did say we should be sure that they are, and I'm not sure we
are quite there yet. But I do believe that we should first make
absolutely sure that these are parameters that are not better
calculated, before deciding to just expose them to the user.

> The current set of named rtt-target combinations for example seem to
> gotten it slightly wrong...

Not a big fan of those either.

> I am all for not leaving the user to figure it out, I just argue for
> making it possible to do something we had not thought about; I believe
> hindsight is 20/20 not fore-sight ;)

Well, if we add a parameter we can't remove it later, because it then
becomes part of the stable API. Whereas adding a new parameter later
*is* possible.

-Toke

Re: [Cake] memory

[Sebastian Moeller]

Hi Toke,

On Oct 29, 2015, at 16:40 , Toke Høiland-Jørgensen <toke@toke.dk> wrote:

> Sebastian Moeller <moeller0@gmx.de> writes:
> 
>> Hi Toke,
>> 
>> On Oct 29, 2015, at 12:02 , Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> 
>>> Sebastian Moeller <moeller0@gmx.de> writes:
>>> 
>>>> Yes, while trying to be funny I glossed over that part a bit (the
>>>> loooong RTTs part was aiming at that), sorry. But really I am wonder
>>>> what it is that makes my position that cake should expose and honor a
>>>> packet limit for its queueing so controversial;
>>> 
>>> The suggestion in itself is not necessarily controversial. However, the
>>> road to bloated interfaces is covered in good intentions. Each addition
>>> may be worthwhile in itself, but the sum of them becomes a horrible mess
>>> to configure (I'm being a bit hyperbolic here, but I'm sure you get my
>>> gist). So I'd rather say that for each possible configuration variable,
>>> we should figure out if there is a way to avoid exposing it, and only
>>> exposing it if we can't find a way to do that. Rather than going with
>>> the "easy" solution of just making it configurable and leaving the user
>>> to figure it out.
>> 
>> 	I am all for empowering the users to change the settings (well, the few
>> settings we talk about here, target, interval, limit, encapsulation and
>> overhead, I believe bandwidth and diffserv-scheme not contentious ;) ) in
>> COMBINATION with having sane defaults that make twiddling not necessary AND
>> potentially scary warnings if users configure something cake considers insane.
>> Exposing parameters that by all probability will not go away does not seem
>> dangerous to me, but the right thing to do. I venture the guess that if cake
>> should give up using codel underneath it is time for a rename, and cake2 or
>> cookie can have different exposed parameters than cake, no?
> 
>> 	We avoid the “horrible mess” part by basically requiring none of
>> the parameters (or just few if none is impossible) to be specified and
>> fill in all not specified parameters using the current auto-setting
>> methods. I am fine with not exposing implementation details, but the
>> parameters that will stay with us until we change the algorithms like
>> target and interval should be exposed (unless we are dead certain that
>> our automatic methods will configure the objectively best parameters
>> for all conditions).
> 
> Well, that's the thing: It is not clear to me that limit and target are
> not exactly that: implementation details.

	I guess we can agree to disagree, from my vintage point interval and target are the two core tunables that the codel mechanism cake builds upon uses and exposes; so with cake being in some sense a successor of fq_codel, I see no reason why exposing those will cause legacy issues. As already said if we switch away from codel under the hood there is no reason to keep the cake name and the parameters. Limit however absolutely needs to be exposed as there is no real way to divine the optimal limit for each deployed instance, given that some of the variables in the optimization space are most likely not available to cake (think a memory reserve the user might require to run another application on the router), or to come back to our example, a user on a high bandwidth satellite link but struck with a puny router might still prefer to sacrifice some good put if that helps avoid un-planned reboots… Or put differently limit is a policy parameter and hence should be exposed to the user; and in a sense so are is the target to interval ratio, according to the codel RFC this controls the bandwidth sacrifice and the resulting median queuing delay, which are also policy decisions the user should be able to control.
	Again "should be able to control" is not incompatible with “should not need to touch as the defaults are sane” in my view ;)


> 
> Note that I did say we should be sure that they are, and I'm not sure we
> are quite there yet. But I do believe that we should first make
> absolutely sure that these are parameters that are not better
> calculated, before deciding to just expose them to the user.

	I believe these two thanks are rather orthogonal, the beauty of exposing these is that it will allows us more leeway in changing the defaults later on (I believe after upstreaming even changing defaults results in discussions whether that does not catch current users by surprise...)

> 
>> The current set of named rtt-target combinations for example seem to
>> gotten it slightly wrong...
> 
> Not a big fan of those either.

	To play devil’s advocate, in the ideal case they will make configuring cake simpler: user on a 10Mbps satellite link might just use:
sudo tc qdisc add dev eth0 root cake bandwidth 10Mbit satellite
	as compare to a potential alternative:
sudo tc qdisc add dev eth0 root cake bandwidth 10Mbit rtt 1000ms target 62.5ms

Prtesonally I would prefer the second, but I like numbers ;)

> 
>> I am all for not leaving the user to figure it out, I just argue for
>> making it possible to do something we had not thought about; I believe
>> hindsight is 20/20 not fore-sight ;)
> 
> Well, if we add a parameter we can't remove it later, because it then
> becomes part of the stable API. Whereas adding a new parameter later
> *is* possible.

	I totally buy this argument; all I question is do we really see cake switch away from using codel underneath? If not then exposing target and interval should do no harm. And limit is something all qdisc should require by law ;) (I believe most actually honor this already). Please note that I do not propose to expose variables to modify the different diffserv schemes, I am totally fine with those being hard coded ;)


Bet Regards
	Sebastian

> 
> -Toke

Re: [Cake] memory

[Dave Taht]

I am scarred by nearly every academic paper even semi published,
mucking with the target, rather than interval parameter. I really
think the evidence is conclusive we do not need to expose the target,
only the interval. Target should be a calculated value, always, in
particular adjusted to 1MTU, and in a range from 5-10%. I had done
much testing at the (interval >> 4) 6.2% figure which seemed to
interact with the cable mac mildly better (6ms acquisition time) back
in the day.

PS: I do not think we have ever done enough analysis of tcp window
sizes in captures in the rrul and heavier duty tests to understand
fully the actual latencies and loss tcp flows experienced.


Dave Täht
I just invested five years of my life to making wifi better. And,
now... the FCC wants to make my work, illegal for people to install.
https://www.gofundme.com/savewifi


On Fri, Oct 30, 2015 at 5:13 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
> Hi Toke,
>
> On Oct 29, 2015, at 16:40 , Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
>> Sebastian Moeller <moeller0@gmx.de> writes:
>>
>>> Hi Toke,
>>>
>>> On Oct 29, 2015, at 12:02 , Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>>
>>>> Sebastian Moeller <moeller0@gmx.de> writes:
>>>>
>>>>> Yes, while trying to be funny I glossed over that part a bit (the
>>>>> loooong RTTs part was aiming at that), sorry. But really I am wonder
>>>>> what it is that makes my position that cake should expose and honor a
>>>>> packet limit for its queueing so controversial;
>>>>
>>>> The suggestion in itself is not necessarily controversial. However, the
>>>> road to bloated interfaces is covered in good intentions. Each addition
>>>> may be worthwhile in itself, but the sum of them becomes a horrible mess
>>>> to configure (I'm being a bit hyperbolic here, but I'm sure you get my
>>>> gist). So I'd rather say that for each possible configuration variable,
>>>> we should figure out if there is a way to avoid exposing it, and only
>>>> exposing it if we can't find a way to do that. Rather than going with
>>>> the "easy" solution of just making it configurable and leaving the user
>>>> to figure it out.
>>>
>>>      I am all for empowering the users to change the settings (well, the few
>>> settings we talk about here, target, interval, limit, encapsulation and
>>> overhead, I believe bandwidth and diffserv-scheme not contentious ;) ) in
>>> COMBINATION with having sane defaults that make twiddling not necessary AND
>>> potentially scary warnings if users configure something cake considers insane.
>>> Exposing parameters that by all probability will not go away does not seem
>>> dangerous to me, but the right thing to do. I venture the guess that if cake
>>> should give up using codel underneath it is time for a rename, and cake2 or
>>> cookie can have different exposed parameters than cake, no?
>>
>>>      We avoid the “horrible mess” part by basically requiring none of
>>> the parameters (or just few if none is impossible) to be specified and
>>> fill in all not specified parameters using the current auto-setting
>>> methods. I am fine with not exposing implementation details, but the
>>> parameters that will stay with us until we change the algorithms like
>>> target and interval should be exposed (unless we are dead certain that
>>> our automatic methods will configure the objectively best parameters
>>> for all conditions).
>>
>> Well, that's the thing: It is not clear to me that limit and target are
>> not exactly that: implementation details.
>
>         I guess we can agree to disagree, from my vintage point interval and target are the two core tunables that the codel mechanism cake builds upon uses and exposes; so with cake being in some sense a successor of fq_codel, I see no reason why exposing those will cause legacy issues. As already said if we switch away from codel under the hood there is no reason to keep the cake name and the parameters. Limit however absolutely needs to be exposed as there is no real way to divine the optimal limit for each deployed instance, given that some of the variables in the optimization space are most likely not available to cake (think a memory reserve the user might require to run another application on the router), or to come back to our example, a user on a high bandwidth satellite link but struck with a puny router might still prefer to sacrifice some good put if that helps avoid un-planned reboots… Or put differently limit is a policy parameter and hence should be exposed to the user; and in a sense so are is the target to interval ratio, according to the codel RFC this controls the bandwidth sacrifice and the resulting median queuing delay, which are also policy decisions the user should be able to control.
>         Again "should be able to control" is not incompatible with “should not need to touch as the defaults are sane” in my view ;)
>
>
>>
>> Note that I did say we should be sure that they are, and I'm not sure we
>> are quite there yet. But I do believe that we should first make
>> absolutely sure that these are parameters that are not better
>> calculated, before deciding to just expose them to the user.
>
>         I believe these two thanks are rather orthogonal, the beauty of exposing these is that it will allows us more leeway in changing the defaults later on (I believe after upstreaming even changing defaults results in discussions whether that does not catch current users by surprise...)
>
>>
>>> The current set of named rtt-target combinations for example seem to
>>> gotten it slightly wrong...
>>
>> Not a big fan of those either.
>
>         To play devil’s advocate, in the ideal case they will make configuring cake simpler: user on a 10Mbps satellite link might just use:
> sudo tc qdisc add dev eth0 root cake bandwidth 10Mbit satellite
>         as compare to a potential alternative:
> sudo tc qdisc add dev eth0 root cake bandwidth 10Mbit rtt 1000ms target 62.5ms
>
> Prtesonally I would prefer the second, but I like numbers ;)
>
>>
>>> I am all for not leaving the user to figure it out, I just argue for
>>> making it possible to do something we had not thought about; I believe
>>> hindsight is 20/20 not fore-sight ;)
>>
>> Well, if we add a parameter we can't remove it later, because it then
>> becomes part of the stable API. Whereas adding a new parameter later
>> *is* possible.
>
>         I totally buy this argument; all I question is do we really see cake switch away from using codel underneath? If not then exposing target and interval should do no harm. And limit is something all qdisc should require by law ;) (I believe most actually honor this already). Please note that I do not propose to expose variables to modify the different diffserv schemes, I am totally fine with those being hard coded ;)
>
>
> Bet Regards
>         Sebastian
>
>>
>> -Toke
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] memory

[Sebastian Moeller]

Hi Dave,

On Oct 30, 2015, at 13:07 , Dave Taht <dave.taht@gmail.com> wrote:

> I am scarred by nearly every academic paper even semi published,
> mucking with the target, rather than interval parameter. I really
> think the evidence is conclusive we do not need to expose the target,
> only the interval.

	I disagree, based on mere gut feeling and the conviction that “trust us we know what is best for you” is not a strategy to success. Let’s supply a great working even optimal default target and allow anybody to shoot their own feet if they want to. Our own history of target in cake does not fill me with confidence that we will get it right; for example we all seem to agree that target should maximum 5-10% of interval or the transfer time required for of one full MTU plus encapsulation packet worth of bytes, but we have not reached consensus whether the target increase on slow links should also result in an increases of interval to keep target at X% or not. Exposing target will allow to remedy this (by recommending users to manually change a sub-optimal default) with already deployed cakes instead of forcing a rebuild on our users.
	Also I think it totally par for the course if tc/cake would heavily complain about un-reasonable interval target combinations so the user knows without doubt that are on their own by their own volition.

> Target should be a calculated value, always, in
> particular adjusted to 1MTU, and in a range from 5-10%. I had done
> much testing at the (interval >> 4) 6.2% figure which seemed to
> interact with the cable mac mildly better (6ms acquisition time) back
> in the day.

	The >>4 thing is a bit of a puzzle for me, I believe we only ever calculate target during intiallisation and not repeatedly during operation so a division would just be as well, heck even a cast to float an FP-division, truncation of the fractional part and cast back to int would be in the noise (I am not proposing this and the kernel might not really want to use the FPU, all I want to illustrate is that I am not sure whether the shift really buys us that much ;) ). I like your rationale for the 6ms for cable though, and given the rather approximate rationale for 5% in the codel RFC I believe that is a good justification for >>4 ;) How would you test this, by the way (just curious, I have no cable link otherwise I would volunteer to re-confirm whether >>4 beats /20). But what if DOCSIS 4.0 changes the temporal grant granularity to say 1ms?

> 
> PS: I do not think we have ever done enough analysis of tcp window
> sizes in captures in the rrul and heavier duty tests to understand
> fully the actual latencies and loss tcp flows experienced.

	I certainly have not, all of this is outside of my (current) skill set, which might explain my almost stubborn insistence on minor points…

Best Regards
	Sebastian

> 
> 
> Dave Täht
> I just invested five years of my life to making wifi better. And,
> now... the FCC wants to make my work, illegal for people to install.
> https://www.gofundme.com/savewifi
> 
> 
> On Fri, Oct 30, 2015 at 5:13 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
>> Hi Toke,
>> 
>> On Oct 29, 2015, at 16:40 , Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> 
>>> Sebastian Moeller <moeller0@gmx.de> writes:
>>> 
>>>> Hi Toke,
>>>> 
>>>> On Oct 29, 2015, at 12:02 , Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>>> 
>>>>> Sebastian Moeller <moeller0@gmx.de> writes:
>>>>> 
>>>>>> Yes, while trying to be funny I glossed over that part a bit (the
>>>>>> loooong RTTs part was aiming at that), sorry. But really I am wonder
>>>>>> what it is that makes my position that cake should expose and honor a
>>>>>> packet limit for its queueing so controversial;
>>>>> 
>>>>> The suggestion in itself is not necessarily controversial. However, the
>>>>> road to bloated interfaces is covered in good intentions. Each addition
>>>>> may be worthwhile in itself, but the sum of them becomes a horrible mess
>>>>> to configure (I'm being a bit hyperbolic here, but I'm sure you get my
>>>>> gist). So I'd rather say that for each possible configuration variable,
>>>>> we should figure out if there is a way to avoid exposing it, and only
>>>>> exposing it if we can't find a way to do that. Rather than going with
>>>>> the "easy" solution of just making it configurable and leaving the user
>>>>> to figure it out.
>>>> 
>>>>     I am all for empowering the users to change the settings (well, the few
>>>> settings we talk about here, target, interval, limit, encapsulation and
>>>> overhead, I believe bandwidth and diffserv-scheme not contentious ;) ) in
>>>> COMBINATION with having sane defaults that make twiddling not necessary AND
>>>> potentially scary warnings if users configure something cake considers insane.
>>>> Exposing parameters that by all probability will not go away does not seem
>>>> dangerous to me, but the right thing to do. I venture the guess that if cake
>>>> should give up using codel underneath it is time for a rename, and cake2 or
>>>> cookie can have different exposed parameters than cake, no?
>>> 
>>>>     We avoid the “horrible mess” part by basically requiring none of
>>>> the parameters (or just few if none is impossible) to be specified and
>>>> fill in all not specified parameters using the current auto-setting
>>>> methods. I am fine with not exposing implementation details, but the
>>>> parameters that will stay with us until we change the algorithms like
>>>> target and interval should be exposed (unless we are dead certain that
>>>> our automatic methods will configure the objectively best parameters
>>>> for all conditions).
>>> 
>>> Well, that's the thing: It is not clear to me that limit and target are
>>> not exactly that: implementation details.
>> 
>>        I guess we can agree to disagree, from my vintage point interval and target are the two core tunables that the codel mechanism cake builds upon uses and exposes; so with cake being in some sense a successor of fq_codel, I see no reason why exposing those will cause legacy issues. As already said if we switch away from codel under the hood there is no reason to keep the cake name and the parameters. Limit however absolutely needs to be exposed as there is no real way to divine the optimal limit for each deployed instance, given that some of the variables in the optimization space are most likely not available to cake (think a memory reserve the user might require to run another application on the router), or to come back to our example, a user on a high bandwidth satellite link but struck with a puny router might still prefer to sacrifice some good put if that helps avoid un-planned reboots… Or put differently limit is a policy parameter and hence should be exposed to the user; and in a sense so are is the target to interval ratio, according to the codel RFC this controls the bandwidth sacrifice and the resulting median queuing delay, which are also policy decisions the user should be able to control.
>>        Again "should be able to control" is not incompatible with “should not need to touch as the defaults are sane” in my view ;)
>> 
>> 
>>> 
>>> Note that I did say we should be sure that they are, and I'm not sure we
>>> are quite there yet. But I do believe that we should first make
>>> absolutely sure that these are parameters that are not better
>>> calculated, before deciding to just expose them to the user.
>> 
>>        I believe these two thanks are rather orthogonal, the beauty of exposing these is that it will allows us more leeway in changing the defaults later on (I believe after upstreaming even changing defaults results in discussions whether that does not catch current users by surprise...)
>> 
>>> 
>>>> The current set of named rtt-target combinations for example seem to
>>>> gotten it slightly wrong...
>>> 
>>> Not a big fan of those either.
>> 
>>        To play devil’s advocate, in the ideal case they will make configuring cake simpler: user on a 10Mbps satellite link might just use:
>> sudo tc qdisc add dev eth0 root cake bandwidth 10Mbit satellite
>>        as compare to a potential alternative:
>> sudo tc qdisc add dev eth0 root cake bandwidth 10Mbit rtt 1000ms target 62.5ms
>> 
>> Prtesonally I would prefer the second, but I like numbers ;)
>> 
>>> 
>>>> I am all for not leaving the user to figure it out, I just argue for
>>>> making it possible to do something we had not thought about; I believe
>>>> hindsight is 20/20 not fore-sight ;)
>>> 
>>> Well, if we add a parameter we can't remove it later, because it then
>>> becomes part of the stable API. Whereas adding a new parameter later
>>> *is* possible.
>> 
>>        I totally buy this argument; all I question is do we really see cake switch away from using codel underneath? If not then exposing target and interval should do no harm. And limit is something all qdisc should require by law ;) (I believe most actually honor this already). Please note that I do not propose to expose variables to modify the different diffserv schemes, I am totally fine with those being hard coded ;)
>> 
>> 
>> Bet Regards
>>        Sebastian
>> 
>>> 
>>> -Toke
>> 
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake