[Cake] Pre-print of Cake paper available

[Cake] Pre-print of Cake paper available

[Toke Høiland-Jørgensen]

Last week we submitted an academic paper describing Cake. A pre-print is
now available on arXiv: https://arxiv.org/abs/1804.07617

Comments welcome, of course :)

-Toke

Re: [Cake] Pre-print of Cake paper available

[Jonathan Morton]

> Last week we submitted an academic paper describing Cake. A pre-print is
> now available on arXiv: https://arxiv.org/abs/1804.07617
> 
> Comments welcome, of course :)

I just forwarded a link to my Canadian relatives who are involved in a rural community ISP.  Lots of microwave links in the sub-Gbps category, I understand.  Will be interesting to hear what they think, and whether they find Cake (or something like it) useful in practice.

 - Jonathan Morton

Re: [Cake] Pre-print of Cake paper available

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 2249 bytes --]


> On Apr 23, 2018, at 10:39 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Last week we submitted an academic paper describing Cake. A pre-print is
> now available on arXiv: https://arxiv.org/abs/1804.07617
> 
> Comments welcome, of course :)


Nice work overall… :) Below is some feedback on content, and attached is a marked up PDF with some feedback on grammar and wording. Click the vanilla squares to show the notes.

Content:

- I wish there were some reference on how widespread of a problem bufferbloat actually is on the current Internet. That would bolster the initial assertion in the introduction.

- Thank you, I finally “get" triple-isolate. :) But I find it easier to understand the behavior of dual-srchost and dual-dsthost, and I think most would prefer its behavior, despite the fact that it needs to be configured manually. Just a thought, knowing that cake currently targets home gateways, and that there are now the egress and ingress keywords, could host isolation default to dual-srchost for egress mode and dual-dsthost for ingress mode? Or since using the keywords would be fragile, is there a better way to know the proper sense for dual-srchost and dual-dsthost?

- If ‘nat’ is not the default, won’t host isolation not work by default for most home gateways, almost all of which do nat? (Untested assumption.)

- Not in the paper, but is the ‘wash’ keyword really needed?

- Is it worth mentioning that when the home gateway’s uplink is WiFi, shaping is hard to do reliably, overhead and framing compensation can’t even be implemented, and that this is all more properly handled in the WiFi specific work?

- One of the biggest deployment challenges (not unique to cake) is that most people have to use shaping, since deploying cake on their gateway’s external interface isn’t practical. But setting the rate properly for shaping isn’t always straightforward. This is sheer speculation, but could observed latency (obtained by passively measuring TCP RTT, for instance) be used as a signal to control the rate? I can only imagine this might be difficult to get right (though I would’ve thought what BBR does is also), so just take this as food for thought...


[-- Attachment #2: 1804.07617v1_pete.pdf --]
[-- Type: application/pdf, Size: 468172 bytes --]

Re: [Cake] Pre-print of Cake paper available

[Jonathan Morton]

> - Thank you, I finally “get" triple-isolate. :) But I find it easier to understand the behavior of dual-srchost and dual-dsthost, and I think most would prefer its behavior, despite the fact that it needs to be configured manually. Just a thought, knowing that cake currently targets home gateways, and that there are now the egress and ingress keywords, could host isolation default to dual-srchost for egress mode and dual-dsthost for ingress mode? Or since using the keywords would be fragile, is there a better way to know the proper sense for dual-srchost and dual-dsthost?

This is covered in the tc-cake manpage:

.B dual-srchost
.br
	Flows are defined by the 5-tuple, and fairness is applied first over
source addresses, then over individual flows.  Good for use on egress traffic
from a LAN to the internet, where it'll prevent any one LAN host from
monopolising the uplink, regardless of the number of flows they use.
.PP
.B dual-dsthost
.br
	Flows are defined by the 5-tuple, and fairness is applied first over
destination addresses, then over individual flows.  Good for use on ingress
traffic to a LAN from the internet, where it'll prevent any one LAN host from
monopolising the downlink, regardless of the number of flows they use.
.PP

I'm not terribly keen on overloading the ingress and egress keywords as you suggest; certain people tend to get a bit worked up about it, and it could be confusing unless done very carefully.  Already, for the command-line interface, we have to explain what "ingress" and "egress" mean in themselves, when some of the target audience might have trouble setting the clock on their VCR.  (Also, they still have a VCR!)

> - If ‘nat’ is not the default, won’t host isolation not work by default for most home gateways, almost all of which do nat? (Untested assumption.)

Following on from the above, home gateways tend to have a GUI which can be made to handle this sort of thing more intuitively.  It helps GUI programmers if the API is reasonably stable and orthogonal.

> - Not in the paper, but is the ‘wash’ keyword really needed?

You'll have to ask Dave about that - he's the one who insists on having it!  The only coherent explanation I can immediately think of is to bypass wifi's builtin medium-grant prioritisation.

> - Is it worth mentioning that when the home gateway’s uplink is WiFi, shaping is hard to do reliably, overhead and framing compensation can’t even be implemented, and that this is all more properly handled in the WiFi specific work?

Wifi *or* 3G, etc.  I'm very conscious of this problem, but it's hard to solve in the qdisc itself.  It's definitely better to solve it at the upstream end of the link and with real-time medium awareness.  Wifi can have that today, with the right hardware, but good luck getting the 2G/3G/4G vendors on side.

Sensing the link rate via changes in flow latency is an attractive idea, but there are subtleties which I haven't found a satisfactory solution to yet.  By nature, such an algorithm would need to be very conservative and could not react quickly enough to guarantee QoS.

 - Jonathan Morton

Re: [Cake] Pre-print of Cake paper available

[Pete Heist]

> On Apr 24, 2018, at 1:31 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> Or since using the keywords would be fragile, is there a better way to know the proper sense for dual-srchost and dual-dsthost?
> 
> This is covered in the tc-cake manage: …

Oops, poor wording on my part. I meant is there a way for cake to detect which of the two modes is most likely intended automatically and default to one of them? I do understand why triple-isolate is an attractive default, and since a typical home gateway routes a few source IPs to many destination IPs, it will usually do what people expect without any configuration (except for when two clients are communicating with the same server, when it may not). It's just feedback that one of the first things I do for a cake setup is set dual-srchost / dual-dsthost as appropriate, and I was just searching for ways to make my personal preference automatic. So yeah, leave it. :)

> I'm not terribly keen on overloading the ingress and egress keywords as you suggest; certain people tend to get a bit worked up about it, and it could be confusing unless done very carefully.  Already, for the command-line interface, we have to explain what "ingress" and "egress" mean in themselves, when some of the target audience might have trouble setting the clock on their VCR.  (Also, they still have a VCR!)

I’m not either. This (along with ’nat’) were part of an overall wish that ‘cake’ without any keywords would do the “right thing” by default as often as possible, even if it means adding auto-detection logic. I don’t expect keyword changes at this point unless it’s really clear how to implement something. I’m old enough to get the VCR joke, and to remember my tiny H and M buttons with substandard tactile qualities...

>> - Not in the paper, but is the ‘wash’ keyword really needed?
> 
> You'll have to ask Dave about that - he's the one who insists on having it!  The only coherent explanation I can immediately think of is to bypass wifi's builtin medium-grant prioritization.

There’s an recommendation in the man page about Comcast that I didn’t follow, but that that second one is also true. I was experimenting with sending traffic over IPIP tunnels, which hides the original markings from the WiFi stack, which might be desirable in a WiFi backhaul, for example. ‘wash' would be another way to do that, although then the original markings wouldn’t be preserved, so the end result is a bit different.

>> - Is it worth mentioning that when the home gateway’s uplink is WiFi, shaping is hard to do reliably, overhead and framing compensation can’t even be implemented, and that this is all more properly handled in the WiFi specific work?
> 
> Wifi *or* 3G, etc.  I'm very conscious of this problem, but it's hard to solve in the qdisc itself.  It's definitely better to solve it at the upstream end of the link and with real-time medium awareness.  Wifi can have that today, with the right hardware, but good luck getting the 2G/3G/4G vendors on side.

It might still be worth mentioning in an academic paper, but this (all of this) is only feedback… :)

Re: [Cake] Pre-print of Cake paper available

[Jonathan Morton]

> This (along with ’nat’) were part of an overall wish that ‘cake’ without any keywords would do the “right thing” by default as often as possible

Turning NAT support on by default might actually be reasonable, since it doesn't really break anything if it's not needed - it just eats a bit of CPU with unnecessary conntrack lookups.

For the flowmodes, basically triple-isolate's raison d'être is to be a reasonable default which (usually) gives most of the benefits of the "dual" modes, without needing to know a-priori anything about network topology.  In the most typical application, the distinction can be seen in whether the qdisc is attached to an IFB or a physical interface, but in deployments that we'd *like* to see, the opposite cases easily occur.  To do anything more sophisticated, we'd need to watch some traffic and guess after a while, and that doesn't feel right.

 - Jonathan Morton

Re: [Cake] Pre-print of Cake paper available

[Pete Heist]

> On Apr 24, 2018, at 7:58 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
> Turning NAT support on by default might actually be reasonable, since it doesn't really break anything if it's not needed - it just eats a bit of CPU with unnecessary conntrack lookups.

I would be for it, if it eats say < 1% additional CPU, and preferably less. I expect the impact to increase with packet rates.

> For the flowmodes, basically triple-isolate's raison d'être is to be a reasonable default which (usually) gives most of the benefits of the "dual" modes, without needing to know a-priori anything about network topology.  In the most typical application, the distinction can be seen in whether the qdisc is attached to an IFB or a physical interface, but in deployments that we'd *like* to see, the opposite cases easily occur.  To do anything more sophisticated, we'd need to watch some traffic and guess after a while, and that doesn't feel right.

Yeah, I see. The same could be done with nat. There could be an auto-detect phase where nat lookups are performed and not to determine if it’s needed. But if these detections didn’t work with near-perfect reliability, it would complicate troubleshooting.

Re: [Cake] Pre-print of Cake paper available

[Sebastian Moeller]

> On Apr 24, 2018, at 09:15, Pete Heist <pete@eventide.io> wrote:
> 
> 
>> On Apr 24, 2018, at 7:58 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>> 
>> Turning NAT support on by default might actually be reasonable, since it doesn't really break anything if it's not needed - it just eats a bit of CPU with unnecessary conntrack lookups.
> 
> I would be for it, if it eats say < 1% additional CPU, and preferably less. I expect the impact to increase with packet rates.
> 
>> For the flowmodes, basically triple-isolate's raison d'être is to be a reasonable default which (usually) gives most of the benefits of the "dual" modes, without needing to know a-priori anything about network topology.  In the most typical application, the distinction can be seen in whether the qdisc is attached to an IFB or a physical interface, but in deployments that we'd *like* to see, the opposite cases easily occur.  To do anything more sophisticated, we'd need to watch some traffic and guess after a while, and that doesn't feel right.
> 
> Yeah, I see. The same could be done with nat. There could be an auto-detect phase where nat lookups are performed and not to determine if it’s needed. But if these detections didn’t work with near-perfect reliability, it would complicate troubleshooting.


IMHO, auto-detection would at least require a dedicated keyword, like "detect-nat" that would be reported in the tc -s qdisc output for cake as long as the detection phase is active and would be replaced by the appropriate "nat" or "nonat"
keyword in the tc -s qdisc output after the phase is done (there should be no ambiguity). But overall I believe that this is adding more confusion. That said I am all for making nat the default, even though it would be nice to have measurements showing its cost. 


> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Pre-print of Cake paper available

[Toke Høiland-Jørgensen]

Pete Heist <pete@eventide.io> writes:

>> On Apr 24, 2018, at 7:58 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>> 
>> Turning NAT support on by default might actually be reasonable, since
>> it doesn't really break anything if it's not needed - it just eats a
>> bit of CPU with unnecessary conntrack lookups.
>
> I would be for it, if it eats say < 1% additional CPU, and preferably
> less. I expect the impact to increase with packet rates.

I'm a bit worried that the way it is implemented now, if we turn it on
by default we risk activating conntrack even when it was otherwise
disabled... That would be a bad side effect, so I think it's better to
be safe and leave it for userspace to enable (which, again, we could do
by default in sqm-scripts).

-Toke

Re: [Cake] Pre-print of Cake paper available

[Pete Heist]

> On Apr 24, 2018, at 10:45 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Pete Heist <pete@eventide.io> writes:
> 
>>> On Apr 24, 2018, at 7:58 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>>> 
>>> Turning NAT support on by default might actually be reasonable, since
>>> it doesn't really break anything if it's not needed - it just eats a
>>> bit of CPU with unnecessary conntrack lookups.
>> 
>> I would be for it, if it eats say < 1% additional CPU, and preferably
>> less. I expect the impact to increase with packet rates.
> 
> I'm a bit worried that the way it is implemented now, if we turn it on
> by default we risk activating conntrack even when it was otherwise
> disabled... That would be a bad side effect, so I think it's better to
> be safe and leave it for userspace to enable (which, again, we could do
> by default in sqm-scripts).

Fair enough. It sounds like cake’s configuration policy is to be more “predictable and consistent” by default than “ready to use in your home gateway” by default, which is understandable if it’s anticipated that more people will use it through some router’s UI or setup scripts than will set it up by hand from the command line…

Re: [Cake] Pre-print of Cake paper available

[David Lang]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1187 bytes --]

On Tue, 24 Apr 2018, Toke Høiland-Jørgensen wrote:

> Pete Heist <pete@eventide.io> writes:
>
>>> On Apr 24, 2018, at 7:58 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>>> 
>>> Turning NAT support on by default might actually be reasonable, since
>>> it doesn't really break anything if it's not needed - it just eats a
>>> bit of CPU with unnecessary conntrack lookups.
>>
>> I would be for it, if it eats say < 1% additional CPU, and preferably
>> less. I expect the impact to increase with packet rates.
>
> I'm a bit worried that the way it is implemented now, if we turn it on
> by default we risk activating conntrack even when it was otherwise
> disabled...

I will say that just about every system ships with conntrack enabled, and 
disabling it can be pretty difficult (especially in LEDE/OpenWRT), there are so 
many things that require it that tracking them all down and disabling them is 
very difficult.

There are not that many places where Cake is going to be used that NAT or some 
other thing that requires connection tracking is not also going to be used, in 
the remaining cases, can it be disabled manually in configs after it's been 
sucked in automatically?

Re: [Cake] Pre-print of Cake paper available

[Toke Høiland-Jørgensen]

David Lang <david@lang.hm> writes:

> On Tue, 24 Apr 2018, Toke Høiland-Jørgensen wrote:
>
>> Pete Heist <pete@eventide.io> writes:
>>
>>>> On Apr 24, 2018, at 7:58 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>>>> 
>>>> Turning NAT support on by default might actually be reasonable, since
>>>> it doesn't really break anything if it's not needed - it just eats a
>>>> bit of CPU with unnecessary conntrack lookups.
>>>
>>> I would be for it, if it eats say < 1% additional CPU, and preferably
>>> less. I expect the impact to increase with packet rates.
>>
>> I'm a bit worried that the way it is implemented now, if we turn it on
>> by default we risk activating conntrack even when it was otherwise
>> disabled...
>
> I will say that just about every system ships with conntrack enabled, and 
> disabling it can be pretty difficult (especially in LEDE/OpenWRT), there are so 
> many things that require it that tracking them all down and disabling them is 
> very difficult.
>
> There are not that many places where Cake is going to be used that NAT or some 
> other thing that requires connection tracking is not also going to be used, in 
> the remaining cases, can it be disabled manually in configs after it's been 
> sucked in automatically?

Hmm, actually it looks like just compiling against the conntrack code
adds a module dependency on conntrack. And as far as I can tell, the
code doesn't initiate any new conntrack state if it doesn't already
exist. So I think it's safe to turn on NAT mode by default. Will add
that :)

-Toke

Re: [Cake] Pre-print of Cake paper available

[Pete Heist]

> On Apr 25, 2018, at 10:28 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Hmm, actually it looks like just compiling against the conntrack code
> adds a module dependency on conntrack. And as far as I can tell, the
> code doesn't initiate any new conntrack state if it doesn't already
> exist. So I think it's safe to turn on NAT mode by default. Will add
> that :)

nat vs nonat CPU load for flent’s rrul_be / "cpu_stats_localhost::load" on APU2:

<limit> <nonat avg cpu load> <nat avg cpu load>
10mbit  0.07  0.07
20mbit  0.09  0.09
30mbit  0.10  0.10
40mbit  0.11  0.11
50mbit  0.13  0.13
100mbit  0.19  0.20
150mbit  0.27  0.28
200mbit  0.33  0.35
250mbit  0.39  0.41
300mbit  0.44  0.45
350mbit  0.47  0.47
400mbit  0.50  0.49
450mbit  0.50  0.51
500mbit  0.53  0.52
none  0.37  0.43 (1864 mbit total up/down)

It looks like the largest impact is when there’s no rate limiting, probably when higher packet rates are reached and the relative proportion of CPU taken is greater. I suppose the backwards results (where nonat takes more CPU than nat) at 400mbit and 500mbit are just outliers. This isn’t a perfect way to measure.

I’ll leave it to you what to do with this information. Rough estimation: nat may be +2% CPU with rate limiting, and +15% without… :)

Re: [Cake] Pre-print of Cake paper available

[Toke Høiland-Jørgensen]

Pete Heist <pete@eventide.io> writes:

>> On Apr 25, 2018, at 10:28 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> 
>> Hmm, actually it looks like just compiling against the conntrack code
>> adds a module dependency on conntrack. And as far as I can tell, the
>> code doesn't initiate any new conntrack state if it doesn't already
>> exist. So I think it's safe to turn on NAT mode by default. Will add
>> that :)
>
> nat vs nonat CPU load for flent’s rrul_be / "cpu_stats_localhost::load" on APU2:
>
> <limit> <nonat avg cpu load> <nat avg cpu load>
> 10mbit  0.07  0.07
> 20mbit  0.09  0.09
> 30mbit  0.10  0.10
> 40mbit  0.11  0.11
> 50mbit  0.13  0.13
> 100mbit  0.19  0.20
> 150mbit  0.27  0.28
> 200mbit  0.33  0.35
> 250mbit  0.39  0.41
> 300mbit  0.44  0.45
> 350mbit  0.47  0.47
> 400mbit  0.50  0.49
> 450mbit  0.50  0.51
> 500mbit  0.53  0.52
> none  0.37  0.43 (1864 mbit total up/down)
>
> It looks like the largest impact is when there’s no rate limiting,
> probably when higher packet rates are reached and the relative
> proportion of CPU taken is greater. I suppose the backwards results
> (where nonat takes more CPU than nat) at 400mbit and 500mbit are just
> outliers. This isn’t a perfect way to measure.
>
> I’ll leave it to you what to do with this information. Rough
> estimation: nat may be +2% CPU with rate limiting, and +15% without…

Huh, that is maybe a bit much for a default; I guess it's better to just
set the NAT flag as needed from sqm-scripts, then...

-Toke

Re: [Cake] Pre-print of Cake paper available

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 516 bytes --]


> On Apr 27, 2018, at 1:08 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> 
>> I’ll leave it to you what to do with this information. Rough
>> estimation: nat may be +2% CPU with rate limiting, and +15% without…
> 
> Huh, that is maybe a bit much for a default; I guess it's better to just
> set the NAT flag as needed from sqm-scripts, then...

True, also given the upstream sensitivity to CPU usage, we might not want to invite another argument by analogy, or an argumentum ad baculum… :)


[-- Attachment #2: Type: text/html, Size: 2968 bytes --]

Re: [Cake] Pre-print of Cake paper available

[Sebastian Moeller]

[-- Attachment #1: Type: text/plain, Size: 5049 bytes --]



> On Apr 24, 2018, at 01:01, Pete Heist <pete@eventide.io> wrote:
> 
> 
>> On Apr 23, 2018, at 10:39 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> 
>> Last week we submitted an academic paper describing Cake. A pre-print is
>> now available on arXiv: https://arxiv.org/abs/1804.07617
>> 
>> Comments welcome, of course :)
> 
> 
> Nice work overall… :) Below is some feedback on content, and attached is a marked up PDF with some feedback on grammar and wording. Click the vanilla squares to show the notes.
> 
> Content:
> 
> - I wish there were some reference on how widespread of a problem bufferbloat actually is on the current Internet. That would bolster the initial assertion in the introduction.
> 
> - Thank you, I finally “get" triple-isolate. :) But I find it easier to understand the behavior of dual-srchost and dual-dsthost, and I think most would prefer its behavior, despite the fact that it needs to be configured manually. Just a thought, knowing that cake currently targets home gateways, and that there are now the egress and ingress keywords, could host isolation default to dual-srchost for egress mode and dual-dsthost for ingress mode? Or since using the keywords would be fragile, is there a better way to know the proper sense for dual-srchost and dual-dsthost?

	The challenge is to find a heuristic that covers all reasonable use cases and does no harm in unexpected cases. I could envision setting up a cake instance on the upstream end of say a microwave link, there "ingress" seems like the appropriate keyword (as the goal would be to keep the link non-congested), but for customer fairness "dual-srchost" would be the appropriate keyword (or just srchost if all the ISP cares for is inter-customer fairness). Sure this will not work with IPv6 (for that we would either need to llok at the MACs or IMHO preferably the IPv6 prefix (or the partially masked IPv6 IP-address, I believe this to be better than MAC adresses as the ISP can easily control the prefix, but I digress)).


> 
> - If ‘nat’ is not the default, won’t host isolation not work by default for most home gateways, almost all of which do nat? (Untested assumption.)

	It will for IPv6 out of the box, but for IPv4 it will regress to per flow fairness

> 
> - Not in the paper, but is the ‘wash’ keyword really needed?

	I believe that it would be preferable to have; one use case is not to leak one's internal DSCPs to the ISP as well as to protect the internal network from inapprpriate dscp marks coming from upstream. Why do this  in cake? It turns out cake already has all the required information at hand and setting up another qdisc to do this seems rather comlpicated. From an enduser's persprective it is nice if cake simply offers to do this as well.

> 
> - Is it worth mentioning that when the home gateway’s uplink is WiFi, shaping is hard to do reliably, overhead and framing compensation can’t even be implemented, and that this is all more properly handled in the WiFi specific work?
> 
> - One of the biggest deployment challenges (not unique to cake) is that most people have to use shaping, since deploying cake on their gateway’s external interface isn’t practical. But setting the rate properly for shaping isn’t always straightforward.

	That is true to some degree, but the overall algorithm is not that hard: Set the shaper at 50% of contracted rate and measure the bufferbloat (depending on the expertise of the user either via flent or the dslreports speedtest); if bufferbloat acceptable set shaper higher (by 50% of the remaining difference to 100% contracted rate) or if unaceptable lower. Really just a binary search for the acceptable limits. Now this should be done individually for each shaper direction. The final proof of the pudding is to see how this shaper copes with a bi-directional saturating load (like flent's rrul). (Okay okay, the real test would be a saturating load consisting out of really small packets, but that will be to esoteric for most users and should have only little relevance for normal use as saturating loads typically use larger packets...)

> This is sheer speculation, but could observed latency (obtained by passively measuring TCP RTT, for instance) be used as a signal to control the rate?

	It can to some degree, gargoyle does something like that, but it really is just approximate unless one has reliable robust one-way latency measurements, otherwise increasing RTT could be caused by either ingress, egress or even both making it somewhat hard to decide which rate to reduce. TCP RTT seems okay, as long as the endhost is close enough, as attempting to fix bad peering/transits by shaping on the last mile might be a bit optimistic. (Or rather altruistic as in that case all one gets is less bandwidth but most likely no latency improvement).


> I can only imagine this might be difficult to get right (though I would’ve thought what BBR does is also), so just take this as food for thought...
> 

[-- Attachment #2: 1804.07617v1_pete.pdf --]
[-- Type: application/pdf, Size: 468172 bytes --]

[-- Attachment #3: Type: text/plain, Size: 146 bytes --]

> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Pre-print of Cake paper available

[Toke Høiland-Jørgensen]

Sebastian Moeller <moeller0@gmx.de> writes:

>> On Apr 24, 2018, at 01:01, Pete Heist <pete@eventide.io> wrote:
>> 
>> 
>>> On Apr 23, 2018, at 10:39 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>> 
>>> Last week we submitted an academic paper describing Cake. A pre-print is
>>> now available on arXiv: https://arxiv.org/abs/1804.07617
>>> 
>>> Comments welcome, of course :)
>> 
>> 
>> Nice work overall… :) Below is some feedback on content, and attached is a marked up PDF with some feedback on grammar and wording. Click the vanilla squares to show the notes.
>> 
>> Content:
>> 
>> - I wish there were some reference on how widespread of a problem bufferbloat actually is on the current Internet. That would bolster the initial assertion in the introduction.
>> 
>> - Thank you, I finally “get" triple-isolate. :) But I find it easier
>> to understand the behavior of dual-srchost and dual-dsthost, and I
>> think most would prefer its behavior, despite the fact that it needs
>> to be configured manually. Just a thought, knowing that cake
>> currently targets home gateways, and that there are now the egress
>> and ingress keywords, could host isolation default to dual-srchost
>> for egress mode and dual-dsthost for ingress mode? Or since using the
>> keywords would be fragile, is there a better way to know the proper
>> sense for dual-srchost and dual-dsthost?
>
> 	The challenge is to find a heuristic that covers all reasonable
> 	use cases and does no harm in unexpected cases. I could envision
> 	setting up a cake instance on the upstream end of say a
> 	microwave link, there "ingress" seems like the appropriate
> 	keyword (as the goal would be to keep the link non-congested),
> 	but for customer fairness "dual-srchost" would be the
> 	appropriate keyword (or just srchost if all the ISP cares for is
> 	inter-customer fairness). Sure this will not work with IPv6 (for
> 	that we would either need to llok at the MACs or IMHO preferably
> 	the IPv6 prefix (or the partially masked IPv6 IP-address, I
> 	believe this to be better than MAC adresses as the ISP can
> 	easily control the prefix, but I digress)).

I don't think we can make assumptions on ISP deployments. The shaper may
or may not be at the point of NAT, and per-customer prefix size can
vary. To properly support the ISP per-customer fairness use case, we'd
probably need to support arbitrary filtering (like what FQ-CoDel
supports with 'tc filter'). And I think, if we wanted to support the ISP
case, that a per-customer *shaper* is more useful...

-Toke

Re: [Cake] Pre-print of Cake paper available

[Jonathan Morton]

> I think, if we wanted to support the ISP case, that a per-customer *shaper* is more useful.

Yes, I think the technology can be recoded to better suit a multi-subscriber environment; it would no longer be Cake, but would use some of the same key algorithms.

 - Jonathan Morton

Re: [Cake] Pre-print of Cake paper available

[Pete Heist]

> On Apr 24, 2018, at 10:50 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> I think, if we wanted to support the ISP case, that a per-customer *shaper* is more useful.
> 
> Yes, I think the technology can be recoded to better suit a multi-subscriber environment; it would no longer be Cake, but would use some of the same key algorithms.

Right, I was going to comment on the paper- where’s the ISP backhaul use case? (Because I might still try to use it that way.) But I was sure that this was already considered, and am not surprised that it might be a different but related project.

Re: [Cake] Pre-print of Cake paper available

[Toke Høiland-Jørgensen]

Pete Heist <pete@eventide.io> writes:

>> On Apr 24, 2018, at 10:50 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>> 
>>> I think, if we wanted to support the ISP case, that a per-customer *shaper* is more useful.
>> 
>> Yes, I think the technology can be recoded to better suit a
>> multi-subscriber environment; it would no longer be Cake, but would
>> use some of the same key algorithms.
>
> Right, I was going to comment on the paper- where’s the ISP backhaul
> use case? (Because I might still try to use it that way.) But I was
> sure that this was already considered, and am not surprised that it
> might be a different but related project.

Well, you could use it on an ISP backhaul by having a separate CAKE
instance per customer, and having another mechanism to assign customer
traffic to each. An HTB tree would be one way to do this; separate
interfaces per customer would be another. The latter works well if
customers are on separate VLANs, for instance...

But yeah, having a single instance of CAKE solve all of your customer
shaping problems is probably not in scope currently... ;)

-Toke

Re: [Cake] Pre-print of Cake paper available

[Pete Heist]

> On Apr 24, 2018, at 11:15 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Well, you could use it on an ISP backhaul by having a separate CAKE
> instance per customer, and having another mechanism to assign customer
> traffic to each.

Yep, am aware of that possibility, and the setup fun that would entail. :)

> But yeah, having a single instance of CAKE solve all of your customer
> shaping problems is probably not in scope currently... ;)

That’s why in the end I think it’s good that the paper makes it clear that the target is home gateways (it’s right in the title). It was designed and implemented with that and mind, and if it were advertised for another purpose without more work, the paper might start to get that “it slices, it dices” feel… :)

Re: [Cake] Pre-print of Cake paper available

[Sebastian Moeller]

Hi Toke,



> On Apr 24, 2018, at 10:47, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Sebastian Moeller <moeller0@gmx.de> writes:
> 
>>> On Apr 24, 2018, at 01:01, Pete Heist <pete@eventide.io> wrote:
>>> 
>>> 
>>>> On Apr 23, 2018, at 10:39 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>>> 
>>>> Last week we submitted an academic paper describing Cake. A pre-print is
>>>> now available on arXiv: https://arxiv.org/abs/1804.07617
>>>> 
>>>> Comments welcome, of course :)
>>> 
>>> 
>>> Nice work overall… :) Below is some feedback on content, and attached is a marked up PDF with some feedback on grammar and wording. Click the vanilla squares to show the notes.
>>> 
>>> Content:
>>> 
>>> - I wish there were some reference on how widespread of a problem bufferbloat actually is on the current Internet. That would bolster the initial assertion in the introduction.
>>> 
>>> - Thank you, I finally “get" triple-isolate. :) But I find it easier
>>> to understand the behavior of dual-srchost and dual-dsthost, and I
>>> think most would prefer its behavior, despite the fact that it needs
>>> to be configured manually. Just a thought, knowing that cake
>>> currently targets home gateways, and that there are now the egress
>>> and ingress keywords, could host isolation default to dual-srchost
>>> for egress mode and dual-dsthost for ingress mode? Or since using the
>>> keywords would be fragile, is there a better way to know the proper
>>> sense for dual-srchost and dual-dsthost?
>> 
>> 	The challenge is to find a heuristic that covers all reasonable
>> 	use cases and does no harm in unexpected cases. I could envision
>> 	setting up a cake instance on the upstream end of say a
>> 	microwave link, there "ingress" seems like the appropriate
>> 	keyword (as the goal would be to keep the link non-congested),
>> 	but for customer fairness "dual-srchost" would be the
>> 	appropriate keyword (or just srchost if all the ISP cares for is
>> 	inter-customer fairness). Sure this will not work with IPv6 (for
>> 	that we would either need to llok at the MACs or IMHO preferably
>> 	the IPv6 prefix (or the partially masked IPv6 IP-address, I
>> 	believe this to be better than MAC adresses as the ISP can
>> 	easily control the prefix, but I digress)).
> 
> I don't think we can make assumptions on ISP deployments.

Sure we do not really need to: https://forum.lede-project.org/t/transparent-cake-box/2161/4?u=moeller0 and https://forum.lede-project.org/t/lede-as-a-dedicated-qos-bufferbloat-appliance/1861/14?u=moeller0
so it looks like one person already use cake in an small ISP context. Now 1 is not a very convincing number, but certainly larger than zero... 


> The shaper may
> or may not be at the point of NAT, and per-customer prefix size can
> vary. To properly support the ISP per-customer fairness use case, we'd
> probably need to support arbitrary filtering (like what FQ-CoDel
> supports with 'tc filter'). And I think, if we wanted to support the ISP
> case, that a per-customer *shaper* is more useful...

Yes, I assume though that these would need to run on the boxes ISPs use to terminate the customer lines; but cake might still make sense for fair sharing of bottlenecks.

Best Regards
	Sebastian

> 
> -Toke

Re: [Cake] Pre-print of Cake paper available

[Toke Høiland-Jørgensen]

Sebastian Moeller <moeller0@gmx.de> writes:

> Hi Toke,
>
>
>
>> On Apr 24, 2018, at 10:47, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> 
>> Sebastian Moeller <moeller0@gmx.de> writes:
>> 
>>>> On Apr 24, 2018, at 01:01, Pete Heist <pete@eventide.io> wrote:
>>>> 
>>>> 
>>>>> On Apr 23, 2018, at 10:39 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>>>> 
>>>>> Last week we submitted an academic paper describing Cake. A pre-print is
>>>>> now available on arXiv: https://arxiv.org/abs/1804.07617
>>>>> 
>>>>> Comments welcome, of course :)
>>>> 
>>>> 
>>>> Nice work overall… :) Below is some feedback on content, and attached is a marked up PDF with some feedback on grammar and wording. Click the vanilla squares to show the notes.
>>>> 
>>>> Content:
>>>> 
>>>> - I wish there were some reference on how widespread of a problem bufferbloat actually is on the current Internet. That would bolster the initial assertion in the introduction.
>>>> 
>>>> - Thank you, I finally “get" triple-isolate. :) But I find it easier
>>>> to understand the behavior of dual-srchost and dual-dsthost, and I
>>>> think most would prefer its behavior, despite the fact that it needs
>>>> to be configured manually. Just a thought, knowing that cake
>>>> currently targets home gateways, and that there are now the egress
>>>> and ingress keywords, could host isolation default to dual-srchost
>>>> for egress mode and dual-dsthost for ingress mode? Or since using the
>>>> keywords would be fragile, is there a better way to know the proper
>>>> sense for dual-srchost and dual-dsthost?
>>> 
>>> 	The challenge is to find a heuristic that covers all reasonable
>>> 	use cases and does no harm in unexpected cases. I could envision
>>> 	setting up a cake instance on the upstream end of say a
>>> 	microwave link, there "ingress" seems like the appropriate
>>> 	keyword (as the goal would be to keep the link non-congested),
>>> 	but for customer fairness "dual-srchost" would be the
>>> 	appropriate keyword (or just srchost if all the ISP cares for is
>>> 	inter-customer fairness). Sure this will not work with IPv6 (for
>>> 	that we would either need to llok at the MACs or IMHO preferably
>>> 	the IPv6 prefix (or the partially masked IPv6 IP-address, I
>>> 	believe this to be better than MAC adresses as the ISP can
>>> 	easily control the prefix, but I digress)).
>> 
>> I don't think we can make assumptions on ISP deployments.
>
> Sure we do not really need to:
> https://forum.lede-project.org/t/transparent-cake-box/2161/4?u=moeller0
> and
> https://forum.lede-project.org/t/lede-as-a-dedicated-qos-bufferbloat-appliance/1861/14?u=moeller0
> so it looks like one person already use cake in an small ISP context.
> Now 1 is not a very convincing number, but certainly larger than
> zero...

Well I just meant that there are many ways to deploy a shaper in an ISP
context (centrally in the network, at the next hop from the customer,
etc).

Looking at those threads, they seem to be increasing the number of
queues. Not sure they need to, but, well, there's nothing in principle
that says this couldn't be configurable (it is in FQ-CoDel). It would
need a bit of a reorg of the current code, though, so that would be a
thing for later I guess...

-Toke

Re: [Cake] Pre-print of Cake paper available

[Sebastian Moeller]

Hi Toke,


> On Apr 24, 2018, at 11:30, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Sebastian Moeller <moeller0@gmx.de> writes:
> [...]
>>> 
>>> I don't think we can make assumptions on ISP deployments.
>> 
>> Sure we do not really need to:
>> https://forum.lede-project.org/t/transparent-cake-box/2161/4?u=moeller0
>> and
>> https://forum.lede-project.org/t/lede-as-a-dedicated-qos-bufferbloat-appliance/1861/14?u=moeller0
>> so it looks like one person already use cake in an small ISP context.
>> Now 1 is not a very convincing number, but certainly larger than
>> zero...
> 
> Well I just meant that there are many ways to deploy a shaper in an ISP
> context (centrally in the network, at the next hop from the customer,
> etc).

	Ah, sorry, I fully agree; and if ISPs are interested they should start talking; I just wanted to report the one case where cake seems to be used in an ISP-ish context.


> 
> Looking at those threads, they seem to be increasing the number of
> queues. Not sure they need to, but, well, there's nothing in principle
> that says this couldn't be configurable (it is in FQ-CoDel). It would
> need a bit of a reorg of the current code, though, so that would be a
> thing for later I guess...

	I had hoped that orangetek could be convinced to do measurements with different number of queues to answer that question, but I guess the current default of 1000 queues is decent for a typical homenet, but will simply become a bit tight with 600 concurrent households (I assume that its the peak usage that would want more queues on the "back-haul", on average 1000 might still be okay, especially with Jonathan's clever set associativity design)

Best Regards
	Sebastian


> 
> -Toke

Re: [Cake] Pre-print of Cake paper available

[Toke Høiland-Jørgensen]

Sebastian Moeller <moeller0@gmx.de> writes:

>> Looking at those threads, they seem to be increasing the number of
>> queues. Not sure they need to, but, well, there's nothing in principle
>> that says this couldn't be configurable (it is in FQ-CoDel). It would
>> need a bit of a reorg of the current code, though, so that would be a
>> thing for later I guess...
>
> 	I had hoped that orangetek could be convinced to do measurements
> 	with different number of queues to answer that question, but I
> 	guess the current default of 1000 queues is decent for a typical
> 	homenet, but will simply become a bit tight with 600 concurrent
> 	households (I assume that its the peak usage that would want
> 	more queues on the "back-haul", on average 1000 might still be
> 	okay, especially with Jonathan's clever set associativity
> 	design)

Well, there's
https://www.researchgate.net/profile/S_Oueslati/publication/247045479_On_the_Scalability_of_Fair_Queueing/links/5451fab10cf285a067c6af0a/On-the-Scalability-of-Fair-Queueing.pdf

-Toke

Re: [Cake] Pre-print of Cake paper available

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 868 bytes --]

On Tue, Apr 24, 2018 at 4:17 AM, Sebastian Moeller <moeller0@gmx.de> wrote:

> That is true to some degree, but the overall algorithm is not that hard:
Set the shaper at 50% of contracted rate and measure the bufferbloat
(depending on the expertise of the user either via flent or the dslreports
speedtest); if bufferbloat acceptable set shaper higher (by 50% of the
remaining difference to 100% contracted rate) or if unaceptable lower.
Really just a binary search for the acceptable limits. Now this should be
done individually for each shaper direction. The final proof of the pudding
is to see how this shaper copes with a bi-directional saturating load (like
flent's rrul).

​Given that the target audience is VCR-owning home users could this be
reduced to a scripts?  Then perhaps hardware vendors could then expose it
in​ their GUI.

/john

[-- Attachment #2: Type: text/html, Size: 1345 bytes --]

Re: [Cake] Pre-print of Cake paper available

[Toke Høiland-Jørgensen]

Pete Heist <pete@eventide.io> writes:

>> On Apr 23, 2018, at 10:39 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> 
>> Last week we submitted an academic paper describing Cake. A pre-print is
>> now available on arXiv: https://arxiv.org/abs/1804.07617
>> 
>> Comments welcome, of course :)
>
>
> Nice work overall… :) Below is some feedback on content, and attached
> is a marked up PDF with some feedback on grammar and wording. Click
> the vanilla squares to show the notes.

Thanks! A lot of those should have been fixed before submission; boy,
did I make a mess of verb tenses... Ah well, I'll incorporate your
fixes, so they will be fixed for the camera ready (assuming it gets
accepted) :)

> Content:
>
> - I wish there were some reference on how widespread of a problem
> bufferbloat actually is on the current Internet. That would bolster
> the initial assertion in the introduction.

Hmm, I do actually have a paper of my own that I could cite for this ;)
The trouble is that we have a pretty tight page limit, and adding
another reference takes us over that, meaning we would have to cut
something else. And I think we can do without in an academic paper
context at least...

> - Thank you, I finally “get" triple-isolate. :) But I find it easier
> to understand the behavior of dual-srchost and dual-dsthost, and I
> think most would prefer its behavior, despite the fact that it needs
> to be configured manually. Just a thought, knowing that cake currently
> targets home gateways, and that there are now the egress and ingress
> keywords, could host isolation default to dual-srchost for egress mode
> and dual-dsthost for ingress mode? Or since using the keywords would
> be fragile, is there a better way to know the proper sense for
> dual-srchost and dual-dsthost?
>
> - If ‘nat’ is not the default, won’t host isolation not work by
> default for most home gateways, almost all of which do nat? (Untested
> assumption.)

I think these questions are actually better handled as userspace policy
issues. For instance, in sqm-scripts we could reasonably default to
dual-srchost on egress and dual-dsthost on ingress, as we know which is
which.

> - Not in the paper, but is the ‘wash’ keyword really needed?

Is anything really *needed*? ;) It's useful in settings where you want to
clear diffserv markings, and not in other places...

> - Is it worth mentioning that when the home gateway’s uplink is WiFi,
> shaping is hard to do reliably, overhead and framing compensation
> can’t even be implemented, and that this is all more properly handled
> in the WiFi specific work?
>
> - One of the biggest deployment challenges (not unique to cake) is
> that most people have to use shaping, since deploying cake on their
> gateway’s external interface isn’t practical. But setting the rate
> properly for shaping isn’t always straightforward. This is sheer
> speculation, but could observed latency (obtained by passively
> measuring TCP RTT, for instance) be used as a signal to control the
> rate? I can only imagine this might be difficult to get right (though
> I would’ve thought what BBR does is also), so just take this as food
> for thought...

I'd categorise these as relevant issues that we don't have space to
discuss in the paper ;)

-Toke