[Cake] Re: help request for cake on a large network

[Sebastian Moeller <moeller0@gmx.de>]

Hi Jonathan...

thanks for commenting on the details.

> On 30. Sep 2025, at 07:18, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> On 28 Sep, 2025, at 8:07 pm, dave seddon <dave.seddon.ca@gmail.com> wrote:
>> 
>> cakeConfig = {
>>   Bandwidth = "990M";  # We currently have 1Gb/s, so this is our limit
>>   OverheadBytes = 38;  # Ethernet overhead (preamble + inter-frame gap + FCS)
>>   MPUBytes = 84;       # Minimum packet unit for Ethernet
>>   NAT = true;
>>   FlowIsolationMode = "triple";
>>   PriorityQueueingPreset = "besteffort";
>> };
> 
> It's perhaps worth talking about these parameters a bit.
> 
> You probably realise that the overhead compensation should be set up for the physical bottleneck link.  Not everything is, or behaves exactly like, an Ethernet cable, even if the connection between your router and the transceiver for that link physically is one.  Or it might really be Ethernet, but with extra headers patched into each Ethernet frame (in a domestic context, that's what PPPoE partially is, but VLANs add an extra couple of words to a frame too).  If in doubt, err on the side of assuming there's more overhead than you know about.  These parameters are, however, completely inadequate to describe wireless links, especially WiFi.

+1; over estimating the true overhead (or MPU) slightly has negligible cost (a small reduction in potential maximum throughput), underestimating however can result in noticeable latency under load increases. Some years ago, I retested my assumed de-bloated DSL link (after a few months of uneventful operation) because somewhat felt just not right under load, and lo and behold, my ISP had increased the overhead by 4 bytes (a VLAN tag) in the interim invalidating my overhead settings by 4 bytes... (That was with an ATM/AAL5 link, where measuring the actual overhead is achievable for end-users).

Long story, short: if in doubt rather add a small safety margin to your best overhead estimate... sqm-scripts came down on recommeding 44 bytes as default. Will not cover all possibilities, but goes a long way in covering those encapsulation seen often on ebd user internet access links.


> 
> I'd also like to remind everyone what "triple" flow isolation is.  It's a way to get something like host-and-flow isolation when you're not certain which side of the link the hosts that want isolating from each other are; there's a heuristic which effectively decides, dynamically for each flow, whether to treat it as being src-host or dst-host isolated.  I put it in mainly to serve as a sensible default, since blindly choosing one or the other would get it wrong exactly half the time.  I don't really like seeing it in a config file that someone's actually worked on.

Mmmh, we might want to tweak the tc-cake man page then to make this clearer? ATM it reads:

dual-srchost
Flows are defined by the 5-tuple, and fairness is applied
first over source addresses, then over individual flows.
Good for use on egress traffic from a LAN to the internet,
where it'll prevent any one LAN host from monopolising the
uplink, regardless of the number of flows they use.

dual-dsthost
Flows are defined by the 5-tuple, and fairness is applied
first over destination addresses, then over individual
flows. Good for use on ingress traffic to a LAN from the
internet, where it'll prevent any one LAN host from
monopolising the downlink, regardless of the number of
flows they use.

triple-isolate (default)
Flows are defined by the 5-tuple, and fairness is applied
over source *and* destination addresses intelligently (ie.
not merely by host-pairs), and also over individual flows.
Use this if you're not certain whether to use dual-srchost
or dual-dsthost; it'll do both jobs at once, preventing any
one host on *either* side of the link from monopolising it
with a large number of flows.


Reading this I would have the impression that selecting triple even on a bespoke config would be fine, no?


> 
> It's better to explicitly configure this rather than using the heuristic.  The tc-cake keywords are "dual-srchost" if the hosts to be isolated are upstream of Cake, and "dual-dsthost" if they are downstream.  In a typical installation where two instances of Cake are managing both directions of traffic, one will be configured each way.
> 
> It's also possible to disable host-isolation entirely, using the "flows" keyword to provide the same kind of flow-isolation as fq_codel (or most other forms of FQ) does.  Cake can also provide only host isolation, relying entirely on statistical multiplexing (and the beneficience of AQM) for managing flows to each host; the keywords are then "srchost" and "dsthost" with the same meaning as above.  If you are seriously worried about the number of active users at critical moments, this could be a useful option for you; the number of hosts Cake can fully handle in this mode is the same as the number of flows it can handle in the normal flow-isolation modes.

I would really like to see some tc -s qdisc output from this network under load, as that should be informative if any twaeks are necessary at all.


> 
> - Jonathan Morton
> _______________________________________________
> Cake mailing list -- cake@lists.bufferbloat.net
> To unsubscribe send an email to cake-leave@lists.bufferbloat.net