[Cake] Fighting bloat in the face of uncertinty

[Justin Kilpatrick]

> What was the scenario that gave you trouble?

We had a 1ms link bloating on a ~6ms path using the default 'internet' profile. The link got very bloaty when loaded (100ms) but only in the download direction.

All traffic in this system has Cake applied on egress, so the sending node was using Cake to place packets onto the 100mbps line, and the receiving node (which was not applying Cake on input) got them up to 300ms later. 

On the other hand during an upload test (which seems like a trivial reversal of the same situation) things where flawless even with the 'internet' rtt value of 100ms. 

Setting the throughput to the link capacity or using the 'metro' /'lan' profiles worked, but was only required on the upstream node to resolve download bloat. 

I think this is an artifact of the traffic in question? Bloat only occurred in the download direction where there where many user streams all happy to munch away at more than the link capacity if you let them. Upload was much less contentious. 

> I note that Cake is not defined in an RFC.  Were you referring to a 
> Codel RFC?  Cake is a bit more sophisticated, with the aim of making it 
> easier to configure.

Yes, at least for definitions of target/interval etc and some tuning guidelines. Although apparently I still get them confused. 

> Should the bandwidth setting correspond to a serialisation delay per 
> packet that approaches the 'target' implied by the above, 'target' will 
> automatically be tuned to avoid the nasty effects that might cause - 
> *unless* you manually override it.  So don't do that.

I can estimate link throughput with more reliability than RTT. But not exactly much, it's easy to be off by 50%. Since setting throughput reduces capacity if I'm wrong I've tried to stay way from it. 

If I set a throughput that's 50% too high should it still help? In my testing it didn't seem to. But I was still using the 'internet' key word otherwise so maybe I was just shooting myself in the foot some other way at the same time. 



-- 
  Justin Kilpatrick
  justin at althea.net

On Sat, Sep 7, 2019, at 7:09 PM, Jonathan Morton wrote:
> > On 8 Sep, 2019, at 1:42 am, Justin Kilpatrick <justin at althea.net> wrote:
> > 
> > I'm using Cake on embedded OpenWRT devices. You probably saw this video on the list a month or two ago. 
> > 
> > https://www.youtube.com/watch?v=G4EKbgShyLw
> 
> I haven't actually watched that one yet...
> 
> > Anyways up until now I've left cake totally untuned and had pretty great results. But we've finally encountered a scenario where untuned Cake allowed for unacceptable bufferbloat on a link.
> > 
> > Hand configuration in accordance with the best practices provided in the RFC works out perfectly, but I need a set of settings I can ship with any device with the expectation that it will be used and abused in many non-standard situations. Producing non-optimal outcomes is fine, producing dramatically degraded outcomes is unacceptable. 
> 
> What was the scenario that gave you trouble?
> 
> I note that Cake is not defined in an RFC.  Were you referring to a 
> Codel RFC?  Cake is a bit more sophisticated, with the aim of making it 
> easier to configure.
> 
> > Which leads to a few questions
> > 
> > 1) What happens if the target is dramatically too low? 
> > 
> > Most of our links can expect latency between 1-10ms, but they may occasionally go much longer than that. What are the consequences of having a 100ms link configured with a target of 10ms?
> 
> The default 'target' parameter is normally 5ms, which goes with a 
> default 'rtt' and 'interval' parameter of 100ms.
> 
> You shouldn't normally need to set 'target' and 'interval' manually, 
> only 'rtt', and there are various keywords to assist with choosing an 
> appropriate 'rtt'.  The default of 100ms is provided by the 'internet' 
> keyword, and this should be able to cope reasonably well with paths 
> down to 10ms.  You could also try "regional" which gives you tuning for 
> 30ms, or "metro" which gives you 10ms, with good behaviour on paths 
> within about an order of magnitude of that.
> 
> Remember, it's the path RTT that matters for this, not the link itself.
> 
> Should the bandwidth setting correspond to a serialisation delay per 
> packet that approaches the 'target' implied by the above, 'target' will 
> automatically be tuned to avoid the nasty effects that might cause - 
> *unless* you manually override it.  So don't do that.
> 
> ECN enabled flows should not easily notice an 'rtt' setting that's too 
> small.  RFC-3168 compliant transports only care about how many RTTs 
> contain at least one CE mark.  Non-ECN flows may see elevated packet 
> loss, however, and thus more retransmissions, but the same congestion 
> control behaviour.  Cake protects these flows from experiencing "tail 
> loss" which could lead to an RTO that the end-user would notice.
> 
> > 2) If interval is dramatically unpredictable is it best to err on the side of under or over estimating?
> > 
> > The user may select an VPN/exit server of their own choosing, the path to it over the network may change or the exit may be much further away. Both 10ms and 80ms would be sane choices of target depending on factors that may change on the fly.
> 
> Generally the default 'rtt' of 100ms is suitable for generic Internet 
> paths, including nearby 10ms hops and 500ms satellite-linked islands.  
> The default 5ms target actually puts a floor on the minimum effective 
> RTT that the marking schedule has to cope with.  There's also a good 
> chance that the "hystart" algorithm in CUBIC will drop it out of 
> slow-start on very short-RTT paths before the AQM triggers.
> 
>  - Jonathan Morton
> 
>