[Cake] Cake vs fq_codel and c/burst on an ER-X bridge

Sebastian Moeller moeller0 at gmx.de
Tue Sep 11 14:28:52 EDT 2018


Hi Pete,


> On Sep 11, 2018, at 20:09, Pete Heist <pete at heistp.net> wrote:
> 
> 
>> On Sep 11, 2018, at 9:54 AM, Sebastian Moeller <moeller0 at gmx.de> wrote:
>>> 
>>> So this has turned info an interesting exercise that produced a result counter to what the common wisdom has been (that fq_codel is “faster” than cake
>> 
>> 	I believe the argument is more about htb+fq_codel versus cake instead of fq_codel versus cake, as it seems to be the shaper functionality that incurs the highest cost.
> 
> I sometimes loosely use fq_codel to mean htb+fq_codel.

	Sorry for being a smart aleck, unfortunately it is my second nature.

> 
>>> It occurs to me that what I really want to know is the maximum set bitrate for the shaper where it still appears to be behaving properly and consistently, meaning, the actual measured TCP throughput is held steady, and at the expected percentage less than the set bitrate. I first find this out by setting a “comfortable” rate of 100Mbit and checking TCP throughput with iperf3, which is typically around 5% less than the set bitrate.
>> 
>> 	So the expected values somewhat depend on the exact configuration, but over all the expected TCP/IPv4 goodput is calculated as follows (I assume you are well aware of that, but I believe this worth repeating to calibrate the expectancy):
> 
> Yes, it’s pretty much right on the money.
> 
>>> Then I increase the shaper’s bitrate 5Mbit at a time and re-run the test until I find the last bitrate at which iperf3 reports a stable (within a few percent) and correct rate over 10 seconds for several runs in a row. See the attached iperf3 results for sample runs around the threshold rates.
>> 
>> 	Except for the 10 seconds this sounds reasonable, I would aim for at least 30, even tough this will be more important once you also monitor the latency under load concurrently to the bandwidth-probing flows…
> 
> I agree, so far I was just trying not to spend an all-nighter on it last night. :) I was actually running 3-5 trials of ten seconds, also because sometimes with fq_codel once it’s slightly above its limit, results would vary from run to run.

	Ah, why is it so often reality that crosses the most diligent plans ;) ?

> 
>>> cake nat dual-srchost / cake nat dual-dsthost ingress: 135 / 145
>> 
>> 	On your box is there actual NAT masquerading happening?
> 
> Yeah, good point, I left nat there because I had one port configured for routing and the other for the bridge and was sometimes swapping between the two. I realize now I actually sent the numbers for routing, not bridging. Bridging without ‘nat’ looks a bit higher (155 Mbit for cake instead of 135 Mbit). I would re-do all these tests for completeness but I’m out of time now.

	Ouch, a ten percent bandwidth cost for the nat feature certainly answeers the question whether nat should be the default...

> 
>> 	The last time we discussed the bust issue, I could not manage to see any difference with or without a specified burst, but I strongly believe I simply did not properly test. Btw, this is unidirectional shaping or with bidirectional saturation?
> 
> Unidirectional. I definitely see a difference, but I wonder what criteria we (and I) used for “out of CPU’ in the past.

	So totally unscientifically me yardstick was as long as throughput increases more or less linearly with configured shaper bandwidth things are fine, and then at the candidate bandwidths I ran "top -d 1" and monitored both idle% ad sirq% with idle falling below 5% being a strong indicator of bottlenecking on cpu cycles. Dlakelan over at github (https://github.com/dlakelan/routerperf) is working on a small side project that aims for tighter multi-core aware logging of cpu usage on a router, but that has not left the early prototype stage.

> 
>> <fq_codel_125.txt><fq_codel_130.txt><cake_135.txt><cake_140.txt>
>> 	I am quite curious about these files, but I seem incapable of downloading/opening them...
> 
> Ok, no more sending attachments to the list I see. If this link doesn’t work I might be replacing a disk at the time… :)
> https://www.heistp.net/downloads/erx-sqm.tgz

Great, that worked; but now I realize that I need to get acquinted with iperf output ;)

Best Regards
	Sebastian

> 



More information about the Cake mailing list