[Bloat] Inaccurate rates with HTB/HFSC+fq_codel on router due to VLAN?

[Sebastian Moeller]

Hi xnor,

> On Mar 17, 2017, at 20:15, xnor <xnoreq at gmail.com> wrote:
> 
> Hey,
> 
> please reply to the list address so that everyone can see it.

	Sure can do, not a secret (at least not intended as a secret). Then again, if it had been you would have spoiled it effectively...

> 
> 
>> Just a guess, but I believe /sys/class/net/eth0.2/statistics/rx_bytes shows the data before HTB/HFSC had a chance to touch the packets.
> Of course. So? HTB/HFSC don't shrink packages, and since I'm only doing TCP and also use fq_codel it should shape ingress quite nicely to my configured rate.

	Erm, so TCP tends to probe the link capacity by increasing its bandwidth repeatedly, given enough flows some will always be touching against the limit. If you look at the rate packets emerge from the shaper instead the rate they rare fed into the shaper you would have IMHO a stronger argument that the shaper does not shape correctly… If after the shaper the rate is as configured but too large before it could mean that some of your senders do not respond nicely to the slow-down signal, or the slow down signal is unintelligible to the senders (like the shaper assumes ECN while the senders do set ECN bits but do not respond properly to the ECN signaling.


> 
>> it would be interesting to repeat the tests but run tcpdump on the interface that delivers the traffic to the test machine on the internal LAN. My prediction is that on that port you will pretty much see the 18000Kbps you expect.
> The LAN is connected through eth0.1 which is part of a bridge interface br-lan (this bridge is the only other interface with an IP address besides eth0.2).
> 
> With 160 download connections I've just measured (also included the average bytes per packet, short bpp):
> 
> eth0.2: 20.3 Mbps download (~1400 bpp)
> eth0: 21.6 Mbps download (~800 bpp), 19 Mbps upload (~780 bpp)
> eth0.1: 18 Mbps upload (~1490 bpp)
> br-lan: 18 Mbps upload (~1490 bpp)
> 
> (all numbers approx. accurate to about +/- 0.1 Mbps)

	I am confused about the reported directions here, but mostly by the average packet sizes on eth0, why are these at 100-100*800/1490 = 46.3087248322 % of the internal packet sizes? Does this indicate massive fragmentation on your wan link?


> 
> This is completely saturating the 20 Mbps link and ruins performance.

	It seems your router only has one port from the SoC to the switch and the WAN port also lives on the switch, so all packets will need to first go from WAN port to CPU via eth0 and then out again via eth0 to the LAN port on the switch, is that a correct interpretation? In that case you can also instantiate the internet download shaper as egress/upload shaper on eth0.1 avoiding the ifb dance... 

> 
> I've tested to decrease the rate to make it work in the above scenario: I had to back off with the rate to about 14 Mbps (!) because then, as you can guess, measured eth0.2 bandwidth drops to below the 20 Mbps link speed.

Well the true link speed should be a hard ceiling...

> 
> With less connections the measured eth0.2 bandwidth is closer to the configured 18 Mbps and so works fine..

	Again, incoming data is first handled and accounted by eth0/eth0.2 before the IFB sees it, so I very much expect to see more than the shaper rate at this point. If the shaper emits the correct rate into your internal network and also drops packets it seems unavoidable that more than the configured rate worth of data arrives into the shaper. But I am not an expert and hence might be wrong.

> 
> 
>> This seems old, if your router is supported you might want to try lede (https://lede-project.org/releases/17.01/start#), then you could also use the cake qdisc which has a few new tricks up its sleeve, nicer than HTB and HFSC...
> I am planning to upgrade, but I highly doubt it'll help and it also doesn't help me clear up the confusion with what is going on here.

	Well you wou;d jump from kernel 3.18? to a more recent 4.4 series kernel which might/should have some bugs fixed. But I understand the desire to understand the current situation better before moving on.

> 
> 
> It's definitely shaping something. The question is: what?

	Packets?

Best Regards

> 
> _______________________________________________
> Bloat mailing list
> Bloat at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat