[Cake] WAN ingress rate with concurrent downloads

[Sebastian Moeller]

Hi Georgios,


> On Oct 28, 2016, at 03:34, Georgios Amanakis <g_amanakis at yahoo.com> wrote:
> 
> Hi Sebastian,
> 
> the ISP is Comcast in MD, USA, residential connection. It is a cable
> connection (i.e. the WAN port on the modem is a coaxial one) and not a
> DSL one.

	Interesting, from the bandwidth I had assumed ADSL ;)

> Maximum ingress rate is ~450kbytes/s which is what the modem
> syncs at.

MMh typically sync-rates are soecified in Kbps or 10^3 Bits per second, not KiloBytes and certainly not 2^10 Bytes; could you clarify what exactly your modem reports and what the best numbers you get from speedtests (like dslreoorts.com) without active AQM/cake/htb, please?


> By using "test_WAN_dual-isolate__piece_of_cake.qos", "tc -d
> qdisc show" reports (enp0s13 is WAN, enp0s14 is LAN):
> -------8<--------
> qdisc noqueue 0: dev lo root refcnt 2 
> qdisc fq_codel 0: dev enp0s14 root refcnt 2 limit 10240p flows 1024 		
> quantum 1514 target 5.0ms interval 100.0ms ecn 
> qdisc cake 8002: dev enp0s13 root refcnt 2 bandwidth 900Kbit besteffort
> 		dual-srchost nat rtt 100.0ms raw 
> qdisc ingress ffff: dev enp0s13 parent ffff:fff1 ---------------- 
> qdisc fq_codel 0: dev ifb0 root refcnt 2 limit 10240p flows 1024 		
> quantum 1514 target 5.0ms interval 100.0ms ecn 
> qdisc cake 8003: dev ifb4enp0s13 root refcnt 2 bandwidth 3300Kbit 		
> besteffort dual-dsthost nat rtt 100.0ms raw 
> -------8<--------
> 
> If one of the hosts on LAN is running bittorrent, bmon on the gateway
> reports:
> 			RX bps 		RXpps	TX bps		TXpps
> enp0s14		23.55KiB		303		402.87KiB	322
> enp0s13		457.26KiB	369		25.04KiB		307
> 

Okay so:
900Kbps = 900 * 1000 / (8 * 1024) = 109.86328125 KiB
3300Kbps = 3300 * 1000 / (8 * 1024) = 402.83203125 KiB

So it looks like enp0s14-TX actually shows the specified internet-download rate, (internet upload is not maxed out)



> 
> Setting the ingress rate at 1900kbps (instead of 3300kbps as above),
> "tc -d qdisc show" reports:
> -------8<--------
> qdisc noqueue 0: dev lo root refcnt 2 
> qdisc fq_codel 0: dev enp0s14 root refcnt 2 limit 10240p flows 1024 		
> quantum 1514 target 5.0ms interval 100.0ms ecn 
> qdisc cake 8008: dev enp0s13 root refcnt 2 bandwidth 900Kbit besteffort
> 		dual-srchost nat rtt 100.0ms raw 
> qdisc ingress ffff: dev enp0s13 parent ffff:fff1 ---------------- 
> qdisc fq_codel 0: dev ifb0 root refcnt 2 limit 10240p flows 1024 		
> quantum 1514 target 5.0ms interval 100.0ms ecn 
> qdisc cake 8009: dev ifb4enp0s13 root refcnt 2 bandwidth 1900Kbit 		
> besteffort dual-dsthost nat rtt 100.0ms raw 
> -------8<--------
> 
> Using bittorrent, bmon on the gateway reports:
> 		RX bps 		RXpps	TX bps		
> TXpps
> enp0s14		19.25KiB	219	232.51KiB	184
> enp0s13		393.36KiB	304	20.25KiB	220
> 

Okay so:
900Kbps = 900 * 1000 / (8 * 1024) = 109.86328125 KiB
3300Kbps = 1900 * 1000 / (8 * 1024) = 231.93359375 KiB

And again enp0s14-TX shows the specified internet-download rate, (internet upload is not maxed out)

For me this looks like bwmon accounts the incoming packets before cake sees and drops them (the enp0s14-TX shows that cake does only transmit the specified rate into the LAN, so cake does its job)). Then the fact that enp0s13 (btw these new interface names have not yet grown on me, yuck) sees more incoming packets simply means that the bit torrent peers are an unruly bunch and do not react nicely to cakes drops and marks (are the packets ECN labeled?). Bit-torrent, when using its µTP protocol tries to use increasing delay as signal to throttle, but cake removes that latency increase under load, which mjight make the torrent peers not scale back their TX rates based on a wrong believe that there is more link capacity available. Not that anybody asks me, but I believe µTP actively is a bad thing and all torrent peers shoud default to TCP instead of µTP?UDP, but I digress.


> In both cases the ingress rate limit on the WAN interface (enp0s13) is
> not obeyed. I verified these data by using tcpdump and doing an IO-
> Graph with wireshark.

	Well that is true in that cake does not run a lower rate on the ethernet interface, but what it does happens after packets are passed from the NIC to the OS. Again I believe you just show bit-torrent misbehaving, a nice illustration why we can’t have nice things, erm why non-latency-inducing shapers, like cake and HTB+fq_codel have a hard time ruling in bit-torrent… BTW, can you see from your packet captures how long a typical torrent-flow actually exists and is active? So are these say a few dozend long-term flows or a swarm of dozens of flows that come and go on a seconds time-frame?

Best Regards
	Sebastian

> 
> Looking forward to your feedback,

P.S.: I am just an amateur in these matters, so other’s on the cake list will surely be able to give better advice. So @experts, please holler if my theory above does not make sense...


> George
> _______________________________________________
> Cake mailing list
> Cake at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake