[Bloat] Curious bufferbloat on DOCSIS connection with many downloads

[Arie <nospam@ariekanarie.nl>]

[-- Attachment #1: Type: text/plain, Size: 2006 bytes --]

Recently I accidentally discovered something quite odd about my home DOCSIS
connection (shaped with cake) while running a flent rrul test.
During the flent rrul test, another machine in my home had started a Steam
game update, this opened somewhere between 8-12 connections and saturated
my downstream.

The interesting part was the resulting rrul graph, showing perfect latency
control during the test, but 100-150ms of bufferbloat at the very start and
end of the rrul graph (when rrul is idle, just pinging).

Instead of a Steam download I started an iperf3 client on a remote machine
that opened 16 connections to a LAN machine. Same behavior as the Steam
download.

At first I thought this was a bug with cake, but no matter how I tuned cake
(limiting my 400/40 connection to 50/20 for example), the behavior was
similar.
I tried fq_codel+htb, and got similar results to cake, high latency before
the rrul test kicked off, great latency while rrul was doing its up and
downloads.

So I swapped out my self-compile LEDE router with an Edgerouter ER-X with
both stock and LEDE firmware, both with fq_codel and cake (on both stock
and LEDE firmware). Same results.
Even without any SQM active, the rrul test still improved the latency of my
connection when it's also downloading from 8+ connections at the same time.

I managed to take the flent rrul test out of the equation by starting a
hping3 "flood" to the hop next of my cable modem.

hping3 -2 -d 0 -s 10080 -k -p 80 -i u100 first-hop-or-ISP-gateway-goes-here
>

This also fixed the latency while many downloads were running by sending
about 300KB/s of empty UDP packets to some unresponsive port on my ISPs
local gateway.

I'm puzzled by this result, somehow sending 300KB/s of empty packets
upstream drastically improves the latency of my connection when it's
receiving many downloads.

Equipment used:
Cisco EPC3212 cable modem (8x4 EuroDOCSIS 3.0)
Linksys WRT1900ACS router with LEDE
Ubiquiti Edgerouter ER-X with both EdgeOS (stock) and LEDE

[-- Attachment #2: Type: text/html, Size: 2496 bytes --]