[Bloat] taking apart BBR's behaviors in flent

[Dave Taht]

I have pushed an updated set of bbr verses cubic tests (formerly
cake/fq_codel/pie/bfifo_256/cake_flowblind)  now including pfifo_100,
pfifo_1000, and a CMTS/cable modem emulation that dates back to the
network conditions we had in 2011: 20Mbit down, 5Mbit up, when the
bufferbloat project began.

http://blog.cerowrt.org/flent/bbr-comprehensive.tgz

I note that my conditions are reversed throughout: "up" is from the
server to the client (20Mbit). Mae Culpa: I dare not change anything
on the clients right now as other testing is going on. So what I'm
measuring and emulating here, is downstream bufferbloat (CMTS to
cablemodem to end user), rather than what Jim Gettys started with
(seeing uplink bloat measured in seconds).

Since 2011, download and upload speeds have improved (I think
100/20Mbit is most common nowadays(?), modem buffering on some models
has declined somewhat, CMTS configurations have got better - but that
emulation remains close to correct and in some cases, is much less
bloated than the results many cable users still get today. I also have
dsl, and other emulations, but I figure this dataset is enough to go
on for a while. (since 2011 Web pages have got "tighter", uTP has
deployed to a larger extent, BQL and other mods to the linux stack are
now common, etc, etc. also)

I did produce a couple graphs.

2011 in all it's cubic bloated glory:

http://blog.cerowrt.org/flent/bbr-comprehensive/bufferbloat_20Mbit_2011_glory_cubic.svg

vs BBR competing with itself:

http://blog.cerowrt.org/flent/bbr-comprehensive/bufferbloat_bbr_2016.svg

And one showing a full fledged simultaneous up/down test with bbr +
cubic on the increasingly misnamed squarewave test.

http://blog.cerowrt.org/flent/bbr-comprehensive/bbr_vs_cubic_20Mbit_cablemodem_emulation.png

I do not have more time or energy to review this further today, nor
look at the captures. Probably. Wet Paint!

Enjoy the data. Happy graphing!

PS: one hysterical thing that happened: when I'd started pushing this
data to my git repository and website, I'd forgotten I'd left the
cable modem emulation on and rtt 48 stuff enabled - as well as cubic -
and I had to endure watching the darn thing slow, stop, resume in a
burst, and so on, for what felt like ages.