A few years back, we looked at flow size distributions of traffic into the Cook Island's Rarotonga via their O3b MEO satellite link (read: RTT > 120 ms for everything).

Roughly speaking, the interesting insight was that around half of the TCP traffic by byte volume was sitting in flows of at most 10 packets. Now 10 packets is the size of the initial cwnd in most currently deployed TCP stack instances, meaning by and large this part of the traffic isn't really subject to congestion control or flow control at all: The up to 10 packets usually get sent out in one go (unless one or more get lost and need to be retransmitted, but that's rare).

Most of this small fry traffic consists of HTTP responses with HTML, CSS and JavaScript files and small / thumbnail images. In fact, a lot of web sites don't contain objects much larger than this. So basically, when dealing with this sort of traffic, TCP never really gets out of slow start mode.

If you ramp up demand on a link like this (=add users), then these small size flows end up squeezing out the larger flows that go through the slow start / back-off / recovery cycle their respective TCP flavour prescribes, which gives them a competitive disadvantage. Typical symptom when you're on a link affected by this: You can check your e-mail OK as long as you don't take an interest in the attachments!

The problem here is that unless you have a way of breaking up that small fry traffic, anything traffic at all that responds to drop- or ECN-induced congestion control of any sort gets put on the back foot. Bufferbloating just delays the congestion control response (helpful for large flows) at the expense of suppressing cwnd growth in slow start (not helpful for large flows). Long RTTs put that right into focus - MEO and GEO sats are a good teacher there. 

Attached is a picture of what slow start looks like on a 100Mbit plan
(acquired via the libreqos testbed, our tests vary, but if you would
like to see many ISP plans tested against (presently) cake, feel free
to click on https://payne.taht.net - it is not up all the time, nor
are the tests the same all the time, for details as to what is
running, please join us in the #libreqos:matrix.org chatroom)

An overall point I have been trying to make is that *at some point*,
any sufficiently long flow will exceed the available fifo queue
length, and drop packets, sometimes quite a lot. That is a point, the
high water mark, worth capturing the bandwidth in, say, the prior
100ms. To me packet behaviors look a lot like musical waveforms,
especially when sampled at the appropriate nyquist rate for the
bandwidth and rtt. Out of any waveform, these days, I can usually pick
out what AQM (if any) is in action. I hope one day soon, more people
see patterns like these, and glean a deeper understanding.

I also keep hoping for someone to lean in, verify, and plot some
results I got recently against mkeown´s theories of buffersizing,
here:

https://blog.cerowrt.org/post/juniper/

I don´t trust my results, especially when they are this good.

_______________________________________________
Starlink mailing list
Starlink@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink

-- 
****************************************************************
Dr. Ulrich Speidel

School of Computer Science

Room 303S.594 (City Campus)

The University of Auckland
u.speidel@auckland.ac.nz 
http://www.cs.auckland.ac.nz/~ulrich/
****************************************************************