[Starlink] FQ_Codel

Wed Jun 8 17:30:57 EDT 2022

On Wed, Jun 8, 2022 at 1:49 PM David P. Reed <dpreed at deepplum.com> wrote:
>
> No, I don't think so. However, a hidden (not often discussed) way that using FQ-codel in a home router works is that you have to artificially restrict the total bitrate in both directions used by your home router to talk to the access provider link.
>
>
>
> Typically, it is recommended to use 95% of the upload/download speeds of that link as the limit. This forces packets to be dropped when the constraint is exceeded. Now this forces congestion control signals (dropped packets) to be observed by both ends. (In a cable DOCSIS system, this allows the edge to manage the throughput of the CMTS for the local endpoint, because the CMTS won't drop packets when it should - because configuring DOCSIS 3.1 CMTS's is often done in a way that causes bufferbloat in CMTS. DOCSIS 2 always had bufferbloat in the CMTS).
>
>
>
> Starlink doesn't sell you a stable "max rate" - instead that rate varies depending on traffic, and can't be easily measured.

I appreciate david simplifying the problem here, but the details are:

On egress... At line rate... ethernet backpressure is provided via the
linux BQL facility ( https://lwn.net/Articles/469652/ ) which
basically buffers up one completion interrupt's worth of bytes
(packets), and punting the complicated FQ and codel drop/mark
decisions to a slightly higher layer. This is typically 3k bytes at
100Mbit, 40k to 128k) (With TSO) at a gbit. There's roughly 1/2 a ms
of needed buffering at the lowest layer on intel chips today. Some arm
chips are capable of doing interrupts quite a bit faster than intel,
100us is feasible on some.

Codel, being "time in queue based" also works to drop packets
intelligently with ethernet pause frames giving you a "variable rate"
link. I'm not big on pause frames, but codel (and pie) can work with
them, where RED cannot.

We long ago succeeded at making a plethora of the very variable rate
wifi devices work (in the driver) by adopting a motto of "one TXOP in
the hardware", "one ready to go", for wifi, leading to a max unmanaged
buffering of about 10ms before a smarter qdisc can kick in. The
biggest problem that wifi had (that I hope starlink doesn't!) was that
wifi packet aggregation is needed to get even close to the rated
bandwidth, and that with a fifo, rather than a per station queue,
performance degrades hugely when flows for more than one station are
active.

If only I could get a few million more people to lose 8 minutes of
their life to this simple and elegant demo of why wifi goes to hell:
https://www.youtube.com/watch?v=Rb-UnHDw02o&t=1550s

Or read up on how well it's solved as of 2016, and use our test suite
for multiple stations at the same time:
https://www.cs.kau.se/tohojo/airtime-fairness/

In both cases with smarter hardware all that could be eliminated, but
I digress. 10ms worst case latency for a sparse flow is much better
than the 7 seconds I first observed on the first wifi router starlink
shipped. I apologize for conflating these things - the principal
wireless gear I'm familiar with is wifi, and a few other TDM style
wireless macs. The principal problem the starlink network has is on
the satellite uplink, followed by the satellite downlink, and the wifi
router problems only show up if you use the router for local things.
The wifi solution we came up with seems generalizable to most forms of
wireless, including LTE/5G and starlink.

> So to configure the dishy or an edge router connected to it correctly, you need to enforce such a limit such that it actually causes FQ-codel to see dropped packets.

So to reiterate, for egress from the client up to the sat:

1) with an interrupts worth of backpressure from the radio, we just
slam cake, fq_codel, or fq_pie on it, and we have a minimum inter-flow
latency of however long that takes (under a ms) for sparse flows, and
a buffering target of 5ms with margin of 100ms for fat flows. (My
guess is that the radio is actually scheduled on intervals higher than
that btw)

or

 2) actual (and accurate) bandwidth stats from the radio as to how
much can fit into the next transmit opportunity, we can set a cake
rate
slightly below that.

The 24Mbit up we've seen would have oh, a 2% cpu impact on these arm
chips running cake with all options enabled. With the shaper, call it
7%

Ingress is harder, ideally, as I've said, they'd be managing the queue
depth and fq at the head end, and not in the sky, doing complicated
things like per-customer fairness, etc, there, and doing just enough
queuing in the sky to cover for 2 tx opportunities. The approach we've
long taken with shaping ingress also at the customer router, well,
that's WIP for variable rate links...

>
> On Wednesday, June 8, 2022 3:12pm, "warren ponder" <wponder11 at gmail.com> said:
>
> So this is really helpful. Is it fair to say then that end users with SQM and fq_codel on a Starlink connection should essentially not turn on SQM.and.just leave it off?

Right now, aside from trying to line up more testers of the autorate
code, it's not worth doing, yes.

> On Wed, Jun 8, 2022, 11:47 AM David P. Reed <dpreed at deepplum.com> wrote:
>>
>> I'm just going to remind folks that fixing bufferbloat in Starlink won't be possible with FQ-Codel in the CPE equipment. If that were possible, it could be fixed entirely in a box sitting between the dishy and the user's "home network".
>>
>>
>>
>> Evidence exists that the bulk of the "bloat" can exist, not just in the dishy, but also in the central "access point" where satellites in a coverage region direct all the traffic from and to the public Internet. This connection from the region becomes bloated if the inbound link and outbound link become "underprovisioned" for peak rates of all the served dishy terminals.
>>
>> That public-Internet to StarLink access point (is there a more distinct, precise name) can develop a very long delay queue.  For the same reason that bufferbloat always gets designed in - memory is cheap and plentiful, so instead of dropping packets to minimize latency, the link just stores packets until multiple seconds worth of traffic build up on one or both ends of that link.
>>
>>
>>
>> This problem can be solved only by dropping packets (with packet drop rate mitigated by ECN-marking) to match the desired round-trip latency across the entire Internet. Typically, this queue size should max out and start dropping packets at about 2 * cross-Internet desired latency * bit-rate of this link.
>>
>> Cross-Internet desired latency can be selected these days by using light-speed in fiber between one side of the North American continent and the other - around 15 msec. is appropriate. (which should be the worst case end-to-end latency observed using Starlink, and is around the 20 msec. number bandied about by Musk - though he really never understood what end-to-end latency means).
>>
>>
>>
>>
>>
>> Now it may be that the dishy itself also has such bloat built in, which would make FQ-Codel in the dishy also important.
>>
>>
>>
>> The problem called bufferbloat occurs whenever ANY router on ANY end-to-end shared path allows such queueing delay to accumulate before shortening the queue.
>>
>>
>>
>> It really frustrates me that memory keeps being added to router outbound buffers anywhere. And it may be that the reason is that almost nobody who designs packet forwarding systems understands Queueing Theory at all! It certainly doesn't help that "packet drops" (even one or two per second) are considered a failure of the equipment.
>>
>>
>>
>> FQ-codel is great, but why it works is that it makes the choice of what packet to drop far better (by being fair and a little bit elastic). However, the lack of FQ-Codel doesn't fix system-level bufferbloat.
>>
>>
>>
>>
>>
>> _______________________________________________
>> Starlink mailing list
>> Starlink at lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/starlink
>
> _______________________________________________
> Starlink mailing list
> Starlink at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink

-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC