From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ej1-x635.google.com (mail-ej1-x635.google.com [IPv6:2a00:1450:4864:20::635]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id DBF853B2A4 for ; Wed, 8 Jun 2022 17:31:11 -0400 (EDT) Received: by mail-ej1-x635.google.com with SMTP id s12so36685092ejx.3 for ; Wed, 08 Jun 2022 14:31:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=teBasKwVO2/jvwJZu8LDdUT2bzsYrw1pbBJUwsMOV+g=; b=N3KdhrFsusSPNyRFY31f6q9bKXjHqyYqrPYDNQetZkp78SDZ8ACJpMXsa5bsPGxcYN 8P6k4ehTHrgpCuIiS8JbqWcdXtNkEhKLlFEjJqtxDbYTvsbcMj174Ms2O1l0pTP3Fark BLOuLZ0wsz50xzMS9ly7bJ2H5Wfiez8d4KFMZdVTqEtZG6kAmz0TIJKscj2tnp3r6QXz yc7bqIUUCNALN1AkMAi6SRZUp60JTyI7GY2KjooSJXGljAizEE7bm0tNe4Jdh1lMnukt JUtsT1EpQi2wEwip1RcnKGKS/Lrg8va/JxMMeDUUwnOAoMqCXPX51kRFmIEcVuP1bDjI 6RKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=teBasKwVO2/jvwJZu8LDdUT2bzsYrw1pbBJUwsMOV+g=; b=M0Uc3GtQ6ceWKFe/AJhm91qmTH/3HFE5k4v3WxBBVX1H/uLjb/tUDUnRXwEBXF7o99 Gi6MuzM26geePu0nd7fD/sMGL4M5OK2eNzwhnj7hXmhQ0ZFUCFKNQSEyd5jkROHI4TSC 0B6R/sFHsXN2TyFB09sj1U1GUhzIgyC60ZOu31YAWMWvOqP/Nyz2cqMiyJi0V8LZndAs 3bZsysE231Wn95O/Os7lkAWbf/6Ju94nY4lkUKU9smoeD9J01Faak+HyzNNNPd2WbU8R lUS1VEOKX46a+3/K1Vxkqx+7jG4ufgg8VkeBW2dOy3p8ykDfHyfbbO3Z/9LC7aJklDL7 R/vw== X-Gm-Message-State: AOAM530Q08b54oOAVKkoUCbzloQ2hah+6FNroB+MhymGLjDb+Ly97FQ3 RvqwYgy7RQrmbkrbbjJ13HVdWxnITNifkEoxHSo= X-Google-Smtp-Source: ABdhPJzOHJoJG+2xGSfQc7RZg/mhksS8aP2u0KUN7bT9p3yRrH9Ervjs4n4ZQFi3zAytF1HhFyUjV4kuoJ8lY5dBfEI= X-Received: by 2002:a17:907:7282:b0:6fa:9365:f922 with SMTP id dt2-20020a170907728200b006fa9365f922mr32599087ejc.262.1654723870423; Wed, 08 Jun 2022 14:31:10 -0700 (PDT) MIME-Version: 1.0 References: <1654714026.72728578@apps.rackspace.com> <1654721363.567314628@apps.rackspace.com> In-Reply-To: <1654721363.567314628@apps.rackspace.com> From: Dave Taht Date: Wed, 8 Jun 2022 14:30:57 -0700 Message-ID: To: "David P. Reed" Cc: warren ponder , starlink@lists.bufferbloat.net Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Starlink] FQ_Codel X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jun 2022 21:31:12 -0000 On Wed, Jun 8, 2022 at 1:49 PM David P. Reed wrote: > > No, I don't think so. However, a hidden (not often discussed) way that us= ing FQ-codel in a home router works is that you have to artificially restri= ct the total bitrate in both directions used by your home router to talk to= the access provider link. > > > > Typically, it is recommended to use 95% of the upload/download speeds of = that link as the limit. This forces packets to be dropped when the constrai= nt is exceeded. Now this forces congestion control signals (dropped packets= ) to be observed by both ends. (In a cable DOCSIS system, this allows the e= dge to manage the throughput of the CMTS for the local endpoint, because th= e CMTS won't drop packets when it should - because configuring DOCSIS 3.1 C= MTS's is often done in a way that causes bufferbloat in CMTS. DOCSIS 2 alwa= ys had bufferbloat in the CMTS). > > > > Starlink doesn't sell you a stable "max rate" - instead that rate varies = depending on traffic, and can't be easily measured. I appreciate david simplifying the problem here, but the details are: On egress... At line rate... ethernet backpressure is provided via the linux BQL facility ( https://lwn.net/Articles/469652/ ) which basically buffers up one completion interrupt's worth of bytes (packets), and punting the complicated FQ and codel drop/mark decisions to a slightly higher layer. This is typically 3k bytes at 100Mbit, 40k to 128k) (With TSO) at a gbit. There's roughly 1/2 a ms of needed buffering at the lowest layer on intel chips today. Some arm chips are capable of doing interrupts quite a bit faster than intel, 100us is feasible on some. Codel, being "time in queue based" also works to drop packets intelligently with ethernet pause frames giving you a "variable rate" link. I'm not big on pause frames, but codel (and pie) can work with them, where RED cannot. We long ago succeeded at making a plethora of the very variable rate wifi devices work (in the driver) by adopting a motto of "one TXOP in the hardware", "one ready to go", for wifi, leading to a max unmanaged buffering of about 10ms before a smarter qdisc can kick in. The biggest problem that wifi had (that I hope starlink doesn't!) was that wifi packet aggregation is needed to get even close to the rated bandwidth, and that with a fifo, rather than a per station queue, performance degrades hugely when flows for more than one station are active. If only I could get a few million more people to lose 8 minutes of their life to this simple and elegant demo of why wifi goes to hell: https://www.youtube.com/watch?v=3DRb-UnHDw02o&t=3D1550s Or read up on how well it's solved as of 2016, and use our test suite for multiple stations at the same time: https://www.cs.kau.se/tohojo/airtime-fairness/ In both cases with smarter hardware all that could be eliminated, but I digress. 10ms worst case latency for a sparse flow is much better than the 7 seconds I first observed on the first wifi router starlink shipped. I apologize for conflating these things - the principal wireless gear I'm familiar with is wifi, and a few other TDM style wireless macs. The principal problem the starlink network has is on the satellite uplink, followed by the satellite downlink, and the wifi router problems only show up if you use the router for local things. The wifi solution we came up with seems generalizable to most forms of wireless, including LTE/5G and starlink. > So to configure the dishy or an edge router connected to it correctly, yo= u need to enforce such a limit such that it actually causes FQ-codel to see= dropped packets. So to reiterate, for egress from the client up to the sat: 1) with an interrupts worth of backpressure from the radio, we just slam cake, fq_codel, or fq_pie on it, and we have a minimum inter-flow latency of however long that takes (under a ms) for sparse flows, and a buffering target of 5ms with margin of 100ms for fat flows. (My guess is that the radio is actually scheduled on intervals higher than that btw) or 2) actual (and accurate) bandwidth stats from the radio as to how much can fit into the next transmit opportunity, we can set a cake rate slightly below that. The 24Mbit up we've seen would have oh, a 2% cpu impact on these arm chips running cake with all options enabled. With the shaper, call it 7% Ingress is harder, ideally, as I've said, they'd be managing the queue depth and fq at the head end, and not in the sky, doing complicated things like per-customer fairness, etc, there, and doing just enough queuing in the sky to cover for 2 tx opportunities. The approach we've long taken with shaping ingress also at the customer router, well, that's WIP for variable rate links... > > On Wednesday, June 8, 2022 3:12pm, "warren ponder" = said: > > So this is really helpful. Is it fair to say then that end users with SQM= and fq_codel on a Starlink connection should essentially not turn on SQM.a= nd.just leave it off? Right now, aside from trying to line up more testers of the autorate code, it's not worth doing, yes. > On Wed, Jun 8, 2022, 11:47 AM David P. Reed wrote: >> >> I'm just going to remind folks that fixing bufferbloat in Starlink won't= be possible with FQ-Codel in the CPE equipment. If that were possible, it = could be fixed entirely in a box sitting between the dishy and the user's "= home network". >> >> >> >> Evidence exists that the bulk of the "bloat" can exist, not just in the = dishy, but also in the central "access point" where satellites in a coverag= e region direct all the traffic from and to the public Internet. This conne= ction from the region becomes bloated if the inbound link and outbound link= become "underprovisioned" for peak rates of all the served dishy terminals= . >> >> That public-Internet to StarLink access point (is there a more distinct,= precise name) can develop a very long delay queue. For the same reason th= at bufferbloat always gets designed in - memory is cheap and plentiful, so = instead of dropping packets to minimize latency, the link just stores packe= ts until multiple seconds worth of traffic build up on one or both ends of = that link. >> >> >> >> This problem can be solved only by dropping packets (with packet drop ra= te mitigated by ECN-marking) to match the desired round-trip latency across= the entire Internet. Typically, this queue size should max out and start d= ropping packets at about 2 * cross-Internet desired latency * bit-rate of t= his link. >> >> Cross-Internet desired latency can be selected these days by using light= -speed in fiber between one side of the North American continent and the ot= her - around 15 msec. is appropriate. (which should be the worst case end-t= o-end latency observed using Starlink, and is around the 20 msec. number ba= ndied about by Musk - though he really never understood what end-to-end lat= ency means). >> >> >> >> >> >> Now it may be that the dishy itself also has such bloat built in, which = would make FQ-Codel in the dishy also important. >> >> >> >> The problem called bufferbloat occurs whenever ANY router on ANY end-to-= end shared path allows such queueing delay to accumulate before shortening = the queue. >> >> >> >> It really frustrates me that memory keeps being added to router outbound= buffers anywhere. And it may be that the reason is that almost nobody who = designs packet forwarding systems understands Queueing Theory at all! It ce= rtainly doesn't help that "packet drops" (even one or two per second) are c= onsidered a failure of the equipment. >> >> >> >> FQ-codel is great, but why it works is that it makes the choice of what = packet to drop far better (by being fair and a little bit elastic). However= , the lack of FQ-Codel doesn't fix system-level bufferbloat. >> >> >> >> >> >> _______________________________________________ >> Starlink mailing list >> Starlink@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/starlink > > _______________________________________________ > Starlink mailing list > Starlink@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/starlink --=20 FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_code= l/ Dave T=C3=A4ht CEO, TekLibre, LLC