From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-il1-x130.google.com (mail-il1-x130.google.com [IPv6:2607:f8b0:4864:20::130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 7E2A93B2A4 for ; Sun, 12 Dec 2021 20:16:51 -0500 (EST) Received: by mail-il1-x130.google.com with SMTP id r2so13646896ilb.10 for ; Sun, 12 Dec 2021 17:16:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=OH11S1LbAQi0lKRL7d51nuqQt3xQ93lHH/ClsrQydx0=; b=fQyalVuDDZByNOZVvr4NSmXV7plaFCzhbuukW+78xXPqGOpxo0ox8hILFRnfa5uUO0 qjWR1+DPoTbSQxpOT/7tHHsrclB1gGSkXBSa3kpxym0Vgel4BQ4suLFjdGQCmWnsy3tv Qx9xTKAxjFLST7MElqdJjXuns0ovh7SN0KFpl033iLHcAJfG+GFekdat8non38gba0Uc GGHh9sMZHR+pBhfHYxC7J7H1gPKsbOMSaTKFbs3MVYZuY3xKWBQiWRwRQ2Arg+XljsMh ZQfCjSYQYiDjzvyjLVs3UcmnWrb+rRwKMudveAJsIeBDWhpCgT707j+5/7QeonQJMrPE KwHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=OH11S1LbAQi0lKRL7d51nuqQt3xQ93lHH/ClsrQydx0=; b=tNt0oqr93Rcr6BsfMYMbn9OaUnvrb47gihXTDs2zIPW89qizB53AWqHAXAZGwoLGpr 1Oa7ARZdJ+xYYQJWJPigAwx83GpaHFxeEbLO/di7BlJQeZHl63zdU4j/gRCdH9kNMXS2 WJCY4kygmHQuj2aH8RUO4sjKhEdAVrF8TNrQK4XkeYzy7BubyEu+quf/V9lszkfqbLbp 6dQgapvkD5NEEs3fW0fVd+5dXqxu1IKiKgCpLD/ff/g5lUil6Cesp0GP5Kd4qw1vKpiJ +0saSv8kzqWOqrNToh4etwhXAXmC3S274IQ2aMPR56SOJQVI0BMNVvdGSh4rHKw+ikKK pbCQ== X-Gm-Message-State: AOAM532oYrHKy6GgU7RZoMB4gMSPntCV5g85rpTlDEKvxDXjIi45Sm5F zMn4IvROYGCcUtbRMAmEHz+xyJuh25/Lnq4fpLQFoKIK X-Google-Smtp-Source: ABdhPJyuBtj0HWeJ1614u37EL2V/OOaWXDBWzRFk7tIWgmAOLNgtDBdnuFrEebPek/WQOqNqz+DtYbLAnmmWFiK6PaE= X-Received: by 2002:a05:6e02:16ca:: with SMTP id 10mr29152138ilx.274.1639358210603; Sun, 12 Dec 2021 17:16:50 -0800 (PST) MIME-Version: 1.0 References: <202107101149.16ABnoRo045497@gndrsh.dnsmgr.net> <20210719155017.GA19783@mail.taht.net> <7A88FC34-46B7-4AB6-BE10-A26E5547CF59@inf.ethz.ch> In-Reply-To: <7A88FC34-46B7-4AB6-BE10-A26E5547CF59@inf.ethz.ch> From: Dave Taht Date: Sun, 12 Dec 2021 17:16:38 -0800 Message-ID: To: Ankit Singla Cc: George Burdell , "starlink@lists.bufferbloat.net" , Sam Kumar Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Starlink] SatNetLab: A call to arms for the next global Internet testbed X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Dec 2021 01:16:51 -0000 On Sun, Dec 12, 2021 at 1:54 PM Ankit Singla wro= te: > > Sorry I didn=E2=80=99t engage with this, folks =E2=80=94 probably came ac= ross as rude, but just had a large and unexpected career shift ongoing (htt= ps://twitter.com/stub_AS/status/1469283183132876809?s=3D20), and didn=E2=80= =99t feel up to it, especially, as I=E2=80=99m largely abandoning my resear= ch along these lines due to these developments. Congrats on your new gig! It'll be very different from what you're used to. I have a tendency to cc the author(s) of any given paper I like in the hope that it sparks discussion. We get very little "hallway conversation" in this covid era, and certainly back in june, with the wild success of that LEO conference, I'd hoped for an explosion and exposition of new ideas for moving our civilization out of LEO finally. I've spent the last few months in my all too few idle moments trying to envision new ideas for new payloads, new communication technologies, and services with a vastly lowered cost to orbit, and newly cost effective means to fling useful small payloads to and from the closer NEOs (my oft-lonely hobby for over 30 years now). > In any case, I have a lot of respect for you folks educating everyone on = latency and buffer bloat, and have been following Dave (Taht)=E2=80=99s gre= at work in the space for awhile. thx. I am really hoping someone cracks the cross-world sat to sat laser congestion problem!!! Maybe you'll find a way to implement some of our new p4 stuff at scale? I'm pretty happy with proposing cake + bql + some sort of packet pair based beam allocator, but without the abiltiy to hack on the dishy directly, have mostly given up. Don't feel like writing a simulator. I am just waiting patiently for an observable change in the measurements and fiddling now a bit with DTN. I too have to admit that the "call of the data center" has been very, very loud (and profitable) of late. But my heart's in space, and teeny routers. > Best, > Ankit > > On Jul 19, 2021, at 17:50, George Burdell wrote: > > On Sat, Jul 10, 2021 at 01:27:28PM -0700, David Lang wrote: > > any buffer sizing based on the number of packets is wrong. Base your buff= er > size on transmit time and you have a chance of being reasonable. > > > This is very true. Packets have a dynamic range of 64 bytes to 64k (GRO) = and > sizing queues in terms of packets leads to bad behavior on mixed up and > down traffic particularly. > > Also... people doing AQM and TCP designs tend to almost always > test one way traffic only, and this leads to less than desirable behavior > on real world traffic. Strike that. Terrible behavior! a pure > single queue AQM struggles mightily to find a good hit rate when there ar= e a > ton of acks, dns, gaming, voip, etc, mixed in with the capacity seeking > flows. > > Nearly every AQM paper you read never tests real, bidir traffic. It's > a huge blind spot, which is why the bufferbloat effort *starts* with > the rrul test and related on testing any new idea we have. > > bfifos are better, but harder to implement in hardware. > > A fun trick: If you are trying to optimize your traffic for R/T communica= tions > rather than speedtest, you can clamp your tcp "mss" to smaller than 600 > bytes *at the router*, and your network gets better. > > (we really should get around to publishing something on that, when you ar= e > plagued by a giant upstream FIFO, filling it with smaller packets really > helps, and it's something a smart user could easily do regardless of the > ISP's preferences) > > > In cases like wifi where packets aren't sent individually, but are sent i= n > blobs of packets going to the same destination, > > > yes... > > you want to buffer at least > a blobs worth of packets to each destination so that when your transmit s= lot > comes up, you can maximize it. > > > Nooooooo! This is one of those harder tradeoffs that is pretty counter > intuitive. You want per station queuing, yes. However the decision as to > how much service time you want to grant each station is absolutely > not in maximizing the transmit slot, but in maximizing the number of > stations you can serve in reasonable time. Simple (and inaccurate) exampl= e: > > 100 stations at 4ms txop each, stuffed full of *udp* data, is 400ms/round= . > (plus usually insane numbers of retries). > > This breaks a lot of things, > and doesn't respect the closely coupled nature of tcp (please re-read > the codel paper!). Cutting the txop in this case to 1ms cuts interstation > service time... at the cost of "bandwidth" that can't be stuffed into > the slow header + wifi data rate equation. > > but what you really want to do is give the sparsest stations quicker > access to the media so they can ramp up to parity (and usually > complete their short flows much faster, and then get off) > > I run with BE 2.4ms txops and announce the same in the beacon. I'd > be willing to bet your scale conference network would work > much better if you did that also. (It would be better if we could > scale txop size to the load, but fq_codel on wifi already > does the sparse station optimization which translates into many > shorter txops than you would see from other wifi schedulers, and > the bulk of the problem I see is the *stations*) > > lastly, you need to defer constructing the blob as long as possible, > so you can shoot at, mark, or reschedule (FQ), the packets in there > at the last moment before they are committed to the hardware. > > Ideally you would not construct any blob at all until a few microseconds > before the transmit opportunity. > > Shifting this back to starlink - they have a marvelous opportunity > to do just this, in the dishy, as they are half duplex and could > defer grabbing the packets from a sch_cake buffer until precisely > before that txop to the sat arrives. > (my guess would be no more than 400us based on what I understand > of the arm chip they are using) > > This would be much better than what we could do in the ath9k > where we were forced to always have "one in the hardware, one > ready to go" due to limitations in that chip. We're making > some progress on the openwifi fpga here, btw... > > > Wifi has the added issue that the blob headers are at a much lower data r= ate > than the dta itself, so you can cram a LOT of data into a blob without > making a significant difference in the airtime used, so you really do wan= t > to be able to send full blobs (not at the cost of delaying tranmission if > you don't have a full blob, a mistake some people make, but you do want t= o > buffer enough to fill the blobs) > > and given that dropped packets results in timeouts and retransmissions th= at > affect the rest of the network, it's not obviously wrong for a lossy hop > like wifi to retry a failed transmission, it just needs to not retry too > many times. > > David Lang > > > On Sat, 10 Jul 2021, Rodney W. Grimes wrote: > > Date: Sat, 10 Jul 2021 04:49:50 -0700 (PDT) > From: Rodney W. Grimes > To: Dave Taht > Cc: starlink@lists.bufferbloat.net, Ankit Singla , > Sam Kumar > Subject: Re: [Starlink] SatNetLab: A call to arms for the next global Int= ernet > testbed > > While it is good to have a call to arms, like this: > > ... much information removed as I only one to reply to 1 very > narrow, but IMHO, very real problem in our networks today ... > > Here's another piece of pre-history - alohanet - the TTL field was the > "time to live" field. The intent was that the packet would indicate > how much time it would be valid before it was discarded. It didn't > work out, and was replaced by hopcount, which of course switched > networks ignore and isonly semi-useful for detecting loops and the > like. > > > TTL works perfectly fine where the original assumptions that a > device along a network path only hangs on to a packet for a > reasonable short duration, and that there is not some "retry" > mechanism in place that is causing this time to explode. BSD, > and as far as I can recall, almost ALL original IP stacks had > a Q depth limit of 50 packets on egress interfaces. Everything > pretty much worked well and the net was happy. Then these base > assumptions got blasted in the name of "measurable bandwidth" and > the concept of packets are so precious we must not loose them, > at almost any cost. Linux crammed the per interface Q up to 1000, > wifi decided that it was reasable to retry at the link layer so > many times that I have seen packets that are >60 seconds old. > > Proposed FIX: Any device that transmits packets that does not > already have an inherit FIXED transmission time MUST consider > the current TTL of that packet and give up if > 10mS * TTL elapses > while it is trying to transmit. AND change the default if Q > size in LINUX to 50 for fifo, the codel, etc AQM stuff is fine > at 1000 as it has delay targets that present the issue that > initially bumping this to 1000 caused. > > ... end of Rods Rant ... > > -- > Rod Grimes rgrimes@freebs= d.org > _______________________________________________ > Starlink mailing list > Starlink@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/starlink > > _______________________________________________ > Starlink mailing list > Starlink@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/starlink > > > _______________________________________________ > Starlink mailing list > Starlink@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/starlink --=20 I tried to build a better future, a few times: https://wayforward.archive.org/?site=3Dhttps%3A%2F%2Fwww.icei.org Dave T=C3=A4ht CEO, TekLibre, LLC