From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 36F143B29D; Fri, 29 Jul 2022 13:41:37 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1659116494; bh=dG9nV1mtXfehQmkdUzzze9FzafCHfnAuICzExo9M0P0=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=V9HvXxTVD4A52ooF9QXhMGXci+ddmGFwJpiq+dxD0UtRznVdzU/ANEVNsCvLBvLim 1rGQ9zFEXdHyCHSe1VTSWOnx4QcVQZkQzIFPRQqlxtNwOLJSn4vQ7y/UlAqfVgchXi NWrSRyYkMsEIP3hzM/7pOhITdNiFoQ+4c9JR9mlM= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from smtpclient.apple ([77.6.182.223]) by mail.gmx.net (mrgmx104 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MoO24-1ngENC2dLM-00oo6h; Fri, 29 Jul 2022 19:41:34 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) From: Sebastian Moeller In-Reply-To: Date: Fri, 29 Jul 2022 19:41:33 +0200 Cc: =?utf-8?Q?Bj=C3=B8rn_Ivar_Teigen?= , Dave Taht via Starlink , codel@lists.bufferbloat.net Content-Transfer-Encoding: quoted-printable Message-Id: <1EC381D2-D85A-48DA-8609-0B1579B65229@gmx.de> References: <63E4019E-958A-4772-8D87-F972B67CD0AE@gmx.de> To: =?utf-8?Q?Dave_T=C3=A4ht?= X-Mailer: Apple Mail (2.3696.120.41.1.1) X-Provags-ID: V03:K1:0ZD33VEoQqWRXVH/qMAGriDBMBqvOhSS7xJfKoGupoosfVdD+ci 5Bm51POYXaRHocuyUDjsnmCLfla596i5ZaK7JeV4pmn39Ira29Kp/z9K4HMdxuJhWho9CnD zm7DV2toNjsVPGtz/N4tgeBMzQCPEvG8/8WLzlTETpBkKOqTRgqAFTaXbJlaa5byAgvncOX MvEy9X3gppIIvn1nrMtQQ== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:vlJ9xtsk50Y=:C2o3NErPi4fH6AKd0qRfd1 rWdhcDD5enaf+V36iMX+pq2iyN8gxxnnqZ6J+lxguW0R96FOcyuIv4La5CCfBjOOvhvP/WhSS 9XHkn+Jj9lpDprmJPRsnL4+IXuhUJgaQ9Q+zHf/TllFKPVcicU+dNt0JFaCkA6lvEmIWLMcyw InKd8dK0u+sqkP8qZt3mbYrYf5oTLLuTWxcBx39zpUvThJP4tm9I+HwdVoDlbMPrNagOymbrE hMwRa32r2oxG/Ba8RiF0+Z7aBKX7/8W6Z15cOEeS/kS0gWS/3UrBmyCjWUe+n7hS/6MAnvyyc oOf8tCzpbIS3up1SvvpHaFj2SeOvrpaF1IN+3k7bZP0pzXnpnycuD2jGRjylKtCX7+WcvyD/w iUJ4vDCrzQTHLjERSni0/B3/jM+W4UxhmLldCsh0VxWtY5eDwpe5N+OKOZ42WA9hz7VTACnu6 wvB4D5XFUdd8KswACg/s+/JM3HU8uuz2O7zu3qwRLQfi0PBIeLfOTulyDlnvbEq0y2qCCpnIr ARGlTK21A9K80FfIdPUWDQtYiVK3vCy5SLarf9wAgR8/LaLNwW98YrDzO9fod2KUfV/gH/+7u DORF81lIFTlbxcOhKyYCzKLTtXLB7lzufejWhhKpqgaIRRlAkRE8dGmJzJIvA7gZGvrUWiXzF P+ZlFRfxmbVzTB+9XejGBp+TH6tn2fjCfZZqQAFRv65zg4nq8iWYMJ76qf6chkvfCQtldoEPx Jper+8IFofFbd0bAb7L6Ais6orCCkGAVmB+E2tbn7C5oZdf30aakx3q606AQ6refanSO0KoGu CcvL+UGxwAx1fyRdEWU9cMetC23mWE07HtOT78jrAOOuuav32bT9u4ej2Rmh0aUsAUfgrw0b2 qFZ96W9SVet1UtbHFNPj752zfduXp0vc2cox3WsnVyRaKPiYFUhWPPER+6CBvcba++oEN4ytX WAlSDcthCq29cVnyNk4oYH8SsO2ewc3TELZ8Lv8RSMvz71+Ic3/e35wE/3UxKQ7lov2lsDqE/ nL8IFF8ReFSLrv9VGt7+4WWpAgdSb5hhdwIUDX86vdhBEhPNEv/OX9gatJdFmh+3RKHfJXNB9 w7ZwdMqlzwWOI66ywh1OvCDY1xBDHRvkOW4G2t/ANoW5fMCYi/w8tWcRw== Subject: Re: [Codel] [Starlink] Finite-Buffer M/G/1 Queues with Time and Space Priorities X-BeenThere: codel@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: CoDel AQM discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jul 2022 17:41:37 -0000 Hi Dave, > On Jul 29, 2022, at 19:08, Dave Taht wrote: >=20 > On Fri, Jul 29, 2022 at 8:29 AM Sebastian Moeller = wrote: >>=20 >> Hi Dave, >>=20 >> I thought it was accepted knowledge that inter-packet delays across = the internet are not reliable? Why are paced chirps immune from that = problem? Or asked differently after all noise filtering required for = robust and reliable operation over the existing internet, is this still = going to be noticeably faster than current slow-start? In spite of the = slow in the name doubling every RTT is IMHO already a pretty aggressive = growth function.... >=20 > I would like to be able to slow "slow start" based on interpacket = arrival times. But that still tries to reason based on shitty^W noisy data... = packet inter-arrival times are not good clean actionable data points, = Cleaning them up is possible, sure, but will cost time, hence my = question how much faster would a robust and reliable paced-chirping slow = start be over "exponential growths"? But I like your idea of "sith pacing", always in tight pairs to tickle = out ACKs faster. Side-note one way of sanity checking capacity estimates from packet = pairs is to also send packets of different sizes but that still leaves = the "it takes time to extract a decent estimate of the true capacity out = of the junk data" problem pure packet inter-arrival times analysis has = maybe reducing the time a bit. In the context of TCP it would probably = also require deep surgery to release smaller than MSS segments, let = alone what happens if carefully crafted packets hit GRO/GSO and friends. >=20 > Elsewhere cloudflare is enabling packet pacing at IW1000.... > = https://blog.cloudflare.com/when-the-window-is-not-fully-open-your-tcp-sta= ck-is-doing-more-than-you-think/ Holy beverage, that is a lot, at that point they might consider = increasing the gain factor instead of the offset from which to grow, so = instead doubling evert RTT starting from 1000, start from 20 but grow by = a factor of 4... I bet they are too chicken for that ;) (looking at = coarse plots it looks like it takes only a few rounds ~ 7 for = IW10_quadruple to leave IW1000_double in the dust)... > Increasingly it feels like there is going to be one network for the > data center, and another for everyone else. That happens if you hand data center guys the key to the = internet, I guess... >> Sidenote, you really think the second paper nailed PCs coffin shut? >=20 > No. It did outline the problems well. Knowing the sources, I have my reservations, they seems = typically strong on the potential upsides and a bit dismissive/cavalier = on the negative side-effects part. Also very little reception for paced = chirping so far, but that can just mean that things are a bit slower in = reaching production code and I got spoiled how fast codel/fq_codel made = it into the kernel (both probably considerably less challenging that = modifying the TCP state machine). > With a shift in perspective, instead of trying to smooth over the > difference in behaviors in the underlying network to fit into an 90s > isochronous packet switched network model, in the face of overwhelming > differences in real world (overwhemingly wireless) behaviors... Not convinced that is the case. I really believe the slow start = is too slow argument is showing unsustainable expectations, as I said = exponential window growths is pretty aggressive... the goal that every = flow independent of size should achieve speeds close to true capacity = simply seems to require an oracle scheduler on all senders injecting = into the same path. Color me sceptical... > ... It seems easier to improve the transports than fix the routers at > this point. Much as bbr models policers today perhaps it's best to try > and model more network types. +1; that seems preferable to all the approaches of putting more = smarts into the network itself... Thanks for the discussion ;) Regards Sebastian >=20 >>=20 >> On 29 July 2022 16:55:42 CEST, Dave Taht wrote: >>>=20 >>> To give credit where credit is due, "packet chirping" had been >>> explored before in the context >>> of the l4s early marking ecn effort: >>>=20 >>> = https://www.bobbriscoe.net/presents/1802paced-chirps/1802paced-chirps.pdf >>>=20 >>> It died here: = https://bobbriscoe.net/projects/netsvc_i-f/chirp_pfldnet10.pdf >>> For now. >>>=20 >>> On Thu, Jul 28, 2022 at 6:50 AM Dave Taht = wrote: >>>>=20 >>>>=20 >>>> thx for the comments everyone! >>>>=20 >>>> On Thu, Jul 28, 2022 at 3:16 AM Bj=C3=B8rn Ivar Teigen via Starlink >>>> wrote: >>>>>=20 >>>>>=20 >>>>> Very good point. Perhaps we can think of it as "at what point does = delay equal loss?". As you say, too much latency (causing reordering for = instance, or triggering an algorithm to smooth over missing data), is = functionally equivalent to loss, and therefore buffering beyond that = point is making things worse by delaying more traffic. The point at = which this kicks in varies a lot between applications though, so some = kind of classification might still make sense. >>>>>=20 >>>>> In a way, I think FQ_Codel does this classification implicitly by = treating sparse and non-sparse flows differently. >>>>=20 >>>>=20 >>>> the implicit flow analysis of fq_codel paper toke did is here: >>>> http://www.diva-portal.org/smash/get/diva2:1251687/FULLTEXT01.pdf >>>> It's a really nice feature!, and helps a lot when also applied to = wifi >>>> station scheduling. >>>>=20 >>>> I have sometimes thought that increasing to quantum to account for = two >>>> paced packets in a row (at high rates) was a good idea, >>>> other times having paced transports analyze the "beat frequency" of >>>> sending packets through fq_codel vs a vs the ack flow = characteristics >>>> (for example, filtering) might be useful. >>>>=20 >>>> Imagine that instead of sending packets on a fixed but increasing >>>> pacing schedule within an RTT thusly >>>>=20 >>>> PPPPPPPPPP # IW10 burst >>>> PP PP PP PP PP # often about 24 packets in what we >>>> think the RTT is >>>>=20 >>>> PP PP PP PP PP PP PP >>>>=20 >>>> PPPPPPPPPPPPPPPPPP >>>>=20 >>>> PPPPPPPPPPPPPPPPPPPPPPP stready buffering and ultimately a drop = (and >>>> yes this is inaccurate a model in a zillion ways, forgive me for >>>> purposes of extrapolation in ascii text) >>>>=20 >>>> If instead... >>>>=20 >>>> You broke up the pacing within an RTT on an actual curve, selecting >>>> some random segment out of PI as your actual starting point, say, = at >>>> 3.14596 here. >>>>=20 >>>> PPPPPP PPPPP PPP >>>> PPPPP PPPPPPPP >>>> PPPPPPPPP PPP PP >>>>=20 >>>> 3.14159265358979323846264338327950288419716939937510 >>>> 58209749445923078164062862089986280348253421170679 >>>> 82148086513282306647093844609550582231725359408128 >>>> 48111745028410270193852110555964462294895493038196 >>>> 44288109756659334461284756482337867831652712019091 >>>> 45648566923460348610454326648213393607260249141273 >>>> 72458700660631558817488152092096282925409171536436 >>>> 78925903600113305305488204665213841469519415116094 >>>> 33057270365759591953092186117381932611793105118548 >>>> 07446237996274956735188575272489122793818301194912 >>>> 98336733624406566430860213949463952247371907021798 >>>> 60943702770539217176293176752384674818467669405132 >>>> 00056812714526356082778577134275778960917363717872 >>>> 14684409012249534301465495853710507922796892589235 >>>> 42019956112129021960864034418159813629774771309960 >>>> 51870721134999999837297804995105973173281609631859 >>>> 50244594553469083026425223082533446850352619311881 >>>> 71010003137838752886587533208381420617177669147303 >>>> 59825349042875546873115956286388235378759375195778 >>>> 18577805321712268066130019278766111959092164201989 >>>>=20 >>>> what could you learn? >>>>=20 >>>>=20 >>>>> - Bj=C3=B8rn Ivar >>>>>=20 >>>>> On Thu, 28 Jul 2022 at 11:55, Sebastian Moeller = wrote: >>>>>>=20 >>>>>>=20 >>>>>> Hi all, >>>>>>=20 >>>>>>=20 >>>>>>> On Jul 28, 2022, at 11:26, Bj=C3=B8rn Ivar Teigen via Starlink = wrote: >>>>>>>=20 >>>>>>> Hi everyone, >>>>>>>=20 >>>>>>> Interesting paper Dave, I've got a few thoughts: >>>>>>>=20 >>>>>>> I like the split into delay-sensitive and loss-sensitive data. >>>>>>=20 >>>>>>=20 >>>>>> However often real data is slightly different (e.g. not nicely = either delay- or loss-sensitive)... e.g. for "real-time" games you have = both delay and loss sensitivity (similarly for VoIP), however both can = deal with occasional lost or delayed packets (if the delay is large = enough to say be re-ordered with the temporally next data packet (voice = sample in VoIP, server-tick update in games), that packet's data will = likely not be evaluated at all). And large scale bulk downloads are both = tolerant to delay and occasional loss. So if we think about a state = space spanned by a delay and a loss-sensitivity axis, I predict most = real traffic types will cluster somewhat around the diagonal (more or = less closely). >>>>>>=20 >>>>>> About the rest of the paper I have nothing to contribute, since I = did not spend the time to work though it. >>>>>>=20 >>>>>> Regards >>>>>> Sebastian >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>> Different applications can have different needs and this split = allows a queuing algorithm to take those differences into account. Not = the first time I've seen this kind of split, but the other one I've seen = used M/M/1/k queues (document here: = https://www.researchgate.net/publication/2452029_A_Queueing_Theory_Model_t= hat_Enables_Control_of_Loss_and_Delay_of_Traffic_at_a_Network_Switch) >>>>>>>=20 >>>>>>> That said, the performance metrics are derived from the embedded = Markov chain of the queuing system. This means the metrics are averages = over *all of time*, and thus there can be shorter periods (seconds, = minutes, hours) of much worse than average performance. Therefore the = conclusions of the paper should be taken with a grain of salt in my = opinion. >>>>>>>=20 >>>>>>> On Thu, 28 Jul 2022 at 10:45, Bless, Roland (TM) via Starlink = wrote: >>>>>>> Hi Dave, >>>>>>>=20 >>>>>>> IMHO the problem w.r.t the applicability of most models from >>>>>>> queueing theory is that they only work for load < 1, whereas >>>>>>> we are using the network with load values ~1 (i.e., around one) = due to >>>>>>> congestion control feedback loops that drive the bottleneck link >>>>>>> to saturation (unless you consider application limited traffic = sources). >>>>>>>=20 >>>>>>> To be fair there are queuing theory models that include packet = loss (which is the case for the paper Dave is asking about here), and = these can work perfectly well for load > 1. Agree about the CC feedback = loops affecting the results though. Even if the distributions are = general in the paper, they still assume samples are IID which is not = true for real networks. Feedback loops make real traffic = self-correlated, which makes the short periods of worse than average = performance worse and more frequent than IID models might suggest. >>>>>>>=20 >>>>>>> Regards, >>>>>>> Bj=C3=B8rn Ivar >>>>>>>=20 >>>>>>>=20 >>>>>>> Regards, >>>>>>> Roland >>>>>>>=20 >>>>>>> On 27.07.22 at 17:34 Dave Taht via Starlink wrote: >>>>>>>>=20 >>>>>>>> Occasionally I pass along a recent paper that I don't = understand in >>>>>>>> the hope that someone can enlighten me. >>>>>>>> This is one of those occasions, where I am trying to leverage = what I >>>>>>>> understand of existing FQ-codel behaviors against real traffic. >>>>>>>>=20 >>>>>>>> https://www.hindawi.com/journals/mpe/2022/4539940/ >>>>>>>>=20 >>>>>>>> Compared to the previous study on finite-buffer M/M/1 priority = queues >>>>>>>> with time and space priority, where service times are identical = and >>>>>>>> exponentially distributed for both types of traffic, in our = model we >>>>>>>> assume that service times are different and are generally = distributed >>>>>>>> for different types of traffic. As a result, our model is more >>>>>>>> suitable for the performance analysis of communication systems >>>>>>>> accommodating multiple types of traffic with different = service-time >>>>>>>> distributions. For the proposed queueing model, we derive the >>>>>>>> queue-length distributions, loss probabilities, and mean = waiting times >>>>>>>> of both types of traffic, as well as the push-out probability = of >>>>>>>> delay-sensitive traffic. >>>>>>>=20 >>>>>>> ________________________________ >>>>>>> Starlink mailing list >>>>>>> Starlink@lists.bufferbloat.net >>>>>>> https://lists.bufferbloat.net/listinfo/starlink >>>>>>>=20 >>>>>>>=20 >>>>>>> -- >>>>>>> Bj=C3=B8rn Ivar Teigen >>>>>>> Head of Research >>>>>>> +47 47335952 | bjorn@domos.no | www.domos.no >>>>>>> ________________________________ >>>>>>> Starlink mailing list >>>>>>> Starlink@lists.bufferbloat.net >>>>>>> https://lists.bufferbloat.net/listinfo/starlink >>>>>>=20 >>>>>>=20 >>>>>=20 >>>>>=20 >>>>> -- >>>>> Bj=C3=B8rn Ivar Teigen >>>>> Head of Research >>>>> +47 47335952 | bjorn@domos.no | www.domos.no >>>>> ________________________________ >>>>> Starlink mailing list >>>>> Starlink@lists.bufferbloat.net >>>>> https://lists.bufferbloat.net/listinfo/starlink >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>> -- >>>> FQ World Domination pending: = https://blog.cerowrt.org/post/state_of_fq_codel/ >>>> Dave T=C3=A4ht CEO, TekLibre, LLC >>>=20 >>>=20 >>>=20 >>>=20 >>> -- >>> FQ World Domination pending: = https://blog.cerowrt.org/post/state_of_fq_codel/ >>> Dave T=C3=A4ht CEO, TekLibre, LLC >>=20 >> -- >> Sent from my Android device with K-9 Mail. Please excuse my brevity. >=20 >=20 >=20 > --=20 > FQ World Domination pending: = https://blog.cerowrt.org/post/state_of_fq_codel/ > Dave T=C3=A4ht CEO, TekLibre, LLC