From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 781803B29D for ; Thu, 14 Apr 2022 13:08:34 -0400 (EDT) Received: by mail-ed1-x534.google.com with SMTP id z99so7185484ede.5 for ; Thu, 14 Apr 2022 10:08:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=GstecQ6c3ZBeEEqCmjEUkqxu4rTW7vAs6nqG/4WsdXE=; b=egTFjJSBKOFsnHunktHGo+f9dgDWqSAkH8yb78dPfOtJD1zE/73rO1gYJh1ZptAeg8 JVPyYW1ILy7I4J4CPUvs0NozngcUTmK7VrcU7i/x2CNXvfmCxe3T1YNkWs0CDWy5oCEo sfMoSeV/LTty+8bNrNsE86X8oI6JKGxrAA2vGGTSuCex4iIGqF3dNcUxDswCnE5nU2xr Ty88bBB1FxliBAz30bjwKqaJLYqEJHKOImYiP7qkiUltz2gSDMJzh0nrnQqOUCCevPmf vF9vnimvcQDf446CoGAyWGByF0H0D1uHRkv+SF8VIIuDRPTy8x6FaXFwPuxZdkmMwCbL EfkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=GstecQ6c3ZBeEEqCmjEUkqxu4rTW7vAs6nqG/4WsdXE=; b=TFwtbBUuGhMwVGbIL5erhsnLjpP8tD3ybP22of/ImjIGU/KzsrgNOFYCxDqle/F7Ma XmRx5WRbv5nylBg+d+oFz2oytMC0Xq6+s7+UpY92M7vIXCYgFEF6hBrZgF/kQTnFso1t xjTxRN6b7ANrgRxAXcuj7TAIfzi0pSoktRC/56Mbvv7Xhl+FhriSvtf38GgwZwKryqE3 eQuXeePlU3wKwInDp/qcxTMWmypUbfpdS0cnPmB9eAi5JSLRDzoRwC+CFnKFdZ+sHxUy BnECPFdeWzI9un0qvgz+X+Hd3O/2guBTsVBLS6M4DsXONH/bQmElob/HDiGfpYO376MY Bwjw== X-Gm-Message-State: AOAM530hGAvzDXRI4p14U8EK+x4fPTkNbkP9tZYd2g2rAxh/8/qxYyyM gds7BKQxYqo5V01yqZo1/G/2vY4bZ/KjEP2vt3TpdSG+3IiGog== X-Google-Smtp-Source: ABdhPJy9Go2DktwTJwE0JW3ZGn0dnaDsTKtN/nu1BwW8/bMSMG9fmLHxFHohg5fOmvGjK8OgEmsP9qju9DguYEIFJYE= X-Received: by 2002:aa7:cac5:0:b0:41c:c1fb:f5cc with SMTP id l5-20020aa7cac5000000b0041cc1fbf5ccmr3995645edt.219.1649956113137; Thu, 14 Apr 2022 10:08:33 -0700 (PDT) MIME-Version: 1.0 References: <4430DD9F-2556-4D38-8BE2-6609265319AF@ifi.uio.no> <1649778681.721621839@apps.rackspace.com> <0026CF35-46DF-4C0C-8FEE-B5309246C1B7@ifi.uio.no> <08F92DA0-1D59-4E58-A289-3D35103CF78B@gmx.de> <1649955272.49298319@apps.rackspace.com> In-Reply-To: <1649955272.49298319@apps.rackspace.com> From: Dave Taht Date: Thu, 14 Apr 2022 10:08:21 -0700 Message-ID: To: "David P. Reed" Cc: Michael Welzl , ECN-Sane Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Ecn-sane] rtt-fairness question X-BeenThere: ecn-sane@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of explicit congestion notification's impact on the Internet List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Apr 2022 17:08:34 -0000 I guess to try and clarify, the objective function here is fq_codel on the bottleneck link, which, starting with exceeding the 5ms target threshold consistently, hammers down on drops starting after 100ms on an invsqrt interval until it finds the right drop rate for the rtt to fill the pipe and not the queue. It is NOT aware of the actual RTT, there are no bits in the header inspected, the flow IS isolated by the 5 tuple. Very long thread with a ton of graphs (if you login) here: https://forum.mikrotik.com/viewtopic.php?t=3D179307 On Thu, Apr 14, 2022 at 9:54 AM David P. Reed wrote: > > Am I to assume, then, that routers need not pay any attention to RTT to a= chieve RTT-fairness? > > > > How does a server or client (at the endpoint) adjust RTT so that it is fa= ir? > > > > Now RTT, technically, is just the sum of the instantaneous queue lengths = in bytes along the path and the reverse path, plus a fixed wire-level delay= . And routers along any path do not have correlated queue sizes. > > > > It seems to me that RTT adjustment requires collective real-time cooperat= ion among all-or-most future users of that path. The path is partially sha= red by many servers and many users, none of whom directly speak to each oth= er. > > > > And routers have very limited memory compared to their throughput-RTdelay= product. So calculating the RTT using spin bits and UIDs for packets seems= a bit much to expect all routers to do. I don't have much of an opinion of the spin bit. I was originally describing achieving near-perfect RTT fairness using cake in the real world on the string of tests on the link above, in a range of 20ms to 260ms. So we are having two different conversations here. > > > So, what process measures the cross-interactions among all the users of a= ll the paths, and what control-loop (presumably stable and TCP-compatible) = actually converges to RTT fairness IRL. I am temped to fork this in light of the sqm-autorate work, but let's stick to two confusing conflations at a time. I was talking about fq_codels interactions with tcp's control loops. Michael, quic. > > > Today, the basis of congestion control in the Internet is that each route= r is a controller of all endpoint flows that share a link, and each router = is free to do whatever it takes to reduce its queue length to near zero as = an average on all timescales larger than about 1/10 of a second (a magic nu= mber that is directly derived from measured human brain time resolution). Actually I thought the 100ms came from the coast-to-coast of the USA in the 70s, much like ATM had cell sizes of 48 bytes (USA) vs 32 bytes (europe) that were optimal for typical distances here, in order to fill a BDP for a single reno-style flow. As this starting limit is disconcerting for videoconfernecing and voice traffic I've often thought that starting to actively manage a queue at 60ms would be better more generally, and in fact a cap on fifos of that much due to the stanford thinking of sqrt flows bdp. I don't care all that much about single flow utilization, but about interactive voice/video conversations. > > > So, for any two machines separated by less than 1/10 of a light-second in= distance, the total queueing delay has to stabilize in about 1/10 of a sec= ond. (I'm using a light-second in a fiber medium, not free-space, as the sp= eed of light in fiber is a lot slower than the speed of light on microwaves= , as Wall Street has recently started recoginizing and investing in). > > > > I don't see how RTT-fairness can be achieved by some set of bits in the I= P header. You can't shorten RTT below about 2/10 of a second in that desire= d system state. You can only "lengthen" RTT by delaying packets in source o= r endpoint buffers, because it's unreasonable to manage all the routers. > > > > And the endpoints that share a path can't talk to each other and reach a = decision in on the order of 2/10 of a second. > > > > So at the very highest level, what is RTT-fairness's objective function o= ptimizing, and how can it work? > > > > Can it be done without any change to routers? > > > > > > > > > > On Tuesday, April 12, 2022 3:07pm, "Michael Welzl" s= aid: > > > > On Apr 12, 2022, at 8:52 PM, Sebastian Moeller wrote: > Question: is QUIC actually using the spin bit as an essential part of the= protocol? > > The spec says it=E2=80=99s optional: https://www.rfc-editor.org/rfc/rfc9= 000.html#name-latency-spin-bit > > Otherwise endpoints might just game this if faking their RTT at a router = yields an advantage... > > This was certainly discussed in the QUIC WG. Probably perceived as an unc= lear incentive, but I didn=E2=80=99t really follow this. > Cheers, > Michael > > This is why pping's use of tcp timestamps is elegant, little incentive fo= r the endpoints to fudge.... > > Regards > Sebastian > > > On 12 April 2022 18:00:15 CEST, Michael Welzl wrote: >> >> Hi, >> Who or what are you objecting against? At least nothing that I describ= ed does what you suggest. >> BTW, just as a side point, for QUIC, routers can know the RTT today - us= ing the spin bit, which was designed for that specific purpose. >> Cheers, >> Michael >> >> >> On Apr 12, 2022, at 5:51 PM, David P. Reed wrote: >> I strongly object to congestion control *in the network* attempting to m= easure RTT (which is an end-to-end comparative metric). Unless the current = RTT is passed in each packet a router cannot enforce fairness. Period. >> >> >> >> Today, by packet drops and fair marking, information is passed to the se= nding nodes (eventually) about congestion. But the router can't know RTT to= day. >> >> >> >> The result of *requiring* RTT fairness would be to put the random bottle= neck router (chosen because it is the slowest forwarder on a contended path= ) become the endpoint controller. >> >> >> >> That's the opposite of an "end-to-end resource sharing protocol". >> >> >> >> Now, I'm not saying it is impossible - what I'm saying it is asking all = endpoints to register with an "Internet-wide" RTT real-time tracking and co= ntrol service. >> >> >> >> This would be the technical equivalent of an ITU central control point. >> >> >> >> So, either someone will invent something I cannot imagine (a distributed= , rapid-convergence algortithm that rellects to *every potential user* of a= shared router along the current path the RTT's of ALL other users (and pot= ential users). >> >> >> >> IMHO, the wish for RTT fairness is like saying that the entire solar sys= tem's gravitational pull should be equalized so that all planets and astero= ids have fair access to 1G gravity. >> >> >> >> >> >> On Friday, April 8, 2022 2:03pm, "Michael Welzl" sa= id: >> >> Hi, >> FWIW, we have done some analysis of fairness and convergence of DCTCP in= : >> Peyman Teymoori, David Hayes, Michael Welzl, Stein Gjessing: "Estimating= an Additive Path Cost with Explicit Congestion Notification", IEEE Transac= tions on Control of Network Systems, 8(2), pp. 859-871, June 2021. DOI 10.1= 109/TCNS.2021.3053179 >> Technical report (longer version): >> https://folk.universitetetioslo.no/michawe/research/publications/NUM-ECN= _report_2019.pdf >> and there=E2=80=99s also some in this paper, which first introduced our = LGC mechanism: >> https://ieeexplore.ieee.org/document/7796757 >> See the technical report on page 9, section D: a simple trick can improv= e DCTCP=E2=80=99s fairness (if that=E2=80=99s really the mechanism to stay= with=E2=80=A6 I=E2=80=99m getting quite happy with the results we get wi= th our LGC scheme :-) ) >> >> Cheers, >> Michael >> >> On Apr 8, 2022, at 6:33 PM, Dave Taht wrote: >> I have managed to drop most of my state regarding the state of various >> dctcp-like solutions. At one level it's good to have not been keeping >> up, washing my brain clean, as it were. For some reason or another I >> went back to the original paper last week, and have been pounding >> through this one again: >> >> Analysis of DCTCP: Stability, Convergence, and Fairness >> >> "Instead, we propose subtracting =CE=B1/2 from the window size for each = marked ACK, >> resulting in the following simple window update equation: >> >> One result of which I was most proud recently was of demonstrating >> perfect rtt fairness in a range of 20ms to 260ms with fq_codel >> https://forum.mikrotik.com/viewtopic.php?t=3D179307 )- and I'm pretty >> interested in 2-260ms, but haven't got around to it. >> >> Now, one early result from the sce vs l4s testing I recall was severe >> latecomer convergence problems - something like 40s to come into flow >> balance - but I can't remember what presentation, paper, or rtt that >> was from. ? >> >> Another one has been various claims towards some level of rtt >> unfairness being ok, but not the actual ratio, nor (going up to the >> paper's proposal above) whether that method had been tried. >> >> My opinion has long been that any form of marking should look more >> closely at the observed RTT than any fixed rate reduction method, and >> compensate the paced rate to suit. But that's presently just reduced >> to an opinion, not having kept up with progress on prague, dctcp-sce, >> or bbrv2. As one example of ignorance, are 2 packets still paced back >> to back? DRR++ + early marking seems to lead to one packet being >> consistently unmarked and the other marked. >> >> -- >> I tried to build a better future, a few times: >> https://wayforward.archive.org/?site=3Dhttps%3A%2F%2Fwww.icei.org >> >> Dave T=C3=A4ht CEO, TekLibre, LLC >> _______________________________________________ >> Ecn-sane mailing list >> Ecn-sane@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/ecn-sane > > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. > > _______________________________________________ > Ecn-sane mailing list > Ecn-sane@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/ecn-sane --=20 I tried to build a better future, a few times: https://wayforward.archive.org/?site=3Dhttps%3A%2F%2Fwww.icei.org Dave T=C3=A4ht CEO, TekLibre, LLC