From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.15.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 615AF3B29D for ; Wed, 20 Apr 2022 08:54:18 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1650459255; bh=ka7V5mM5qfDOOZ08rzt8wFSCCM64Ji9/i0VhbgEGEN8=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=JJPvJn2M60UQUIFUvIOxcmZv6E8pbpFjhJmWLMRHa2aUJxpcuTAXPX65jcZVaz+qv V0jiGJ6KW2Mf37yWvO2uheDsh++UvoLX96sLB0A7WD57mmJOPpSjd2q6RSibvprCMz QKbRpBPZ+rzZso0xhthaRHMnjbx58SBiFD88WvyY= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from smtpclient.apple ([134.76.241.253]) by mail.gmx.net (mrgmx005 [212.227.17.190]) with ESMTPSA (Nemesis) id 1MXXuH-1nPPpY0kiV-00Z3JZ; Wed, 20 Apr 2022 14:54:15 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) From: Sebastian Moeller X-Priority: 3 (Normal) In-Reply-To: <1650400809.579413230@apps.rackspace.com> Date: Wed, 20 Apr 2022 14:54:13 +0200 Cc: Michael Welzl , ecn-sane@lists.bufferbloat.net Content-Transfer-Encoding: quoted-printable Message-Id: References: <4430DD9F-2556-4D38-8BE2-6609265319AF@ifi.uio.no> <1649778681.721621839@apps.rackspace.com> <0026CF35-46DF-4C0C-8FEE-B5309246C1B7@ifi.uio.no> <08F92DA0-1D59-4E58-A289-3D35103CF78B@gmx.de> <1649955272.49298319@apps.rackspace.com> <1650400809.579413230@apps.rackspace.com> To: "David P. Reed" X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Provags-ID: V03:K1:UugVijyA6G8qSr08StMlI4OfPcy8pjlF4R1Ho5K3Oxkayee1nJA KocrZ1ML7rQGi8uszB/f1L/PzHhxkW35zS/et5CFjSKcIRgZclYWLTB3zo0S7inpaDbrmuz jp7aKE/oBb/shNkXkEmWBlUi2KtapBHRyyOVy9wXm/bqFc1qukcJ8yxTREYPfVrwvjV+MqN AVaKVtOMtSN6LtumGermw== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:J+6GrZWJ9oY=:AIxRQO0GODvpOQ10ZWCbRZ W/FbtR4Qi4S67nniDir7krLait0Xcq5Xg2mmIs22duwzhlMelgTQRm6zT8MsSm+pJcEECdxVT CW++MckHdg8d55ihM0YNmE/tle4EM6Ad8UGIobP5Msm4kZVGezQObOqBmncDUWH0j9Rkse5yd Lc1JqjrMo8qwGrfjHpU0PPtNms/fx9cdWcCScdFRQ3W4EpKGGlTZKjmYOCPrRr8RiZc+HQ7o2 eg7/C6+nZatPZx6tqxwTePe/7GF+pTXZ/M/jUifG8SCfcLmnffkN7lTjJ3idr1rdW2cLnwT6F VK48I8TY0bN1JYjZRiBp+9Z5nlCbxdzn/F+E7GONLT4DRUTb9btRqHr9SevCqvinsrvQBU1z3 hmu4wj2L55L8fwv5HmmSZPg+wztH7d52k47MjGZMpTIVLUbrmxUBAdGtAPidIA+Nm9l5ztbK7 E0stCPLDG1YXsIkWxCeJ+8Y9lBsO+5AEmYJIqnzMSIwEaglIizWbiRZKl2t1QDHV0wobqs6S1 j6SKM+y+JmcyUENd2TxZSf//XJ13/t40F0+dF2xxrR+UwIgKhMKuhecUZ2QL74Kdv5aLBx0BE PGFKAxJLy7RfdJmVNgNHZ3bS2sIP63eF3owiIpAfFfJXmB+VS5Srpa1P3L4gWc475tH4K0By0 Uws9soRAnth8KiwgP3Ifb1Yicz6xbZZydXdiNy2rIk8zB3lgfxT+/4NIhfhereF7Pz9OZ1DjX c8xQyg3YhKcH4GVqISrLfcLu5WSv7GlbW49/hy+aKzbXPwHrIW+2quTA1/auNhJmmXKoHtIxb QMQQyzsvMWmVOX9DT5aXm0UePwbMxrqEeNAZkVEl0GpTZ57+MYD3QFoph3bNsQ0MsC8PdKV5r MmrITZC04FiDa1FYk67CvXvCFVWsx+Zh/NRGPef1FHiuAuohuohhwHXUPLgB9/EP4fHDHutOV XDyT983OnLlbfBE+RNoM6sgUyETUFxCU7UudJVpjnZk+cwcXwvPCYj91zj5YDJAHmlHAJJ+zp ry3ht1gLWrc3rQYI05IY2VjUT0wsPGLFZv4+leTbB7rZG4z48bbd3Ot3vzW98C66faRUhoFil Qu5YsvyqCgMGe4= Subject: Re: [Ecn-sane] rtt-fairness question X-BeenThere: ecn-sane@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of explicit congestion notification's impact on the Internet List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Apr 2022 12:54:18 -0000 Hi David, > On Apr 19, 2022, at 22:40, David P. Reed wrote: >=20 > Sebastian - all your thoughts here seem reasonable. > =20 > I would point out only two things: > =20 > 1) 100 ms. is a magic number for human perception. It's basically the = order of magnitude of humans' ability to respond to unpredictable events = outside the human. Yes, with this I fully agree, "order of magnitude", the actual = numerical value of 100 is for convenience and has no real significance = IMHO. Which I should have phrased better. Side-note such experiments = typically require the subject to create a measurable response, which = will take additional time to the initial event detection, but that still = fits within the 100ms order of magnitude much better than a hypothetical = 10ms. (for visual events at 10ms the frontal lobe will not even have the = information available that something changed, vision is amazingly slow*) > That's why it is magic. Now humans can actually perceive intervals = much, much shorter (depending on how we pay attention), but usually it = is by comparing two events' time ordering. We can even synchronize to = external, predictable events with finer resolution (as in Jazz improv or = just good chamber music playing). A century of careful scientific = research supports this, niot just one experiment. Quite a number of experiments however are misinterpreted (or = rather interpreted without the required nuance) on the internet (yes, I = know shocking ;) that the internet can be factually imprecise). > Which is why one should take it seriously as a useful target. (the = fact that one can achieve it across the planet with digital signalling = networks makes it a desirable goal for anything interactive between a = human and any entity, be it computer or human). If one can do better, of = course, that's great. I like that from my home computer I can get lots = of places in under 8 msec (15 msec RTT). > =20 > 2) given that a particular heavily utilized link might be shared for = paths where the light-speed-in-fiber round trip for active flows varies = by an order of magnitude, why does one try to make fair RTT (as opposed = to all other possible metrics on each flow) among flows. I think the measure that is equalized here is throughput per = flow, it is just that if done competently this will also alleviate the = inherent disadvantage that longer RTT flows have compared to shorter RTT = flows. But then again, other measures are possible as well assuming the = bottleneck can get at these easily.=20 > It doesn't make any sense to me why. Going back to human interaction = times, it makes sense to me that you might want to be unfair so that = most flows get faster than 200 ms. RTT, for example, penalizing those = who are really close to each other anyway. > If the RTT is already low because congestion has been controlled, you = can't make it lower. Basically, the ideal queue state is < 1 packet in = the bottleneck outbound queues, no matter what the RTT through that = queue is. Well, why RTT-fairness? My answer is similar as for why I like = FQ, because equitable sharing is the one strategy that without = information about the flows relative importance avoids the pitfall of = starving important flows that just happen to have a long RTT or a less = aggressive controller... So IMHO RTT fairness does not need to be = absolute but simply good enough to keep all flows at making decent = forward progress. The very moment someone comes in knowing more about = the different flows' importance, more optimal capacity sharing becomes = possible (like in Vint's example)... in a sense neither FQ nor the = "accidental" RTT-fairness it offers are likely optimal but they are IMHO = considerably less likely to be pessimal than any uninformed inequitable = sharing. Regards Sebastian *) Given that vision is essentially our long-range sense** that internal = latency typically is not an issue, since events/objects will often be = far enough away that detection can afford that extra time **) In space and time, just look at the stars ;)=20 > =20 > =20 > =20 > On Thursday, April 14, 2022 5:25pm, "Sebastian Moeller" = said: >=20 > > Just indulge me here for a few crazy ideas ;) > >=20 > > > On Apr 14, 2022, at 18:54, David P. Reed = wrote: > > > > > > Am I to assume, then, that routers need not pay any attention to = RTT to > > achieve RTT-fairness? > >=20 > > Part of RTT-bias seems caused by the simple fact that tight control = loops work > > better than sloppy ones ;) > >=20 > > There seem to be three ways to try to remedy that to some degree: > > 1) the daft one: > > define a reference RTT (larger than typically encountered) and have = all TCPs > > respond as if encountering that delay -> until the path RTT exceeds = that > > reference TCP things should be reasonably fair > >=20 > > 2) the flows communicate with the bottleneck honestly: > > if flows would communicate their RTT to the bottleneck the = bottleneck could > > partition its resources such that signaling (mark/drop) and puffer = size is > > bespoke per-flow. In theory that can work, but relies on either the = RTT > > information being non-gameably linked to the protocol's operation* = or everybody > > being fully veridical and honest > > *) think a protocol that will only work if the best estimate of the = RTT is > > communicated between the two sides continuously > >=20 > > 3) the router being verbose: > > If routers communicate the fill-state of their queue (global or = per-flow does not > > matter all that much) flows in theory can do a better job at not = putting way too > > much data in flight remedying the cost of drops/marks that affects = high RTT flows > > more than the shorter ones. (The router has little incentive to lie = here, if it > > wanted to punish a flow it would be easier to simply drop its = packets and be done > > with). > >=20 > >=20 > > IMHO 3, while theoretically the least effective of the three is the = only one that > > has a reasonable chance of being employed... or rather is already = deployed in the > > form of ECN (with mild effects). > >=20 > > > How does a server or client (at the endpoint) adjust RTT so that = it is fair? > >=20 > > See 1) above, but who in their right mind would actually implement = something like > > that (TCP Prague did that, but IMHO never in earnest but just to = "address" the > > L4S bullet point RTT-bias reduction). > >=20 > > > Now RTT, technically, is just the sum of the instantaneous queue = lengths in > > bytes along the path and the reverse path, plus a fixed wire-level = delay. And > > routers along any path do not have correlated queue sizes. > > > > > > It seems to me that RTT adjustment requires collective real-time = cooperation > > among all-or-most future users of that path. The path is partially = shared by many > > servers and many users, none of whom directly speak to each other. > > > > > > And routers have very limited memory compared to their = throughput-RTdelay > > product. So calculating the RTT using spin bits and UIDs for packets = seems a bit > > much to expect all routers to do. > >=20 > > If posed like this, I guess the better question is, what can/should = routers be > > expected to do here: either equitably share their queues or share = queue > > inequitably such that throughput is equitable. =46rom a pure router = point of the > > view the first seems "fairest", but as fq_codel and cake show, = within reason > > equitable capacity sharing is possible (so not perfectly and not for = every > > possible RTT spread). > >=20 > > > > > > So, what process measures the cross-interactions among all the = users of all > > the paths, and what control-loop (presumably stable and = TCP-compatible) actually > > converges to RTT fairness IRL. > >=20 > > Theoretically nothing, in reality on a home link FQ+competent AQM = goes a long way > > in that direction. > >=20 > >=20 > > > > > > Today, the basis of congestion control in the Internet is that = each router is > > a controller of all endpoint flows that share a link, and each = router is free to > > do whatever it takes to reduce its queue length to near zero as an = average on all > > timescales larger than about 1/10 of a second (a magic number that = is directly > > derived from measured human brain time resolution). > >=20 > > The typical applies, be suspicious of too round numbers.... 100ms is = in no way > > magic and also not "correct" it is however a decent description of = reaction times > > in a number of perceptul tasks that can be mis-interpreted as = showing things like > > the brain runs at 10Hz or similar... > >=20 > >=20 > > > > > > So, for any two machines separated by less than 1/10 of a = light-second in > > distance, the total queueing delay has to stabilize in about 1/10 of = a second. > > (I'm using a light-second in a fiber medium, not free-space, as the = speed of light > > in fiber is a lot slower than the speed of light on microwaves, as = Wall Street has > > recently started recoginizing and investing in). > > > > > > I don't see how RTT-fairness can be achieved by some set of bits = in the IP > > header. You can't shorten RTT below about 2/10 of a second in that = desired system > > state. You can only "lengthen" RTT by delaying packets in source or = endpoint > > buffers, because it's unreasonable to manage all the routers. > > > > > > And the endpoints that share a path can't talk to each other and = reach a > > decision in on the order of 2/10 of a second. > > > > > > So at the very highest level, what is RTT-fairness's objective = function > > optimizing, and how can it work? > > > > > > Can it be done without any change to routers? > >=20 > > Well the goal here seems to undo the RTT-dependence of throughput so = a router can > > equalize per flow throughput and thereby (from its own vantage = point) enforce RTT > > independence, within the amount of memory available. And that = already works today > > for all identifiable flows, but apparently at a computational cost = that larger > > routers do not want to pay. But you knew all that > >=20 > >=20 > > > > > > > > > > > > > > > On Tuesday, April 12, 2022 3:07pm, "Michael Welzl" = > > said: > > > > > > > > > > > > On Apr 12, 2022, at 8:52 PM, Sebastian Moeller > > wrote: > > > Question: is QUIC actually using the spin bit as an essential part = of the > > protocol? > > > The spec says it=E2=80=99s optional:=20 > > https://www.rfc-editor.org/rfc/rfc9000.html#name-latency-spin-bit > > > Otherwise endpoints might just game this if faking their RTT at a = router > > yields an advantage... > > > This was certainly discussed in the QUIC WG. Probably perceived as = an unclear > > incentive, but I didn=E2=80=99t really follow this. > > > Cheers, > > > Michael > > > > > > This is why pping's use of tcp timestamps is elegant, little = incentive for > > the endpoints to fudge.... > > > > > > Regards > > > Sebastian > > > > > > > > > On 12 April 2022 18:00:15 CEST, Michael Welzl > > wrote: > > > Hi, > > > Who or what are you objecting against? At least nothing that I = described > > does what you suggest. > > > BTW, just as a side point, for QUIC, routers can know the RTT = today - using > > the spin bit, which was designed for that specific purpose. > > > Cheers, > > > Michael > > > > > > > > > On Apr 12, 2022, at 5:51 PM, David P. Reed > > wrote: > > > I strongly object to congestion control *in the network* = attempting to > > measure RTT (which is an end-to-end comparative metric). Unless the = current RTT is > > passed in each packet a router cannot enforce fairness. Period. > > > > > > Today, by packet drops and fair marking, information is passed to = the sending > > nodes (eventually) about congestion. But the router can't know RTT = today. > > > > > > The result of *requiring* RTT fairness would be to put the random = bottleneck > > router (chosen because it is the slowest forwarder on a contended = path) become the > > endpoint controller. > > > > > > That's the opposite of an "end-to-end resource sharing protocol". > > > > > > Now, I'm not saying it is impossible - what I'm saying it is = asking all > > endpoints to register with an "Internet-wide" RTT real-time tracking = and control > > service. > > > > > > This would be the technical equivalent of an ITU central control = point. > > > > > > So, either someone will invent something I cannot imagine (a = distributed, > > rapid-convergence algortithm that rellects to *every potential user* = of a shared > > router along the current path the RTT's of ALL other users (and = potential users). > > > > > > IMHO, the wish for RTT fairness is like saying that the entire = solar system's > > gravitational pull should be equalized so that all planets and = asteroids have fair > > access to 1G gravity. > > > > > > > > > On Friday, April 8, 2022 2:03pm, "Michael Welzl" = > > said: > > > > > > Hi, > > > FWIW, we have done some analysis of fairness and convergence of = DCTCP in: > > > Peyman Teymoori, David Hayes, Michael Welzl, Stein Gjessing: = "Estimating an > > Additive Path Cost with Explicit Congestion Notification", IEEE = Transactions on > > Control of Network Systems, 8(2), pp. 859-871, June 2021. DOI > > 10.1109/TCNS.2021.3053179 > > > Technical report (longer version): > > > > > = https://folk.universitetetioslo.no/michawe/research/publications/NUM-ECN_r= eport_2019.pdf > > > and there=E2=80=99s also some in this paper, which first = introduced our LGC > > mechanism: > > > https://ieeexplore.ieee.org/document/7796757 > > > See the technical report on page 9, section D: a simple trick can = improve > > DCTCP=E2=80=99s fairness (if that=E2=80=99s really the mechanism to = stay with=E2=80=A6 > > I=E2=80=99m getting quite happy with the results we get with our LGC = scheme :-)=20 > > ) > > > > > > Cheers, > > > Michael > > > > > > On Apr 8, 2022, at 6:33 PM, Dave Taht wrote: > > > I have managed to drop most of my state regarding the state of = various > > > dctcp-like solutions. At one level it's good to have not been = keeping > > > up, washing my brain clean, as it were. For some reason or another = I > > > went back to the original paper last week, and have been pounding > > > through this one again: > > > > > > Analysis of DCTCP: Stability, Convergence, and Fairness > > > > > > "Instead, we propose subtracting =CE=B1/2 from the window size for = each > > marked ACK, > > > resulting in the following simple window update equation: > > > > > > One result of which I was most proud recently was of demonstrating > > > perfect rtt fairness in a range of 20ms to 260ms with fq_codel > > > https://forum.mikrotik.com/viewtopic.php?t=3D179307 )- and I'm = pretty > > > interested in 2-260ms, but haven't got around to it. > > > > > > Now, one early result from the sce vs l4s testing I recall was = severe > > > latecomer convergence problems - something like 40s to come into = flow > > > balance - but I can't remember what presentation, paper, or rtt = that > > > was from. ? > > > > > > Another one has been various claims towards some level of rtt > > > unfairness being ok, but not the actual ratio, nor (going up to = the > > > paper's proposal above) whether that method had been tried. > > > > > > My opinion has long been that any form of marking should look more > > > closely at the observed RTT than any fixed rate reduction method, = and > > > compensate the paced rate to suit. But that's presently just = reduced > > > to an opinion, not having kept up with progress on prague, = dctcp-sce, > > > or bbrv2. As one example of ignorance, are 2 packets still paced = back > > > to back? DRR++ + early marking seems to lead to one packet being > > > consistently unmarked and the other marked. > > > > > > -- > > > I tried to build a better future, a few times: > > > https://wayforward.archive.org/?site=3Dhttps%3A%2F%2Fwww.icei.org > > > > > > Dave T=C3=A4ht CEO, TekLibre, LLC > > > _______________________________________________ > > > Ecn-sane mailing list > > > Ecn-sane@lists.bufferbloat.net > > > https://lists.bufferbloat.net/listinfo/ecn-sane > > > > > > -- > > > Sent from my Android device with K-9 Mail. Please excuse my = brevity. > > > > >=20 > >