From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.15.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 160C83B29D for ; Mon, 11 Oct 2021 03:31:19 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1633937476; bh=g2Yj88lRjDRyoOW5TTbINrNmYVtb0xOjjYzJxhKzxQ8=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=gFIgmGF7myXQZHDseUeFJHynOWRVgw1U7Zb94BZQZtkgr032LU5+mLZD3P5GrwUzi ZVlIWQAac+iMm8DLK+zvGntM4bVSkiLKpbb9Zg4hp7wUD/tqO4bczegSxhDWYojTJv kvlRWY4blqfKv+SZxzyTOXKCTvXKJEYGNvljIJxE= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from [192.168.250.101] ([134.76.241.253]) by mail.gmx.net (mrgmx004 [212.227.17.190]) with ESMTPSA (Nemesis) id 1Mf07E-1nExIH3OLy-00gUnV; Mon, 11 Oct 2021 09:31:15 +0200 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.7\)) From: Sebastian Moeller In-Reply-To: Date: Mon, 11 Oct 2021 09:31:14 +0200 Cc: Jonathan Morton , Rpm Content-Transfer-Encoding: quoted-printable Message-Id: References: <45E18E9B-DD71-4F8A-92C2-AB5AA4439DC0@gmx.de> To: Christoph Paasch X-Mailer: Apple Mail (2.3608.120.23.2.7) X-Provags-ID: V03:K1:vgCeU6wM7MRwfSmXscQWIUTv0+/bzDe6S9xBDUx6fbpemqXaDoS se8JsXIjsEa/zpLq/LKKtqsTTETMFRyhmyNrdF4MMO24EJRN8Hm24GIiBhwk9sBrgMrw5BQ oqbSYQc1DLbgwBgS2JSKeMC67Gp7zgOlwJ9OqToIwGmXTRM/rz6dX5f4FAX9f0YkqcEx+MZ ZP1xswzNYQ6iOF6H1bSSg== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:I8EEN+chm60=:RoScGADhMN3YvM0WlJGIFP xBEV5pWqoG53erwuXyrORBv9i92mK2KPjCj4TKHIE+PdrlLqJN+pzmkkST5zOLG3kam5oSGPm /uPpOI9cyp+ww35YNwuSNbVdsxXfVBiHDKUNw7F28G+3/1LMp47e2ZMzrbDzQnP1dzSBA4jrE x9MjFLUjJpunO7AOsACl3mbauPBxfeg9UY2gWfM5NOwNC8C9F6Ju7YOjfouNKWUvmwnh2W8cE J97p6ldninNvjUaOLsdeLfyqTa2EHbWsEWJ5GGtWsnckVr8BSxLMy1BlJzuvTyutNQyHmPbGh Z/WIgbgROncguDbGGkWUosiHykfQoXXfDcKPTh0FCIlE1faKPIT1FEIQIqwbFjCoVBJV1EAAY RspEFCMI8Xv3zBXlradedbvYadD+yLqV9gmFOpADSKW5tvTq6KdBpfBCA8WZuHb1GCYnNIGOZ H9kuhmFmn44uYpWkxPeBnJBO2KVEZcKKdg+ucuDqaR8jzwpurvi8GP5/kD6uQsLKSA7STxSsK dM1EpGY92l8/SDUEeOTLd4AEoKMN/3Wax48F2NlwIvquRsG0DSe3H4rkcjWYsUlJe4k/D3J2E zDe/yfYlNIhYi/U+6QcyD/ixpz+1S9FBLRr3ed3bpRfEZOeJA07h1FlCKEwELv21OxB254No8 YWHh9dNbsenfBTrg4Hqw/rCmKc1htoqF4MVXbXQvYgIOaop4cMb31LQRCnK7uWmFKBd05lSUG FgsB8p6KVWOOJB0rAlXlaP0XeFpWLkMQZeLS0m6mk40VOGKrj6jLpS7ooEVQycwSWGTyWg5YT 0cWoHY9KXAkw7+kvWC1zAXfpddu4L7pR2yYMufdp4Wa5oNX28Efg1AaeH5VgFgvKeOAMc6HkQ HdXq8Ve4lWg3isRBGOKvXwe3G1a88bVQ7/bnEDQuD4gOvKwnrwQF4ln7Cr3ktWKc3xQqkht8G mltR+1ONM9EFE/y75HL4JVzv2CSEvfrAPAlu6BH2LvGuDLk5IhYXfTqgm6EAJ/QlMvliWUy2u x/KkT1W04WFrn1HRT/G9KQYPAEILPBfYeJuJRAmjR8Ryd86htS38WOYZFrv+XfoMnFtUWZOq4 mpyFss7VGzGMAM= Subject: Re: [Rpm] Alternate definitions of "working condition" - unnecessary? X-BeenThere: rpm@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: revolutions per minute - a new metric for measuring responsiveness List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Oct 2021 07:31:20 -0000 Hi Christoph, > On Oct 9, 2021, at 01:32, Christoph Paasch wrote: >=20 > On 10/07/21 - 12:30, Sebastian Moeller wrote: >> Hi Christoph, >>=20 >>> On Oct 7, 2021, at 02:11, Christoph Paasch via Rpm >>> wrote: >>>=20 >>> On 10/07/21 - 02:18, Jonathan Morton via Rpm wrote: >>>>> On 7 Oct, 2021, at 12:22 am, Dave Taht via Rpm >>>>> wrote: There are additional cases = where, >>>>> perhaps, the fq component works, and the aqm doesn't. >>>>=20 >>>> Such as Apple's version of FQ-Codel? The source code is public, so = we >>>> might as well talk about it. >>>=20 >>> Let's not just talk about it, but actually read it ;-) >>>=20 >>>> There are two deviations I know about in the AQM portion of that. >>>> First is that they do the marking and/or dropping at the tail of = the >>>> queue, not the head. Second is that the marking/dropping frequency = is >>>> fixed, instead of increasing during a continuous period of = congestion >>>> as real Codel does. >>>=20 >>> We don't drop/mark locally generated traffic (which is the use-case = we >>> care abhout). >>=20 >> In this discussion probably true, but I recall that one reason = why >> sch_fq_codel is a more versatile qdisc compared to sch_fq under >> Linux is that fq excels for locally generated traffic, while >> fq_codel is also working well for forwarded traffic. And I use >> "forwarding" here to encompass things like VMs running on a = host, >> where direct "back-pressure" will not work...=20 >=20 > Our main use-case is iOS. This is by far the most common case and thus = there > are no VMs or alike. All traffic is generated locally by our TCP > implementation. Ah, explains your priorities. My only iOS device is 11 years = old, and as far as I understand does not support fq_codel at all, so my = testing is restricted to my macbooks and was under mojave and catalina: macbook:~ user$ sudo netstat -I en4 -qq en4: [ sched: FQ_CODEL qlength: 0/128 ] [ pkts: 126518 bytes: 60151318 dropped pkts: 0 bytes: = 0 ] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D [ pri: CTL (0) srv_cl: 0x480190 quantum: 600 drr_max: = 8 ] [ queued pkts: 0 bytes: 0 ] [ dequeued pkts: 16969 bytes: 1144841 ] [ budget: 0 target qdelay: 5.00 msec update = interval:100.00 msec ] [ flow control: 0 feedback: 0 stalls: 0 failed: 0 ] [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] [ flows total: 0 new: 0 old: 0 ] [ throttle on: 0 off: 0 drop: 0 ] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D [ pri: VO (1) srv_cl: 0x400180 quantum: 600 drr_max: = 8 ] [ queued pkts: 0 bytes: 0 ] [ dequeued pkts: 0 bytes: 0 ] [ budget: 0 target qdelay: 5.00 msec update = interval:100.00 msec ] [ flow control: 0 feedback: 0 stalls: 0 failed: 0 ] [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] [ flows total: 0 new: 0 old: 0 ] [ throttle on: 0 off: 0 drop: 0 ] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D [ pri: VI (2) srv_cl: 0x380100 quantum: 3000 drr_max: = 6 ] [ queued pkts: 0 bytes: 0 ] [ dequeued pkts: 0 bytes: 0 ] [ budget: 0 target qdelay: 5.00 msec update = interval:100.00 msec ] [ flow control: 0 feedback: 0 stalls: 0 failed: 0 ] [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] [ flows total: 0 new: 0 old: 0 ] [ throttle on: 0 off: 0 drop: 0 ] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D [ pri: RV (3) srv_cl: 0x300110 quantum: 3000 drr_max: = 6 ] [ queued pkts: 0 bytes: 0 ] [ dequeued pkts: 0 bytes: 0 ] [ budget: 0 target qdelay: 5.00 msec update = interval:100.00 msec ] [ flow control: 0 feedback: 0 stalls: 0 failed: 0 ] [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] [ flows total: 0 new: 0 old: 0 ] [ throttle on: 0 off: 0 drop: 0 ] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D [ pri: AV (4) srv_cl: 0x280120 quantum: 3000 drr_max: = 6 ] [ queued pkts: 0 bytes: 0 ] [ dequeued pkts: 0 bytes: 0 ] [ budget: 0 target qdelay: 5.00 msec update = interval:100.00 msec ] [ flow control: 0 feedback: 0 stalls: 0 failed: 0 ] [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] [ flows total: 0 new: 0 old: 0 ] [ throttle on: 0 off: 0 drop: 0 ] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D [ pri: OAM (5) srv_cl: 0x200020 quantum: 1500 drr_max: = 4 ] [ queued pkts: 0 bytes: 0 ] [ dequeued pkts: 0 bytes: 0 ] [ budget: 0 target qdelay: 5.00 msec update = interval:100.00 msec ] [ flow control: 0 feedback: 0 stalls: 0 failed: 0 ] [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] [ flows total: 0 new: 0 old: 0 ] [ throttle on: 0 off: 0 drop: 0 ] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D [ pri: RD (6) srv_cl: 0x180010 quantum: 1500 drr_max: = 4 ] [ queued pkts: 0 bytes: 0 ] [ dequeued pkts: 78 bytes: 13943 ] [ budget: 0 target qdelay: 5.00 msec update = interval:100.00 msec ] [ flow control: 0 feedback: 0 stalls: 0 failed: 0 ] [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] [ flows total: 0 new: 0 old: 0 ] [ throttle on: 0 off: 0 drop: 0 ] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D [ pri: BE (7) srv_cl: 0x0 quantum: 1500 drr_max: 4 ] [ queued pkts: 0 bytes: 0 ] [ dequeued pkts: 98857 bytes: 56860512 ] [ budget: 0 target qdelay: 5.00 msec update = interval:100.00 msec ] [ flow control: 0 feedback: 0 stalls: 0 failed: 0 ] [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] [ flows total: 0 new: 0 old: 0 ] [ throttle on: 0 off: 0 drop: 0 ] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D [ pri: BK (8) srv_cl: 0x100080 quantum: 1500 drr_max: = 2 ] [ queued pkts: 0 bytes: 0 ] [ dequeued pkts: 10565 bytes: 2126520 ] [ budget: 0 target qdelay: 5.00 msec update = interval:100.00 msec ] [ flow control: 0 feedback: 0 stalls: 0 failed: 0 ] [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] [ flows total: 0 new: 0 old: 0 ] [ throttle on: 0 off: 0 drop: 0 ] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D [ pri: BK_SYS (9) srv_cl: 0x80090 quantum: 1500 drr_max: 2 ] [ queued pkts: 0 bytes: 0 ] [ dequeued pkts: 49 bytes: 5502 ] [ budget: 0 target qdelay: 5.00 msec update = interval:100.00 msec ] [ flow control: 0 feedback: 0 stalls: 0 failed: 0 ] [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] [ flows total: 0 new: 0 old: 0 ] [ throttle on: 0 off: 0 drop: 0 ] hms-beagle2:~ smoeller$=20 macbook:~ user$=20 >=20 >>>> We signal flow-control straight back to the TCP-stack at which = point the >>> queue is entirely drained before TCP starts transmitting again. >>>=20 >>> So, drop-frequency really doesn't matter because there is no drop. >>=20 >> But is it still codel/fq_codel if it does not implement head = drop >> (as described in >> https://datatracker.ietf.org/doc/html/rfc8290#section-4.2) and = if >> the control loop >> (https://datatracker.ietf.org/doc/html/rfc8289#section-3.3) is >> changed? (I am also wondering how reducing the default number of >> sub-queues from 1024 to 128 behaves on the background of the >> birthday paradox). >=20 > Not sure where the 128 comes from ? See above: [ sched: FQ_CODEL qlength: 0/128 ] but I might simply be misinterpreting the number here, because reading = this again instead of relaying on memory indicates that 128 is the = length of each individual queue and not the number of queues? Or is that = the length of the hardware queue sitting below fq_codel?=20 Anyway, any hints how to query/configure the fq_codel instance under = macos (I am not fluent in BSDeses)? > And birthday paradox does not apply. The magic happens in = inp_calc_flowhash() ;-) Thanks will need to spend time reading and understanding the = code obviously. Regards Sebastian >=20 >=20 > Cheers, > Christoph >=20 >=20 >> Best Regards Sebastian >>=20 >> P.S.: My definition of working conditions entails bidirectionally >> saturating traffic with responsive and (transiently) under-responsive >> flows. Something like a few long running TCP transfers to generate >> "base-load" and a higher number of TCP flows in IW or slow start to = add >> some spice to the whole. In the future, once QUIC actually takes = off*, >> adding more well defined/behaved UDP flows to the mix seems = reasonable. My >> off the cuff test for the effect of IW used to be to start a browser = and >> open a collection of (30-50) tabs getting a nice "thundering herd" of = TCP >> flows starting around the same time. But it seems browser makers got = too >> smart for me and will not do what I want any more but temporally = space the >> different sites in the tabs so that my nice thundering herd is less >> obnoxious (which IMHO is actually the right thing to do for actual = usage, >> but for testing it sucks). >>=20 >> *) Occasionally browsing the NANOG archives makes me wonder how the = move >> from HTTP/TCP to QUIC/UDP is going to play with operators propensity = to >> rate-limit UDP, but that is a different kettle of fish... >>=20 >>=20 >>>=20 >>>=20 >>> Christoph >>>=20 >>>>=20 >>>> I predict the consequences of these mistakes will differ according = to >>>> the type of traffic applied: >>>>=20 >>>> With TCP traffic over an Internet-scale path, the consequences are = not >>>> serious. The tail-drop means that the response at the end of >>>> slow-start will be slower, with a higher peak of intra-flow induced >>>> delay, and there is also a small but measurable risk of tail-loss >>>> causing a more serious application-level delay. These alone = *should* >>>> be enough to prompt a fix, if Apple are actually serious about >>>> improving application responsiveness. The fixed marking frequency, >>>> however, is probably invisible for this traffic. >>>>=20 >>>> With TCP traffic over a short-RTT path, the effects are more >>>> pronounced. The delay excursion at the end of slow-start will be >>>> larger in comparison to the baseline RTT, and when the latter is = short >>>> enough, the fixed congestion signalling frequency means there will = be >>>> some standing queue that real Codel would get rid of. This = standing >>>> queue will influence the TCP stack's RTT estimator and thus RTO = value, >>>> increasing the delay consequent to tail loss. >>>>=20 >>>> Similar effects to the above can be expected with other reliable = stream >>>> transports (SCTP, QUIC), though the details may differ. >>>>=20 >>>> The consequences with non-congestion-controlled traffic could be = much >>>> more serious. Real Codel will increase its drop frequency = continuously >>>> when faced with overload, eventually gaining control of the queue = depth >>>> as long as the load remains finite and reasonably constant. = Because >>>> Apple's AQM doesn't increase its drop frequency, the queue depth = for >>>> such a flow will increase continuously until either a = delay-sensitive >>>> rate selection mechanism is triggered at the sender, or the queue >>>> overflows and triggers burst losses. >>>>=20 >>>> So in the context of this discussion, is it worth generating a type = of >>>> load that specifically exercises this failure mode? If so, what = does >>>> it look like? >>>>=20 >>>> - Jonathan Morton _______________________________________________ = Rpm >>>> mailing list Rpm@lists.bufferbloat.net >>>> https://lists.bufferbloat.net/listinfo/rpm >>> _______________________________________________ Rpm mailing list >>> Rpm@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/rpm