From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.15.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id BF4523B29E for ; Tue, 12 Oct 2021 06:23:35 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1634034201; bh=EBdsIVT/mqD2NFf5pzf9aIzbDUs2IzGqd/RaXuMuQ1Q=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=g/jUF7l+1c4HtT9AedfgBwf7fKfkPm+v9GIOXUzCTLrVvpnMgnx4oBHPGh7Gq8dOs pgmgiQxx9ZFkeVVep6Dhb4iw8E+3d8TwBxqxMwMwoPc49pVjeA0MZvnFPq8/clUoAw jvFGKhN1RvLl/GVtEQNNhX3SIsp+/6+aqhtiaISE= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from [192.168.250.101] ([134.76.241.253]) by mail.gmx.net (mrgmx005 [212.227.17.190]) with ESMTPSA (Nemesis) id 1M26rD-1mcAsa2exl-002Ylv; Tue, 12 Oct 2021 12:23:21 +0200 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.7\)) From: Sebastian Moeller In-Reply-To: Date: Tue, 12 Oct 2021 12:23:20 +0200 Cc: Jonathan Morton , Rpm Content-Transfer-Encoding: quoted-printable Message-Id: <9478EC7E-7DEF-4AC9-B8EE-574818FA5EF0@gmx.de> References: <45E18E9B-DD71-4F8A-92C2-AB5AA4439DC0@gmx.de> To: Christoph Paasch X-Mailer: Apple Mail (2.3608.120.23.2.7) X-Provags-ID: V03:K1:jSK33eYCnVLrUkm09cdaTYVW4FemH6yH0kNAefWF7PdQXRvp8cw zQzpQsEebZRucAgzYaU+9xjztXvOk3tV4uBliETG4tRc4ZRnclSJMzdZUkNG2YIORFudK+J Qib/sm/qK7XX2TGEy8E9YVDWS+o15yl9qa0fyCDuulgnOB8NkYAFGsUqjR+mdCVYrMcRIBn 2PnY512DMdPQ96GojRzyg== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:7N8bZtQQye0=:0mRwwEZS9KjebCnfI/BXEc c9n+LQrTKcnkYO0oFbL01RjZ+JFKXhWCQg9MvKBKQ5ni15/Zos7X9JELk4LRsSiAWgNudxWGI Dthcui3eQjilVual+CNSHzh8rawNT7625EXFhqoDC209M7jDhZ9AavXMAqcz3FZEcvB0KBjh7 xefQ0vu0SQte/Gh4DfpUndLa5oUpvimsdNpuwHsqAY4uz6u4hgDNYtFd5AW8ZLWA9BqSrRyeb vxSmU7WzJqOUPB56dl0Ch2Ygt0huhJq4k+dq7UWJwwShZO0lPVl1LxEEqGr1YY4yo7Uagn2YJ jnw6numWxHiTNoGDHk/l8K7CsuUqaumSH0+0DOEuoggh3p/LvUWbeMRP7p4odn/mYxySnOUpP tzxG4hwR5sJkOxKHMpogQJT87yBe0bpmAeyGbtN+WIc1zHQNJ2DZ8rXQOimIvbKN9wXBMrUXl mignzngRa9J1rVsljRa/N5F6Jdn6cwZSxVaGH7gPrKHsXs7OmsLmCJcO9NUjF9iB6AgMK3I7J T6IF4mm9zS5DQywbu6hLihwjQHC5o64vWi3PVpluQ3CxlWU4R6vJwBh+tUhoYF0EvmRX+PoEP zF3K1QiARansATREMISWGPfeFfk5McDvtRJthQkcc05cH7llXInHl+ACqjMwWQKf45q1nOtrM sGk1XuM7LGYOy6xMcq8CqD+RrS+JJQb+LsTSOhYP6UKojsqZxvUoRnilNqzGZJPZlCJy8M6XC sQ5i6G8dnPCMLZNXRQlQWtK2CtX6bgZIeGyGufReLkoWOasCytw1gk07CNdPD65cSAkZamQxT iJRX0Bo447VexwW0chNbYR+0SJ83iruoyHD0Vy1LjzUsNc60JlyfRF3cgKvJO6FEVEew0GpXZ EL3NG9NQnbkEiZMyD3omS5IHnIUzDjaXm1N3umlIDuzDc/70GMKoE2HEbRIVg/soFELAL098P qeYy8ueRqsdEwLqV7jopsdu/azDvurXeORj/DRSeMYI+rnbsUdRAHM9M7XyQLv2Ko+d1bY4lD E58PSZObF/YGvWGtgVE50Dosos+K4w+ViQXPNWre01jTblEYCro5u0xX0EHlpPsoZ5ot2HyGu NhGFgWNPL0Ybwc= Subject: Re: [Rpm] Alternate definitions of "working condition" - unnecessary? X-BeenThere: rpm@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: revolutions per minute - a new metric for measuring responsiveness List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Oct 2021 10:23:36 -0000 Hi Christoph, > On Oct 11, 2021, at 19:34, Christoph Paasch wrote: >=20 > On 10/11/21 - 09:31, Sebastian Moeller wrote: >>> On Oct 9, 2021, at 01:32, Christoph Paasch = wrote: >>>=20 >>> On 10/07/21 - 12:30, Sebastian Moeller wrote: >>>> Hi Christoph, >>>>=20 >>>>> On Oct 7, 2021, at 02:11, Christoph Paasch via Rpm >>>>> wrote: >>>>>=20 >>>>> On 10/07/21 - 02:18, Jonathan Morton via Rpm wrote: >>>>>>> On 7 Oct, 2021, at 12:22 am, Dave Taht via Rpm >>>>>>> wrote: There are additional cases = where, >>>>>>> perhaps, the fq component works, and the aqm doesn't. >>>>>>=20 >>>>>> Such as Apple's version of FQ-Codel? The source code is public, = so we >>>>>> might as well talk about it. >>>>>=20 >>>>> Let's not just talk about it, but actually read it ;-) >>>>>=20 >>>>>> There are two deviations I know about in the AQM portion of that. >>>>>> First is that they do the marking and/or dropping at the tail of = the >>>>>> queue, not the head. Second is that the marking/dropping = frequency is >>>>>> fixed, instead of increasing during a continuous period of = congestion >>>>>> as real Codel does. >>>>>=20 >>>>> We don't drop/mark locally generated traffic (which is the = use-case we >>>>> care abhout). >>>>=20 >>>> In this discussion probably true, but I recall that one reason = why >>>> sch_fq_codel is a more versatile qdisc compared to sch_fq under >>>> Linux is that fq excels for locally generated traffic, while >>>> fq_codel is also working well for forwarded traffic. And I use >>>> "forwarding" here to encompass things like VMs running on a = host, >>>> where direct "back-pressure" will not work...=20 >>>=20 >>> Our main use-case is iOS. This is by far the most common case and = thus there >>> are no VMs or alike. All traffic is generated locally by our TCP >>> implementation. >>=20 >> Ah, explains your priorities. >=20 > Yes - we are aware of these issues for forwarding or VM-generated = traffic. >=20 > But the amount of traffic is so much lower compared to the other = use-cases > that it is not even a drop in the bucket. [SM] One last try ;), how does this affect using an iOS device = mobile AP to get other devices online? >=20 >> My only iOS device is 11 years old, and as far as I understand does = not support fq_codel at all, so my testing is restricted to my macbooks = and was under mojave and catalina: >>=20 >> macbook:~ user$ sudo netstat -I en4 -qq >> en4: >> [ sched: FQ_CODEL qlength: 0/128 ] >> [ pkts: 126518 bytes: 60151318 dropped pkts: 0 = bytes: 0 ] >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D >> [ pri: CTL (0) srv_cl: 0x480190 quantum: 600 drr_max: = 8 ] >> [ queued pkts: 0 bytes: 0 ] >> [ dequeued pkts: 16969 bytes: 1144841 ] >> [ budget: 0 target qdelay: 5.00 msec update = interval:100.00 msec ] >> [ flow control: 0 feedback: 0 stalls: 0 failed: = 0 ] >> [ drop overflow: 0 early: 0 memfail: 0 = duprexmt:0 ] >> [ flows total: 0 new: 0 old: 0 ] >> [ throttle on: 0 off: 0 drop: 0 ] >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D >> [ pri: VO (1) srv_cl: 0x400180 quantum: 600 drr_max: = 8 ] >> [ queued pkts: 0 bytes: 0 ] >> [ dequeued pkts: 0 bytes: 0 ] >> [ budget: 0 target qdelay: 5.00 msec update = interval:100.00 msec ] >> [ flow control: 0 feedback: 0 stalls: 0 failed: = 0 ] >> [ drop overflow: 0 early: 0 memfail: 0 = duprexmt:0 ] >> [ flows total: 0 new: 0 old: 0 ] >> [ throttle on: 0 off: 0 drop: 0 ] >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D >> [ pri: VI (2) srv_cl: 0x380100 quantum: 3000 drr_max: = 6 ] >> [ queued pkts: 0 bytes: 0 ] >> [ dequeued pkts: 0 bytes: 0 ] >> [ budget: 0 target qdelay: 5.00 msec update = interval:100.00 msec ] >> [ flow control: 0 feedback: 0 stalls: 0 failed: = 0 ] >> [ drop overflow: 0 early: 0 memfail: 0 = duprexmt:0 ] >> [ flows total: 0 new: 0 old: 0 ] >> [ throttle on: 0 off: 0 drop: 0 ] >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D >> [ pri: RV (3) srv_cl: 0x300110 quantum: 3000 drr_max: = 6 ] >> [ queued pkts: 0 bytes: 0 ] >> [ dequeued pkts: 0 bytes: 0 ] >> [ budget: 0 target qdelay: 5.00 msec update = interval:100.00 msec ] >> [ flow control: 0 feedback: 0 stalls: 0 failed: = 0 ] >> [ drop overflow: 0 early: 0 memfail: 0 = duprexmt:0 ] >> [ flows total: 0 new: 0 old: 0 ] >> [ throttle on: 0 off: 0 drop: 0 ] >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D >> [ pri: AV (4) srv_cl: 0x280120 quantum: 3000 drr_max: = 6 ] >> [ queued pkts: 0 bytes: 0 ] >> [ dequeued pkts: 0 bytes: 0 ] >> [ budget: 0 target qdelay: 5.00 msec update = interval:100.00 msec ] >> [ flow control: 0 feedback: 0 stalls: 0 failed: = 0 ] >> [ drop overflow: 0 early: 0 memfail: 0 = duprexmt:0 ] >> [ flows total: 0 new: 0 old: 0 ] >> [ throttle on: 0 off: 0 drop: 0 ] >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D >> [ pri: OAM (5) srv_cl: 0x200020 quantum: 1500 drr_max: = 4 ] >> [ queued pkts: 0 bytes: 0 ] >> [ dequeued pkts: 0 bytes: 0 ] >> [ budget: 0 target qdelay: 5.00 msec update = interval:100.00 msec ] >> [ flow control: 0 feedback: 0 stalls: 0 failed: = 0 ] >> [ drop overflow: 0 early: 0 memfail: 0 = duprexmt:0 ] >> [ flows total: 0 new: 0 old: 0 ] >> [ throttle on: 0 off: 0 drop: 0 ] >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D >> [ pri: RD (6) srv_cl: 0x180010 quantum: 1500 drr_max: = 4 ] >> [ queued pkts: 0 bytes: 0 ] >> [ dequeued pkts: 78 bytes: 13943 ] >> [ budget: 0 target qdelay: 5.00 msec update = interval:100.00 msec ] >> [ flow control: 0 feedback: 0 stalls: 0 failed: = 0 ] >> [ drop overflow: 0 early: 0 memfail: 0 = duprexmt:0 ] >> [ flows total: 0 new: 0 old: 0 ] >> [ throttle on: 0 off: 0 drop: 0 ] >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D >> [ pri: BE (7) srv_cl: 0x0 quantum: 1500 drr_max: 4 ] >> [ queued pkts: 0 bytes: 0 ] >> [ dequeued pkts: 98857 bytes: 56860512 ] >> [ budget: 0 target qdelay: 5.00 msec update = interval:100.00 msec ] >> [ flow control: 0 feedback: 0 stalls: 0 failed: = 0 ] >> [ drop overflow: 0 early: 0 memfail: 0 = duprexmt:0 ] >> [ flows total: 0 new: 0 old: 0 ] >> [ throttle on: 0 off: 0 drop: 0 ] >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D >> [ pri: BK (8) srv_cl: 0x100080 quantum: 1500 drr_max: = 2 ] >> [ queued pkts: 0 bytes: 0 ] >> [ dequeued pkts: 10565 bytes: 2126520 ] >> [ budget: 0 target qdelay: 5.00 msec update = interval:100.00 msec ] >> [ flow control: 0 feedback: 0 stalls: 0 failed: = 0 ] >> [ drop overflow: 0 early: 0 memfail: 0 = duprexmt:0 ] >> [ flows total: 0 new: 0 old: 0 ] >> [ throttle on: 0 off: 0 drop: 0 ] >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D >> [ pri: BK_SYS (9) srv_cl: 0x80090 quantum: 1500 drr_max: = 2 ] >> [ queued pkts: 0 bytes: 0 ] >> [ dequeued pkts: 49 bytes: 5502 ] >> [ budget: 0 target qdelay: 5.00 msec update = interval:100.00 msec ] >> [ flow control: 0 feedback: 0 stalls: 0 failed: = 0 ] >> [ drop overflow: 0 early: 0 memfail: 0 = duprexmt:0 ] >> [ flows total: 0 new: 0 old: 0 ] >> [ throttle on: 0 off: 0 drop: 0 ] >> hms-beagle2:~ smoeller$=20 >>=20 >> macbook:~ user$=20 >>=20 >>=20 >>=20 >>=20 >>=20 >>>=20 >>>>>> We signal flow-control straight back to the TCP-stack at which = point the >>>>> queue is entirely drained before TCP starts transmitting again. >>>>>=20 >>>>> So, drop-frequency really doesn't matter because there is no drop. >>>>=20 >>>> But is it still codel/fq_codel if it does not implement head = drop >>>> (as described in >>>> https://datatracker.ietf.org/doc/html/rfc8290#section-4.2) and = if >>>> the control loop >>>> (https://datatracker.ietf.org/doc/html/rfc8289#section-3.3) is >>>> changed? (I am also wondering how reducing the default number of >>>> sub-queues from 1024 to 128 behaves on the background of the >>>> birthday paradox). >>>=20 >>> Not sure where the 128 comes from ? >>=20 >> See above: >> [ sched: FQ_CODEL qlength: 0/128 ] >> but I might simply be misinterpreting the number here, because = reading this again instead of relaying on memory indicates that 128 is = the length of each individual queue and not the number of queues? Or is = that the length of the hardware queue sitting below fq_codel?=20 >=20 > This 128 can be safely ignored. It has no meaning :) [SM] Thanks! and excuse my confusion. >=20 >> Anyway, any hints how to query/configure the fq_codel instance under = macos (I am not fluent in BSDeses)? >=20 > For querying, netstat -qq as you already saw. >=20 > There is not much to configure... Just two sysctls: >=20 > net.classq.target_qdelay: 0 > net.classq.update_interval: 0 >=20 > 0 means that the system's default is in use. [SM] Okay that is a refreshing uncomplicated UI, a bit hidden, = but since the default 5/100 work well, this seems a decent choice. Thanks again Regards Sebastian >=20 >=20 > Christoph >=20 >>> And birthday paradox does not apply. The magic happens in = inp_calc_flowhash() ;-) >>=20 >> Thanks will need to spend time reading and understanding the = code obviously. >>=20 >> Regards >> Sebastian >>=20 >>>=20 >>>=20 >>> Cheers, >>> Christoph >>>=20 >>>=20 >>>> Best Regards Sebastian >>>>=20 >>>> P.S.: My definition of working conditions entails bidirectionally >>>> saturating traffic with responsive and (transiently) = under-responsive >>>> flows. Something like a few long running TCP transfers to generate >>>> "base-load" and a higher number of TCP flows in IW or slow start to = add >>>> some spice to the whole. In the future, once QUIC actually takes = off*, >>>> adding more well defined/behaved UDP flows to the mix seems = reasonable. My >>>> off the cuff test for the effect of IW used to be to start a = browser and >>>> open a collection of (30-50) tabs getting a nice "thundering herd" = of TCP >>>> flows starting around the same time. But it seems browser makers = got too >>>> smart for me and will not do what I want any more but temporally = space the >>>> different sites in the tabs so that my nice thundering herd is less >>>> obnoxious (which IMHO is actually the right thing to do for actual = usage, >>>> but for testing it sucks). >>>>=20 >>>> *) Occasionally browsing the NANOG archives makes me wonder how the = move >>>> from HTTP/TCP to QUIC/UDP is going to play with operators = propensity to >>>> rate-limit UDP, but that is a different kettle of fish... >>>>=20 >>>>=20 >>>>>=20 >>>>>=20 >>>>> Christoph >>>>>=20 >>>>>>=20 >>>>>> I predict the consequences of these mistakes will differ = according to >>>>>> the type of traffic applied: >>>>>>=20 >>>>>> With TCP traffic over an Internet-scale path, the consequences = are not >>>>>> serious. The tail-drop means that the response at the end of >>>>>> slow-start will be slower, with a higher peak of intra-flow = induced >>>>>> delay, and there is also a small but measurable risk of tail-loss >>>>>> causing a more serious application-level delay. These alone = *should* >>>>>> be enough to prompt a fix, if Apple are actually serious about >>>>>> improving application responsiveness. The fixed marking = frequency, >>>>>> however, is probably invisible for this traffic. >>>>>>=20 >>>>>> With TCP traffic over a short-RTT path, the effects are more >>>>>> pronounced. The delay excursion at the end of slow-start will be >>>>>> larger in comparison to the baseline RTT, and when the latter is = short >>>>>> enough, the fixed congestion signalling frequency means there = will be >>>>>> some standing queue that real Codel would get rid of. This = standing >>>>>> queue will influence the TCP stack's RTT estimator and thus RTO = value, >>>>>> increasing the delay consequent to tail loss. >>>>>>=20 >>>>>> Similar effects to the above can be expected with other reliable = stream >>>>>> transports (SCTP, QUIC), though the details may differ. >>>>>>=20 >>>>>> The consequences with non-congestion-controlled traffic could be = much >>>>>> more serious. Real Codel will increase its drop frequency = continuously >>>>>> when faced with overload, eventually gaining control of the queue = depth >>>>>> as long as the load remains finite and reasonably constant. = Because >>>>>> Apple's AQM doesn't increase its drop frequency, the queue depth = for >>>>>> such a flow will increase continuously until either a = delay-sensitive >>>>>> rate selection mechanism is triggered at the sender, or the queue >>>>>> overflows and triggers burst losses. >>>>>>=20 >>>>>> So in the context of this discussion, is it worth generating a = type of >>>>>> load that specifically exercises this failure mode? If so, what = does >>>>>> it look like? >>>>>>=20 >>>>>> - Jonathan Morton _______________________________________________ = Rpm >>>>>> mailing list Rpm@lists.bufferbloat.net >>>>>> https://lists.bufferbloat.net/listinfo/rpm >>>>> _______________________________________________ Rpm mailing list >>>>> Rpm@lists.bufferbloat.net = https://lists.bufferbloat.net/listinfo/rpm >>=20