From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.15.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 33E403CB37; Sat, 14 Oct 2023 15:44:19 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de; s=s31663417; t=1697312657; x=1697917457; i=moeller0@gmx.de; bh=8nKyqGDxxe9L/abKQcrKmVyq9UsIrzzg7G5F9mVwQwI=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=i74jsaUnvyMyTa5G/Ab2Y/e6S73vkEP+QB1dRvtQbouDfeOM6tJPFgFnSt2omcpvG5DVKXCGFUh ZuvJXycSqL82xJtwdgH8lpwDyryW5GgxJE1FCDoOA8g5MLMRqJDHv+MTNCmP7TyOo+cOXqRPOytG6 l25bSiw0q+ePw25sDVerw9pstAupe3jWTx+fzCfKruiFOfJLPKjQIFwtCSqCyt2AggnoGAsU6s0ox m3yYCUqIFODZKidzzNZ9K9qCyVYCOD5gY5EzgGaW425nFGkhWT73yRcRFQpqqOYBnn1wk2w4yWXm7 wyhHhpnLJ8gdUdbCfnzsekhoKpYJ9b9gJXag== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Received: from smtpclient.apple ([77.10.49.66]) by mail.gmx.net (mrgmx005 [212.227.17.190]) with ESMTPSA (Nemesis) id 1MbAh0-1rObKz0cWm-00bXzd; Sat, 14 Oct 2023 21:44:17 +0200 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.4\)) From: Sebastian Moeller In-Reply-To: Date: Sat, 14 Oct 2023 21:44:16 +0200 Cc: Jonathan Morton , bloat , codel@lists.bufferbloat.net Content-Transfer-Encoding: quoted-printable Message-Id: <2F0C6067-1E4E-4CBA-962D-71B3F1F9B9CC@gmx.de> References: <3E02B7F5-D0C6-4C33-AFA9-30A2E54B155F@gmail.com> To: =?utf-8?Q?Dave_T=C3=A4ht?= X-Mailer: Apple Mail (2.3696.120.41.1.4) X-Provags-ID: V03:K1:Z22o9JzPUVB0ySZoJ/AClw+4zrbpz0ZTmaKGbhb4ZgXncR5xyhc OudHBs1K+ATRsmkT7041fTtqPYHiGo+VwFeFbrlA1dviq+R8eNasNrnxeQ3t7GH56egr8zK yPj38Vpv1MdhyhY8fXyO1XCy5+yRbg7zrTK/MzONi3YcnFIeqOUCZRV0j6AQRpH3EmToEC7 PH/xWrZm8gPmDa8Guz31A== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:GQZgbtCyad0=;/N9azTi03LSXRnSOQIxry9sdMs0 9SFQXrV7UKY9IEaK9xrGsA1COnYn3nSzmq0NOCHzG1f8FKD756hjbl4HlGA1yol2t9k1ysrPG p1n5cwh4RlkYrkcKqmvJhflX/PvXpVIWNUfMuJ3X+Co+sHzsc+TMmRrQE3sNsXKDpwszOlcAs tmKTIPcZSDpY6BMX+fyRVK0UhNiEY+5psNMzi43mbGrrdR1JCxxzuNp5AiJ2M4ZAzKunjXlpf x4sK8ULRH32H6HzdybUqQwaY7IE7V6tpzqRlPd1WP9b2oYvMKOwp80Z+LPHyBLZsy+ozfKsRx HF6tFi2pU/ohhnjCixks9qNqKIZcxragDE3fD6SAJXHti9IXALRrHsTCIv1C0uYTZChezkBCI 9ABYho3eWszQik+UafBWxTSNmuaPcRywba5OMf1vXPtuU0/DrQ/U3mx7VtgaOZj/TOm4XZ6HA Gc51jPLwh04tzrMrLh59lvUt+aUJFNyyLYHp5/YhPxMmGabMtwdyM93yg2HcjB6qqQACRbLlZ CUMhzfqy98sWr2KHSMv5GZTe28Rvrwef2johypZDbkYe3pyzO+qORT6otysZ+xpuPEWxcuK6h DbtO888PX0VeGWY1HGQE+fVjwQJzPbUR8gD+dHT7SbansgRzGL6nWSjmeUtU9YK4RhfdUqVHR +FsgkdhV823j9S98Kw9tGIrEP6nt9OipVbCuBbtsEkqBvBJ9+mhDSe0Y36QgHrUXOrrufU3n6 6wXZWOco+eQi6TrX5scAsnWdsMhv5kjWenGbPueEGoeo7UwJGi7mIqpAomGDsEfFqe1KYUWwK /tlnkh8jbMog1gTTJe8odTyupIge6jIEVCnHFfolaUaGafEssDtNHMu9DoiDdMKM741JqOfEh zoVb/AF6b6BGM65p5/My9IT14cp4I+BkxJ1ba3VN6zBQ3nGe+Ox2txSMwfHVOFpwPpyffhK+i 82htvA== Subject: Re: [Codel] fq_"codel" in apple's gear X-BeenThere: codel@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: CoDel AQM discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Oct 2023 19:44:19 -0000 Hi Dave, > On Oct 14, 2023, at 21:01, Dave Taht wrote: >=20 > netstat -I en0 -qq Here is a bit of output from macosx monterey 12.7 (on intel hardware). I = know this is not really recent, ut it is the most recent I have = available... This is on an realtek based USB3 ethernet dongle that used to support 1 = Gbps, but at from macosx update on it is stuck on 100Mbps = FastEthernet... Here is the output after a flent test: Out of my "--test-parameter markings=3DCS0,CS1,CS2,CS3,CS4,CS5,CS6,CS7" = it seems only CS1 was honored and put into BK the rest all ended up in = BE... I have zero clue what macos uses to steer packets into the different = classes, might be DSCP alone, but then at least using some more exotic = ones than boring CS6/CS7 No idea what CTL is, the average packet size mist be around 67 bytes... macbook:CAKE-autorate user$ netstat -I en3 -qq en3: [ sched: FQ_CODEL qlength: 0/128 ] [ pkts: 522515 bytes: 334464372 dropped pkts: 0 bytes: = 0 ] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D [ pri: CTL (0) srv_cl: 0x480190 quantum: 605 drr_max: = 8 ] [ queued pkts: 0 bytes: 0 ] [ dequeued pkts: 292918 bytes: 19787064 ] [ budget: 0 target qdelay: 10.00 msec update = interval:100.00 msec ] [ flow control: 0 feedback: 0 stalls: 0 failed: 0 = overwhelming: 0 ] [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] [ flows total: 0 new: 0 old: 0 ] [ throttle on: 0 off: 0 drop: 0 ] [ compressible pkts: 0 compressed pkts: 0] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D [ pri: VO (1) srv_cl: 0x400180 quantum: 605 drr_max: = 8 ] [ queued pkts: 0 bytes: 0 ] [ dequeued pkts: 0 bytes: 0 ] [ budget: 0 target qdelay: 10.00 msec update = interval:100.00 msec ] [ flow control: 0 feedback: 0 stalls: 0 failed: 0 = overwhelming: 0 ] [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] [ flows total: 0 new: 0 old: 0 ] [ throttle on: 0 off: 0 drop: 0 ] [ compressible pkts: 0 compressed pkts: 0] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D [ pri: VI (2) srv_cl: 0x380100 quantum: 3028 drr_max: = 6 ] [ queued pkts: 0 bytes: 0 ] [ dequeued pkts: 0 bytes: 0 ] [ budget: 0 target qdelay: 10.00 msec update = interval:100.00 msec ] [ flow control: 0 feedback: 0 stalls: 0 failed: 0 = overwhelming: 0 ] [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] [ flows total: 0 new: 0 old: 0 ] [ throttle on: 0 off: 0 drop: 0 ] [ compressible pkts: 0 compressed pkts: 0] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D [ pri: RV (3) srv_cl: 0x300110 quantum: 3028 drr_max: = 6 ] [ queued pkts: 0 bytes: 0 ] [ dequeued pkts: 0 bytes: 0 ] [ budget: 0 target qdelay: 10.00 msec update = interval:100.00 msec ] [ flow control: 0 feedback: 0 stalls: 0 failed: 0 = overwhelming: 0 ] [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] [ flows total: 0 new: 0 old: 0 ] [ throttle on: 0 off: 0 drop: 0 ] [ compressible pkts: 0 compressed pkts: 0] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D [ pri: AV (4) srv_cl: 0x280120 quantum: 3028 drr_max: = 6 ] [ queued pkts: 0 bytes: 0 ] [ dequeued pkts: 0 bytes: 0 ] [ budget: 0 target qdelay: 10.00 msec update = interval:100.00 msec ] [ flow control: 0 feedback: 0 stalls: 0 failed: 0 = overwhelming: 0 ] [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] [ flows total: 0 new: 0 old: 0 ] [ throttle on: 0 off: 0 drop: 0 ] [ compressible pkts: 0 compressed pkts: 0] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D [ pri: OAM (5) srv_cl: 0x200020 quantum: 1514 drr_max: = 4 ] [ queued pkts: 0 bytes: 0 ] [ dequeued pkts: 0 bytes: 0 ] [ budget: 0 target qdelay: 10.00 msec update = interval:100.00 msec ] [ flow control: 0 feedback: 0 stalls: 0 failed: 0 = overwhelming: 0 ] [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] [ flows total: 0 new: 0 old: 0 ] [ throttle on: 0 off: 0 drop: 0 ] [ compressible pkts: 0 compressed pkts: 0] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D [ pri: RD (6) srv_cl: 0x180010 quantum: 1514 drr_max: = 4 ] [ queued pkts: 0 bytes: 0 ] [ dequeued pkts: 13 bytes: 6684 ] [ budget: 0 target qdelay: 10.00 msec update = interval:100.00 msec ] [ flow control: 0 feedback: 0 stalls: 0 failed: 0 = overwhelming: 0 ] [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] [ flows total: 0 new: 0 old: 0 ] [ throttle on: 0 off: 0 drop: 0 ] [ compressible pkts: 0 compressed pkts: 0] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D [ pri: BE (7) srv_cl: 0x0 quantum: 1514 drr_max: 4 ] [ queued pkts: 0 bytes: 0 ] [ dequeued pkts: 225368 bytes: 314071428 ] [ budget: 0 target qdelay: 10.00 msec update = interval:100.00 msec ] [ flow control: 0 feedback: 0 stalls: 0 failed: 0 = overwhelming: 0 ] [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] [ flows total: 0 new: 0 old: 0 ] [ throttle on: 0 off: 0 drop: 0 ] [ compressible pkts: 0 compressed pkts: 0] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D [ pri: BK (8) srv_cl: 0x100080 quantum: 1514 drr_max: = 2 ] [ queued pkts: 0 bytes: 0 ] [ dequeued pkts: 3831 bytes: 576846 ] [ budget: 0 target qdelay: 10.00 msec update = interval:100.00 msec ] [ flow control: 0 feedback: 0 stalls: 0 failed: 0 = overwhelming: 0 ] [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] [ flows total: 0 new: 0 old: 0 ] [ throttle on: 0 off: 0 drop: 0 ] [ compressible pkts: 0 compressed pkts: 0] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D [ pri: BK_SYS (9) srv_cl: 0x80090 quantum: 1514 drr_max: 2 ] [ queued pkts: 0 bytes: 0 ] [ dequeued pkts: 385 bytes: 22350 ] [ budget: 0 target qdelay: 10.00 msec update = interval:100.00 msec ] [ flow control: 0 feedback: 0 stalls: 0 failed: 0 = overwhelming: 0 ] [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] [ flows total: 0 new: 0 old: 0 ] [ throttle on: 0 off: 0 drop: 0 ] [ compressible pkts: 375 compressed pkts: 0] macbook:CAKE-autorate user$ uname -a Darwin macbook.local 21.6.0 Darwin Kernel Version 21.6.0: Fri Sep 15 = 16:17:23 PDT 2023; root:xnu-8020.240.18.703.5~1/RELEASE_X86_64 x86_64 Here is the flent output: bash-3.2$ echo "IPv4" ; date ; ping -c 10 netperf-eu.bufferbloat.net ; = ./run-flent --ipv4 -l 60 -H netperf-eu.bufferbloat.net rrul_var = --remote-metadata=3Droot@192.168.42.1 = --test-parameter=3Dcpu_stats_hosts=3Droot@192.168.42.1 --step-size=3D.05 = --test-parameter bidir_streams=3D8 --test-parameter = markings=3DCS0,CS1,CS2,CS3,CS4,CS5,CS6,CS7 --test-parameter = ping_hosts=3D1.1.1.1 -D . -t = IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct45500of46719K-D090pct105000of= 116789K_hms-beagle2_en4TB2_TurrisOmnia-TurrisOS.6.3.2-pppoe-wan-eth2.7_2_b= ridged-FB7520-OpenWrt-r22256-14334c222e-OutdoorMSAN-VDSL100_2_netperf-eu.b= ufferbloat.net --log-file IPv4 Sat Oct 14 21:19:24 CEST 2023 PING demo.tohojo.dk (193.10.227.30): 56 data bytes 64 bytes from 193.10.227.30: icmp_seq=3D0 ttl=3D48 time=3D24.259 ms 64 bytes from 193.10.227.30: icmp_seq=3D1 ttl=3D48 time=3D22.789 ms 64 bytes from 193.10.227.30: icmp_seq=3D2 ttl=3D48 time=3D24.097 ms 64 bytes from 193.10.227.30: icmp_seq=3D3 ttl=3D48 time=3D24.433 ms 64 bytes from 193.10.227.30: icmp_seq=3D4 ttl=3D48 time=3D24.342 ms 64 bytes from 193.10.227.30: icmp_seq=3D5 ttl=3D48 time=3D24.325 ms 64 bytes from 193.10.227.30: icmp_seq=3D6 ttl=3D48 time=3D23.456 ms 64 bytes from 193.10.227.30: icmp_seq=3D7 ttl=3D48 time=3D22.628 ms 64 bytes from 193.10.227.30: icmp_seq=3D8 ttl=3D48 time=3D24.396 ms 64 bytes from 193.10.227.30: icmp_seq=3D9 ttl=3D48 time=3D23.521 ms --- demo.tohojo.dk ping statistics --- 10 packets transmitted, 10 packets received, 0.0% packet loss round-trip min/avg/max/stddev =3D 22.628/23.825/24.433/0.649 ms Starting Flent 2.1.1+git.7b931915 using Python 3.10.8. Starting rrul_var test. Expected run time: 70 seconds. Data file written to = ./rrul_var-2023-10-14T211934.512820.IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_= U097pct45500of46719K-D090pct105000of116789K_hms-beagle2_en4TB2_TurrisOmnia= -TurrisOS_6_3_2-pppoe-wan-eth2_7_2_bridged-FB7520-OpenWrt-r22256-14334c222= e-OutdoorMSAN-VDSL100.flent.gz Summary of rrul_var test run from 2023-10-14 19:19:34.512820 Title: = 'IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct45500of46719K-D090pct105000o= f116789K_hms-beagle2_en4TB2_TurrisOmnia-TurrisOS.6.3.2-pppoe-wan-eth2.7_2_= bridged-FB7520-OpenWrt-r22256-14334c222e-OutdoorMSAN-VDSL100_2_netperf-eu.= bufferbloat.net' avg median = 99th % # data pts Ping (ms) ICMP 1.1.1.1 (extra) : 13.10 12.90 = 17.40 ms 1396 Ping (ms) avg : 27.07 N/A = N/A ms 1639 Ping (ms)::ICMP : 26.15 26.10 = 30.70 ms 1399 Ping (ms)::UDP 0 (CS0) : 27.78 27.55 = 32.13 ms 1639 Ping (ms)::UDP 1 (CS1) : 29.34 29.51 = 35.09 ms 1639 Ping (ms)::UDP 2 (CS2) : 26.18 26.06 = 30.56 ms 1639 Ping (ms)::UDP 3 (CS3) : 26.22 26.06 = 30.77 ms 1639 Ping (ms)::UDP 4 (CS4) : 26.20 25.98 = 30.93 ms 1639 Ping (ms)::UDP 5 (CS5) : 26.20 26.06 = 31.33 ms 1639 Ping (ms)::UDP 6 (CS6) : 28.13 28.11 = 32.86 ms 1639 Ping (ms)::UDP 7 (CS7) : 26.51 26.44 = 31.39 ms 1639 TCP download avg : 11.33 N/A = N/A Mbits/s 1639 TCP download sum : 90.62 N/A = N/A Mbits/s 1639 TCP download::0 (CS0) : 11.39 11.25 = 17.48 Mbits/s 1639 TCP download::1 (CS1) : 10.66 10.21 = 28.97 Mbits/s 1639 TCP download::2 (CS2) : 12.17 11.99 = 18.87 Mbits/s 1639 TCP download::3 (CS3) : 11.29 11.06 = 18.27 Mbits/s 1639 TCP download::4 (CS4) : 11.87 11.34 = 23.51 Mbits/s 1639 TCP download::5 (CS5) : 10.51 10.49 = 16.58 Mbits/s 1639 TCP download::6 (CS6) : 11.37 11.15 = 18.32 Mbits/s 1639 TCP download::7 (CS7) : 11.36 11.25 = 16.99 Mbits/s 1639 TCP totals : 130.00 N/A = N/A Mbits/s 1639 TCP upload avg : 4.92 N/A = N/A Mbits/s 1639 TCP upload sum : 39.38 N/A = N/A Mbits/s 1639 TCP upload::0 (CS0) : 5.50 5.48 = 7.94 Mbits/s 1639 TCP upload::1 (CS1) : 2.13 2.19 = 2.64 Mbits/s 1639 TCP upload::2 (CS2) : 5.47 5.47 = 7.13 Mbits/s 1639 TCP upload::3 (CS3) : 5.47 5.48 = 6.97 Mbits/s 1639 TCP upload::4 (CS4) : 5.47 5.49 = 7.23 Mbits/s 1639 TCP upload::5 (CS5) : 5.46 5.46 = 6.83 Mbits/s 1639 TCP upload::6 (CS6) : 4.96 4.95 = 6.25 Mbits/s 1639 TCP upload::7 (CS7) : 4.92 4.94 = 6.18 Mbits/s 1639 cpu_stats_root@192.168.42.1::load : 0.40 0.42 = 0.83 1243 bash-3.2$