From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pg1-x52c.google.com (mail-pg1-x52c.google.com [IPv6:2607:f8b0:4864:20::52c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 43F5B3CB37; Sat, 14 Oct 2023 15:47:06 -0400 (EDT) Received: by mail-pg1-x52c.google.com with SMTP id 41be03b00d2f7-5a9bf4fbd3fso1485871a12.1; Sat, 14 Oct 2023 12:47:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697312825; x=1697917625; darn=lists.bufferbloat.net; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=Bu4DEkiZsoE6wriKl1KA5LlHbDG2KnltlMVwjC9sR4Y=; b=NU774/5VGxDGCQBDMfEP6JsiW+BXHZyPN35tsy60aB5xSghrV6Iug3XeK3M7k6m9a2 9ZhkeUWZz4c+g/jEF5Ru2VgEbVTKqtit9wwjFKuyaD414gc/UK6dAnYazM3E2mH90PCq yEJ26DsdGGY77USPF/dHgp8hm71m2fbne3a9kJ5WhzanWnTN+iNgLrZ6H7WMGYrOv+Ax ydgXLkFcg0edY0jOBlDyReJwIar1KCvx9ktxAkkEPAYjlXeczSdalpUVzi4UWkZJOrVB UfFR1+waYNvZfkoEPKPkybUmwVlpFvyrG6avKU4cu5LIhwskkAcupS0ERwHiumMx0bya M24g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697312825; x=1697917625; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Bu4DEkiZsoE6wriKl1KA5LlHbDG2KnltlMVwjC9sR4Y=; b=VMeTLJ1u+n7fYuMkQpRxz+nFSa4SI7uJGA4YS3XWEszUBkk2NURzF0+X2iXgChRPg8 lOMz0awb5fTKtE573zUltGjGf1iaGUJnXy+44Fjl/yo8kyPlgXx2npbgu+AMDZZzihLk H/s1/bODkuUJ6HgkaCbkdWP5RAjcxSs6rHVlQMIxQGWmy0oubJNhoH8usUmYaQwmWDun iAf3U7/XHEC4VOoF5Fvw6WWT1xKre6WpqESamwvUq7ZFqAiG8nydJvfPxpIUW4f06NQ8 4Oka3H2wRRWU5G041NNTU4Wk/Jx8eFu7yScGWwvQCNl+EuRvN4w23x95OW07Ns7/3CSP Y4Dg== X-Gm-Message-State: AOJu0YzIOB7US2AC5EHdAY+j1pb8BrI7sVvJQR+HL91QYtds1XdWiq8E LdX7kmcUZo9WYD7/r+73AWA6JTilwWcKQ19aurs= X-Google-Smtp-Source: AGHT+IHilRhsC1HpsWx5aDxeIy0vQyu/4iGHEWsz1d1cSRYMiMJYDiA2TIPb++f9afbwhQGGqb1fBIjJ6iR4wuS4m6w= X-Received: by 2002:a17:90a:d917:b0:27d:4686:73b2 with SMTP id c23-20020a17090ad91700b0027d468673b2mr3952026pjv.20.1697312824983; Sat, 14 Oct 2023 12:47:04 -0700 (PDT) MIME-Version: 1.0 References: <3E02B7F5-D0C6-4C33-AFA9-30A2E54B155F@gmail.com> <2F0C6067-1E4E-4CBA-962D-71B3F1F9B9CC@gmx.de> In-Reply-To: <2F0C6067-1E4E-4CBA-962D-71B3F1F9B9CC@gmx.de> From: Dave Taht Date: Sat, 14 Oct 2023 12:46:54 -0700 Message-ID: To: Sebastian Moeller Cc: Jonathan Morton , bloat , codel@lists.bufferbloat.net Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Codel] fq_"codel" in apple's gear X-BeenThere: codel@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: CoDel AQM discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Oct 2023 19:47:06 -0000 thank you! A packet capture, however, would be more revealing... On Sat, Oct 14, 2023 at 12:44=E2=80=AFPM Sebastian Moeller wrote: > > Hi Dave, > > > > On Oct 14, 2023, at 21:01, Dave Taht wrote: > > > > netstat -I en0 -qq > > Here is a bit of output from macosx monterey 12.7 (on intel hardware). I = know this is not really recent, ut it is the most recent I have available.= .. > > This is on an realtek based USB3 ethernet dongle that used to support 1 G= bps, but at from macosx update on it is stuck on 100Mbps FastEthernet... > > > Here is the output after a flent test: > > Out of my "--test-parameter markings=3DCS0,CS1,CS2,CS3,CS4,CS5,CS6,CS7" i= t seems only CS1 was honored and put into BK the rest all ended up in BE... > I have zero clue what macos uses to steer packets into the different clas= ses, might be DSCP alone, but then at least using some more exotic ones tha= n boring CS6/CS7 > No idea what CTL is, the average packet size mist be around 67 bytes... > > > macbook:CAKE-autorate user$ netstat -I en3 -qq > en3: > [ sched: FQ_CODEL qlength: 0/128 ] > [ pkts: 522515 bytes: 334464372 dropped pkts: 0 bytes: = 0 ] > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D > [ pri: CTL (0) srv_cl: 0x480190 quantum: 605 drr_max: = 8 ] > [ queued pkts: 0 bytes: 0 ] > [ dequeued pkts: 292918 bytes: 19787064 ] > [ budget: 0 target qdelay: 10.00 msec update interval:1= 00.00 msec ] > [ flow control: 0 feedback: 0 stalls: 0 failed: 0 o= verwhelming: 0 ] > [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] > [ flows total: 0 new: 0 old: 0 ] > [ throttle on: 0 off: 0 drop: 0 ] > [ compressible pkts: 0 compressed pkts: 0] > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D > [ pri: VO (1) srv_cl: 0x400180 quantum: 605 drr_max: = 8 ] > [ queued pkts: 0 bytes: 0 ] > [ dequeued pkts: 0 bytes: 0 ] > [ budget: 0 target qdelay: 10.00 msec update interval:1= 00.00 msec ] > [ flow control: 0 feedback: 0 stalls: 0 failed: 0 o= verwhelming: 0 ] > [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] > [ flows total: 0 new: 0 old: 0 ] > [ throttle on: 0 off: 0 drop: 0 ] > [ compressible pkts: 0 compressed pkts: 0] > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D > [ pri: VI (2) srv_cl: 0x380100 quantum: 3028 drr_max: = 6 ] > [ queued pkts: 0 bytes: 0 ] > [ dequeued pkts: 0 bytes: 0 ] > [ budget: 0 target qdelay: 10.00 msec update interval:1= 00.00 msec ] > [ flow control: 0 feedback: 0 stalls: 0 failed: 0 o= verwhelming: 0 ] > [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] > [ flows total: 0 new: 0 old: 0 ] > [ throttle on: 0 off: 0 drop: 0 ] > [ compressible pkts: 0 compressed pkts: 0] > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D > [ pri: RV (3) srv_cl: 0x300110 quantum: 3028 drr_max: = 6 ] > [ queued pkts: 0 bytes: 0 ] > [ dequeued pkts: 0 bytes: 0 ] > [ budget: 0 target qdelay: 10.00 msec update interval:1= 00.00 msec ] > [ flow control: 0 feedback: 0 stalls: 0 failed: 0 o= verwhelming: 0 ] > [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] > [ flows total: 0 new: 0 old: 0 ] > [ throttle on: 0 off: 0 drop: 0 ] > [ compressible pkts: 0 compressed pkts: 0] > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D > [ pri: AV (4) srv_cl: 0x280120 quantum: 3028 drr_max: = 6 ] > [ queued pkts: 0 bytes: 0 ] > [ dequeued pkts: 0 bytes: 0 ] > [ budget: 0 target qdelay: 10.00 msec update interval:1= 00.00 msec ] > [ flow control: 0 feedback: 0 stalls: 0 failed: 0 o= verwhelming: 0 ] > [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] > [ flows total: 0 new: 0 old: 0 ] > [ throttle on: 0 off: 0 drop: 0 ] > [ compressible pkts: 0 compressed pkts: 0] > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D > [ pri: OAM (5) srv_cl: 0x200020 quantum: 1514 drr_max: = 4 ] > [ queued pkts: 0 bytes: 0 ] > [ dequeued pkts: 0 bytes: 0 ] > [ budget: 0 target qdelay: 10.00 msec update interval:1= 00.00 msec ] > [ flow control: 0 feedback: 0 stalls: 0 failed: 0 o= verwhelming: 0 ] > [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] > [ flows total: 0 new: 0 old: 0 ] > [ throttle on: 0 off: 0 drop: 0 ] > [ compressible pkts: 0 compressed pkts: 0] > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D > [ pri: RD (6) srv_cl: 0x180010 quantum: 1514 drr_max: = 4 ] > [ queued pkts: 0 bytes: 0 ] > [ dequeued pkts: 13 bytes: 6684 ] > [ budget: 0 target qdelay: 10.00 msec update interval:1= 00.00 msec ] > [ flow control: 0 feedback: 0 stalls: 0 failed: 0 o= verwhelming: 0 ] > [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] > [ flows total: 0 new: 0 old: 0 ] > [ throttle on: 0 off: 0 drop: 0 ] > [ compressible pkts: 0 compressed pkts: 0] > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D > [ pri: BE (7) srv_cl: 0x0 quantum: 1514 drr_max: 4 ] > [ queued pkts: 0 bytes: 0 ] > [ dequeued pkts: 225368 bytes: 314071428 ] > [ budget: 0 target qdelay: 10.00 msec update interval:1= 00.00 msec ] > [ flow control: 0 feedback: 0 stalls: 0 failed: 0 o= verwhelming: 0 ] > [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] > [ flows total: 0 new: 0 old: 0 ] > [ throttle on: 0 off: 0 drop: 0 ] > [ compressible pkts: 0 compressed pkts: 0] > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D > [ pri: BK (8) srv_cl: 0x100080 quantum: 1514 drr_max: = 2 ] > [ queued pkts: 0 bytes: 0 ] > [ dequeued pkts: 3831 bytes: 576846 ] > [ budget: 0 target qdelay: 10.00 msec update interval:1= 00.00 msec ] > [ flow control: 0 feedback: 0 stalls: 0 failed: 0 o= verwhelming: 0 ] > [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] > [ flows total: 0 new: 0 old: 0 ] > [ throttle on: 0 off: 0 drop: 0 ] > [ compressible pkts: 0 compressed pkts: 0] > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D > [ pri: BK_SYS (9) srv_cl: 0x80090 quantum: 1514 drr_max: 2 ] > [ queued pkts: 0 bytes: 0 ] > [ dequeued pkts: 385 bytes: 22350 ] > [ budget: 0 target qdelay: 10.00 msec update interval:1= 00.00 msec ] > [ flow control: 0 feedback: 0 stalls: 0 failed: 0 o= verwhelming: 0 ] > [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ] > [ flows total: 0 new: 0 old: 0 ] > [ throttle on: 0 off: 0 drop: 0 ] > [ compressible pkts: 375 compressed pkts: 0] > macbook:CAKE-autorate user$ uname -a > Darwin macbook.local 21.6.0 Darwin Kernel Version 21.6.0: Fri Sep 15 16:1= 7:23 PDT 2023; root:xnu-8020.240.18.703.5~1/RELEASE_X86_64 x86_64 > > > Here is the flent output: > > bash-3.2$ echo "IPv4" ; date ; ping -c 10 netperf-eu.bufferbloat.net ; ./= run-flent --ipv4 -l 60 -H netperf-eu.bufferbloat.net rrul_var --remote-meta= data=3Droot@192.168.42.1 --test-parameter=3Dcpu_stats_hosts=3Droot@192.168.= 42.1 --step-size=3D.05 --test-parameter bidir_streams=3D8 --test-parameter = markings=3DCS0,CS1,CS2,CS3,CS4,CS5,CS6,CS7 --test-parameter ping_hosts=3D1.= 1.1.1 -D . -t IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct45500of46719K-D0= 90pct105000of116789K_hms-beagle2_en4TB2_TurrisOmnia-TurrisOS.6.3.2-pppoe-wa= n-eth2.7_2_bridged-FB7520-OpenWrt-r22256-14334c222e-OutdoorMSAN-VDSL100_2_n= etperf-eu.bufferbloat.net --log-file > IPv4 > Sat Oct 14 21:19:24 CEST 2023 > PING demo.tohojo.dk (193.10.227.30): 56 data bytes > 64 bytes from 193.10.227.30: icmp_seq=3D0 ttl=3D48 time=3D24.259 ms > 64 bytes from 193.10.227.30: icmp_seq=3D1 ttl=3D48 time=3D22.789 ms > 64 bytes from 193.10.227.30: icmp_seq=3D2 ttl=3D48 time=3D24.097 ms > 64 bytes from 193.10.227.30: icmp_seq=3D3 ttl=3D48 time=3D24.433 ms > 64 bytes from 193.10.227.30: icmp_seq=3D4 ttl=3D48 time=3D24.342 ms > 64 bytes from 193.10.227.30: icmp_seq=3D5 ttl=3D48 time=3D24.325 ms > 64 bytes from 193.10.227.30: icmp_seq=3D6 ttl=3D48 time=3D23.456 ms > 64 bytes from 193.10.227.30: icmp_seq=3D7 ttl=3D48 time=3D22.628 ms > 64 bytes from 193.10.227.30: icmp_seq=3D8 ttl=3D48 time=3D24.396 ms > 64 bytes from 193.10.227.30: icmp_seq=3D9 ttl=3D48 time=3D23.521 ms > > --- demo.tohojo.dk ping statistics --- > 10 packets transmitted, 10 packets received, 0.0% packet loss > round-trip min/avg/max/stddev =3D 22.628/23.825/24.433/0.649 ms > Starting Flent 2.1.1+git.7b931915 using Python 3.10.8. > Starting rrul_var test. Expected run time: 70 seconds. > Data file written to ./rrul_var-2023-10-14T211934.512820.IPv4_SQM_cake_la= yer-cake_LLA-ETH_OH34_U097pct45500of46719K-D090pct105000of116789K_hms-beagl= e2_en4TB2_TurrisOmnia-TurrisOS_6_3_2-pppoe-wan-eth2_7_2_bridged-FB7520-Open= Wrt-r22256-14334c222e-OutdoorMSAN-VDSL100.flent.gz > > Summary of rrul_var test run from 2023-10-14 19:19:34.512820 > Title: 'IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct45500of46719K-D090= pct105000of116789K_hms-beagle2_en4TB2_TurrisOmnia-TurrisOS.6.3.2-pppoe-wan-= eth2.7_2_bridged-FB7520-OpenWrt-r22256-14334c222e-OutdoorMSAN-VDSL100_2_net= perf-eu.bufferbloat.net' > > avg median 99th= % # data pts > Ping (ms) ICMP 1.1.1.1 (extra) : 13.10 12.90 17.= 40 ms 1396 > Ping (ms) avg : 27.07 N/A N= /A ms 1639 > Ping (ms)::ICMP : 26.15 26.10 30.= 70 ms 1399 > Ping (ms)::UDP 0 (CS0) : 27.78 27.55 32.= 13 ms 1639 > Ping (ms)::UDP 1 (CS1) : 29.34 29.51 35.= 09 ms 1639 > Ping (ms)::UDP 2 (CS2) : 26.18 26.06 30.= 56 ms 1639 > Ping (ms)::UDP 3 (CS3) : 26.22 26.06 30.= 77 ms 1639 > Ping (ms)::UDP 4 (CS4) : 26.20 25.98 30.= 93 ms 1639 > Ping (ms)::UDP 5 (CS5) : 26.20 26.06 31.= 33 ms 1639 > Ping (ms)::UDP 6 (CS6) : 28.13 28.11 32.= 86 ms 1639 > Ping (ms)::UDP 7 (CS7) : 26.51 26.44 31.= 39 ms 1639 > TCP download avg : 11.33 N/A N= /A Mbits/s 1639 > TCP download sum : 90.62 N/A N= /A Mbits/s 1639 > TCP download::0 (CS0) : 11.39 11.25 17.= 48 Mbits/s 1639 > TCP download::1 (CS1) : 10.66 10.21 28.= 97 Mbits/s 1639 > TCP download::2 (CS2) : 12.17 11.99 18.= 87 Mbits/s 1639 > TCP download::3 (CS3) : 11.29 11.06 18.= 27 Mbits/s 1639 > TCP download::4 (CS4) : 11.87 11.34 23.= 51 Mbits/s 1639 > TCP download::5 (CS5) : 10.51 10.49 16.= 58 Mbits/s 1639 > TCP download::6 (CS6) : 11.37 11.15 18.= 32 Mbits/s 1639 > TCP download::7 (CS7) : 11.36 11.25 16.= 99 Mbits/s 1639 > TCP totals : 130.00 N/A N= /A Mbits/s 1639 > TCP upload avg : 4.92 N/A N= /A Mbits/s 1639 > TCP upload sum : 39.38 N/A N= /A Mbits/s 1639 > TCP upload::0 (CS0) : 5.50 5.48 7.= 94 Mbits/s 1639 > TCP upload::1 (CS1) : 2.13 2.19 2.= 64 Mbits/s 1639 > TCP upload::2 (CS2) : 5.47 5.47 7.= 13 Mbits/s 1639 > TCP upload::3 (CS3) : 5.47 5.48 6.= 97 Mbits/s 1639 > TCP upload::4 (CS4) : 5.47 5.49 7.= 23 Mbits/s 1639 > TCP upload::5 (CS5) : 5.46 5.46 6.= 83 Mbits/s 1639 > TCP upload::6 (CS6) : 4.96 4.95 6.= 25 Mbits/s 1639 > TCP upload::7 (CS7) : 4.92 4.94 6.= 18 Mbits/s 1639 > cpu_stats_root@192.168.42.1::load : 0.40 0.42 0.= 83 1243 > bash-3.2$ > > > --=20 Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.htm= l Dave T=C3=A4ht CSO, LibreQos