* Re: [Bloat] [Codel] fq_"codel" in apple's gear
[not found] ` <CAA93jw5VcHe0dpSVFu9zXd58JUhNiA1Z=gVASohfHctHTC+EpA@mail.gmail.com>
@ 2023-10-14 19:01 ` Dave Taht
2023-10-14 19:44 ` Sebastian Moeller
0 siblings, 1 reply; 4+ messages in thread
From: Dave Taht @ 2023-10-14 19:01 UTC (permalink / raw)
To: Jonathan Morton, bloat; +Cc: codel
I would be curious if fq_"codel" was still in use on the current apple
products. On an OSX box, the magic command to look at the stats was:
netstat -I en0 -qq # or whatever the right interface is
And you can just poll that while slamming traffic through it.
On Sat, Oct 14, 2023 at 11:50 AM Dave Taht <dave.taht@gmail.com> wrote:
>
> On Sat, Oct 14, 2023 at 11:17 AM Jonathan Morton <chromatix99@gmail.com> wrote:
> >
> > > On 14 Oct, 2023, at 8:23 pm, Dave Taht <dave.taht@gmail.com> wrote:
> > >
> > > I would really like someone with a fresh set of eyeballs to review the
> > > "codel" portion of apple's latest code drop here:
> > >
> > > https://www.linkedin.com/feed/update/urn:li:activity:7118969694987935744/
> >
> > So far as I can tell, it still implements DRR++ (with a comment referencing RFC-8290), but not Codel (there's also no comment referencing RFC-8289). There's at least one function for handling ECN which is simply not called, at least not in that source file.
>
> Thank you for summoning the gumption to look again. I was having blood
> pressure problems.
>
> > In fact, I looked in the header to see where an ECN mark counter was defined. It is referenced in essentially three places: the definition, an extraction for reporting to userspace, and an increment - in a function implementing L4S marking, not Codel.
> >
> > A similar counter for "early dropped" packets is incremented only when packets have to be dropped in an emergency because the queue is *full*. That's the other counter I would expect to be touched by an AQM implementation.
>
> Yep. I had run some tests then and saw the rtts climbing alarmingly,
> while polling their stack and only seeing drops in big bursty batches,
> total resets of the tcp flow, and so on.
>
> > So, no tangible improvement that I can find easily.
> >
> > - Jonathan Morton
>
>
>
> --
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos
--
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Bloat] [Codel] fq_"codel" in apple's gear
2023-10-14 19:01 ` [Bloat] [Codel] fq_"codel" in apple's gear Dave Taht
@ 2023-10-14 19:44 ` Sebastian Moeller
2023-10-14 19:46 ` Dave Taht
0 siblings, 1 reply; 4+ messages in thread
From: Sebastian Moeller @ 2023-10-14 19:44 UTC (permalink / raw)
To: Dave Täht; +Cc: Jonathan Morton, bloat, codel
Hi Dave,
> On Oct 14, 2023, at 21:01, Dave Taht <dave.taht@gmail.com> wrote:
>
> netstat -I en0 -qq
Here is a bit of output from macosx monterey 12.7 (on intel hardware). I know this is not really recent, ut it is the most recent I have available...
This is on an realtek based USB3 ethernet dongle that used to support 1 Gbps, but at from macosx update on it is stuck on 100Mbps FastEthernet...
Here is the output after a flent test:
Out of my "--test-parameter markings=CS0,CS1,CS2,CS3,CS4,CS5,CS6,CS7" it seems only CS1 was honored and put into BK the rest all ended up in BE...
I have zero clue what macos uses to steer packets into the different classes, might be DSCP alone, but then at least using some more exotic ones than boring CS6/CS7
No idea what CTL is, the average packet size mist be around 67 bytes...
macbook:CAKE-autorate user$ netstat -I en3 -qq
en3:
[ sched: FQ_CODEL qlength: 0/128 ]
[ pkts: 522515 bytes: 334464372 dropped pkts: 0 bytes: 0 ]
=====================================================
[ pri: CTL (0) srv_cl: 0x480190 quantum: 605 drr_max: 8 ]
[ queued pkts: 0 bytes: 0 ]
[ dequeued pkts: 292918 bytes: 19787064 ]
[ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
[ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
[ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
[ flows total: 0 new: 0 old: 0 ]
[ throttle on: 0 off: 0 drop: 0 ]
[ compressible pkts: 0 compressed pkts: 0]
=====================================================
[ pri: VO (1) srv_cl: 0x400180 quantum: 605 drr_max: 8 ]
[ queued pkts: 0 bytes: 0 ]
[ dequeued pkts: 0 bytes: 0 ]
[ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
[ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
[ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
[ flows total: 0 new: 0 old: 0 ]
[ throttle on: 0 off: 0 drop: 0 ]
[ compressible pkts: 0 compressed pkts: 0]
=====================================================
[ pri: VI (2) srv_cl: 0x380100 quantum: 3028 drr_max: 6 ]
[ queued pkts: 0 bytes: 0 ]
[ dequeued pkts: 0 bytes: 0 ]
[ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
[ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
[ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
[ flows total: 0 new: 0 old: 0 ]
[ throttle on: 0 off: 0 drop: 0 ]
[ compressible pkts: 0 compressed pkts: 0]
=====================================================
[ pri: RV (3) srv_cl: 0x300110 quantum: 3028 drr_max: 6 ]
[ queued pkts: 0 bytes: 0 ]
[ dequeued pkts: 0 bytes: 0 ]
[ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
[ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
[ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
[ flows total: 0 new: 0 old: 0 ]
[ throttle on: 0 off: 0 drop: 0 ]
[ compressible pkts: 0 compressed pkts: 0]
=====================================================
[ pri: AV (4) srv_cl: 0x280120 quantum: 3028 drr_max: 6 ]
[ queued pkts: 0 bytes: 0 ]
[ dequeued pkts: 0 bytes: 0 ]
[ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
[ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
[ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
[ flows total: 0 new: 0 old: 0 ]
[ throttle on: 0 off: 0 drop: 0 ]
[ compressible pkts: 0 compressed pkts: 0]
=====================================================
[ pri: OAM (5) srv_cl: 0x200020 quantum: 1514 drr_max: 4 ]
[ queued pkts: 0 bytes: 0 ]
[ dequeued pkts: 0 bytes: 0 ]
[ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
[ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
[ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
[ flows total: 0 new: 0 old: 0 ]
[ throttle on: 0 off: 0 drop: 0 ]
[ compressible pkts: 0 compressed pkts: 0]
=====================================================
[ pri: RD (6) srv_cl: 0x180010 quantum: 1514 drr_max: 4 ]
[ queued pkts: 0 bytes: 0 ]
[ dequeued pkts: 13 bytes: 6684 ]
[ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
[ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
[ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
[ flows total: 0 new: 0 old: 0 ]
[ throttle on: 0 off: 0 drop: 0 ]
[ compressible pkts: 0 compressed pkts: 0]
=====================================================
[ pri: BE (7) srv_cl: 0x0 quantum: 1514 drr_max: 4 ]
[ queued pkts: 0 bytes: 0 ]
[ dequeued pkts: 225368 bytes: 314071428 ]
[ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
[ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
[ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
[ flows total: 0 new: 0 old: 0 ]
[ throttle on: 0 off: 0 drop: 0 ]
[ compressible pkts: 0 compressed pkts: 0]
=====================================================
[ pri: BK (8) srv_cl: 0x100080 quantum: 1514 drr_max: 2 ]
[ queued pkts: 0 bytes: 0 ]
[ dequeued pkts: 3831 bytes: 576846 ]
[ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
[ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
[ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
[ flows total: 0 new: 0 old: 0 ]
[ throttle on: 0 off: 0 drop: 0 ]
[ compressible pkts: 0 compressed pkts: 0]
=====================================================
[ pri: BK_SYS (9) srv_cl: 0x80090 quantum: 1514 drr_max: 2 ]
[ queued pkts: 0 bytes: 0 ]
[ dequeued pkts: 385 bytes: 22350 ]
[ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
[ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
[ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
[ flows total: 0 new: 0 old: 0 ]
[ throttle on: 0 off: 0 drop: 0 ]
[ compressible pkts: 375 compressed pkts: 0]
macbook:CAKE-autorate user$ uname -a
Darwin macbook.local 21.6.0 Darwin Kernel Version 21.6.0: Fri Sep 15 16:17:23 PDT 2023; root:xnu-8020.240.18.703.5~1/RELEASE_X86_64 x86_64
Here is the flent output:
bash-3.2$ echo "IPv4" ; date ; ping -c 10 netperf-eu.bufferbloat.net ; ./run-flent --ipv4 -l 60 -H netperf-eu.bufferbloat.net rrul_var --remote-metadata=root@192.168.42.1 --test-parameter=cpu_stats_hosts=root@192.168.42.1 --step-size=.05 --test-parameter bidir_streams=8 --test-parameter markings=CS0,CS1,CS2,CS3,CS4,CS5,CS6,CS7 --test-parameter ping_hosts=1.1.1.1 -D . -t IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct45500of46719K-D090pct105000of116789K_hms-beagle2_en4TB2_TurrisOmnia-TurrisOS.6.3.2-pppoe-wan-eth2.7_2_bridged-FB7520-OpenWrt-r22256-14334c222e-OutdoorMSAN-VDSL100_2_netperf-eu.bufferbloat.net --log-file
IPv4
Sat Oct 14 21:19:24 CEST 2023
PING demo.tohojo.dk (193.10.227.30): 56 data bytes
64 bytes from 193.10.227.30: icmp_seq=0 ttl=48 time=24.259 ms
64 bytes from 193.10.227.30: icmp_seq=1 ttl=48 time=22.789 ms
64 bytes from 193.10.227.30: icmp_seq=2 ttl=48 time=24.097 ms
64 bytes from 193.10.227.30: icmp_seq=3 ttl=48 time=24.433 ms
64 bytes from 193.10.227.30: icmp_seq=4 ttl=48 time=24.342 ms
64 bytes from 193.10.227.30: icmp_seq=5 ttl=48 time=24.325 ms
64 bytes from 193.10.227.30: icmp_seq=6 ttl=48 time=23.456 ms
64 bytes from 193.10.227.30: icmp_seq=7 ttl=48 time=22.628 ms
64 bytes from 193.10.227.30: icmp_seq=8 ttl=48 time=24.396 ms
64 bytes from 193.10.227.30: icmp_seq=9 ttl=48 time=23.521 ms
--- demo.tohojo.dk ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 22.628/23.825/24.433/0.649 ms
Starting Flent 2.1.1+git.7b931915 using Python 3.10.8.
Starting rrul_var test. Expected run time: 70 seconds.
Data file written to ./rrul_var-2023-10-14T211934.512820.IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct45500of46719K-D090pct105000of116789K_hms-beagle2_en4TB2_TurrisOmnia-TurrisOS_6_3_2-pppoe-wan-eth2_7_2_bridged-FB7520-OpenWrt-r22256-14334c222e-OutdoorMSAN-VDSL100.flent.gz
Summary of rrul_var test run from 2023-10-14 19:19:34.512820
Title: 'IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct45500of46719K-D090pct105000of116789K_hms-beagle2_en4TB2_TurrisOmnia-TurrisOS.6.3.2-pppoe-wan-eth2.7_2_bridged-FB7520-OpenWrt-r22256-14334c222e-OutdoorMSAN-VDSL100_2_netperf-eu.bufferbloat.net'
avg median 99th % # data pts
Ping (ms) ICMP 1.1.1.1 (extra) : 13.10 12.90 17.40 ms 1396
Ping (ms) avg : 27.07 N/A N/A ms 1639
Ping (ms)::ICMP : 26.15 26.10 30.70 ms 1399
Ping (ms)::UDP 0 (CS0) : 27.78 27.55 32.13 ms 1639
Ping (ms)::UDP 1 (CS1) : 29.34 29.51 35.09 ms 1639
Ping (ms)::UDP 2 (CS2) : 26.18 26.06 30.56 ms 1639
Ping (ms)::UDP 3 (CS3) : 26.22 26.06 30.77 ms 1639
Ping (ms)::UDP 4 (CS4) : 26.20 25.98 30.93 ms 1639
Ping (ms)::UDP 5 (CS5) : 26.20 26.06 31.33 ms 1639
Ping (ms)::UDP 6 (CS6) : 28.13 28.11 32.86 ms 1639
Ping (ms)::UDP 7 (CS7) : 26.51 26.44 31.39 ms 1639
TCP download avg : 11.33 N/A N/A Mbits/s 1639
TCP download sum : 90.62 N/A N/A Mbits/s 1639
TCP download::0 (CS0) : 11.39 11.25 17.48 Mbits/s 1639
TCP download::1 (CS1) : 10.66 10.21 28.97 Mbits/s 1639
TCP download::2 (CS2) : 12.17 11.99 18.87 Mbits/s 1639
TCP download::3 (CS3) : 11.29 11.06 18.27 Mbits/s 1639
TCP download::4 (CS4) : 11.87 11.34 23.51 Mbits/s 1639
TCP download::5 (CS5) : 10.51 10.49 16.58 Mbits/s 1639
TCP download::6 (CS6) : 11.37 11.15 18.32 Mbits/s 1639
TCP download::7 (CS7) : 11.36 11.25 16.99 Mbits/s 1639
TCP totals : 130.00 N/A N/A Mbits/s 1639
TCP upload avg : 4.92 N/A N/A Mbits/s 1639
TCP upload sum : 39.38 N/A N/A Mbits/s 1639
TCP upload::0 (CS0) : 5.50 5.48 7.94 Mbits/s 1639
TCP upload::1 (CS1) : 2.13 2.19 2.64 Mbits/s 1639
TCP upload::2 (CS2) : 5.47 5.47 7.13 Mbits/s 1639
TCP upload::3 (CS3) : 5.47 5.48 6.97 Mbits/s 1639
TCP upload::4 (CS4) : 5.47 5.49 7.23 Mbits/s 1639
TCP upload::5 (CS5) : 5.46 5.46 6.83 Mbits/s 1639
TCP upload::6 (CS6) : 4.96 4.95 6.25 Mbits/s 1639
TCP upload::7 (CS7) : 4.92 4.94 6.18 Mbits/s 1639
cpu_stats_root@192.168.42.1::load : 0.40 0.42 0.83 1243
bash-3.2$
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Bloat] [Codel] fq_"codel" in apple's gear
2023-10-14 19:44 ` Sebastian Moeller
@ 2023-10-14 19:46 ` Dave Taht
2023-10-15 13:09 ` Sebastian Moeller
0 siblings, 1 reply; 4+ messages in thread
From: Dave Taht @ 2023-10-14 19:46 UTC (permalink / raw)
To: Sebastian Moeller; +Cc: Jonathan Morton, bloat, codel
thank you! A packet capture, however, would be more revealing...
On Sat, Oct 14, 2023 at 12:44 PM Sebastian Moeller <moeller0@gmx.de> wrote:
>
> Hi Dave,
>
>
> > On Oct 14, 2023, at 21:01, Dave Taht <dave.taht@gmail.com> wrote:
> >
> > netstat -I en0 -qq
>
> Here is a bit of output from macosx monterey 12.7 (on intel hardware). I know this is not really recent, ut it is the most recent I have available...
>
> This is on an realtek based USB3 ethernet dongle that used to support 1 Gbps, but at from macosx update on it is stuck on 100Mbps FastEthernet...
>
>
> Here is the output after a flent test:
>
> Out of my "--test-parameter markings=CS0,CS1,CS2,CS3,CS4,CS5,CS6,CS7" it seems only CS1 was honored and put into BK the rest all ended up in BE...
> I have zero clue what macos uses to steer packets into the different classes, might be DSCP alone, but then at least using some more exotic ones than boring CS6/CS7
> No idea what CTL is, the average packet size mist be around 67 bytes...
>
>
> macbook:CAKE-autorate user$ netstat -I en3 -qq
> en3:
> [ sched: FQ_CODEL qlength: 0/128 ]
> [ pkts: 522515 bytes: 334464372 dropped pkts: 0 bytes: 0 ]
> =====================================================
> [ pri: CTL (0) srv_cl: 0x480190 quantum: 605 drr_max: 8 ]
> [ queued pkts: 0 bytes: 0 ]
> [ dequeued pkts: 292918 bytes: 19787064 ]
> [ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
> [ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
> [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
> [ flows total: 0 new: 0 old: 0 ]
> [ throttle on: 0 off: 0 drop: 0 ]
> [ compressible pkts: 0 compressed pkts: 0]
> =====================================================
> [ pri: VO (1) srv_cl: 0x400180 quantum: 605 drr_max: 8 ]
> [ queued pkts: 0 bytes: 0 ]
> [ dequeued pkts: 0 bytes: 0 ]
> [ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
> [ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
> [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
> [ flows total: 0 new: 0 old: 0 ]
> [ throttle on: 0 off: 0 drop: 0 ]
> [ compressible pkts: 0 compressed pkts: 0]
> =====================================================
> [ pri: VI (2) srv_cl: 0x380100 quantum: 3028 drr_max: 6 ]
> [ queued pkts: 0 bytes: 0 ]
> [ dequeued pkts: 0 bytes: 0 ]
> [ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
> [ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
> [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
> [ flows total: 0 new: 0 old: 0 ]
> [ throttle on: 0 off: 0 drop: 0 ]
> [ compressible pkts: 0 compressed pkts: 0]
> =====================================================
> [ pri: RV (3) srv_cl: 0x300110 quantum: 3028 drr_max: 6 ]
> [ queued pkts: 0 bytes: 0 ]
> [ dequeued pkts: 0 bytes: 0 ]
> [ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
> [ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
> [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
> [ flows total: 0 new: 0 old: 0 ]
> [ throttle on: 0 off: 0 drop: 0 ]
> [ compressible pkts: 0 compressed pkts: 0]
> =====================================================
> [ pri: AV (4) srv_cl: 0x280120 quantum: 3028 drr_max: 6 ]
> [ queued pkts: 0 bytes: 0 ]
> [ dequeued pkts: 0 bytes: 0 ]
> [ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
> [ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
> [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
> [ flows total: 0 new: 0 old: 0 ]
> [ throttle on: 0 off: 0 drop: 0 ]
> [ compressible pkts: 0 compressed pkts: 0]
> =====================================================
> [ pri: OAM (5) srv_cl: 0x200020 quantum: 1514 drr_max: 4 ]
> [ queued pkts: 0 bytes: 0 ]
> [ dequeued pkts: 0 bytes: 0 ]
> [ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
> [ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
> [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
> [ flows total: 0 new: 0 old: 0 ]
> [ throttle on: 0 off: 0 drop: 0 ]
> [ compressible pkts: 0 compressed pkts: 0]
> =====================================================
> [ pri: RD (6) srv_cl: 0x180010 quantum: 1514 drr_max: 4 ]
> [ queued pkts: 0 bytes: 0 ]
> [ dequeued pkts: 13 bytes: 6684 ]
> [ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
> [ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
> [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
> [ flows total: 0 new: 0 old: 0 ]
> [ throttle on: 0 off: 0 drop: 0 ]
> [ compressible pkts: 0 compressed pkts: 0]
> =====================================================
> [ pri: BE (7) srv_cl: 0x0 quantum: 1514 drr_max: 4 ]
> [ queued pkts: 0 bytes: 0 ]
> [ dequeued pkts: 225368 bytes: 314071428 ]
> [ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
> [ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
> [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
> [ flows total: 0 new: 0 old: 0 ]
> [ throttle on: 0 off: 0 drop: 0 ]
> [ compressible pkts: 0 compressed pkts: 0]
> =====================================================
> [ pri: BK (8) srv_cl: 0x100080 quantum: 1514 drr_max: 2 ]
> [ queued pkts: 0 bytes: 0 ]
> [ dequeued pkts: 3831 bytes: 576846 ]
> [ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
> [ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
> [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
> [ flows total: 0 new: 0 old: 0 ]
> [ throttle on: 0 off: 0 drop: 0 ]
> [ compressible pkts: 0 compressed pkts: 0]
> =====================================================
> [ pri: BK_SYS (9) srv_cl: 0x80090 quantum: 1514 drr_max: 2 ]
> [ queued pkts: 0 bytes: 0 ]
> [ dequeued pkts: 385 bytes: 22350 ]
> [ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
> [ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
> [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
> [ flows total: 0 new: 0 old: 0 ]
> [ throttle on: 0 off: 0 drop: 0 ]
> [ compressible pkts: 375 compressed pkts: 0]
> macbook:CAKE-autorate user$ uname -a
> Darwin macbook.local 21.6.0 Darwin Kernel Version 21.6.0: Fri Sep 15 16:17:23 PDT 2023; root:xnu-8020.240.18.703.5~1/RELEASE_X86_64 x86_64
>
>
> Here is the flent output:
>
> bash-3.2$ echo "IPv4" ; date ; ping -c 10 netperf-eu.bufferbloat.net ; ./run-flent --ipv4 -l 60 -H netperf-eu.bufferbloat.net rrul_var --remote-metadata=root@192.168.42.1 --test-parameter=cpu_stats_hosts=root@192.168.42.1 --step-size=.05 --test-parameter bidir_streams=8 --test-parameter markings=CS0,CS1,CS2,CS3,CS4,CS5,CS6,CS7 --test-parameter ping_hosts=1.1.1.1 -D . -t IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct45500of46719K-D090pct105000of116789K_hms-beagle2_en4TB2_TurrisOmnia-TurrisOS.6.3.2-pppoe-wan-eth2.7_2_bridged-FB7520-OpenWrt-r22256-14334c222e-OutdoorMSAN-VDSL100_2_netperf-eu.bufferbloat.net --log-file
> IPv4
> Sat Oct 14 21:19:24 CEST 2023
> PING demo.tohojo.dk (193.10.227.30): 56 data bytes
> 64 bytes from 193.10.227.30: icmp_seq=0 ttl=48 time=24.259 ms
> 64 bytes from 193.10.227.30: icmp_seq=1 ttl=48 time=22.789 ms
> 64 bytes from 193.10.227.30: icmp_seq=2 ttl=48 time=24.097 ms
> 64 bytes from 193.10.227.30: icmp_seq=3 ttl=48 time=24.433 ms
> 64 bytes from 193.10.227.30: icmp_seq=4 ttl=48 time=24.342 ms
> 64 bytes from 193.10.227.30: icmp_seq=5 ttl=48 time=24.325 ms
> 64 bytes from 193.10.227.30: icmp_seq=6 ttl=48 time=23.456 ms
> 64 bytes from 193.10.227.30: icmp_seq=7 ttl=48 time=22.628 ms
> 64 bytes from 193.10.227.30: icmp_seq=8 ttl=48 time=24.396 ms
> 64 bytes from 193.10.227.30: icmp_seq=9 ttl=48 time=23.521 ms
>
> --- demo.tohojo.dk ping statistics ---
> 10 packets transmitted, 10 packets received, 0.0% packet loss
> round-trip min/avg/max/stddev = 22.628/23.825/24.433/0.649 ms
> Starting Flent 2.1.1+git.7b931915 using Python 3.10.8.
> Starting rrul_var test. Expected run time: 70 seconds.
> Data file written to ./rrul_var-2023-10-14T211934.512820.IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct45500of46719K-D090pct105000of116789K_hms-beagle2_en4TB2_TurrisOmnia-TurrisOS_6_3_2-pppoe-wan-eth2_7_2_bridged-FB7520-OpenWrt-r22256-14334c222e-OutdoorMSAN-VDSL100.flent.gz
>
> Summary of rrul_var test run from 2023-10-14 19:19:34.512820
> Title: 'IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct45500of46719K-D090pct105000of116789K_hms-beagle2_en4TB2_TurrisOmnia-TurrisOS.6.3.2-pppoe-wan-eth2.7_2_bridged-FB7520-OpenWrt-r22256-14334c222e-OutdoorMSAN-VDSL100_2_netperf-eu.bufferbloat.net'
>
> avg median 99th % # data pts
> Ping (ms) ICMP 1.1.1.1 (extra) : 13.10 12.90 17.40 ms 1396
> Ping (ms) avg : 27.07 N/A N/A ms 1639
> Ping (ms)::ICMP : 26.15 26.10 30.70 ms 1399
> Ping (ms)::UDP 0 (CS0) : 27.78 27.55 32.13 ms 1639
> Ping (ms)::UDP 1 (CS1) : 29.34 29.51 35.09 ms 1639
> Ping (ms)::UDP 2 (CS2) : 26.18 26.06 30.56 ms 1639
> Ping (ms)::UDP 3 (CS3) : 26.22 26.06 30.77 ms 1639
> Ping (ms)::UDP 4 (CS4) : 26.20 25.98 30.93 ms 1639
> Ping (ms)::UDP 5 (CS5) : 26.20 26.06 31.33 ms 1639
> Ping (ms)::UDP 6 (CS6) : 28.13 28.11 32.86 ms 1639
> Ping (ms)::UDP 7 (CS7) : 26.51 26.44 31.39 ms 1639
> TCP download avg : 11.33 N/A N/A Mbits/s 1639
> TCP download sum : 90.62 N/A N/A Mbits/s 1639
> TCP download::0 (CS0) : 11.39 11.25 17.48 Mbits/s 1639
> TCP download::1 (CS1) : 10.66 10.21 28.97 Mbits/s 1639
> TCP download::2 (CS2) : 12.17 11.99 18.87 Mbits/s 1639
> TCP download::3 (CS3) : 11.29 11.06 18.27 Mbits/s 1639
> TCP download::4 (CS4) : 11.87 11.34 23.51 Mbits/s 1639
> TCP download::5 (CS5) : 10.51 10.49 16.58 Mbits/s 1639
> TCP download::6 (CS6) : 11.37 11.15 18.32 Mbits/s 1639
> TCP download::7 (CS7) : 11.36 11.25 16.99 Mbits/s 1639
> TCP totals : 130.00 N/A N/A Mbits/s 1639
> TCP upload avg : 4.92 N/A N/A Mbits/s 1639
> TCP upload sum : 39.38 N/A N/A Mbits/s 1639
> TCP upload::0 (CS0) : 5.50 5.48 7.94 Mbits/s 1639
> TCP upload::1 (CS1) : 2.13 2.19 2.64 Mbits/s 1639
> TCP upload::2 (CS2) : 5.47 5.47 7.13 Mbits/s 1639
> TCP upload::3 (CS3) : 5.47 5.48 6.97 Mbits/s 1639
> TCP upload::4 (CS4) : 5.47 5.49 7.23 Mbits/s 1639
> TCP upload::5 (CS5) : 5.46 5.46 6.83 Mbits/s 1639
> TCP upload::6 (CS6) : 4.96 4.95 6.25 Mbits/s 1639
> TCP upload::7 (CS7) : 4.92 4.94 6.18 Mbits/s 1639
> cpu_stats_root@192.168.42.1::load : 0.40 0.42 0.83 1243
> bash-3.2$
>
>
>
--
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Bloat] [Codel] fq_"codel" in apple's gear
2023-10-14 19:46 ` Dave Taht
@ 2023-10-15 13:09 ` Sebastian Moeller
0 siblings, 0 replies; 4+ messages in thread
From: Sebastian Moeller @ 2023-10-15 13:09 UTC (permalink / raw)
To: Dave Täht; +Cc: Jonathan Morton, bloat, codel
Hi Dave,
> On Oct 14, 2023, at 21:46, Dave Taht <dave.taht@gmail.com> wrote:
>
> thank you! A packet capture, however, would be more revealing...
will try to get a packet capture later, when the rest of the users are a sleep...
But I want to re-iterate 12.7 is not a recent macos version by any account, so the data will not be all that extrapolatable...
Regards
Sebastian
>
> On Sat, Oct 14, 2023 at 12:44 PM Sebastian Moeller <moeller0@gmx.de> wrote:
>>
>> Hi Dave,
>>
>>
>>> On Oct 14, 2023, at 21:01, Dave Taht <dave.taht@gmail.com> wrote:
>>>
>>> netstat -I en0 -qq
>>
>> Here is a bit of output from macosx monterey 12.7 (on intel hardware). I know this is not really recent, ut it is the most recent I have available...
>>
>> This is on an realtek based USB3 ethernet dongle that used to support 1 Gbps, but at from macosx update on it is stuck on 100Mbps FastEthernet...
>>
>>
>> Here is the output after a flent test:
>>
>> Out of my "--test-parameter markings=CS0,CS1,CS2,CS3,CS4,CS5,CS6,CS7" it seems only CS1 was honored and put into BK the rest all ended up in BE...
>> I have zero clue what macos uses to steer packets into the different classes, might be DSCP alone, but then at least using some more exotic ones than boring CS6/CS7
>> No idea what CTL is, the average packet size mist be around 67 bytes...
>>
>>
>> macbook:CAKE-autorate user$ netstat -I en3 -qq
>> en3:
>> [ sched: FQ_CODEL qlength: 0/128 ]
>> [ pkts: 522515 bytes: 334464372 dropped pkts: 0 bytes: 0 ]
>> =====================================================
>> [ pri: CTL (0) srv_cl: 0x480190 quantum: 605 drr_max: 8 ]
>> [ queued pkts: 0 bytes: 0 ]
>> [ dequeued pkts: 292918 bytes: 19787064 ]
>> [ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
>> [ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
>> [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
>> [ flows total: 0 new: 0 old: 0 ]
>> [ throttle on: 0 off: 0 drop: 0 ]
>> [ compressible pkts: 0 compressed pkts: 0]
>> =====================================================
>> [ pri: VO (1) srv_cl: 0x400180 quantum: 605 drr_max: 8 ]
>> [ queued pkts: 0 bytes: 0 ]
>> [ dequeued pkts: 0 bytes: 0 ]
>> [ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
>> [ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
>> [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
>> [ flows total: 0 new: 0 old: 0 ]
>> [ throttle on: 0 off: 0 drop: 0 ]
>> [ compressible pkts: 0 compressed pkts: 0]
>> =====================================================
>> [ pri: VI (2) srv_cl: 0x380100 quantum: 3028 drr_max: 6 ]
>> [ queued pkts: 0 bytes: 0 ]
>> [ dequeued pkts: 0 bytes: 0 ]
>> [ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
>> [ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
>> [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
>> [ flows total: 0 new: 0 old: 0 ]
>> [ throttle on: 0 off: 0 drop: 0 ]
>> [ compressible pkts: 0 compressed pkts: 0]
>> =====================================================
>> [ pri: RV (3) srv_cl: 0x300110 quantum: 3028 drr_max: 6 ]
>> [ queued pkts: 0 bytes: 0 ]
>> [ dequeued pkts: 0 bytes: 0 ]
>> [ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
>> [ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
>> [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
>> [ flows total: 0 new: 0 old: 0 ]
>> [ throttle on: 0 off: 0 drop: 0 ]
>> [ compressible pkts: 0 compressed pkts: 0]
>> =====================================================
>> [ pri: AV (4) srv_cl: 0x280120 quantum: 3028 drr_max: 6 ]
>> [ queued pkts: 0 bytes: 0 ]
>> [ dequeued pkts: 0 bytes: 0 ]
>> [ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
>> [ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
>> [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
>> [ flows total: 0 new: 0 old: 0 ]
>> [ throttle on: 0 off: 0 drop: 0 ]
>> [ compressible pkts: 0 compressed pkts: 0]
>> =====================================================
>> [ pri: OAM (5) srv_cl: 0x200020 quantum: 1514 drr_max: 4 ]
>> [ queued pkts: 0 bytes: 0 ]
>> [ dequeued pkts: 0 bytes: 0 ]
>> [ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
>> [ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
>> [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
>> [ flows total: 0 new: 0 old: 0 ]
>> [ throttle on: 0 off: 0 drop: 0 ]
>> [ compressible pkts: 0 compressed pkts: 0]
>> =====================================================
>> [ pri: RD (6) srv_cl: 0x180010 quantum: 1514 drr_max: 4 ]
>> [ queued pkts: 0 bytes: 0 ]
>> [ dequeued pkts: 13 bytes: 6684 ]
>> [ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
>> [ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
>> [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
>> [ flows total: 0 new: 0 old: 0 ]
>> [ throttle on: 0 off: 0 drop: 0 ]
>> [ compressible pkts: 0 compressed pkts: 0]
>> =====================================================
>> [ pri: BE (7) srv_cl: 0x0 quantum: 1514 drr_max: 4 ]
>> [ queued pkts: 0 bytes: 0 ]
>> [ dequeued pkts: 225368 bytes: 314071428 ]
>> [ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
>> [ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
>> [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
>> [ flows total: 0 new: 0 old: 0 ]
>> [ throttle on: 0 off: 0 drop: 0 ]
>> [ compressible pkts: 0 compressed pkts: 0]
>> =====================================================
>> [ pri: BK (8) srv_cl: 0x100080 quantum: 1514 drr_max: 2 ]
>> [ queued pkts: 0 bytes: 0 ]
>> [ dequeued pkts: 3831 bytes: 576846 ]
>> [ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
>> [ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
>> [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
>> [ flows total: 0 new: 0 old: 0 ]
>> [ throttle on: 0 off: 0 drop: 0 ]
>> [ compressible pkts: 0 compressed pkts: 0]
>> =====================================================
>> [ pri: BK_SYS (9) srv_cl: 0x80090 quantum: 1514 drr_max: 2 ]
>> [ queued pkts: 0 bytes: 0 ]
>> [ dequeued pkts: 385 bytes: 22350 ]
>> [ budget: 0 target qdelay: 10.00 msec update interval:100.00 msec ]
>> [ flow control: 0 feedback: 0 stalls: 0 failed: 0 overwhelming: 0 ]
>> [ drop overflow: 0 early: 0 memfail: 0 duprexmt:0 ]
>> [ flows total: 0 new: 0 old: 0 ]
>> [ throttle on: 0 off: 0 drop: 0 ]
>> [ compressible pkts: 375 compressed pkts: 0]
>> macbook:CAKE-autorate user$ uname -a
>> Darwin macbook.local 21.6.0 Darwin Kernel Version 21.6.0: Fri Sep 15 16:17:23 PDT 2023; root:xnu-8020.240.18.703.5~1/RELEASE_X86_64 x86_64
>>
>>
>> Here is the flent output:
>>
>> bash-3.2$ echo "IPv4" ; date ; ping -c 10 netperf-eu.bufferbloat.net ; ./run-flent --ipv4 -l 60 -H netperf-eu.bufferbloat.net rrul_var --remote-metadata=root@192.168.42.1 --test-parameter=cpu_stats_hosts=root@192.168.42.1 --step-size=.05 --test-parameter bidir_streams=8 --test-parameter markings=CS0,CS1,CS2,CS3,CS4,CS5,CS6,CS7 --test-parameter ping_hosts=1.1.1.1 -D . -t IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct45500of46719K-D090pct105000of116789K_hms-beagle2_en4TB2_TurrisOmnia-TurrisOS.6.3.2-pppoe-wan-eth2.7_2_bridged-FB7520-OpenWrt-r22256-14334c222e-OutdoorMSAN-VDSL100_2_netperf-eu.bufferbloat.net --log-file
>> IPv4
>> Sat Oct 14 21:19:24 CEST 2023
>> PING demo.tohojo.dk (193.10.227.30): 56 data bytes
>> 64 bytes from 193.10.227.30: icmp_seq=0 ttl=48 time=24.259 ms
>> 64 bytes from 193.10.227.30: icmp_seq=1 ttl=48 time=22.789 ms
>> 64 bytes from 193.10.227.30: icmp_seq=2 ttl=48 time=24.097 ms
>> 64 bytes from 193.10.227.30: icmp_seq=3 ttl=48 time=24.433 ms
>> 64 bytes from 193.10.227.30: icmp_seq=4 ttl=48 time=24.342 ms
>> 64 bytes from 193.10.227.30: icmp_seq=5 ttl=48 time=24.325 ms
>> 64 bytes from 193.10.227.30: icmp_seq=6 ttl=48 time=23.456 ms
>> 64 bytes from 193.10.227.30: icmp_seq=7 ttl=48 time=22.628 ms
>> 64 bytes from 193.10.227.30: icmp_seq=8 ttl=48 time=24.396 ms
>> 64 bytes from 193.10.227.30: icmp_seq=9 ttl=48 time=23.521 ms
>>
>> --- demo.tohojo.dk ping statistics ---
>> 10 packets transmitted, 10 packets received, 0.0% packet loss
>> round-trip min/avg/max/stddev = 22.628/23.825/24.433/0.649 ms
>> Starting Flent 2.1.1+git.7b931915 using Python 3.10.8.
>> Starting rrul_var test. Expected run time: 70 seconds.
>> Data file written to ./rrul_var-2023-10-14T211934.512820.IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct45500of46719K-D090pct105000of116789K_hms-beagle2_en4TB2_TurrisOmnia-TurrisOS_6_3_2-pppoe-wan-eth2_7_2_bridged-FB7520-OpenWrt-r22256-14334c222e-OutdoorMSAN-VDSL100.flent.gz
>>
>> Summary of rrul_var test run from 2023-10-14 19:19:34.512820
>> Title: 'IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct45500of46719K-D090pct105000of116789K_hms-beagle2_en4TB2_TurrisOmnia-TurrisOS.6.3.2-pppoe-wan-eth2.7_2_bridged-FB7520-OpenWrt-r22256-14334c222e-OutdoorMSAN-VDSL100_2_netperf-eu.bufferbloat.net'
>>
>> avg median 99th % # data pts
>> Ping (ms) ICMP 1.1.1.1 (extra) : 13.10 12.90 17.40 ms 1396
>> Ping (ms) avg : 27.07 N/A N/A ms 1639
>> Ping (ms)::ICMP : 26.15 26.10 30.70 ms 1399
>> Ping (ms)::UDP 0 (CS0) : 27.78 27.55 32.13 ms 1639
>> Ping (ms)::UDP 1 (CS1) : 29.34 29.51 35.09 ms 1639
>> Ping (ms)::UDP 2 (CS2) : 26.18 26.06 30.56 ms 1639
>> Ping (ms)::UDP 3 (CS3) : 26.22 26.06 30.77 ms 1639
>> Ping (ms)::UDP 4 (CS4) : 26.20 25.98 30.93 ms 1639
>> Ping (ms)::UDP 5 (CS5) : 26.20 26.06 31.33 ms 1639
>> Ping (ms)::UDP 6 (CS6) : 28.13 28.11 32.86 ms 1639
>> Ping (ms)::UDP 7 (CS7) : 26.51 26.44 31.39 ms 1639
>> TCP download avg : 11.33 N/A N/A Mbits/s 1639
>> TCP download sum : 90.62 N/A N/A Mbits/s 1639
>> TCP download::0 (CS0) : 11.39 11.25 17.48 Mbits/s 1639
>> TCP download::1 (CS1) : 10.66 10.21 28.97 Mbits/s 1639
>> TCP download::2 (CS2) : 12.17 11.99 18.87 Mbits/s 1639
>> TCP download::3 (CS3) : 11.29 11.06 18.27 Mbits/s 1639
>> TCP download::4 (CS4) : 11.87 11.34 23.51 Mbits/s 1639
>> TCP download::5 (CS5) : 10.51 10.49 16.58 Mbits/s 1639
>> TCP download::6 (CS6) : 11.37 11.15 18.32 Mbits/s 1639
>> TCP download::7 (CS7) : 11.36 11.25 16.99 Mbits/s 1639
>> TCP totals : 130.00 N/A N/A Mbits/s 1639
>> TCP upload avg : 4.92 N/A N/A Mbits/s 1639
>> TCP upload sum : 39.38 N/A N/A Mbits/s 1639
>> TCP upload::0 (CS0) : 5.50 5.48 7.94 Mbits/s 1639
>> TCP upload::1 (CS1) : 2.13 2.19 2.64 Mbits/s 1639
>> TCP upload::2 (CS2) : 5.47 5.47 7.13 Mbits/s 1639
>> TCP upload::3 (CS3) : 5.47 5.48 6.97 Mbits/s 1639
>> TCP upload::4 (CS4) : 5.47 5.49 7.23 Mbits/s 1639
>> TCP upload::5 (CS5) : 5.46 5.46 6.83 Mbits/s 1639
>> TCP upload::6 (CS6) : 4.96 4.95 6.25 Mbits/s 1639
>> TCP upload::7 (CS7) : 4.92 4.94 6.18 Mbits/s 1639
>> cpu_stats_root@192.168.42.1::load : 0.40 0.42 0.83 1243
>> bash-3.2$
>>
>>
>>
>
>
> --
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2023-10-15 13:09 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <CAA93jw7+3nAiNieOFaeUhojYHPE3q33mhrN3zvdbJRePxzRzWA@mail.gmail.com>
[not found] ` <3E02B7F5-D0C6-4C33-AFA9-30A2E54B155F@gmail.com>
[not found] ` <CAA93jw5VcHe0dpSVFu9zXd58JUhNiA1Z=gVASohfHctHTC+EpA@mail.gmail.com>
2023-10-14 19:01 ` [Bloat] [Codel] fq_"codel" in apple's gear Dave Taht
2023-10-14 19:44 ` Sebastian Moeller
2023-10-14 19:46 ` Dave Taht
2023-10-15 13:09 ` Sebastian Moeller
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox