<div dir="ltr"><div>G'day Dave,</div><div><br></div><div>Regarding the devices under test with Byte Queue Limits (BQL).</div><div><br></div><div></div><div></div><div>lychee pi ( <a href="https://wiki.sipeed.com/hardware/en/lichee/th1520/lp4a.html">https://wiki.sipeed.com/hardware/en/lichee/th1520/lp4a.html</a> ) with driver "st_gmac" does NOT have BQL. ( not a great surprise )<br></div><div><br></div><div>Driver is:<br></div><div><a href="https://github.com/randomizedcoder/cake/blob/main/device_info/lpi4a/ethtool_driver_end0">https://github.com/randomizedcoder/cake/blob/main/device_info/lpi4a/ethtool_driver_end0</a> <br></div><div><br></div><div>das@lpi4a:~$ /usr/sbin/ethtool -i end0<br>driver: st_gmac <-----------<br>version: Jan_2016<br>firmware-version: <br>expansion-rom-version: <br>bus-info: <br>supports-statistics: yes<br>supports-test: no<br>supports-eeprom-access: no<br>supports-register-dump: yes<br>supports-priv-flags: no</div><div><br></div><div>Jetson-nano has driver r8169 and is also not BQL enabled.</div><div><a href="https://github.com/randomizedcoder/cake/blob/main/device_info/jetson-nano/ethtool_driver_eth0">https://github.com/randomizedcoder/cake/blob/main/device_info/jetson-nano/ethtool_driver_eth0</a></div><div><br></div><div>Based on doc here: <a href="https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/">https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/</a>, the current drivers with BQL are:<br></div><div><br></div><div>das@t:~/Downloads/linux$ find drivers/net/ -name '*.c' -print | \<br>xargs fgrep -l netdev_completed_queue<br>drivers/net/can/spi/mcp251xfd/mcp251xfd-tef.c<br>drivers/net/can/usb/etas_es58x/es58x_core.c<br>drivers/net/can/dev/length.c<br>drivers/net/ipa/ipa_gsi.c<br>drivers/net/wan/fsl_ucc_hdlc.c<br>drivers/net/ethernet/freescale/ucc_geth.c<br>drivers/net/ethernet/marvell/skge.c<br>drivers/net/ethernet/marvell/sky2.c<br>drivers/net/ethernet/qualcomm/emac/emac-mac.c<br>drivers/net/ethernet/socionext/netsec.c<br>drivers/net/ethernet/atheros/ag71xx.c<br>drivers/net/ethernet/mediatek/mtk_star_emac.c<br>drivers/net/ethernet/3com/3c59x.c<br>drivers/net/ethernet/broadcom/bcm4908_enet.c<br>drivers/net/ethernet/broadcom/bgmac.c<br>drivers/net/ethernet/broadcom/b44.c<br>drivers/net/ethernet/broadcom/bcm63xx_enet.c<br>drivers/net/ethernet/realtek/8139cp.c <------- close, but NOT r8169<br>drivers/net/ethernet/lantiq_xrx200.c<br>drivers/net/ethernet/via/via-rhine.c<br>drivers/net/ethernet/hisilicon/hix5hd2_gmac.c<br>drivers/net/ethernet/hisilicon/hip04_eth.c<br>drivers/net/ethernet/hisilicon/hisi_femac.c<br>drivers/net/ethernet/intel/e1000e/netdev.c<br>drivers/net/ethernet/intel/e1000/e1000_main.c<br>drivers/net/ethernet/nvidia/forcedeth.c<br></div><div><br></div><div>The st_gmac driver can also be seen here:<br></div><div><a href="https://github.com/torvalds/linux/blob/4f82870119a46b0d04d91ef4697ac4977a255a9d/drivers/net/ethernet/stmicro/stmmac/stmmac_ethtool.c#L26">https://github.com/torvalds/linux/blob/4f82870119a46b0d04d91ef4697ac4977a255a9d/drivers/net/ethernet/stmicro/stmmac/stmmac_ethtool.c#L26</a></div><div><br></div><div>I guess I can take a look at creating an MR or two (2).</div><div><br></div><div>Regards,</div><div>Dave Seddon<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Oct 24, 2023 at 9:27 AM dave seddon <<a href="mailto:dave.seddon.ca@gmail.com">dave.seddon.ca@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>G'day,</div><div><br></div><div>Just to make sure the results are repeatable, I ran the flent tests again for 600 seconds.<br></div><div><br></div><div>Directory here<br></div><div><a href="https://github.com/randomizedcoder/qdisc_results/tree/main/qdisc/2023-10-24T05%3A16%3A21" target="_blank">https://github.com/randomizedcoder/qdisc_results/tree/main/qdisc/2023-10-24T05%3A16%3A21</a></div><div><br></div><div>Example direct links to the flent files:<br></div><div><a href="https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-24T05%3A16%3A21/pi4/cake20/flent/test/16_flent/rrul-2023-10-24T061153.944938.2023-10-24T05_16_21_pi4_cake20.flent.gz" target="_blank">https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-24T05%3A16%3A21/pi4/cake20/flent/test/16_flent/rrul-2023-10-24T061153.944938.2023-10-24T05_16_21_pi4_cake20.flent.gz</a> <br></div><div></div><div><a href="https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-24T05%3A16%3A21/jetson-nano/fq_codel/flent/test/16_flent/rrul-2023-10-24T081332.189258.2023-10-24T05_16_21_jetson-nano_fq_codel.flent.gz" target="_blank">https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-24T05%3A16%3A21/jetson-nano/fq_codel/flent/test/16_flent/rrul-2023-10-24T081332.189258.2023-10-24T05_16_21_jetson-nano_fq_codel.flent.gz</a></div><div><br></div><div>I'm not experienced at reading these flent reports, but it looks to me like fq_codel has similar latency, but higher Mb/s than cake.<br></div><div><br></div><div>Jetson nano fq_codel<br></div><div><img src="cid:ii_lo4jafgv1" alt="image.png" width="578" height="361"></div><div><br></div><div><br></div><div>Jetson nano cake20</div><div><img src="cid:ii_lo4jc4mn2" alt="image.png" width="578" height="372"></div><div><br></div><div>The new network card arrived, so I need to work out how to integrate the latency injection. ( I kind of wish I had multiple switches, so I could do q-in-q, because this could dramatically simply the linux latency injecting bridge machine's config. )</div><div><br></div><div>Kind regards,</div><div>Dave Seddon<br></div><div><br><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Oct 23, 2023 at 1:31 PM dave seddon <<a href="mailto:dave.seddon.ca@gmail.com" target="_blank">dave.seddon.ca@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>G'day,</div><div><br></div><div>Dave Taht and I have had a couple of phone conversations now, and he's convinced me that rather than inserting the netem delay on each laptop, that latency should be added by a seperate device. To this end, I've got another little PC and a NIC coming, so that I can repeat all the tests with seperate latency injection.</div><div><br></div><div>However, I've also completed the flent tests with the laptops adding latency at each end.<br></div><div><br></div><div>Full test runs here:<br></div><div><a href="https://github.com/randomizedcoder/qdisc_results/tree/main/qdisc/2023-10-23T16%3A49%3A10" target="_blank">https://github.com/randomizedcoder/qdisc_results/tree/main/qdisc/2023-10-23T16%3A49%3A10</a></div><div><br></div><div>You can find the actual rrul flent .tar.gz results for each test.</div><div><br></div><div>e.g</div><div>Pi4 fq is here:</div><div><a href="https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-23T16%3A49%3A10/pi4/fq/flent/test/16_flent/rrul-2023-10-23T170016.068273.2023-10-23T16_49_10_pi4_fq.flent.gz" target="_blank">https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-23T16%3A49%3A10/pi4/fq/flent/test/16_flent/rrul-2023-10-23T170016.068273.2023-10-23T16_49_10_pi4_fq.flent.gz</a></div><div><br></div><div>Lychee Pi Risv with cake qdisc:<br></div><div><a href="https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-23T16%3A49%3A10/lpi4a/cake20/flent/test/16_flent/rrul-2023-10-23T201354.818316.2023-10-23T16_49_10_lpi4a_cake20.flent.gz" target="_blank">https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-23T16%3A49%3A10/lpi4a/cake20/flent/test/16_flent/rrul-2023-10-23T201354.818316.2023-10-23T16_49_10_lpi4a_cake20.flent.gz</a></div><div><br></div><div>Just take these with a grain of salt until the new latency injection is in place.</div><div><br></div><div>... I'll see if I can script up the generation of all the pretty graphs soon<br></div><div><br></div><div>Thanks,</div><div>Dave Seddon<br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Oct 15, 2023 at 8:11 AM dave seddon <<a href="mailto:dave.seddon.ca@gmail.com" target="_blank">dave.seddon.ca@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>G'day,</div><div><br></div><div>I've put more work into a test framework around the qdisc tests, but unfortunately flent doesn't work easily with Ubuntu LTS ( <a href="https://github.com/tohojo/flent/issues/232" target="_blank">https://github.com/tohojo/flent/issues/232</a>, which I think is an issue with flent parsing the fping output ).<br></div><div><br></div><div><div><div>Results and graphs in this sheet:<br></div><div><a href="https://docs.google.com/spreadsheets/d/1T59QwEdNwJFm4TgDFA_NY98gicOm8ABXKvDsSIMz9ag/edit#gid=1203641125" target="_blank">https://docs.google.com/spreadsheets/d/1T59QwEdNwJFm4TgDFA_NY98gicOm8ABXKvDsSIMz9ag/edit#gid=1203641125</a></div></div></div><div><br></div><div>Raw results of x2 test runs are here:<br></div><div><a href="https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/report.csv" target="_blank">https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/report.csv</a></div><div><br></div><div>Each run:<br></div><div><a href="https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/report.csv" target="_blank">https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/report.csv</a></div><div><a href="https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-14T14%3A22%3A53/report.csv" target="_blank">https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-14T14%3A22%3A53/report.csv</a></div><div><br></div><div>Full iperf outputs are available too, for example: <a href="https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/nanopi-r2s/fq_codel/iperf/test/16_iperf/stdout" target="_blank">https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/nanopi-r2s/fq_codel/iperf/test/16_iperf/stdout </a><br></div><div><br></div><div>Logs for each run are also available, for example: <a href="https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/log.json" target="_blank">https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/log.json</a></div><div><br></div><div>The code repo updated here: <a href="https://github.com/randomizedcoder/cake" target="_blank">https://github.com/randomizedcoder/cake</a> , with the<a href="https://github.com/randomizedcoder/cake/blob/main/README.md" target="_blank">https://github.com/randomizedcoder/cake/blob/main/README.md</a> which explains how the test work.</div><div>Updated google doc is started here: <a href="https://docs.google.com/document/d/1fYKj3BS89aB9drg_DsSq289xSdVQhn1zUJYCj0WuCs0/edit?usp=sharing" target="_blank">https://docs.google.com/document/d/1fYKj3BS89aB9drg_DsSq289xSdVQhn1zUJYCj0WuCs0/edit?usp=sharing</a></div><div><br></div><div>Based on the questions on this list earlier, there is a folder with device information for each of the devices<br></div><div><a href="https://github.com/randomizedcoder/cake/tree/main/device_info" target="_blank">https://github.com/randomizedcoder/cake/tree/main/device_info</a></div><div><br></div><div>For example, the Pi4 and the Lichee Pi (risc-v) hardware layout is here:</div><div>- <a href="https://github.com/randomizedcoder/cake/blob/main/device_info/pi4/hwloc-ls-pi4.png" target="_blank">https://github.com/randomizedcoder/cake/blob/main/device_info/pi4/hwloc-ls-pi4.png <br></a></div><div>- <a href="https://github.com/randomizedcoder/cake/blob/main/device_info/lpi4a/hwloc-ls-lpi4a.png" target="_blank">https://github.com/randomizedcoder/cake/blob/main/device_info/lpi4a/hwloc-ls-lpi4a.png</a><br></div><div><br></div><div>The switch has also been upgraded to a Cisco 3750x, which I think based on the "show interface" output has a max queue size of 40 frames. The test process clears the counters before each test and gathers the "show interface" output at the end.<br></div><div><br></div><div>The Lichee Pi 4A doesn't look good ( <a href="https://wiki.sipeed.com/hardware/en/lichee/th1520/lp4a.html" target="_blank">https://wiki.sipeed.com/hardware/en/lichee/th1520/lp4a.html</a> )<br></div><div><br></div><div><img src="cid:ii_lnrlip390" alt="image.png" width="578" height="356"><br></div><div>I really wish the flent was working, so I'll probably see if I can work out the parsing.</div><div><br></div><div>Thanks,</div><div>Dave Seddon<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Oct 13, 2023 at 10:25 AM dave seddon <<a href="mailto:dave.seddon.ca@gmail.com" target="_blank">dave.seddon.ca@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>My bad. There's a bug for this.... Looks like I have to downgrade fping<br></div><div><br></div><div><a href="https://github.com/tohojo/flent/issues/232" target="_blank">https://github.com/tohojo/flent/issues/232</a></div><div><a href="https://github.com/schweikert/fping/issues/203" target="_blank">https://github.com/schweikert/fping/issues/203</a></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Oct 13, 2023 at 8:59 AM dave seddon <<a href="mailto:dave.seddon.ca@gmail.com" target="_blank">dave.seddon.ca@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>G'day,</div><div><br></div><div>I've been working away on automation of the tests. Pretty close to having much nicer tests with a lot more details. I've also got the risc-v device working.<br></div><div><br></div><div>However, I've run into something funny with flent. Flent is not happy with fping or ping.<br></div><div><br></div><div>das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ /usr/bin/sudo /usr/sbin/ip netns exec network101 /usr/bin/flent rrul --output /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/flent_pi4_noqueue.png --data-dir /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/ --format summary --plot all_scaled --title-extra 2023-10-13T15:53:21_pi4_noqueue --note 2023-10-13T15:53:21_pi4_noqueue --extended-metadata --host 172.17.51.10 --length 60 --ipv4 --socket-stats<br>Starting Flent 2.0.1 using Python 3.10.12.<br>Starting rrul test. Expected run time: 70 seconds.<br>WARNING: Found fping, but couldn't parse its output. Not using. <---------------- ???<br>ERROR: Runner Ping (ms) ICMP failed check: Cannot parse output of the system ping binary (/usr/bin/ping). Please install fping v3.5+. <----- ??</div><div><br></div><div>das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ dpkg --list | grep ping<br>ii fping 5.1-1 amd64 sends ICMP ECHO_REQUEST packets to network hosts<br>ii iputils-ping 3:20211215-1 amd64 Tools to test the reachability of network hosts<br>ii kpartx 0.8.8-1ubuntu1.22.04.1 amd64 create device mappings for partitions<br>ii libharfbuzz0b:amd64 2.7.4-1ubuntu3.1 amd64 OpenType text shaping engine (shared library)<br>das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ fping --version<br>fping: Version 5.1</div><div>das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ ping -V<br>ping from iputils 20211215</div><div><br></div><div>das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ cat /etc/lsb-release <br>DISTRIB_ID=Ubuntu<br>DISTRIB_RELEASE=22.04<br>DISTRIB_CODENAME=jammy<br>DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"</div><div><br></div><div>I did install via "apt install fping"<br></div><div><br></div><div>Any thoughts please?</div><div><br></div><div>Kind regards,</div><div>Dave<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Sep 28, 2023 at 6:27 AM Sebastian Moeller via Cake <<a href="mailto:cake@lists.bufferbloat.net" target="_blank">cake@lists.bufferbloat.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
<br>
> On Sep 28, 2023, at 15:19, David Lang <<a href="mailto:david@lang.hm" target="_blank">david@lang.hm</a>> wrote:<br>
> <br>
> On Thu, 28 Sep 2023, Sebastian Moeller via Cake wrote:<br>
> <br>
>> P.S.: I am tempted, but will likely wait until they are available in quantity and hope that the street price comes down a bit before getting one ;)<br>
> <br>
> They aren't available at all yet, and it's not clear when they will be available.<br>
<br>
The announcement was end of October, but I think I could pre-order right now if I was feeling an urge. You are right though, announced != available or delivered.<br>
<br>
Regards<br>
Sebastian<br>
<br>
P.S.: I have a pi400 in use as "desktop" for my oldest kid, this is close to be actually generally usable, I would guess that changing a potential p500 from the pi400's 4GB to 8 GB together with the other imprivements the 5 brings might push it over the threshold into the truly useful category. Which probably means that either a potential pi500 will come late and probably with only 4 GB, but let's see how this works out now that the supply situation is less problematic.<br>
And I understand that there are other capable ARM based SoCs for homerouter/desktop duty, I just happen ot have a soft spot for the raspberry project ;)<br>
<br>
> <br>
> David Lang<br>
<br>
_______________________________________________<br>
Cake mailing list<br>
<a href="mailto:Cake@lists.bufferbloat.net" target="_blank">Cake@lists.bufferbloat.net</a><br>
<a href="https://lists.bufferbloat.net/listinfo/cake" rel="noreferrer" target="_blank">https://lists.bufferbloat.net/listinfo/cake</a><br>
</blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Regards,<br></div>Dave Seddon<br>+1 415 857 5102<br></div></div></div></div></div></div>
</blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Regards,<br></div>Dave Seddon<br>+1 415 857 5102<br></div></div></div></div></div></div>
</blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Regards,<br></div>Dave Seddon<br>+1 415 857 5102<br></div></div></div></div></div></div>
</blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Regards,<br></div>Dave Seddon<br>+1 415 857 5102<br></div></div></div></div></div></div>
</blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Regards,<br></div>Dave Seddon<br>+1 415 857 5102<br></div></div></div></div></div></div>
</blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Regards,<br></div>Dave Seddon<br>+1 415 857 5102<br></div></div></div></div></div></div>