[Cake] some comprehensive arm64 w/cake results

[Cake] some comprehensive arm64 w/cake results

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 336 bytes --]

A huge thanks to dave seddon for buckling down and doing some comprehensive
testing of a variety of arm64 gear!

https://docs.google.com/document/d/1HxIU_TEBI6xG9jRHlr8rzyyxFEN43zMcJXUFlRuhiUI/edit#heading=h.bpvv3vr500nw

-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

[-- Attachment #2: Type: text/html, Size: 810 bytes --]

Re: [Cake] some comprehensive arm64 w/cake results

[Jonathan Morton]

On Monday, 18 September 2023, Dave Taht via Cake wrote:
> A huge thanks to dave seddon for buckling down and doing some comprehensive
> testing of a variety of arm64 gear!
> 
> https://docs.google.com/document/d/1HxIU_TEBI6xG9jRHlr8rzyyxFEN43zMcJXUFlRuhiUI/edit#heading=h.bpvv3vr500nw

I'm really not sure it's valid to compare fq_codel on default settings (equivalent to rtt 100ms) to Cake on 1ms and 3ms rtt settings.  At such tight settings you're asking Cake to hold the queue down to a fifth of a millisecond, and we already know from L4S that's a fool's errand.  The internal overheads of the Linux kernel will give you delays of that order already.

At least include a run with Cake similarly on rtt 100ms, which is the default and what most Internet facing CPE would run.  You may well find that throughput improves significantly. 

  -  Jonathan Morton
Sent from my Sailfish device

Re: [Cake] some comprehensive arm64 w/cake results

[David P. Reed]

I appreciate the effort that went into this testing. However, there are some signficant concerns regarding saying that this is typical ARM64 performance. (ARM64 easily beats many Intel x86_64 CPUs, if the motherboards are designed for that - I have a very nice SolidRun 16 core ARM64 board based on the NXP ARMs, and the Chinese make a lot of servers)

It should be noted that all the Pi's tested are configured with the *lousy* Ethernet interface that is standard on Pi's (USB). I don't know about the Jetsons (I own a Jetson, but have never tested its networking or even looked at the design of the I/O).

Now the Pi CM4 *module* is capable of connecting to a PCIe Ethernet adapter (using 1 lane PCIe gen 2 x1, which supports 4 Gb/sec transfers, enough for GigE wirespeed). Jeff Geerling has demonstrated various cards get superior performance.
https://www.hackster.io/news/jeff-geerling-squeezes-4-15gb-s-from-a-raspberry-pi-compute-module-4-using-a-pcie-network-card-bb373ca52966

https://www.tomshardware.com/news/raspberry-pi-compute-module-4-four-pcie-slots is a nice "motherboard" for the CM4 module that carries 4 PCIe slots.

So even the Pi CM4 should be able to beat up the Intel processors in a fair fight!

I should also note that Armbian's kernel isn't a particularly high performance kernel build.

Again, good job on getting results, but as somebody who has worked on measuring  Linux OS performance on various CPU architectures, I'd be very, very cautious about drawing conclusions about *ARM* from this.

If you want to test Cake on ARM64, you should perhaps set up an AWS ARM64 machine (Graviton CPUs, and good Ethernet adapters) which won't cost much money when you are charged by the hour.

The same caution applies to RISC-V systems. It's NOT the cpu architecture that you want to measure - it's tne networking architectures that are almost always crippled in some way on small boards.

Let's not reinforce the distortion that Intel is so great!

BTW, I don't know if a Pi CM4 with a couple of GigE ports would make a good home router at a reasonable price point. But it seems to me it could be a great candidate. It's a pretty fast machine and its network can definitely support two GigE ports.


On Sunday, September 17, 2023 9:05pm, "Dave Taht via Cake" <cake@lists.bufferbloat.net> said:

> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
> A huge thanks to dave seddon for buckling down and doing some comprehensive
> testing of a variety of arm64 gear!
> 
> https://docs.google.com/document/d/1HxIU_TEBI6xG9jRHlr8rzyyxFEN43zMcJXUFlRuhiUI/edit#heading=h.bpvv3vr500nw
> 
> --
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos
> 

Re: [Cake] some comprehensive arm64 w/cake results

[dave seddon]

[-- Attachment #1: Type: text/plain, Size: 4548 bytes --]

G'day Mr David Reed,

Thanks for the comments.

Definitely agree with your sentiments and the tests definitely do NOT
simply represent Intel verse ARM.

Perhaps I should have been more clear about the objectives of the testing:

I'm curious to understand the performance of these lower end SoC devices,
because these are the types of devices that act as home gateway routers, as
access points, and such.  There are many many millions of these devices out
there and I don't know how well understood their performance is:
e.g. How bad is my Spectrum Internet cable modem?
e.g. I have a Unifi security gateway and it's "smart queue" performance is
pretty poor ( <200 Mb/s ).  Why is it so poor?

Obviously, with real servers ( and even virtual AWS ones ) which have real
NICs, you get things like multi-queues with RSS, and a lot more tuning
knobs, and so they can go a lot faster.

In the tests so far, the Asus CN60 device with the r8169 performs pretty
well, where the NIC is likely to be contributing positively.  The default
configuration has a bunch of off-loading enabled:

root@asus-cn60-2:/home/das# ethtool --show-features enp1s0 | grep ": on"
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: on
tx-checksum-ipv6: on
generic-receive-offload: on
rx-vlan-offload: on
tx-vlan-offload: on
highdma: on [fixed]

However, based on these initial tests, which are not complete, it's
certainly curious that the Pi4 is doing ~923Mbit/s with pfifo_fast and then
doing significantly less ( ~621 Mbits/sec ) with cake.  I'm interested to
understand this in more detail, where DaveT has recommended adding 20ms or
40ms.  The cake tests so far had rtt 1ms and rtt 3ms, which might be too
low.  ( If it is too low, then maybe it would make sense to remove "rtt lan
= rtt 1ms" option, as it's a misleading configuration option? )

Definitely, during the testing these little devices have the NIC IRQs all
going through core 0, so I want to explore tuning options.

root@rpi4b:/home/das# cat /proc/interrupts | grep -E '(CPU0|eth0)'
           CPU0       CPU1       CPU2       CPU3
 30:   38651749          0          0          0     GICv2 189 Level
eth0      <--- IRQs only going to CPU0
 31:   20418643          0          0          0     GICv2 190 Level
eth0

Some ideas include:
- Moving most processes of core0. e.g. Configure all the systemd slices NOT
to use core0, so core0 is essentially freed to only service the IRQs
- RPS (
https://www.kernel.org/doc/html/latest/networking/scaling.html#rps-receive-packet-steering
). e.g. Can the other cores get more involved?
- Tuning ideas from here:
https://github.com/leandromoreira/linux-network-performance-parameters.
Specifically, I was wondering about increasing netdev_budget sysctls.

The defaults are shown here

root@rpi4b:/home/das# sysctl -a | grep netdev_budget
net.core.netdev_budget = 300
net.core.netdev_budget_usecs = 8000

"Armbian's kernel isn't a particularly high performance kernel build."

Happy to discuss any recommended tuning.  Armbrian is very easy to install
on the microSD card.  ( Actually, I have the LicheePi 4A RISC-V, but can't
find a easy image to just load on a microSD card. )


Over the weekend, I reconfigured the testing setup using a lot more VLANs.
Now each device has ALL the different qdiscs configured on different VLANs
and IPs, allowing the iperf/flent tests to be run one after the other with
no need to change the qdiscs between tests.  I'm currently repeating every
combination of test, before adding the netem 20/40ms latency as DaveT
suggested.  ( Test take a while: 8 devices * 6 qdiscs = 48 tests, by 10
minute tests = 480 minutes = 8 hours )

Roughly the plan is:
1. Retest all combinations.  This is to confirm the starting position. <---
running now
2. Add netem latency 20 and 40ms, and retest all combinations.  I'm hoping
Pi4 cake performance will be closer to > 900 Mb/s
3. Apply some tuning options, and retest all combinations

Kind regards,
Dave Seddon

On Sun, Sep 17, 2023 at 6:05 PM Dave Taht <dave.taht@gmail.com> wrote:

>
> A huge thanks to dave seddon for buckling down and doing some
> comprehensive testing of a variety of arm64 gear!
>
>
> https://docs.google.com/document/d/1HxIU_TEBI6xG9jRHlr8rzyyxFEN43zMcJXUFlRuhiUI/edit#heading=h.bpvv3vr500nw
>
> --
> Oct 30:
> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos
>


-- 
Regards,
Dave Seddon
+1 415 857 5102

[-- Attachment #2: Type: text/html, Size: 6445 bytes --]

Re: [Cake] some comprehensive arm64 w/cake results

[David P. Reed]

On Monday, September 18, 2023 3:50pm, "dave seddon via Cake" <cake@lists.bufferbloat.net> said:

> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
> G'day Mr David Reed,
> 
> Thanks for the comments.
> 
> Definitely agree with your sentiments and the tests definitely do NOT
> simply represent Intel verse ARM.
> 
> Perhaps I should have been more clear about the objectives of the testing:

It's just an issue I'm sensitive to, because throughout my career I've read "Brand X is slow" when the test was actually testing something else. (An annoying post popped up on Medium today that claimed "WebAssembly doesn't speed up Web applications" based on a badly designed Linux Foundation-commissioned study that the poster misunderstood. The poster also seemed to think that running web applications using the laptop's cycles is bad compared to running web applications exclusively on the server in the cloud). This already had me in a sour mood.

> 
> I'm curious to understand the performance of these lower end SoC devices,
> because these are the types of devices that act as home gateway routers, as
> access points, and such.  There are many many millions of these devices out
> there and I don't know how well understood their performance is:
> e.g. How bad is my Spectrum Internet cable modem?
> e.g. I have a Unifi security gateway and it's "smart queue" performance is
> pretty poor ( <200 Mb/s ).  Why is it so poor?

I'm curious, too! We know that on older home routers, with really slow MIPS processors, Cake struggles with GigE. As these old MIPS designs get phased out and replaced by ARM designs, it will matter.
Raspberry Pi 4's just aren't very good at networking because of their I/O architecture on the board, just as they are slow at USB in general. That's why the CM4 is interesting. It's interesting that the PiHole has gotten so popular - it would run better on an Pi with a better network architecture.

> 
> Obviously, with real servers ( and even virtual AWS ones ) which have real
> NICs, you get things like multi-queues with RSS, and a lot more tuning
> knobs, and so they can go a lot faster.
> 
> In the tests so far, the Asus CN60 device with the r8169 performs pretty
> well, where the NIC is likely to be contributing positively.  The default
> configuration has a bunch of off-loading enabled:
> 
> root@asus-cn60-2:/home/das# ethtool --show-features enp1s0 | grep ": on"
> rx-checksumming: on
> tx-checksumming: on
> tx-checksum-ipv4: on
> tx-checksum-ipv6: on
> generic-receive-offload: on
> rx-vlan-offload: on
> tx-vlan-offload: on
> highdma: on [fixed]
> 
> However, based on these initial tests, which are not complete, it's
> certainly curious that the Pi4 is doing ~923Mbit/s with pfifo_fast and then
> doing significantly less ( ~621 Mbits/sec ) with cake.  I'm interested to
> understand this in more detail, where DaveT has recommended adding 20ms or
> 40ms.  The cake tests so far had rtt 1ms and rtt 3ms, which might be too
> low.  ( If it is too low, then maybe it would make sense to remove "rtt lan
> = rtt 1ms" option, as it's a misleading configuration option? )
> 
> Definitely, during the testing these little devices have the NIC IRQs all
> going through core 0, so I want to explore tuning options.
> 
> root@rpi4b:/home/das# cat /proc/interrupts | grep -E '(CPU0|eth0)'
>            CPU0       CPU1       CPU2       CPU3
>  30:   38651749          0          0          0     GICv2 189 Level
> eth0      <--- IRQs only going to CPU0
>  31:   20418643          0          0          0     GICv2 190 Level
> eth0
> 
> Some ideas include:
> - Moving most processes of core0. e.g. Configure all the systemd slices NOT
> to use core0, so core0 is essentially freed to only service the IRQs
> - RPS (
> https://www.kernel.org/doc/html/latest/networking/scaling.html#rps-receive-packet-steering
> ). e.g. Can the other cores get more involved?
> - Tuning ideas from here:
> https://github.com/leandromoreira/linux-network-performance-parameters.
> Specifically, I was wondering about increasing netdev_budget sysctls.
> 
> The defaults are shown here
> 
> root@rpi4b:/home/das# sysctl -a | grep netdev_budget
> net.core.netdev_budget = 300
> net.core.netdev_budget_usecs = 8000
> 
> "Armbian's kernel isn't a particularly high performance kernel build."
> 
> Happy to discuss any recommended tuning.  Armbrian is very easy to install
> on the microSD card.  ( Actually, I have the LicheePi 4A RISC-V, but can't
> find a easy image to just load on a microSD card. )
> 
> 
> Over the weekend, I reconfigured the testing setup using a lot more VLANs.
> Now each device has ALL the different qdiscs configured on different VLANs
> and IPs, allowing the iperf/flent tests to be run one after the other with
> no need to change the qdiscs between tests.  I'm currently repeating every
> combination of test, before adding the netem 20/40ms latency as DaveT
> suggested.  ( Test take a while: 8 devices * 6 qdiscs = 48 tests, by 10
> minute tests = 480 minutes = 8 hours )
> 
> Roughly the plan is:
> 1. Retest all combinations.  This is to confirm the starting position. <---
> running now
> 2. Add netem latency 20 and 40ms, and retest all combinations.  I'm hoping
> Pi4 cake performance will be closer to > 900 Mb/s
> 3. Apply some tuning options, and retest all combinations
>
I'm very interested in seeing your results after this.
Grat job so far.
 
> Kind regards,
> Dave Seddon
> 
> On Sun, Sep 17, 2023 at 6:05 PM Dave Taht <dave.taht@gmail.com> wrote:
> 
>>
>> A huge thanks to dave seddon for buckling down and doing some
>> comprehensive testing of a variety of arm64 gear!
>>
>>
>> https://docs.google.com/document/d/1HxIU_TEBI6xG9jRHlr8rzyyxFEN43zMcJXUFlRuhiUI/edit#heading=h.bpvv3vr500nw
>>
>> --
>> Oct 30:
>> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
>> Dave Täht CSO, LibreQos
>>
> 
> 
> --
> Regards,
> Dave Seddon
> +1 415 857 5102
> 

Re: [Cake] some comprehensive arm64 w/cake results

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 1714 bytes --]

> On 18 Sep, 2023, at 11:24 pm, David P. Reed via Cake <cake@lists.bufferbloat.net> wrote:
> 
> I'm curious, too! We know that on older home routers, with really slow MIPS processors, Cake struggles with GigE. As these old MIPS designs get phased out and replaced by ARM designs, it will matter.

All qdiscs struggle with gigabit speeds on these SoCs.  It's because their networking architecture is optimised for passing packets straight through on the switch fabric, not pulling them into the CPU on the way.  The bandwidth to the CPU is comparable to an old PCI 32b 33MHz bus.  At 100Mbps or so each way, they can keep up just fine.

> Raspberry Pi 4's just aren't very good at networking because of their I/O architecture on the board, just as they are slow at USB in general. That's why the CM4 is interesting. It's interesting that the PiHole has gotten so popular - it would run better on an Pi with a better network architecture.

On the contrary, the Pi 4 has an excellent I/O architecture compared to most of its peers, and especially compared to the previous Pis.  The built-in NIC is internal to the SoC and *NOT* attached via USB any more, so it can genuinely support gigabit speeds.  The USB interface is also fast enough to support a second GigE NIC, though the latency wouldn't be as good as one attached over PCIe.  That's with a standard, off-the-shelf Pi 4B.

Here are some single-armed (just the internal NIC) results we ran a while ago.  Note that Cake is actually faster than fq_codel - or rather, Cake's internal shaper is faster than HTB shaping with an fq_codel leaf queue - and (not shown here) both are faster on the Pi 4 than on an x86-based APU2.



 - Jonathan Morton


[-- Attachment #2.1: Type: text/html, Size: 2463 bytes --]

[-- Attachment #2.2: Screenshot 2023-09-19 at 1.02.17 am.png --]
[-- Type: image/png, Size: 83584 bytes --]

Re: [Cake] some comprehensive arm64 w/cake results

[Jonathan Morton]

> On 18 Sep, 2023, at 10:50 pm, dave seddon via Cake <cake@lists.bufferbloat.net> wrote:
> 
> The cake tests so far had rtt 1ms and rtt 3ms, which might be too low.  ( If it is too low, then maybe it would make sense to remove "rtt lan = rtt 1ms" option, as it's a misleading configuration option? )

If all your traffic is over the LAN, and you have a machine and application tuned for the extra-low latencies that a LAN can offer, then setting LAN-grade targets for Cake might make sense.  But most people's traffic is a mixture, with the performance of Internet traffic being more important, and that is better served by the *default* settings.

You ran fq_codel at its default settings.  These are equivalent to Cake's default settings, so far as the AQM activity is concerned.  I'm just asking for a like-to-like comparison.  You could be pleasantly surprised.

 - Jonathan Morton

Re: [Cake] some comprehensive arm64 w/cake results

[dave seddon]

[-- Attachment #1: Type: text/plain, Size: 1477 bytes --]

Thanks Jonathan!

Curious and curiouser

I'd love to understand the difference between the tests I've been doing and
your tests.

   - How many TCP flows did you have please ( cake performance seems to
   drop significantly with increased number of TCP flows, although I need to
   do more testing to understand why )?
   - What was the RTT?
   - Load tool?
   - ... so many questions :)


On Mon, Sep 18, 2023 at 3:13 PM Jonathan Morton <chromatix99@gmail.com>
wrote:

> > On 18 Sep, 2023, at 10:50 pm, dave seddon via Cake <
> cake@lists.bufferbloat.net> wrote:
> >
> > The cake tests so far had rtt 1ms and rtt 3ms, which might be too low.
> ( If it is too low, then maybe it would make sense to remove "rtt lan = rtt
> 1ms" option, as it's a misleading configuration option? )
>
> If all your traffic is over the LAN, and you have a machine and
> application tuned for the extra-low latencies that a LAN can offer, then
> setting LAN-grade targets for Cake might make sense.  But most people's
> traffic is a mixture, with the performance of Internet traffic being more
> important, and that is better served by the *default* settings.
>
> You ran fq_codel at its default settings.  These are equivalent to Cake's
> default settings, so far as the AQM activity is concerned.  I'm just asking
> for a like-to-like comparison.  You could be pleasantly surprised.
>
>  - Jonathan Morton



-- 
Regards,
Dave Seddon
+1 415 857 5102

[-- Attachment #2: Type: text/html, Size: 2142 bytes --]

Re: [Cake] some comprehensive arm64 w/cake results

[Jonathan Morton]

> On 19 Sep, 2023, at 1:52 am, dave seddon <dave.seddon.ca@gmail.com> wrote:
> 
> I'd love to understand the difference between the tests I've been doing and your tests.
> 	• How many TCP flows did you have please ( cake performance seems to drop significantly with increased number of TCP flows, although I need to do more testing to understand why )?

It's almost certainly related to the extremely tight AQM settings you imposed on it, and on no other AQM you tested.

> 	• What was the RTT?

I think this was just in a LAN environment, to gauge the capabilities of the machine just as you're doing.  We insert a delay in the middle when testing other things, such as the performance and fine-scale behaviour of a qdisc/CC combination.

> 	• Load tool?

I believe Pete was using iperf3 for this, or possibly Flent which delegates to netperf.  These days he's developing something new along the same lines.

A simple test using one or more of Flent's standard tests should be enough to replicate these results, at least approximately.  I would also recommend using ECN, but you should get at least reasonable results without it, provided the AQM is set sanely.

 - Jonathan Morton

Re: [Cake] some comprehensive arm64 w/cake results

[Jonathan Morton]

> On 19 Sep, 2023, at 1:07 am, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> Raspberry Pi 4's just aren't very good at networking because of their I/O architecture on the board, just as they are slow at USB in general. That's why the CM4 is interesting. It's interesting that the PiHole has gotten so popular - it would run better on an Pi with a better network architecture.
> 
> On the contrary, the Pi 4 has an excellent I/O architecture compared to most of its peers, and especially compared to the previous Pis.  The built-in NIC is internal to the SoC and *NOT* attached via USB any more, so it can genuinely support gigabit speeds.  The USB interface is also fast enough to support a second GigE NIC, though the latency wouldn't be as good as one attached over PCIe.  That's with a standard, off-the-shelf Pi 4B.

Timely breaking news:  the Raspberry Pi 5 has just been announced.

The important new feature here (for us) is that it exposes a PCIe bus lane on the standard model, so you don't have to mess around with the Compute Module just to get access to that.  The built-in Ethernet port is now implemented in a PCIe-attached "southbridge" chip, and the WiFi performance has been improved by accelerating the interface by which the radio is attached.

On the downside, the price has gone up.

 - Jonathan Morton

Re: [Cake] some comprehensive arm64 w/cake results

[Sebastian Moeller]

Hi Jonathan

> On Sep 28, 2023, at 13:44, Jonathan Morton via Cake <cake@lists.bufferbloat.net> wrote:
> 
>> On 19 Sep, 2023, at 1:07 am, Jonathan Morton <chromatix99@gmail.com> wrote:
>> 
>>> Raspberry Pi 4's just aren't very good at networking because of their I/O architecture on the board, just as they are slow at USB in general. That's why the CM4 is interesting. It's interesting that the PiHole has gotten so popular - it would run better on an Pi with a better network architecture.
>> 
>> On the contrary, the Pi 4 has an excellent I/O architecture compared to most of its peers, and especially compared to the previous Pis.  The built-in NIC is internal to the SoC and *NOT* attached via USB any more, so it can genuinely support gigabit speeds.  The USB interface is also fast enough to support a second GigE NIC, though the latency wouldn't be as good as one attached over PCIe.  That's with a standard, off-the-shelf Pi 4B.
> 
> Timely breaking news:  the Raspberry Pi 5 has just been announced.
> 
> The important new feature here (for us) is that it exposes a PCIe bus lane on the standard model, so you don't have to mess around with the Compute Module just to get access to that.  The built-in Ethernet port is now implemented in a PCIe-attached "southbridge" chip, and the WiFi performance has been improved by accelerating the interface by which the radio is attached.

	[SM] More things on the plus side:
4 A76 cores up to 2.4 GHz (versus 4 A72 cores up to 1.8 Ghz)
4 x 512 KB L2 cache plus shared 2 MB L3 cache (versus 1 MB shared L2)
LPDDR4X-4266 memory (versus LPDDR4-2400)

This promises even better performance for loads like cake than the already pretty nifty pi4B


> 
> On the downside, the price has gone up.

	[SM] As well as the power consumption...


> 
> - Jonathan Morton
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] some comprehensive arm64 w/cake results

[Jonathan Morton]

> On 28 Sep, 2023, at 3:15 pm, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> This promises even better performance for loads like cake than the already pretty nifty pi4B

Well, increased computing performance is always welcome - but as I've said before, in most cases I don't think CPU performance is the limiting factor for CAKE.

When the CPU load goes up as networking throughput reaches the physical limit of the interface (or the I/O subsystem), what you're seeing is the CPU just spinning its wheels while waiting for a mutex to unblock.  Spinning faster doesn't make the mutex unblock sooner!

 - Jonathan Morton

Re: [Cake] some comprehensive arm64 w/cake results

[David Lang]

On Thu, 28 Sep 2023, Jonathan Morton via Cake wrote:

>> On 28 Sep, 2023, at 3:15 pm, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> This promises even better performance for loads like cake than the already pretty nifty pi4B
>
> Well, increased computing performance is always welcome - but as I've said before, in most cases I don't think CPU performance is the limiting factor for CAKE.
>
> When the CPU load goes up as networking throughput reaches the physical limit of the interface (or the I/O subsystem), what you're seeing is the CPU just spinning its wheels while waiting for a mutex to unblock.  Spinning faster doesn't make the mutex unblock sooner!

but a faster I/O system should help.

David Lang

Re: [Cake] some comprehensive arm64 w/cake results

[Sebastian Moeller]

Hi Jonathan,


> On Sep 28, 2023, at 14:33, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> On 28 Sep, 2023, at 3:15 pm, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> This promises even better performance for loads like cake than the already pretty nifty pi4B
> 
> Well, increased computing performance is always welcome - but as I've said before, in most cases I don't think CPU performance is the limiting factor for CAKE.
> 
> When the CPU load goes up as networking throughput reaches the physical limit of the interface (or the I/O subsystem), what you're seeing is the CPU just spinning its wheels while waiting for a mutex to unblock.  Spinning faster doesn't make the mutex unblock sooner!

	[SM] I think that the improvements of cache and memory hierarchy and throughput will be helpful, currently some people report odd issues with rpi4Bs depending on how many and which cores are used, I hope that the rpi5 ameliorates these issues. The gigabit ethernet adapter was already connected well to the SoC starting with the rpi4 when they ditched the USB2 bus used by earlier pis to connect the ethernet. But I agree it will require real benchmarks to see if the newer design truly delivers more cake performance and if yes, how much more. (And I also note that the rpi4B is not doing badly at 1 Gbps ethernet either, at least for "normal" numbers of parallel flows.
	Regarding a full dual (or triple) interface router, I have hopes that a PCIe connected NIC might be generally better than a USB3 dongle (even though USB3 on paper has plenty capacity for gigabit ethernet).

Regards
	Sebastian

P.S.: I am tempted, but will likely wait until they are available in quantity and hope that the street price comes down a bit before getting one ;)



> 
> - Jonathan Morton

Re: [Cake] some comprehensive arm64 w/cake results

[David Lang]

On Thu, 28 Sep 2023, Sebastian Moeller via Cake wrote:

> P.S.: I am tempted, but will likely wait until they are available in quantity and hope that the street price comes down a bit before getting one ;)

They aren't available at all yet, and it's not clear when they will be 
available.

David Lang

Re: [Cake] some comprehensive arm64 w/cake results

[Sebastian Moeller]

> On Sep 28, 2023, at 15:19, David Lang <david@lang.hm> wrote:
> 
> On Thu, 28 Sep 2023, Sebastian Moeller via Cake wrote:
> 
>> P.S.: I am tempted, but will likely wait until they are available in quantity and hope that the street price comes down a bit before getting one ;)
> 
> They aren't available at all yet, and it's not clear when they will be available.

	The announcement was end of October, but I think I could pre-order right now if I was feeling an urge. You are right though, announced != available or delivered.

Regards
	Sebastian

P.S.: I have a pi400 in use as "desktop" for my oldest kid, this is close to be actually generally usable, I would guess that changing a potential p500 from the pi400's 4GB to 8 GB together with the other imprivements the 5 brings might push it over the threshold into the truly useful category. Which probably means that either a potential pi500 will come late and probably with only 4 GB, but let's see how this works out now that the supply situation is less problematic.
And I understand that there are other capable ARM based SoCs for homerouter/desktop duty, I just happen ot have a soft spot for the raspberry project ;)

> 
> David Lang

Re: [Cake] some comprehensive arm64 w/cake results

[dave seddon]

[-- Attachment #1: Type: text/plain, Size: 3713 bytes --]

G'day,

I've been working away on automation of the tests.  Pretty close to having
much nicer tests with a lot more details.  I've also got the risc-v device
working.

However, I've run into something funny with flent.  Flent is not happy with
fping or ping.

das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ /usr/bin/sudo /usr/sbin/ip
netns exec network101 /usr/bin/flent rrul --output
 /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/flent_pi4_noqueue.png
--data-dir /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/
--format summary --plot all_scaled --title-extra
2023-10-13T15:53:21_pi4_noqueue --note 2023-10-13T15:53:21_pi4_noqueue
--extended-metadata --host 172.17.51.10 --length 60 --ipv4 --socket-stats
Starting Flent 2.0.1 using Python 3.10.12.
Starting rrul test. Expected run time: 70 seconds.
WARNING: Found fping, but couldn't parse its output. Not
using.              <---------------- ???
ERROR: Runner Ping (ms) ICMP failed check: Cannot parse output of the
system ping binary (/usr/bin/ping). Please install fping v3.5+.    <----- ??

das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ dpkg --list | grep ping
ii  fping                                 5.1-1
      amd64        sends ICMP ECHO_REQUEST packets to network hosts
ii  iputils-ping                          3:20211215-1
       amd64        Tools to test the reachability of network hosts
ii  kpartx                                0.8.8-1ubuntu1.22.04.1
       amd64        create device mappings for partitions
ii  libharfbuzz0b:amd64                   2.7.4-1ubuntu3.1
       amd64        OpenType text shaping engine (shared library)
das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ fping --version
fping: Version 5.1
das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ ping -V
ping from iputils 20211215

das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"

I did install via "apt install fping"

Any thoughts please?

Kind regards,
Dave

On Thu, Sep 28, 2023 at 6:27 AM Sebastian Moeller via Cake <
cake@lists.bufferbloat.net> wrote:

>
>
> > On Sep 28, 2023, at 15:19, David Lang <david@lang.hm> wrote:
> >
> > On Thu, 28 Sep 2023, Sebastian Moeller via Cake wrote:
> >
> >> P.S.: I am tempted, but will likely wait until they are available in
> quantity and hope that the street price comes down a bit before getting one
> ;)
> >
> > They aren't available at all yet, and it's not clear when they will be
> available.
>
>         The announcement was end of October, but I think I could pre-order
> right now if I was feeling an urge. You are right though, announced !=
> available or delivered.
>
> Regards
>         Sebastian
>
> P.S.: I have a pi400 in use as "desktop" for my oldest kid, this is close
> to be actually generally usable, I would guess that changing a potential
> p500 from the pi400's 4GB to 8 GB together with the other imprivements the
> 5 brings might push it over the threshold into the truly useful category.
> Which probably means that either a potential pi500 will come late and
> probably with only 4 GB, but let's see how this works out now that the
> supply situation is less problematic.
> And I understand that there are other capable ARM based SoCs for
> homerouter/desktop duty, I just happen ot have a soft spot for the
> raspberry project ;)
>
> >
> > David Lang
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>


-- 
Regards,
Dave Seddon
+1 415 857 5102

[-- Attachment #2: Type: text/html, Size: 5000 bytes --]

Re: [Cake] some comprehensive arm64 w/cake results

[dave seddon]

[-- Attachment #1: Type: text/plain, Size: 4165 bytes --]

My bad.  There's a bug for this.... Looks like I have to downgrade fping

https://github.com/tohojo/flent/issues/232
https://github.com/schweikert/fping/issues/203

On Fri, Oct 13, 2023 at 8:59 AM dave seddon <dave.seddon.ca@gmail.com>
wrote:

> G'day,
>
> I've been working away on automation of the tests.  Pretty close to having
> much nicer tests with a lot more details.  I've also got the risc-v device
> working.
>
> However, I've run into something funny with flent.  Flent is not happy
> with fping or ping.
>
> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ /usr/bin/sudo /usr/sbin/ip
> netns exec network101 /usr/bin/flent rrul --output
>  /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/flent_pi4_noqueue.png
> --data-dir /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/
> --format summary --plot all_scaled --title-extra
> 2023-10-13T15:53:21_pi4_noqueue --note 2023-10-13T15:53:21_pi4_noqueue
> --extended-metadata --host 172.17.51.10 --length 60 --ipv4 --socket-stats
> Starting Flent 2.0.1 using Python 3.10.12.
> Starting rrul test. Expected run time: 70 seconds.
> WARNING: Found fping, but couldn't parse its output. Not
> using.              <---------------- ???
> ERROR: Runner Ping (ms) ICMP failed check: Cannot parse output of the
> system ping binary (/usr/bin/ping). Please install fping v3.5+.    <----- ??
>
> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ dpkg --list | grep ping
> ii  fping                                 5.1-1
>         amd64        sends ICMP ECHO_REQUEST packets to network hosts
> ii  iputils-ping                          3:20211215-1
>        amd64        Tools to test the reachability of network hosts
> ii  kpartx                                0.8.8-1ubuntu1.22.04.1
>        amd64        create device mappings for partitions
> ii  libharfbuzz0b:amd64                   2.7.4-1ubuntu3.1
>        amd64        OpenType text shaping engine (shared library)
> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ fping --version
> fping: Version 5.1
> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ ping -V
> ping from iputils 20211215
>
> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=22.04
> DISTRIB_CODENAME=jammy
> DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
>
> I did install via "apt install fping"
>
> Any thoughts please?
>
> Kind regards,
> Dave
>
> On Thu, Sep 28, 2023 at 6:27 AM Sebastian Moeller via Cake <
> cake@lists.bufferbloat.net> wrote:
>
>>
>>
>> > On Sep 28, 2023, at 15:19, David Lang <david@lang.hm> wrote:
>> >
>> > On Thu, 28 Sep 2023, Sebastian Moeller via Cake wrote:
>> >
>> >> P.S.: I am tempted, but will likely wait until they are available in
>> quantity and hope that the street price comes down a bit before getting one
>> ;)
>> >
>> > They aren't available at all yet, and it's not clear when they will be
>> available.
>>
>>         The announcement was end of October, but I think I could
>> pre-order right now if I was feeling an urge. You are right though,
>> announced != available or delivered.
>>
>> Regards
>>         Sebastian
>>
>> P.S.: I have a pi400 in use as "desktop" for my oldest kid, this is close
>> to be actually generally usable, I would guess that changing a potential
>> p500 from the pi400's 4GB to 8 GB together with the other imprivements the
>> 5 brings might push it over the threshold into the truly useful category.
>> Which probably means that either a potential pi500 will come late and
>> probably with only 4 GB, but let's see how this works out now that the
>> supply situation is less problematic.
>> And I understand that there are other capable ARM based SoCs for
>> homerouter/desktop duty, I just happen ot have a soft spot for the
>> raspberry project ;)
>>
>> >
>> > David Lang
>>
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
>>
>
>
> --
> Regards,
> Dave Seddon
> +1 415 857 5102
>


-- 
Regards,
Dave Seddon
+1 415 857 5102

[-- Attachment #2: Type: text/html, Size: 5976 bytes --]

Re: [Cake] some comprehensive arm64 w/cake results

[dave seddon]

[-- Attachment #1.1: Type: text/plain, Size: 6738 bytes --]

G'day,

I've put more work into a test framework around the qdisc tests, but
unfortunately flent doesn't work easily with Ubuntu LTS (
https://github.com/tohojo/flent/issues/232, which I think is an issue with
flent parsing the fping output ).

Results and graphs in this sheet:
https://docs.google.com/spreadsheets/d/1T59QwEdNwJFm4TgDFA_NY98gicOm8ABXKvDsSIMz9ag/edit#gid=1203641125

Raw results of x2 test runs are here:
https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/report.csv

Each run:
https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/report.csv
https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-14T14%3A22%3A53/report.csv

Full iperf outputs are available too, for example:
https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/nanopi-r2s/fq_codel/iperf/test/16_iperf/stdout


Logs for each run are also available, for example:
https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/log.json

The code repo updated here: https://github.com/randomizedcoder/cake , with
thehttps://github.com/randomizedcoder/cake/blob/main/README.md which
explains how the test work.
Updated google doc is started here:
https://docs.google.com/document/d/1fYKj3BS89aB9drg_DsSq289xSdVQhn1zUJYCj0WuCs0/edit?usp=sharing

Based on the questions on this list earlier, there is a folder with device
information for each of the devices
https://github.com/randomizedcoder/cake/tree/main/device_info

For example, the Pi4 and the Lichee Pi (risc-v) hardware layout is here:
- https://github.com/randomizedcoder/cake/blob/main/device_info/pi4/hwloc-ls-pi4.png

-
https://github.com/randomizedcoder/cake/blob/main/device_info/lpi4a/hwloc-ls-lpi4a.png

The switch has also been upgraded to a Cisco 3750x, which I think based on
the "show interface" output has a max queue size of 40 frames.  The test
process clears the counters before each test and gathers the "show
interface" output at the end.

The Lichee Pi 4A doesn't look good (
https://wiki.sipeed.com/hardware/en/lichee/th1520/lp4a.html )

[image: image.png]
I really wish the flent was working, so I'll probably see if I can work out
the parsing.

Thanks,
Dave Seddon

On Fri, Oct 13, 2023 at 10:25 AM dave seddon <dave.seddon.ca@gmail.com>
wrote:

> My bad.  There's a bug for this.... Looks like I have to downgrade fping
>
> https://github.com/tohojo/flent/issues/232
> https://github.com/schweikert/fping/issues/203
>
> On Fri, Oct 13, 2023 at 8:59 AM dave seddon <dave.seddon.ca@gmail.com>
> wrote:
>
>> G'day,
>>
>> I've been working away on automation of the tests.  Pretty close to
>> having much nicer tests with a lot more details.  I've also got the risc-v
>> device working.
>>
>> However, I've run into something funny with flent.  Flent is not happy
>> with fping or ping.
>>
>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ /usr/bin/sudo
>> /usr/sbin/ip netns exec network101 /usr/bin/flent rrul --output
>>  /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/flent_pi4_noqueue.png
>> --data-dir /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/
>> --format summary --plot all_scaled --title-extra
>> 2023-10-13T15:53:21_pi4_noqueue --note 2023-10-13T15:53:21_pi4_noqueue
>> --extended-metadata --host 172.17.51.10 --length 60 --ipv4 --socket-stats
>> Starting Flent 2.0.1 using Python 3.10.12.
>> Starting rrul test. Expected run time: 70 seconds.
>> WARNING: Found fping, but couldn't parse its output. Not
>> using.              <---------------- ???
>> ERROR: Runner Ping (ms) ICMP failed check: Cannot parse output of the
>> system ping binary (/usr/bin/ping). Please install fping v3.5+.    <----- ??
>>
>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ dpkg --list | grep ping
>> ii  fping                                 5.1-1
>>         amd64        sends ICMP ECHO_REQUEST packets to network hosts
>> ii  iputils-ping                          3:20211215-1
>>          amd64        Tools to test the reachability of network hosts
>> ii  kpartx                                0.8.8-1ubuntu1.22.04.1
>>          amd64        create device mappings for partitions
>> ii  libharfbuzz0b:amd64                   2.7.4-1ubuntu3.1
>>          amd64        OpenType text shaping engine (shared library)
>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ fping --version
>> fping: Version 5.1
>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ ping -V
>> ping from iputils 20211215
>>
>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ cat /etc/lsb-release
>> DISTRIB_ID=Ubuntu
>> DISTRIB_RELEASE=22.04
>> DISTRIB_CODENAME=jammy
>> DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
>>
>> I did install via "apt install fping"
>>
>> Any thoughts please?
>>
>> Kind regards,
>> Dave
>>
>> On Thu, Sep 28, 2023 at 6:27 AM Sebastian Moeller via Cake <
>> cake@lists.bufferbloat.net> wrote:
>>
>>>
>>>
>>> > On Sep 28, 2023, at 15:19, David Lang <david@lang.hm> wrote:
>>> >
>>> > On Thu, 28 Sep 2023, Sebastian Moeller via Cake wrote:
>>> >
>>> >> P.S.: I am tempted, but will likely wait until they are available in
>>> quantity and hope that the street price comes down a bit before getting one
>>> ;)
>>> >
>>> > They aren't available at all yet, and it's not clear when they will be
>>> available.
>>>
>>>         The announcement was end of October, but I think I could
>>> pre-order right now if I was feeling an urge. You are right though,
>>> announced != available or delivered.
>>>
>>> Regards
>>>         Sebastian
>>>
>>> P.S.: I have a pi400 in use as "desktop" for my oldest kid, this is
>>> close to be actually generally usable, I would guess that changing a
>>> potential p500 from the pi400's 4GB to 8 GB together with the other
>>> imprivements the 5 brings might push it over the threshold into the truly
>>> useful category. Which probably means that either a potential pi500 will
>>> come late and probably with only 4 GB, but let's see how this works out now
>>> that the supply situation is less problematic.
>>> And I understand that there are other capable ARM based SoCs for
>>> homerouter/desktop duty, I just happen ot have a soft spot for the
>>> raspberry project ;)
>>>
>>> >
>>> > David Lang
>>>
>>> _______________________________________________
>>> Cake mailing list
>>> Cake@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cake
>>>
>>
>>
>> --
>> Regards,
>> Dave Seddon
>> +1 415 857 5102
>>
>
>
> --
> Regards,
> Dave Seddon
> +1 415 857 5102
>


-- 
Regards,
Dave Seddon
+1 415 857 5102

[-- Attachment #1.2: Type: text/html, Size: 10941 bytes --]

[-- Attachment #2: image.png --]
[-- Type: image/png, Size: 59679 bytes --]

Re: [Cake] some comprehensive arm64 w/cake results

[Sebastian Moeller]

[-- Attachment #1: Type: text/plain, Size: 7284 bytes --]

If I recall correctly, flent will use irtt for its delay probes if available on both ends. Sure fixing fping seems like a good thing longer term, but to get data in quickly, maybe try irtt instead?

On 15 October 2023 17:11:23 CEST, dave seddon via Cake <cake@lists.bufferbloat.net> wrote:
>G'day,
>
>I've put more work into a test framework around the qdisc tests, but
>unfortunately flent doesn't work easily with Ubuntu LTS (
>https://github.com/tohojo/flent/issues/232, which I think is an issue with
>flent parsing the fping output ).
>
>Results and graphs in this sheet:
>https://docs.google.com/spreadsheets/d/1T59QwEdNwJFm4TgDFA_NY98gicOm8ABXKvDsSIMz9ag/edit#gid=1203641125
>
>Raw results of x2 test runs are here:
>https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/report.csv
>
>Each run:
>https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/report.csv
>https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-14T14%3A22%3A53/report.csv
>
>Full iperf outputs are available too, for example:
>https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/nanopi-r2s/fq_codel/iperf/test/16_iperf/stdout
>
>
>Logs for each run are also available, for example:
>https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/log.json
>
>The code repo updated here: https://github.com/randomizedcoder/cake , with
>thehttps://github.com/randomizedcoder/cake/blob/main/README.md which
>explains how the test work.
>Updated google doc is started here:
>https://docs.google.com/document/d/1fYKj3BS89aB9drg_DsSq289xSdVQhn1zUJYCj0WuCs0/edit?usp=sharing
>
>Based on the questions on this list earlier, there is a folder with device
>information for each of the devices
>https://github.com/randomizedcoder/cake/tree/main/device_info
>
>For example, the Pi4 and the Lichee Pi (risc-v) hardware layout is here:
>- https://github.com/randomizedcoder/cake/blob/main/device_info/pi4/hwloc-ls-pi4.png
>
>-
>https://github.com/randomizedcoder/cake/blob/main/device_info/lpi4a/hwloc-ls-lpi4a.png
>
>The switch has also been upgraded to a Cisco 3750x, which I think based on
>the "show interface" output has a max queue size of 40 frames.  The test
>process clears the counters before each test and gathers the "show
>interface" output at the end.
>
>The Lichee Pi 4A doesn't look good (
>https://wiki.sipeed.com/hardware/en/lichee/th1520/lp4a.html )
>
>[image: image.png]
>I really wish the flent was working, so I'll probably see if I can work out
>the parsing.
>
>Thanks,
>Dave Seddon
>
>On Fri, Oct 13, 2023 at 10:25 AM dave seddon <dave.seddon.ca@gmail.com>
>wrote:
>
>> My bad.  There's a bug for this.... Looks like I have to downgrade fping
>>
>> https://github.com/tohojo/flent/issues/232
>> https://github.com/schweikert/fping/issues/203
>>
>> On Fri, Oct 13, 2023 at 8:59 AM dave seddon <dave.seddon.ca@gmail.com>
>> wrote:
>>
>>> G'day,
>>>
>>> I've been working away on automation of the tests.  Pretty close to
>>> having much nicer tests with a lot more details.  I've also got the risc-v
>>> device working.
>>>
>>> However, I've run into something funny with flent.  Flent is not happy
>>> with fping or ping.
>>>
>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ /usr/bin/sudo
>>> /usr/sbin/ip netns exec network101 /usr/bin/flent rrul --output
>>>  /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/flent_pi4_noqueue.png
>>> --data-dir /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/
>>> --format summary --plot all_scaled --title-extra
>>> 2023-10-13T15:53:21_pi4_noqueue --note 2023-10-13T15:53:21_pi4_noqueue
>>> --extended-metadata --host 172.17.51.10 --length 60 --ipv4 --socket-stats
>>> Starting Flent 2.0.1 using Python 3.10.12.
>>> Starting rrul test. Expected run time: 70 seconds.
>>> WARNING: Found fping, but couldn't parse its output. Not
>>> using.              <---------------- ???
>>> ERROR: Runner Ping (ms) ICMP failed check: Cannot parse output of the
>>> system ping binary (/usr/bin/ping). Please install fping v3.5+.    <----- ??
>>>
>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ dpkg --list | grep ping
>>> ii  fping                                 5.1-1
>>>         amd64        sends ICMP ECHO_REQUEST packets to network hosts
>>> ii  iputils-ping                          3:20211215-1
>>>          amd64        Tools to test the reachability of network hosts
>>> ii  kpartx                                0.8.8-1ubuntu1.22.04.1
>>>          amd64        create device mappings for partitions
>>> ii  libharfbuzz0b:amd64                   2.7.4-1ubuntu3.1
>>>          amd64        OpenType text shaping engine (shared library)
>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ fping --version
>>> fping: Version 5.1
>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ ping -V
>>> ping from iputils 20211215
>>>
>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ cat /etc/lsb-release
>>> DISTRIB_ID=Ubuntu
>>> DISTRIB_RELEASE=22.04
>>> DISTRIB_CODENAME=jammy
>>> DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
>>>
>>> I did install via "apt install fping"
>>>
>>> Any thoughts please?
>>>
>>> Kind regards,
>>> Dave
>>>
>>> On Thu, Sep 28, 2023 at 6:27 AM Sebastian Moeller via Cake <
>>> cake@lists.bufferbloat.net> wrote:
>>>
>>>>
>>>>
>>>> > On Sep 28, 2023, at 15:19, David Lang <david@lang.hm> wrote:
>>>> >
>>>> > On Thu, 28 Sep 2023, Sebastian Moeller via Cake wrote:
>>>> >
>>>> >> P.S.: I am tempted, but will likely wait until they are available in
>>>> quantity and hope that the street price comes down a bit before getting one
>>>> ;)
>>>> >
>>>> > They aren't available at all yet, and it's not clear when they will be
>>>> available.
>>>>
>>>>         The announcement was end of October, but I think I could
>>>> pre-order right now if I was feeling an urge. You are right though,
>>>> announced != available or delivered.
>>>>
>>>> Regards
>>>>         Sebastian
>>>>
>>>> P.S.: I have a pi400 in use as "desktop" for my oldest kid, this is
>>>> close to be actually generally usable, I would guess that changing a
>>>> potential p500 from the pi400's 4GB to 8 GB together with the other
>>>> imprivements the 5 brings might push it over the threshold into the truly
>>>> useful category. Which probably means that either a potential pi500 will
>>>> come late and probably with only 4 GB, but let's see how this works out now
>>>> that the supply situation is less problematic.
>>>> And I understand that there are other capable ARM based SoCs for
>>>> homerouter/desktop duty, I just happen ot have a soft spot for the
>>>> raspberry project ;)
>>>>
>>>> >
>>>> > David Lang
>>>>
>>>> _______________________________________________
>>>> Cake mailing list
>>>> Cake@lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/cake
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Dave Seddon
>>> +1 415 857 5102
>>>
>>
>>
>> --
>> Regards,
>> Dave Seddon
>> +1 415 857 5102
>>
>
>
>-- 
>Regards,
>Dave Seddon
>+1 415 857 5102

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 11903 bytes --]

Re: [Cake] some comprehensive arm64 w/cake results

[Dave Taht]

really lovely, thank you. I hope we can get flent fixed. That risc
result was awful. Does it have BQL? Are there specs on the ethernet
interface?

This risc-v beaglebone just came out today:
https://www.beagleboard.org/boards/beaglev-ahead

On Sun, Oct 15, 2023 at 8:11 AM dave seddon <dave.seddon.ca@gmail.com> wrote:
>
> G'day,
>
> I've put more work into a test framework around the qdisc tests, but unfortunately flent doesn't work easily with Ubuntu LTS ( https://github.com/tohojo/flent/issues/232, which I think is an issue with flent parsing the fping output ).
>
> Results and graphs in this sheet:
> https://docs.google.com/spreadsheets/d/1T59QwEdNwJFm4TgDFA_NY98gicOm8ABXKvDsSIMz9ag/edit#gid=1203641125
>
> Raw results of x2 test runs are here:
> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/report.csv
>
> Each run:
> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/report.csv
> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-14T14%3A22%3A53/report.csv
>
> Full iperf outputs are available too, for example: https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/nanopi-r2s/fq_codel/iperf/test/16_iperf/stdout
>
> Logs for each run are also available, for example: https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/log.json
>
> The code repo updated here: https://github.com/randomizedcoder/cake , with thehttps://github.com/randomizedcoder/cake/blob/main/README.md which explains how the test work.
> Updated google doc is started here: https://docs.google.com/document/d/1fYKj3BS89aB9drg_DsSq289xSdVQhn1zUJYCj0WuCs0/edit?usp=sharing
>
> Based on the questions on this list earlier, there is a folder with device information for each of the devices
> https://github.com/randomizedcoder/cake/tree/main/device_info
>
> For example, the Pi4 and the Lichee Pi (risc-v) hardware layout is here:
> - https://github.com/randomizedcoder/cake/blob/main/device_info/pi4/hwloc-ls-pi4.png
> - https://github.com/randomizedcoder/cake/blob/main/device_info/lpi4a/hwloc-ls-lpi4a.png
>
> The switch has also been upgraded to a Cisco 3750x, which I think based on the "show interface" output has a max queue size of 40 frames.  The test process clears the counters before each test and gathers the "show interface" output at the end.
>
> The Lichee Pi 4A doesn't look good ( https://wiki.sipeed.com/hardware/en/lichee/th1520/lp4a.html )
>
>
> I really wish the flent was working, so I'll probably see if I can work out the parsing.
>
> Thanks,
> Dave Seddon
>
> On Fri, Oct 13, 2023 at 10:25 AM dave seddon <dave.seddon.ca@gmail.com> wrote:
>>
>> My bad.  There's a bug for this.... Looks like I have to downgrade fping
>>
>> https://github.com/tohojo/flent/issues/232
>> https://github.com/schweikert/fping/issues/203
>>
>> On Fri, Oct 13, 2023 at 8:59 AM dave seddon <dave.seddon.ca@gmail.com> wrote:
>>>
>>> G'day,
>>>
>>> I've been working away on automation of the tests.  Pretty close to having much nicer tests with a lot more details.  I've also got the risc-v device working.
>>>
>>> However, I've run into something funny with flent.  Flent is not happy with fping or ping.
>>>
>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ /usr/bin/sudo /usr/sbin/ip netns exec network101 /usr/bin/flent rrul --output  /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/flent_pi4_noqueue.png --data-dir /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/ --format summary --plot all_scaled --title-extra 2023-10-13T15:53:21_pi4_noqueue --note 2023-10-13T15:53:21_pi4_noqueue --extended-metadata --host 172.17.51.10 --length 60 --ipv4 --socket-stats
>>> Starting Flent 2.0.1 using Python 3.10.12.
>>> Starting rrul test. Expected run time: 70 seconds.
>>> WARNING: Found fping, but couldn't parse its output. Not using.              <---------------- ???
>>> ERROR: Runner Ping (ms) ICMP failed check: Cannot parse output of the system ping binary (/usr/bin/ping). Please install fping v3.5+.    <----- ??
>>>
>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ dpkg --list | grep ping
>>> ii  fping                                 5.1-1                                   amd64        sends ICMP ECHO_REQUEST packets to network hosts
>>> ii  iputils-ping                          3:20211215-1                            amd64        Tools to test the reachability of network hosts
>>> ii  kpartx                                0.8.8-1ubuntu1.22.04.1                  amd64        create device mappings for partitions
>>> ii  libharfbuzz0b:amd64                   2.7.4-1ubuntu3.1                        amd64        OpenType text shaping engine (shared library)
>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ fping --version
>>> fping: Version 5.1
>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ ping -V
>>> ping from iputils 20211215
>>>
>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ cat /etc/lsb-release
>>> DISTRIB_ID=Ubuntu
>>> DISTRIB_RELEASE=22.04
>>> DISTRIB_CODENAME=jammy
>>> DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
>>>
>>> I did install via "apt install fping"
>>>
>>> Any thoughts please?
>>>
>>> Kind regards,
>>> Dave
>>>
>>> On Thu, Sep 28, 2023 at 6:27 AM Sebastian Moeller via Cake <cake@lists.bufferbloat.net> wrote:
>>>>
>>>>
>>>>
>>>> > On Sep 28, 2023, at 15:19, David Lang <david@lang.hm> wrote:
>>>> >
>>>> > On Thu, 28 Sep 2023, Sebastian Moeller via Cake wrote:
>>>> >
>>>> >> P.S.: I am tempted, but will likely wait until they are available in quantity and hope that the street price comes down a bit before getting one ;)
>>>> >
>>>> > They aren't available at all yet, and it's not clear when they will be available.
>>>>
>>>>         The announcement was end of October, but I think I could pre-order right now if I was feeling an urge. You are right though, announced != available or delivered.
>>>>
>>>> Regards
>>>>         Sebastian
>>>>
>>>> P.S.: I have a pi400 in use as "desktop" for my oldest kid, this is close to be actually generally usable, I would guess that changing a potential p500 from the pi400's 4GB to 8 GB together with the other imprivements the 5 brings might push it over the threshold into the truly useful category. Which probably means that either a potential pi500 will come late and probably with only 4 GB, but let's see how this works out now that the supply situation is less problematic.
>>>> And I understand that there are other capable ARM based SoCs for homerouter/desktop duty, I just happen ot have a soft spot for the raspberry project ;)
>>>>
>>>> >
>>>> > David Lang
>>>>
>>>> _______________________________________________
>>>> Cake mailing list
>>>> Cake@lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/cake
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Dave Seddon
>>> +1 415 857 5102
>>
>>
>>
>> --
>> Regards,
>> Dave Seddon
>> +1 415 857 5102
>
>
>
> --
> Regards,
> Dave Seddon
> +1 415 857 5102



-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: [Cake] some comprehensive arm64 w/cake results

[dave seddon]

[-- Attachment #1: Type: text/plain, Size: 7938 bytes --]

Oh thanks Sebastian.  I have irtt installed, but it looks like I need to
start the server.   That's easy.  Doing it now.

( Incidentally, I did write a golang based icmp pinger.  It can ping at
very high rates: https://github.com/edgio/icmpengine.  Really should write
one with rust using io_uring... )



On Sun, Oct 15, 2023 at 8:53 AM Sebastian Moeller <moeller0@gmx.de> wrote:

> If I recall correctly, flent will use irtt for its delay probes if
> available on both ends. Sure fixing fping seems like a good thing longer
> term, but to get data in quickly, maybe try irtt instead?
>
>
> On 15 October 2023 17:11:23 CEST, dave seddon via Cake <
> cake@lists.bufferbloat.net> wrote:
>
>> G'day,
>>
>> I've put more work into a test framework around the qdisc tests, but
>> unfortunately flent doesn't work easily with Ubuntu LTS (
>> https://github.com/tohojo/flent/issues/232, which I think is an issue
>> with flent parsing the fping output ).
>>
>> Results and graphs in this sheet:
>>
>> https://docs.google.com/spreadsheets/d/1T59QwEdNwJFm4TgDFA_NY98gicOm8ABXKvDsSIMz9ag/edit#gid=1203641125
>>
>> Raw results of x2 test runs are here:
>>
>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/report.csv
>>
>> Each run:
>>
>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/report.csv
>>
>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-14T14%3A22%3A53/report.csv
>>
>> Full iperf outputs are available too, for example: https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/nanopi-r2s/fq_codel/iperf/test/16_iperf/stdout
>>
>>
>> Logs for each run are also available, for example:
>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/log.json
>>
>> The code repo updated here: https://github.com/randomizedcoder/cake ,
>> with thehttps://github.com/randomizedcoder/cake/blob/main/README.md
>> which explains how the test work.
>> Updated google doc is started here:
>> https://docs.google.com/document/d/1fYKj3BS89aB9drg_DsSq289xSdVQhn1zUJYCj0WuCs0/edit?usp=sharing
>>
>> Based on the questions on this list earlier, there is a folder with
>> device information for each of the devices
>> https://github.com/randomizedcoder/cake/tree/main/device_info
>>
>> For example, the Pi4 and the Lichee Pi (risc-v) hardware layout is here:
>> - https://github.com/randomizedcoder/cake/blob/main/device_info/pi4/hwloc-ls-pi4.png
>>
>> -
>> https://github.com/randomizedcoder/cake/blob/main/device_info/lpi4a/hwloc-ls-lpi4a.png
>>
>> The switch has also been upgraded to a Cisco 3750x, which I think based
>> on the "show interface" output has a max queue size of 40 frames.  The test
>> process clears the counters before each test and gathers the "show
>> interface" output at the end.
>>
>> The Lichee Pi 4A doesn't look good (
>> https://wiki.sipeed.com/hardware/en/lichee/th1520/lp4a.html )
>>
>> [image: image.png]
>> I really wish the flent was working, so I'll probably see if I can work
>> out the parsing.
>>
>> Thanks,
>> Dave Seddon
>>
>> On Fri, Oct 13, 2023 at 10:25 AM dave seddon <dave.seddon.ca@gmail.com>
>> wrote:
>>
>>> My bad.  There's a bug for this.... Looks like I have to downgrade fping
>>>
>>> https://github.com/tohojo/flent/issues/232
>>> https://github.com/schweikert/fping/issues/203
>>>
>>> On Fri, Oct 13, 2023 at 8:59 AM dave seddon <dave.seddon.ca@gmail.com>
>>> wrote:
>>>
>>>> G'day,
>>>>
>>>> I've been working away on automation of the tests.  Pretty close to
>>>> having much nicer tests with a lot more details.  I've also got the risc-v
>>>> device working.
>>>>
>>>> However, I've run into something funny with flent.  Flent is not happy
>>>> with fping or ping.
>>>>
>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ /usr/bin/sudo
>>>> /usr/sbin/ip netns exec network101 /usr/bin/flent rrul --output
>>>>  /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/flent_pi4_noqueue.png
>>>> --data-dir /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/
>>>> --format summary --plot all_scaled --title-extra
>>>> 2023-10-13T15:53:21_pi4_noqueue --note 2023-10-13T15:53:21_pi4_noqueue
>>>> --extended-metadata --host 172.17.51.10 --length 60 --ipv4 --socket-stats
>>>> Starting Flent 2.0.1 using Python 3.10.12.
>>>> Starting rrul test. Expected run time: 70 seconds.
>>>> WARNING: Found fping, but couldn't parse its output. Not
>>>> using.              <---------------- ???
>>>> ERROR: Runner Ping (ms) ICMP failed check: Cannot parse output of the
>>>> system ping binary (/usr/bin/ping). Please install fping v3.5+.    <----- ??
>>>>
>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ dpkg --list | grep ping
>>>> ii  fping                                 5.1-1
>>>>           amd64        sends ICMP ECHO_REQUEST packets to network hosts
>>>> ii  iputils-ping                          3:20211215-1
>>>>            amd64        Tools to test the reachability of network hosts
>>>> ii  kpartx                                0.8.8-1ubuntu1.22.04.1
>>>>            amd64        create device mappings for partitions
>>>> ii  libharfbuzz0b:amd64                   2.7.4-1ubuntu3.1
>>>>            amd64        OpenType text shaping engine (shared library)
>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ fping --version
>>>> fping: Version 5.1
>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ ping -V
>>>> ping from iputils 20211215
>>>>
>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ cat /etc/lsb-release
>>>> DISTRIB_ID=Ubuntu
>>>> DISTRIB_RELEASE=22.04
>>>> DISTRIB_CODENAME=jammy
>>>> DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
>>>>
>>>> I did install via "apt install fping"
>>>>
>>>> Any thoughts please?
>>>>
>>>> Kind regards,
>>>> Dave
>>>>
>>>> On Thu, Sep 28, 2023 at 6:27 AM Sebastian Moeller via Cake <
>>>> cake@lists.bufferbloat.net> wrote:
>>>>
>>>>>
>>>>>
>>>>> > On Sep 28, 2023, at 15:19, David Lang <david@lang.hm> wrote:
>>>>> >
>>>>> > On Thu, 28 Sep 2023, Sebastian Moeller via Cake wrote:
>>>>> >
>>>>> >> P.S.: I am tempted, but will likely wait until they are available
>>>>> in quantity and hope that the street price comes down a bit before getting
>>>>> one ;)
>>>>> >
>>>>> > They aren't available at all yet, and it's not clear when they will
>>>>> be available.
>>>>>
>>>>>         The announcement was end of October, but I think I could
>>>>> pre-order right now if I was feeling an urge. You are right though,
>>>>> announced != available or delivered.
>>>>>
>>>>> Regards
>>>>>         Sebastian
>>>>>
>>>>> P.S.: I have a pi400 in use as "desktop" for my oldest kid, this is
>>>>> close to be actually generally usable, I would guess that changing a
>>>>> potential p500 from the pi400's 4GB to 8 GB together with the other
>>>>> imprivements the 5 brings might push it over the threshold into the truly
>>>>> useful category. Which probably means that either a potential pi500 will
>>>>> come late and probably with only 4 GB, but let's see how this works out now
>>>>> that the supply situation is less problematic.
>>>>> And I understand that there are other capable ARM based SoCs for
>>>>> homerouter/desktop duty, I just happen ot have a soft spot for the
>>>>> raspberry project ;)
>>>>>
>>>>> >
>>>>> > David Lang
>>>>>
>>>>> _______________________________________________
>>>>> Cake mailing list
>>>>> Cake@lists.bufferbloat.net
>>>>> https://lists.bufferbloat.net/listinfo/cake
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Dave Seddon
>>>> +1 415 857 5102
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Dave Seddon
>>> +1 415 857 5102
>>>
>>
>> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>


-- 
Regards,
Dave Seddon
+1 415 857 5102

[-- Attachment #2: Type: text/html, Size: 12545 bytes --]

Re: [Cake] some comprehensive arm64 w/cake results

[David P. Reed]

60K pings per second? Well, that's probably fast enough for Cake work..., but I'm sure you can do a LOT better... Try AF_XDP and/or DPDK. I think AF_XDP works on ARM.

https://adarsh-kumar-phe15.medium.com/receiving-14-million-network-packets-per-second-a9d5cc1408b6

Now admittedly, my datacenter networking work is focused on single-digit microsecond RTT protocols (and doing them in Linux). But we really need to get clear on what "fast" means in modern computers. It does bug me that the folks working on network performance are still focused as if networks have to be slow.

The same problems can be found throughout the Linux kernel, where assumptions still seem to be holding on to what worked in Linus's original PC (single core, slow rotating disks, ...). I recently found a comment (in the disk page swap code) where the idea that disk cylinders are fast, but seeks are slow, is implicit in the design. It's the world's most "modern" "legacy system" - with lots of design decisions that make no sense anymore, but can't be undone.

Of course, Internet congestion control, in general, is still stuck in the original Van Jacobsen sawtooth era. My guess is it won't get fixed, though I applaud Cake, and despair the hardware folks who keep adding buffers.


On Sunday, October 15, 2023 12:24pm, "dave seddon via Cake" <cake@lists.bufferbloat.net> said:

> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
> Oh thanks Sebastian.  I have irtt installed, but it looks like I need to
> start the server.   That's easy.  Doing it now.
> 
> ( Incidentally, I did write a golang based icmp pinger.  It can ping at
> very high rates: https://github.com/edgio/icmpengine.  Really should write
> one with rust using io_uring... )
> 
> 
> 
> On Sun, Oct 15, 2023 at 8:53 AM Sebastian Moeller <moeller0@gmx.de> wrote:
> 
>> If I recall correctly, flent will use irtt for its delay probes if
>> available on both ends. Sure fixing fping seems like a good thing longer
>> term, but to get data in quickly, maybe try irtt instead?
>>
>>
>> On 15 October 2023 17:11:23 CEST, dave seddon via Cake <
>> cake@lists.bufferbloat.net> wrote:
>>
>>> G'day,
>>>
>>> I've put more work into a test framework around the qdisc tests, but
>>> unfortunately flent doesn't work easily with Ubuntu LTS (
>>> https://github.com/tohojo/flent/issues/232, which I think is an issue
>>> with flent parsing the fping output ).
>>>
>>> Results and graphs in this sheet:
>>>
>>> https://docs.google.com/spreadsheets/d/1T59QwEdNwJFm4TgDFA_NY98gicOm8ABXKvDsSIMz9ag/edit#gid=1203641125
>>>
>>> Raw results of x2 test runs are here:
>>>
>>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/report.csv
>>>
>>> Each run:
>>>
>>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/report.csv
>>>
>>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-14T14%3A22%3A53/report.csv
>>>
>>> Full iperf outputs are available too, for example:
>>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/nanopi-r2s/fq_codel/iperf/test/16_iperf/stdout
>>>
>>>
>>> Logs for each run are also available, for example:
>>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/log.json
>>>
>>> The code repo updated here: https://github.com/randomizedcoder/cake ,
>>> with thehttps://github.com/randomizedcoder/cake/blob/main/README.md
>>> which explains how the test work.
>>> Updated google doc is started here:
>>> https://docs.google.com/document/d/1fYKj3BS89aB9drg_DsSq289xSdVQhn1zUJYCj0WuCs0/edit?usp=sharing
>>>
>>> Based on the questions on this list earlier, there is a folder with
>>> device information for each of the devices
>>> https://github.com/randomizedcoder/cake/tree/main/device_info
>>>
>>> For example, the Pi4 and the Lichee Pi (risc-v) hardware layout is here:
>>> -
>>> https://github.com/randomizedcoder/cake/blob/main/device_info/pi4/hwloc-ls-pi4.png
>>>
>>> -
>>> https://github.com/randomizedcoder/cake/blob/main/device_info/lpi4a/hwloc-ls-lpi4a.png
>>>
>>> The switch has also been upgraded to a Cisco 3750x, which I think based
>>> on the "show interface" output has a max queue size of 40 frames.  The test
>>> process clears the counters before each test and gathers the "show
>>> interface" output at the end.
>>>
>>> The Lichee Pi 4A doesn't look good (
>>> https://wiki.sipeed.com/hardware/en/lichee/th1520/lp4a.html )
>>>
>>> [image: image.png]
>>> I really wish the flent was working, so I'll probably see if I can work
>>> out the parsing.
>>>
>>> Thanks,
>>> Dave Seddon
>>>
>>> On Fri, Oct 13, 2023 at 10:25 AM dave seddon <dave.seddon.ca@gmail.com>
>>> wrote:
>>>
>>>> My bad.  There's a bug for this.... Looks like I have to downgrade fping
>>>>
>>>> https://github.com/tohojo/flent/issues/232
>>>> https://github.com/schweikert/fping/issues/203
>>>>
>>>> On Fri, Oct 13, 2023 at 8:59 AM dave seddon <dave.seddon.ca@gmail.com>
>>>> wrote:
>>>>
>>>>> G'day,
>>>>>
>>>>> I've been working away on automation of the tests.  Pretty close to
>>>>> having much nicer tests with a lot more details.  I've also got the risc-v
>>>>> device working.
>>>>>
>>>>> However, I've run into something funny with flent.  Flent is not happy
>>>>> with fping or ping.
>>>>>
>>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ /usr/bin/sudo
>>>>> /usr/sbin/ip netns exec network101 /usr/bin/flent rrul --output
>>>>>  /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/flent_pi4_noqueue.png
>>>>> --data-dir /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/
>>>>> --format summary --plot all_scaled --title-extra
>>>>> 2023-10-13T15:53:21_pi4_noqueue --note 2023-10-13T15:53:21_pi4_noqueue
>>>>> --extended-metadata --host 172.17.51.10 --length 60 --ipv4 --socket-stats
>>>>> Starting Flent 2.0.1 using Python 3.10.12.
>>>>> Starting rrul test. Expected run time: 70 seconds.
>>>>> WARNING: Found fping, but couldn't parse its output. Not
>>>>> using.              <---------------- ???
>>>>> ERROR: Runner Ping (ms) ICMP failed check: Cannot parse output of the
>>>>> system ping binary (/usr/bin/ping). Please install fping v3.5+.    <----- ??
>>>>>
>>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ dpkg --list | grep ping
>>>>> ii  fping                                 5.1-1
>>>>>           amd64        sends ICMP ECHO_REQUEST packets to network hosts
>>>>> ii  iputils-ping                          3:20211215-1
>>>>>            amd64        Tools to test the reachability of network hosts
>>>>> ii  kpartx                                0.8.8-1ubuntu1.22.04.1
>>>>>            amd64        create device mappings for partitions
>>>>> ii  libharfbuzz0b:amd64                   2.7.4-1ubuntu3.1
>>>>>            amd64        OpenType text shaping engine (shared library)
>>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ fping --version
>>>>> fping: Version 5.1
>>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ ping -V
>>>>> ping from iputils 20211215
>>>>>
>>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ cat /etc/lsb-release
>>>>> DISTRIB_ID=Ubuntu
>>>>> DISTRIB_RELEASE=22.04
>>>>> DISTRIB_CODENAME=jammy
>>>>> DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
>>>>>
>>>>> I did install via "apt install fping"
>>>>>
>>>>> Any thoughts please?
>>>>>
>>>>> Kind regards,
>>>>> Dave
>>>>>
>>>>> On Thu, Sep 28, 2023 at 6:27 AM Sebastian Moeller via Cake <
>>>>> cake@lists.bufferbloat.net> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> > On Sep 28, 2023, at 15:19, David Lang <david@lang.hm> wrote:
>>>>>> >
>>>>>> > On Thu, 28 Sep 2023, Sebastian Moeller via Cake wrote:
>>>>>> >
>>>>>> >> P.S.: I am tempted, but will likely wait until they are available
>>>>>> in quantity and hope that the street price comes down a bit before getting
>>>>>> one ;)
>>>>>> >
>>>>>> > They aren't available at all yet, and it's not clear when they will
>>>>>> be available.
>>>>>>
>>>>>>         The announcement was end of October, but I think I could
>>>>>> pre-order right now if I was feeling an urge. You are right though,
>>>>>> announced != available or delivered.
>>>>>>
>>>>>> Regards
>>>>>>         Sebastian
>>>>>>
>>>>>> P.S.: I have a pi400 in use as "desktop" for my oldest kid, this is
>>>>>> close to be actually generally usable, I would guess that changing a
>>>>>> potential p500 from the pi400's 4GB to 8 GB together with the other
>>>>>> imprivements the 5 brings might push it over the threshold into the truly
>>>>>> useful category. Which probably means that either a potential pi500 will
>>>>>> come late and probably with only 4 GB, but let's see how this works out now
>>>>>> that the supply situation is less problematic.
>>>>>> And I understand that there are other capable ARM based SoCs for
>>>>>> homerouter/desktop duty, I just happen ot have a soft spot for the
>>>>>> raspberry project ;)
>>>>>>
>>>>>> >
>>>>>> > David Lang
>>>>>>
>>>>>> _______________________________________________
>>>>>> Cake mailing list
>>>>>> Cake@lists.bufferbloat.net
>>>>>> https://lists.bufferbloat.net/listinfo/cake
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Dave Seddon
>>>>> +1 415 857 5102
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Dave Seddon
>>>> +1 415 857 5102
>>>>
>>>
>>> --
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>
> 
> 
> --
> Regards,
> Dave Seddon
> +1 415 857 5102
> 

Re: [Cake] some comprehensive arm64 w/cake results

[Jonathan Morton]

> On 15 Oct, 2023, at 11:29 pm, David P. Reed via Cake <cake@lists.bufferbloat.net> wrote:
> 
> Of course, Internet congestion control, in general, is still stuck in the original Van Jacobsen sawtooth era. My guess is it won't get fixed, though I applaud Cake, and despair the hardware folks who keep adding buffers.

I am still working in this area, including on a revised version of SCE which might regain traction in the IETF.

One of the more immediate results of this is a new AQM algorithm which builds on the success of Codel, and a family of qdiscs I'm building around it.  These range from a queueless traffic policer to a "Cake version 2", with an interesting approximate-fairness approach for the middle child.  The AQM itself is already working, though not yet documented in a public-facing form.

I'm also taking a new approach to overlaying the "small" congestion response of SCE over the "big" response of conventional congestion control, which I think is capable of solving several long-standing problems in one go.  I'll say more when we have a working prototype - but think in terms of "no sawtooth" and "naturally max-min fair".  TCP Prague will look positively primitive compared to this - if it works (and it should).

 - Jonathan Morton

Re: [Cake] some comprehensive arm64 w/cake results

[dave seddon]

[-- Attachment #1.1: Type: text/plain, Size: 8348 bytes --]

G'day,

Dave Taht and I have had a couple of phone conversations now, and he's
convinced me that rather than inserting the netem delay on each laptop,
that latency should be added by a seperate device.  To this end, I've got
another little PC and a NIC coming, so that I can repeat all the tests with
seperate latency injection.

However, I've also completed the flent tests with the laptops adding
latency at each end.

Full test runs here:
https://github.com/randomizedcoder/qdisc_results/tree/main/qdisc/2023-10-23T16%3A49%3A10

You can find the actual rrul flent .tar.gz results for each test.

e.g
Pi4 fq is here:
https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-23T16%3A49%3A10/pi4/fq/flent/test/16_flent/rrul-2023-10-23T170016.068273.2023-10-23T16_49_10_pi4_fq.flent.gz

Lychee Pi Risv with cake qdisc:
https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-23T16%3A49%3A10/lpi4a/cake20/flent/test/16_flent/rrul-2023-10-23T201354.818316.2023-10-23T16_49_10_lpi4a_cake20.flent.gz

Just take these with a grain of salt until the new latency injection is in
place.

... I'll see if I can script up the generation of all the pretty graphs soon

Thanks,
Dave Seddon


On Sun, Oct 15, 2023 at 8:11 AM dave seddon <dave.seddon.ca@gmail.com>
wrote:

> G'day,
>
> I've put more work into a test framework around the qdisc tests, but
> unfortunately flent doesn't work easily with Ubuntu LTS (
> https://github.com/tohojo/flent/issues/232, which I think is an issue
> with flent parsing the fping output ).
>
> Results and graphs in this sheet:
>
> https://docs.google.com/spreadsheets/d/1T59QwEdNwJFm4TgDFA_NY98gicOm8ABXKvDsSIMz9ag/edit#gid=1203641125
>
> Raw results of x2 test runs are here:
> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/report.csv
>
> Each run:
>
> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/report.csv
>
> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-14T14%3A22%3A53/report.csv
>
> Full iperf outputs are available too, for example: https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/nanopi-r2s/fq_codel/iperf/test/16_iperf/stdout
>
>
> Logs for each run are also available, for example:
> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/log.json
>
> The code repo updated here: https://github.com/randomizedcoder/cake ,
> with thehttps://github.com/randomizedcoder/cake/blob/main/README.md which
> explains how the test work.
> Updated google doc is started here:
> https://docs.google.com/document/d/1fYKj3BS89aB9drg_DsSq289xSdVQhn1zUJYCj0WuCs0/edit?usp=sharing
>
> Based on the questions on this list earlier, there is a folder with device
> information for each of the devices
> https://github.com/randomizedcoder/cake/tree/main/device_info
>
> For example, the Pi4 and the Lichee Pi (risc-v) hardware layout is here:
> - https://github.com/randomizedcoder/cake/blob/main/device_info/pi4/hwloc-ls-pi4.png
>
> -
> https://github.com/randomizedcoder/cake/blob/main/device_info/lpi4a/hwloc-ls-lpi4a.png
>
> The switch has also been upgraded to a Cisco 3750x, which I think based on
> the "show interface" output has a max queue size of 40 frames.  The test
> process clears the counters before each test and gathers the "show
> interface" output at the end.
>
> The Lichee Pi 4A doesn't look good (
> https://wiki.sipeed.com/hardware/en/lichee/th1520/lp4a.html )
>
> [image: image.png]
> I really wish the flent was working, so I'll probably see if I can work
> out the parsing.
>
> Thanks,
> Dave Seddon
>
> On Fri, Oct 13, 2023 at 10:25 AM dave seddon <dave.seddon.ca@gmail.com>
> wrote:
>
>> My bad.  There's a bug for this.... Looks like I have to downgrade fping
>>
>> https://github.com/tohojo/flent/issues/232
>> https://github.com/schweikert/fping/issues/203
>>
>> On Fri, Oct 13, 2023 at 8:59 AM dave seddon <dave.seddon.ca@gmail.com>
>> wrote:
>>
>>> G'day,
>>>
>>> I've been working away on automation of the tests.  Pretty close to
>>> having much nicer tests with a lot more details.  I've also got the risc-v
>>> device working.
>>>
>>> However, I've run into something funny with flent.  Flent is not happy
>>> with fping or ping.
>>>
>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ /usr/bin/sudo
>>> /usr/sbin/ip netns exec network101 /usr/bin/flent rrul --output
>>>  /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/flent_pi4_noqueue.png
>>> --data-dir /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/
>>> --format summary --plot all_scaled --title-extra
>>> 2023-10-13T15:53:21_pi4_noqueue --note 2023-10-13T15:53:21_pi4_noqueue
>>> --extended-metadata --host 172.17.51.10 --length 60 --ipv4 --socket-stats
>>> Starting Flent 2.0.1 using Python 3.10.12.
>>> Starting rrul test. Expected run time: 70 seconds.
>>> WARNING: Found fping, but couldn't parse its output. Not
>>> using.              <---------------- ???
>>> ERROR: Runner Ping (ms) ICMP failed check: Cannot parse output of the
>>> system ping binary (/usr/bin/ping). Please install fping v3.5+.    <----- ??
>>>
>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ dpkg --list | grep ping
>>> ii  fping                                 5.1-1
>>>           amd64        sends ICMP ECHO_REQUEST packets to network hosts
>>> ii  iputils-ping                          3:20211215-1
>>>          amd64        Tools to test the reachability of network hosts
>>> ii  kpartx                                0.8.8-1ubuntu1.22.04.1
>>>          amd64        create device mappings for partitions
>>> ii  libharfbuzz0b:amd64                   2.7.4-1ubuntu3.1
>>>          amd64        OpenType text shaping engine (shared library)
>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ fping --version
>>> fping: Version 5.1
>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ ping -V
>>> ping from iputils 20211215
>>>
>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ cat /etc/lsb-release
>>> DISTRIB_ID=Ubuntu
>>> DISTRIB_RELEASE=22.04
>>> DISTRIB_CODENAME=jammy
>>> DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
>>>
>>> I did install via "apt install fping"
>>>
>>> Any thoughts please?
>>>
>>> Kind regards,
>>> Dave
>>>
>>> On Thu, Sep 28, 2023 at 6:27 AM Sebastian Moeller via Cake <
>>> cake@lists.bufferbloat.net> wrote:
>>>
>>>>
>>>>
>>>> > On Sep 28, 2023, at 15:19, David Lang <david@lang.hm> wrote:
>>>> >
>>>> > On Thu, 28 Sep 2023, Sebastian Moeller via Cake wrote:
>>>> >
>>>> >> P.S.: I am tempted, but will likely wait until they are available in
>>>> quantity and hope that the street price comes down a bit before getting one
>>>> ;)
>>>> >
>>>> > They aren't available at all yet, and it's not clear when they will
>>>> be available.
>>>>
>>>>         The announcement was end of October, but I think I could
>>>> pre-order right now if I was feeling an urge. You are right though,
>>>> announced != available or delivered.
>>>>
>>>> Regards
>>>>         Sebastian
>>>>
>>>> P.S.: I have a pi400 in use as "desktop" for my oldest kid, this is
>>>> close to be actually generally usable, I would guess that changing a
>>>> potential p500 from the pi400's 4GB to 8 GB together with the other
>>>> imprivements the 5 brings might push it over the threshold into the truly
>>>> useful category. Which probably means that either a potential pi500 will
>>>> come late and probably with only 4 GB, but let's see how this works out now
>>>> that the supply situation is less problematic.
>>>> And I understand that there are other capable ARM based SoCs for
>>>> homerouter/desktop duty, I just happen ot have a soft spot for the
>>>> raspberry project ;)
>>>>
>>>> >
>>>> > David Lang
>>>>
>>>> _______________________________________________
>>>> Cake mailing list
>>>> Cake@lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/cake
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Dave Seddon
>>> +1 415 857 5102
>>>
>>
>>
>> --
>> Regards,
>> Dave Seddon
>> +1 415 857 5102
>>
>
>
> --
> Regards,
> Dave Seddon
> +1 415 857 5102
>


-- 
Regards,
Dave Seddon
+1 415 857 5102

[-- Attachment #1.2: Type: text/html, Size: 13743 bytes --]

[-- Attachment #2: image.png --]
[-- Type: image/png, Size: 59679 bytes --]

Re: [Cake] some comprehensive arm64 w/cake results

[dave seddon]

[-- Attachment #1.1: Type: text/plain, Size: 9047 bytes --]

Pi4 fq doesn't look too bad from a latency perspective

[image: image.png]

And wow this risc-v device is terrible!  >600 milliseconds latency.  Dave
suggested that maybe this NIC driver doesn't have the buffer depth feature,
and I suspect he's correct.

[image: image.png]





On Mon, Oct 23, 2023 at 1:31 PM dave seddon <dave.seddon.ca@gmail.com>
wrote:

> G'day,
>
> Dave Taht and I have had a couple of phone conversations now, and he's
> convinced me that rather than inserting the netem delay on each laptop,
> that latency should be added by a seperate device.  To this end, I've got
> another little PC and a NIC coming, so that I can repeat all the tests with
> seperate latency injection.
>
> However, I've also completed the flent tests with the laptops adding
> latency at each end.
>
> Full test runs here:
>
> https://github.com/randomizedcoder/qdisc_results/tree/main/qdisc/2023-10-23T16%3A49%3A10
>
> You can find the actual rrul flent .tar.gz results for each test.
>
> e.g
> Pi4 fq is here:
>
> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-23T16%3A49%3A10/pi4/fq/flent/test/16_flent/rrul-2023-10-23T170016.068273.2023-10-23T16_49_10_pi4_fq.flent.gz
>
> Lychee Pi Risv with cake qdisc:
>
> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-23T16%3A49%3A10/lpi4a/cake20/flent/test/16_flent/rrul-2023-10-23T201354.818316.2023-10-23T16_49_10_lpi4a_cake20.flent.gz
>
> Just take these with a grain of salt until the new latency injection is in
> place.
>
> ... I'll see if I can script up the generation of all the pretty graphs
> soon
>
> Thanks,
> Dave Seddon
>
>
> On Sun, Oct 15, 2023 at 8:11 AM dave seddon <dave.seddon.ca@gmail.com>
> wrote:
>
>> G'day,
>>
>> I've put more work into a test framework around the qdisc tests, but
>> unfortunately flent doesn't work easily with Ubuntu LTS (
>> https://github.com/tohojo/flent/issues/232, which I think is an issue
>> with flent parsing the fping output ).
>>
>> Results and graphs in this sheet:
>>
>> https://docs.google.com/spreadsheets/d/1T59QwEdNwJFm4TgDFA_NY98gicOm8ABXKvDsSIMz9ag/edit#gid=1203641125
>>
>> Raw results of x2 test runs are here:
>>
>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/report.csv
>>
>> Each run:
>>
>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/report.csv
>>
>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-14T14%3A22%3A53/report.csv
>>
>> Full iperf outputs are available too, for example: https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/nanopi-r2s/fq_codel/iperf/test/16_iperf/stdout
>>
>>
>> Logs for each run are also available, for example:
>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/log.json
>>
>> The code repo updated here: https://github.com/randomizedcoder/cake ,
>> with thehttps://github.com/randomizedcoder/cake/blob/main/README.md
>> which explains how the test work.
>> Updated google doc is started here:
>> https://docs.google.com/document/d/1fYKj3BS89aB9drg_DsSq289xSdVQhn1zUJYCj0WuCs0/edit?usp=sharing
>>
>> Based on the questions on this list earlier, there is a folder with
>> device information for each of the devices
>> https://github.com/randomizedcoder/cake/tree/main/device_info
>>
>> For example, the Pi4 and the Lichee Pi (risc-v) hardware layout is here:
>> - https://github.com/randomizedcoder/cake/blob/main/device_info/pi4/hwloc-ls-pi4.png
>>
>> -
>> https://github.com/randomizedcoder/cake/blob/main/device_info/lpi4a/hwloc-ls-lpi4a.png
>>
>> The switch has also been upgraded to a Cisco 3750x, which I think based
>> on the "show interface" output has a max queue size of 40 frames.  The test
>> process clears the counters before each test and gathers the "show
>> interface" output at the end.
>>
>> The Lichee Pi 4A doesn't look good (
>> https://wiki.sipeed.com/hardware/en/lichee/th1520/lp4a.html )
>>
>> [image: image.png]
>> I really wish the flent was working, so I'll probably see if I can work
>> out the parsing.
>>
>> Thanks,
>> Dave Seddon
>>
>> On Fri, Oct 13, 2023 at 10:25 AM dave seddon <dave.seddon.ca@gmail.com>
>> wrote:
>>
>>> My bad.  There's a bug for this.... Looks like I have to downgrade fping
>>>
>>> https://github.com/tohojo/flent/issues/232
>>> https://github.com/schweikert/fping/issues/203
>>>
>>> On Fri, Oct 13, 2023 at 8:59 AM dave seddon <dave.seddon.ca@gmail.com>
>>> wrote:
>>>
>>>> G'day,
>>>>
>>>> I've been working away on automation of the tests.  Pretty close to
>>>> having much nicer tests with a lot more details.  I've also got the risc-v
>>>> device working.
>>>>
>>>> However, I've run into something funny with flent.  Flent is not happy
>>>> with fping or ping.
>>>>
>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ /usr/bin/sudo
>>>> /usr/sbin/ip netns exec network101 /usr/bin/flent rrul --output
>>>>  /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/flent_pi4_noqueue.png
>>>> --data-dir /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/
>>>> --format summary --plot all_scaled --title-extra
>>>> 2023-10-13T15:53:21_pi4_noqueue --note 2023-10-13T15:53:21_pi4_noqueue
>>>> --extended-metadata --host 172.17.51.10 --length 60 --ipv4 --socket-stats
>>>> Starting Flent 2.0.1 using Python 3.10.12.
>>>> Starting rrul test. Expected run time: 70 seconds.
>>>> WARNING: Found fping, but couldn't parse its output. Not
>>>> using.              <---------------- ???
>>>> ERROR: Runner Ping (ms) ICMP failed check: Cannot parse output of the
>>>> system ping binary (/usr/bin/ping). Please install fping v3.5+.    <----- ??
>>>>
>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ dpkg --list | grep ping
>>>> ii  fping                                 5.1-1
>>>>           amd64        sends ICMP ECHO_REQUEST packets to network hosts
>>>> ii  iputils-ping                          3:20211215-1
>>>>            amd64        Tools to test the reachability of network hosts
>>>> ii  kpartx                                0.8.8-1ubuntu1.22.04.1
>>>>            amd64        create device mappings for partitions
>>>> ii  libharfbuzz0b:amd64                   2.7.4-1ubuntu3.1
>>>>            amd64        OpenType text shaping engine (shared library)
>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ fping --version
>>>> fping: Version 5.1
>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ ping -V
>>>> ping from iputils 20211215
>>>>
>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ cat /etc/lsb-release
>>>> DISTRIB_ID=Ubuntu
>>>> DISTRIB_RELEASE=22.04
>>>> DISTRIB_CODENAME=jammy
>>>> DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
>>>>
>>>> I did install via "apt install fping"
>>>>
>>>> Any thoughts please?
>>>>
>>>> Kind regards,
>>>> Dave
>>>>
>>>> On Thu, Sep 28, 2023 at 6:27 AM Sebastian Moeller via Cake <
>>>> cake@lists.bufferbloat.net> wrote:
>>>>
>>>>>
>>>>>
>>>>> > On Sep 28, 2023, at 15:19, David Lang <david@lang.hm> wrote:
>>>>> >
>>>>> > On Thu, 28 Sep 2023, Sebastian Moeller via Cake wrote:
>>>>> >
>>>>> >> P.S.: I am tempted, but will likely wait until they are available
>>>>> in quantity and hope that the street price comes down a bit before getting
>>>>> one ;)
>>>>> >
>>>>> > They aren't available at all yet, and it's not clear when they will
>>>>> be available.
>>>>>
>>>>>         The announcement was end of October, but I think I could
>>>>> pre-order right now if I was feeling an urge. You are right though,
>>>>> announced != available or delivered.
>>>>>
>>>>> Regards
>>>>>         Sebastian
>>>>>
>>>>> P.S.: I have a pi400 in use as "desktop" for my oldest kid, this is
>>>>> close to be actually generally usable, I would guess that changing a
>>>>> potential p500 from the pi400's 4GB to 8 GB together with the other
>>>>> imprivements the 5 brings might push it over the threshold into the truly
>>>>> useful category. Which probably means that either a potential pi500 will
>>>>> come late and probably with only 4 GB, but let's see how this works out now
>>>>> that the supply situation is less problematic.
>>>>> And I understand that there are other capable ARM based SoCs for
>>>>> homerouter/desktop duty, I just happen ot have a soft spot for the
>>>>> raspberry project ;)
>>>>>
>>>>> >
>>>>> > David Lang
>>>>>
>>>>> _______________________________________________
>>>>> Cake mailing list
>>>>> Cake@lists.bufferbloat.net
>>>>> https://lists.bufferbloat.net/listinfo/cake
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Dave Seddon
>>>> +1 415 857 5102
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Dave Seddon
>>> +1 415 857 5102
>>>
>>
>>
>> --
>> Regards,
>> Dave Seddon
>> +1 415 857 5102
>>
>
>
> --
> Regards,
> Dave Seddon
> +1 415 857 5102
>


-- 
Regards,
Dave Seddon
+1 415 857 5102

[-- Attachment #1.2: Type: text/html, Size: 14957 bytes --]

[-- Attachment #2: image.png --]
[-- Type: image/png, Size: 59679 bytes --]

[-- Attachment #3: image.png --]
[-- Type: image/png, Size: 350233 bytes --]

[-- Attachment #4: image.png --]
[-- Type: image/png, Size: 326697 bytes --]

Re: [Cake] some comprehensive arm64 w/cake results

[dave seddon]

[-- Attachment #1.1: Type: text/plain, Size: 9928 bytes --]

G'day,

Just to make sure the results are repeatable, I ran the flent tests again
for 600 seconds.

Directory here
https://github.com/randomizedcoder/qdisc_results/tree/main/qdisc/2023-10-24T05%3A16%3A21

Example direct links to the flent files:
https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-24T05%3A16%3A21/pi4/cake20/flent/test/16_flent/rrul-2023-10-24T061153.944938.2023-10-24T05_16_21_pi4_cake20.flent.gz
https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-24T05%3A16%3A21/jetson-nano/fq_codel/flent/test/16_flent/rrul-2023-10-24T081332.189258.2023-10-24T05_16_21_jetson-nano_fq_codel.flent.gz

I'm not experienced at reading these flent reports, but it looks to me like
fq_codel has similar latency, but higher Mb/s than cake.

Jetson nano fq_codel
[image: image.png]


Jetson nano cake20
[image: image.png]

The new network card arrived, so I need to work out how to integrate the
latency injection. ( I kind of wish I had multiple switches, so I could do
q-in-q, because this could dramatically simply the linux latency injecting
bridge machine's config. )

Kind regards,
Dave Seddon



On Mon, Oct 23, 2023 at 1:31 PM dave seddon <dave.seddon.ca@gmail.com>
wrote:

> G'day,
>
> Dave Taht and I have had a couple of phone conversations now, and he's
> convinced me that rather than inserting the netem delay on each laptop,
> that latency should be added by a seperate device.  To this end, I've got
> another little PC and a NIC coming, so that I can repeat all the tests with
> seperate latency injection.
>
> However, I've also completed the flent tests with the laptops adding
> latency at each end.
>
> Full test runs here:
>
> https://github.com/randomizedcoder/qdisc_results/tree/main/qdisc/2023-10-23T16%3A49%3A10
>
> You can find the actual rrul flent .tar.gz results for each test.
>
> e.g
> Pi4 fq is here:
>
> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-23T16%3A49%3A10/pi4/fq/flent/test/16_flent/rrul-2023-10-23T170016.068273.2023-10-23T16_49_10_pi4_fq.flent.gz
>
> Lychee Pi Risv with cake qdisc:
>
> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-23T16%3A49%3A10/lpi4a/cake20/flent/test/16_flent/rrul-2023-10-23T201354.818316.2023-10-23T16_49_10_lpi4a_cake20.flent.gz
>
> Just take these with a grain of salt until the new latency injection is in
> place.
>
> ... I'll see if I can script up the generation of all the pretty graphs
> soon
>
> Thanks,
> Dave Seddon
>
>
> On Sun, Oct 15, 2023 at 8:11 AM dave seddon <dave.seddon.ca@gmail.com>
> wrote:
>
>> G'day,
>>
>> I've put more work into a test framework around the qdisc tests, but
>> unfortunately flent doesn't work easily with Ubuntu LTS (
>> https://github.com/tohojo/flent/issues/232, which I think is an issue
>> with flent parsing the fping output ).
>>
>> Results and graphs in this sheet:
>>
>> https://docs.google.com/spreadsheets/d/1T59QwEdNwJFm4TgDFA_NY98gicOm8ABXKvDsSIMz9ag/edit#gid=1203641125
>>
>> Raw results of x2 test runs are here:
>>
>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/report.csv
>>
>> Each run:
>>
>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/report.csv
>>
>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-14T14%3A22%3A53/report.csv
>>
>> Full iperf outputs are available too, for example: https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/nanopi-r2s/fq_codel/iperf/test/16_iperf/stdout
>>
>>
>> Logs for each run are also available, for example:
>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/log.json
>>
>> The code repo updated here: https://github.com/randomizedcoder/cake ,
>> with thehttps://github.com/randomizedcoder/cake/blob/main/README.md
>> which explains how the test work.
>> Updated google doc is started here:
>> https://docs.google.com/document/d/1fYKj3BS89aB9drg_DsSq289xSdVQhn1zUJYCj0WuCs0/edit?usp=sharing
>>
>> Based on the questions on this list earlier, there is a folder with
>> device information for each of the devices
>> https://github.com/randomizedcoder/cake/tree/main/device_info
>>
>> For example, the Pi4 and the Lichee Pi (risc-v) hardware layout is here:
>> - https://github.com/randomizedcoder/cake/blob/main/device_info/pi4/hwloc-ls-pi4.png
>>
>> -
>> https://github.com/randomizedcoder/cake/blob/main/device_info/lpi4a/hwloc-ls-lpi4a.png
>>
>> The switch has also been upgraded to a Cisco 3750x, which I think based
>> on the "show interface" output has a max queue size of 40 frames.  The test
>> process clears the counters before each test and gathers the "show
>> interface" output at the end.
>>
>> The Lichee Pi 4A doesn't look good (
>> https://wiki.sipeed.com/hardware/en/lichee/th1520/lp4a.html )
>>
>> [image: image.png]
>> I really wish the flent was working, so I'll probably see if I can work
>> out the parsing.
>>
>> Thanks,
>> Dave Seddon
>>
>> On Fri, Oct 13, 2023 at 10:25 AM dave seddon <dave.seddon.ca@gmail.com>
>> wrote:
>>
>>> My bad.  There's a bug for this.... Looks like I have to downgrade fping
>>>
>>> https://github.com/tohojo/flent/issues/232
>>> https://github.com/schweikert/fping/issues/203
>>>
>>> On Fri, Oct 13, 2023 at 8:59 AM dave seddon <dave.seddon.ca@gmail.com>
>>> wrote:
>>>
>>>> G'day,
>>>>
>>>> I've been working away on automation of the tests.  Pretty close to
>>>> having much nicer tests with a lot more details.  I've also got the risc-v
>>>> device working.
>>>>
>>>> However, I've run into something funny with flent.  Flent is not happy
>>>> with fping or ping.
>>>>
>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ /usr/bin/sudo
>>>> /usr/sbin/ip netns exec network101 /usr/bin/flent rrul --output
>>>>  /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/flent_pi4_noqueue.png
>>>> --data-dir /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/
>>>> --format summary --plot all_scaled --title-extra
>>>> 2023-10-13T15:53:21_pi4_noqueue --note 2023-10-13T15:53:21_pi4_noqueue
>>>> --extended-metadata --host 172.17.51.10 --length 60 --ipv4 --socket-stats
>>>> Starting Flent 2.0.1 using Python 3.10.12.
>>>> Starting rrul test. Expected run time: 70 seconds.
>>>> WARNING: Found fping, but couldn't parse its output. Not
>>>> using.              <---------------- ???
>>>> ERROR: Runner Ping (ms) ICMP failed check: Cannot parse output of the
>>>> system ping binary (/usr/bin/ping). Please install fping v3.5+.    <----- ??
>>>>
>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ dpkg --list | grep ping
>>>> ii  fping                                 5.1-1
>>>>           amd64        sends ICMP ECHO_REQUEST packets to network hosts
>>>> ii  iputils-ping                          3:20211215-1
>>>>            amd64        Tools to test the reachability of network hosts
>>>> ii  kpartx                                0.8.8-1ubuntu1.22.04.1
>>>>            amd64        create device mappings for partitions
>>>> ii  libharfbuzz0b:amd64                   2.7.4-1ubuntu3.1
>>>>            amd64        OpenType text shaping engine (shared library)
>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ fping --version
>>>> fping: Version 5.1
>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ ping -V
>>>> ping from iputils 20211215
>>>>
>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ cat /etc/lsb-release
>>>> DISTRIB_ID=Ubuntu
>>>> DISTRIB_RELEASE=22.04
>>>> DISTRIB_CODENAME=jammy
>>>> DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
>>>>
>>>> I did install via "apt install fping"
>>>>
>>>> Any thoughts please?
>>>>
>>>> Kind regards,
>>>> Dave
>>>>
>>>> On Thu, Sep 28, 2023 at 6:27 AM Sebastian Moeller via Cake <
>>>> cake@lists.bufferbloat.net> wrote:
>>>>
>>>>>
>>>>>
>>>>> > On Sep 28, 2023, at 15:19, David Lang <david@lang.hm> wrote:
>>>>> >
>>>>> > On Thu, 28 Sep 2023, Sebastian Moeller via Cake wrote:
>>>>> >
>>>>> >> P.S.: I am tempted, but will likely wait until they are available
>>>>> in quantity and hope that the street price comes down a bit before getting
>>>>> one ;)
>>>>> >
>>>>> > They aren't available at all yet, and it's not clear when they will
>>>>> be available.
>>>>>
>>>>>         The announcement was end of October, but I think I could
>>>>> pre-order right now if I was feeling an urge. You are right though,
>>>>> announced != available or delivered.
>>>>>
>>>>> Regards
>>>>>         Sebastian
>>>>>
>>>>> P.S.: I have a pi400 in use as "desktop" for my oldest kid, this is
>>>>> close to be actually generally usable, I would guess that changing a
>>>>> potential p500 from the pi400's 4GB to 8 GB together with the other
>>>>> imprivements the 5 brings might push it over the threshold into the truly
>>>>> useful category. Which probably means that either a potential pi500 will
>>>>> come late and probably with only 4 GB, but let's see how this works out now
>>>>> that the supply situation is less problematic.
>>>>> And I understand that there are other capable ARM based SoCs for
>>>>> homerouter/desktop duty, I just happen ot have a soft spot for the
>>>>> raspberry project ;)
>>>>>
>>>>> >
>>>>> > David Lang
>>>>>
>>>>> _______________________________________________
>>>>> Cake mailing list
>>>>> Cake@lists.bufferbloat.net
>>>>> https://lists.bufferbloat.net/listinfo/cake
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Dave Seddon
>>>> +1 415 857 5102
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Dave Seddon
>>> +1 415 857 5102
>>>
>>
>>
>> --
>> Regards,
>> Dave Seddon
>> +1 415 857 5102
>>
>
>
> --
> Regards,
> Dave Seddon
> +1 415 857 5102
>


-- 
Regards,
Dave Seddon
+1 415 857 5102

[-- Attachment #1.2: Type: text/html, Size: 16551 bytes --]

[-- Attachment #2: image.png --]
[-- Type: image/png, Size: 59679 bytes --]

[-- Attachment #3: image.png --]
[-- Type: image/png, Size: 378056 bytes --]

[-- Attachment #4: image.png --]
[-- Type: image/png, Size: 336208 bytes --]

Re: [Cake] some comprehensive arm64 w/cake results

[dave seddon]

[-- Attachment #1.1: Type: text/plain, Size: 12798 bytes --]

G'day Dave,

Regarding the devices under test with Byte Queue Limits (BQL).

lychee pi ( https://wiki.sipeed.com/hardware/en/lichee/th1520/lp4a.html )
with driver "st_gmac" does NOT have BQL. ( not a great surprise )

Driver is:
https://github.com/randomizedcoder/cake/blob/main/device_info/lpi4a/ethtool_driver_end0

das@lpi4a:~$ /usr/sbin/ethtool -i end0
driver: st_gmac                                <-----------
version: Jan_2016
firmware-version:
expansion-rom-version:
bus-info:
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

Jetson-nano has driver r8169 and is also not BQL enabled.
https://github.com/randomizedcoder/cake/blob/main/device_info/jetson-nano/ethtool_driver_eth0

Based on doc here:
https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/,
the current drivers with BQL are:

das@t:~/Downloads/linux$ find drivers/net/ -name '*.c' -print | \
xargs fgrep -l netdev_completed_queue
drivers/net/can/spi/mcp251xfd/mcp251xfd-tef.c
drivers/net/can/usb/etas_es58x/es58x_core.c
drivers/net/can/dev/length.c
drivers/net/ipa/ipa_gsi.c
drivers/net/wan/fsl_ucc_hdlc.c
drivers/net/ethernet/freescale/ucc_geth.c
drivers/net/ethernet/marvell/skge.c
drivers/net/ethernet/marvell/sky2.c
drivers/net/ethernet/qualcomm/emac/emac-mac.c
drivers/net/ethernet/socionext/netsec.c
drivers/net/ethernet/atheros/ag71xx.c
drivers/net/ethernet/mediatek/mtk_star_emac.c
drivers/net/ethernet/3com/3c59x.c
drivers/net/ethernet/broadcom/bcm4908_enet.c
drivers/net/ethernet/broadcom/bgmac.c
drivers/net/ethernet/broadcom/b44.c
drivers/net/ethernet/broadcom/bcm63xx_enet.c
drivers/net/ethernet/realtek/8139cp.c                    <------- close,
but NOT r8169
drivers/net/ethernet/lantiq_xrx200.c
drivers/net/ethernet/via/via-rhine.c
drivers/net/ethernet/hisilicon/hix5hd2_gmac.c
drivers/net/ethernet/hisilicon/hip04_eth.c
drivers/net/ethernet/hisilicon/hisi_femac.c
drivers/net/ethernet/intel/e1000e/netdev.c
drivers/net/ethernet/intel/e1000/e1000_main.c
drivers/net/ethernet/nvidia/forcedeth.c

The st_gmac driver can also be seen here:
https://github.com/torvalds/linux/blob/4f82870119a46b0d04d91ef4697ac4977a255a9d/drivers/net/ethernet/stmicro/stmmac/stmmac_ethtool.c#L26

I guess I can take a look at creating an MR or two (2).

Regards,
Dave Seddon

On Tue, Oct 24, 2023 at 9:27 AM dave seddon <dave.seddon.ca@gmail.com>
wrote:

> G'day,
>
> Just to make sure the results are repeatable, I ran the flent tests again
> for 600 seconds.
>
> Directory here
>
> https://github.com/randomizedcoder/qdisc_results/tree/main/qdisc/2023-10-24T05%3A16%3A21
>
> Example direct links to the flent files:
>
> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-24T05%3A16%3A21/pi4/cake20/flent/test/16_flent/rrul-2023-10-24T061153.944938.2023-10-24T05_16_21_pi4_cake20.flent.gz
>
> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-24T05%3A16%3A21/jetson-nano/fq_codel/flent/test/16_flent/rrul-2023-10-24T081332.189258.2023-10-24T05_16_21_jetson-nano_fq_codel.flent.gz
>
> I'm not experienced at reading these flent reports, but it looks to me
> like fq_codel has similar latency, but higher Mb/s than cake.
>
> Jetson nano fq_codel
> [image: image.png]
>
>
> Jetson nano cake20
> [image: image.png]
>
> The new network card arrived, so I need to work out how to integrate the
> latency injection. ( I kind of wish I had multiple switches, so I could do
> q-in-q, because this could dramatically simply the linux latency injecting
> bridge machine's config. )
>
> Kind regards,
> Dave Seddon
>
>
>
> On Mon, Oct 23, 2023 at 1:31 PM dave seddon <dave.seddon.ca@gmail.com>
> wrote:
>
>> G'day,
>>
>> Dave Taht and I have had a couple of phone conversations now, and he's
>> convinced me that rather than inserting the netem delay on each laptop,
>> that latency should be added by a seperate device.  To this end, I've got
>> another little PC and a NIC coming, so that I can repeat all the tests with
>> seperate latency injection.
>>
>> However, I've also completed the flent tests with the laptops adding
>> latency at each end.
>>
>> Full test runs here:
>>
>> https://github.com/randomizedcoder/qdisc_results/tree/main/qdisc/2023-10-23T16%3A49%3A10
>>
>> You can find the actual rrul flent .tar.gz results for each test.
>>
>> e.g
>> Pi4 fq is here:
>>
>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-23T16%3A49%3A10/pi4/fq/flent/test/16_flent/rrul-2023-10-23T170016.068273.2023-10-23T16_49_10_pi4_fq.flent.gz
>>
>> Lychee Pi Risv with cake qdisc:
>>
>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-23T16%3A49%3A10/lpi4a/cake20/flent/test/16_flent/rrul-2023-10-23T201354.818316.2023-10-23T16_49_10_lpi4a_cake20.flent.gz
>>
>> Just take these with a grain of salt until the new latency injection is
>> in place.
>>
>> ... I'll see if I can script up the generation of all the pretty graphs
>> soon
>>
>> Thanks,
>> Dave Seddon
>>
>>
>> On Sun, Oct 15, 2023 at 8:11 AM dave seddon <dave.seddon.ca@gmail.com>
>> wrote:
>>
>>> G'day,
>>>
>>> I've put more work into a test framework around the qdisc tests, but
>>> unfortunately flent doesn't work easily with Ubuntu LTS (
>>> https://github.com/tohojo/flent/issues/232, which I think is an issue
>>> with flent parsing the fping output ).
>>>
>>> Results and graphs in this sheet:
>>>
>>> https://docs.google.com/spreadsheets/d/1T59QwEdNwJFm4TgDFA_NY98gicOm8ABXKvDsSIMz9ag/edit#gid=1203641125
>>>
>>> Raw results of x2 test runs are here:
>>>
>>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/report.csv
>>>
>>> Each run:
>>>
>>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/report.csv
>>>
>>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-14T14%3A22%3A53/report.csv
>>>
>>> Full iperf outputs are available too, for example: https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/nanopi-r2s/fq_codel/iperf/test/16_iperf/stdout
>>>
>>>
>>> Logs for each run are also available, for example:
>>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/log.json
>>>
>>> The code repo updated here: https://github.com/randomizedcoder/cake ,
>>> with thehttps://github.com/randomizedcoder/cake/blob/main/README.md
>>> which explains how the test work.
>>> Updated google doc is started here:
>>> https://docs.google.com/document/d/1fYKj3BS89aB9drg_DsSq289xSdVQhn1zUJYCj0WuCs0/edit?usp=sharing
>>>
>>> Based on the questions on this list earlier, there is a folder with
>>> device information for each of the devices
>>> https://github.com/randomizedcoder/cake/tree/main/device_info
>>>
>>> For example, the Pi4 and the Lichee Pi (risc-v) hardware layout is here:
>>> - https://github.com/randomizedcoder/cake/blob/main/device_info/pi4/hwloc-ls-pi4.png
>>>
>>> -
>>> https://github.com/randomizedcoder/cake/blob/main/device_info/lpi4a/hwloc-ls-lpi4a.png
>>>
>>> The switch has also been upgraded to a Cisco 3750x, which I think based
>>> on the "show interface" output has a max queue size of 40 frames.  The test
>>> process clears the counters before each test and gathers the "show
>>> interface" output at the end.
>>>
>>> The Lichee Pi 4A doesn't look good (
>>> https://wiki.sipeed.com/hardware/en/lichee/th1520/lp4a.html )
>>>
>>> [image: image.png]
>>> I really wish the flent was working, so I'll probably see if I can work
>>> out the parsing.
>>>
>>> Thanks,
>>> Dave Seddon
>>>
>>> On Fri, Oct 13, 2023 at 10:25 AM dave seddon <dave.seddon.ca@gmail.com>
>>> wrote:
>>>
>>>> My bad.  There's a bug for this.... Looks like I have to downgrade fping
>>>>
>>>> https://github.com/tohojo/flent/issues/232
>>>> https://github.com/schweikert/fping/issues/203
>>>>
>>>> On Fri, Oct 13, 2023 at 8:59 AM dave seddon <dave.seddon.ca@gmail.com>
>>>> wrote:
>>>>
>>>>> G'day,
>>>>>
>>>>> I've been working away on automation of the tests.  Pretty close to
>>>>> having much nicer tests with a lot more details.  I've also got the risc-v
>>>>> device working.
>>>>>
>>>>> However, I've run into something funny with flent.  Flent is not happy
>>>>> with fping or ping.
>>>>>
>>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ /usr/bin/sudo
>>>>> /usr/sbin/ip netns exec network101 /usr/bin/flent rrul --output
>>>>>  /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/flent_pi4_noqueue.png
>>>>> --data-dir /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/
>>>>> --format summary --plot all_scaled --title-extra
>>>>> 2023-10-13T15:53:21_pi4_noqueue --note 2023-10-13T15:53:21_pi4_noqueue
>>>>> --extended-metadata --host 172.17.51.10 --length 60 --ipv4 --socket-stats
>>>>> Starting Flent 2.0.1 using Python 3.10.12.
>>>>> Starting rrul test. Expected run time: 70 seconds.
>>>>> WARNING: Found fping, but couldn't parse its output. Not
>>>>> using.              <---------------- ???
>>>>> ERROR: Runner Ping (ms) ICMP failed check: Cannot parse output of the
>>>>> system ping binary (/usr/bin/ping). Please install fping v3.5+.    <----- ??
>>>>>
>>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ dpkg --list | grep ping
>>>>> ii  fping                                 5.1-1
>>>>>             amd64        sends ICMP ECHO_REQUEST packets to network hosts
>>>>> ii  iputils-ping                          3:20211215-1
>>>>>            amd64        Tools to test the reachability of network hosts
>>>>> ii  kpartx                                0.8.8-1ubuntu1.22.04.1
>>>>>            amd64        create device mappings for partitions
>>>>> ii  libharfbuzz0b:amd64                   2.7.4-1ubuntu3.1
>>>>>            amd64        OpenType text shaping engine (shared library)
>>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ fping --version
>>>>> fping: Version 5.1
>>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ ping -V
>>>>> ping from iputils 20211215
>>>>>
>>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ cat /etc/lsb-release
>>>>> DISTRIB_ID=Ubuntu
>>>>> DISTRIB_RELEASE=22.04
>>>>> DISTRIB_CODENAME=jammy
>>>>> DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
>>>>>
>>>>> I did install via "apt install fping"
>>>>>
>>>>> Any thoughts please?
>>>>>
>>>>> Kind regards,
>>>>> Dave
>>>>>
>>>>> On Thu, Sep 28, 2023 at 6:27 AM Sebastian Moeller via Cake <
>>>>> cake@lists.bufferbloat.net> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> > On Sep 28, 2023, at 15:19, David Lang <david@lang.hm> wrote:
>>>>>> >
>>>>>> > On Thu, 28 Sep 2023, Sebastian Moeller via Cake wrote:
>>>>>> >
>>>>>> >> P.S.: I am tempted, but will likely wait until they are available
>>>>>> in quantity and hope that the street price comes down a bit before getting
>>>>>> one ;)
>>>>>> >
>>>>>> > They aren't available at all yet, and it's not clear when they will
>>>>>> be available.
>>>>>>
>>>>>>         The announcement was end of October, but I think I could
>>>>>> pre-order right now if I was feeling an urge. You are right though,
>>>>>> announced != available or delivered.
>>>>>>
>>>>>> Regards
>>>>>>         Sebastian
>>>>>>
>>>>>> P.S.: I have a pi400 in use as "desktop" for my oldest kid, this is
>>>>>> close to be actually generally usable, I would guess that changing a
>>>>>> potential p500 from the pi400's 4GB to 8 GB together with the other
>>>>>> imprivements the 5 brings might push it over the threshold into the truly
>>>>>> useful category. Which probably means that either a potential pi500 will
>>>>>> come late and probably with only 4 GB, but let's see how this works out now
>>>>>> that the supply situation is less problematic.
>>>>>> And I understand that there are other capable ARM based SoCs for
>>>>>> homerouter/desktop duty, I just happen ot have a soft spot for the
>>>>>> raspberry project ;)
>>>>>>
>>>>>> >
>>>>>> > David Lang
>>>>>>
>>>>>> _______________________________________________
>>>>>> Cake mailing list
>>>>>> Cake@lists.bufferbloat.net
>>>>>> https://lists.bufferbloat.net/listinfo/cake
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Dave Seddon
>>>>> +1 415 857 5102
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Dave Seddon
>>>> +1 415 857 5102
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Dave Seddon
>>> +1 415 857 5102
>>>
>>
>>
>> --
>> Regards,
>> Dave Seddon
>> +1 415 857 5102
>>
>
>
> --
> Regards,
> Dave Seddon
> +1 415 857 5102
>


-- 
Regards,
Dave Seddon
+1 415 857 5102

[-- Attachment #1.2: Type: text/html, Size: 20684 bytes --]

[-- Attachment #2: image.png --]
[-- Type: image/png, Size: 59679 bytes --]

[-- Attachment #3: image.png --]
[-- Type: image/png, Size: 378056 bytes --]

[-- Attachment #4: image.png --]
[-- Type: image/png, Size: 336208 bytes --]

Re: [Cake] some comprehensive arm64 w/cake results

[Dave Taht]

[-- Attachment #1.1: Type: text/plain, Size: 13856 bytes --]

that is a really out of date doc on the bufferbloat site. Last I ran rg for
the network driver tree it was in the many hundreds depending on how you
count variants, but hundreds more left undone.

There is a guide to creating the six or so lines needed here:

http://www.taht.net/~d/broadcom_aug9_2018.pdf

Somewhere around on some preso of mine was the before/after for the
beaglebone black. It was NICE.

On Tue, Oct 24, 2023 at 2:35 PM dave seddon <dave.seddon.ca@gmail.com>
wrote:

> G'day Dave,
>
> Regarding the devices under test with Byte Queue Limits (BQL).
>
> lychee pi ( https://wiki.sipeed.com/hardware/en/lichee/th1520/lp4a.html )
> with driver "st_gmac" does NOT have BQL. ( not a great surprise )
>
> Driver is:
>
> https://github.com/randomizedcoder/cake/blob/main/device_info/lpi4a/ethtool_driver_end0
>
> das@lpi4a:~$ /usr/sbin/ethtool -i end0
> driver: st_gmac                                <-----------
> version: Jan_2016
> firmware-version:
> expansion-rom-version:
> bus-info:
> supports-statistics: yes
> supports-test: no
> supports-eeprom-access: no
> supports-register-dump: yes
> supports-priv-flags: no
>
> Jetson-nano has driver r8169 and is also not BQL enabled.
>
> https://github.com/randomizedcoder/cake/blob/main/device_info/jetson-nano/ethtool_driver_eth0
>
> Based on doc here:
> https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/,
> the current drivers with BQL are:
>
> das@t:~/Downloads/linux$ find drivers/net/ -name '*.c' -print | \
> xargs fgrep -l netdev_completed_queue
> drivers/net/can/spi/mcp251xfd/mcp251xfd-tef.c
> drivers/net/can/usb/etas_es58x/es58x_core.c
> drivers/net/can/dev/length.c
> drivers/net/ipa/ipa_gsi.c
> drivers/net/wan/fsl_ucc_hdlc.c
> drivers/net/ethernet/freescale/ucc_geth.c
> drivers/net/ethernet/marvell/skge.c
> drivers/net/ethernet/marvell/sky2.c
> drivers/net/ethernet/qualcomm/emac/emac-mac.c
> drivers/net/ethernet/socionext/netsec.c
> drivers/net/ethernet/atheros/ag71xx.c
> drivers/net/ethernet/mediatek/mtk_star_emac.c
> drivers/net/ethernet/3com/3c59x.c
> drivers/net/ethernet/broadcom/bcm4908_enet.c
> drivers/net/ethernet/broadcom/bgmac.c
> drivers/net/ethernet/broadcom/b44.c
> drivers/net/ethernet/broadcom/bcm63xx_enet.c
> drivers/net/ethernet/realtek/8139cp.c                    <------- close,
> but NOT r8169
> drivers/net/ethernet/lantiq_xrx200.c
> drivers/net/ethernet/via/via-rhine.c
> drivers/net/ethernet/hisilicon/hix5hd2_gmac.c
> drivers/net/ethernet/hisilicon/hip04_eth.c
> drivers/net/ethernet/hisilicon/hisi_femac.c
> drivers/net/ethernet/intel/e1000e/netdev.c
> drivers/net/ethernet/intel/e1000/e1000_main.c
> drivers/net/ethernet/nvidia/forcedeth.c
>
> The st_gmac driver can also be seen here:
>
> https://github.com/torvalds/linux/blob/4f82870119a46b0d04d91ef4697ac4977a255a9d/drivers/net/ethernet/stmicro/stmmac/stmmac_ethtool.c#L26
>
> I guess I can take a look at creating an MR or two (2).
>
> Regards,
> Dave Seddon
>
> On Tue, Oct 24, 2023 at 9:27 AM dave seddon <dave.seddon.ca@gmail.com>
> wrote:
>
>> G'day,
>>
>> Just to make sure the results are repeatable, I ran the flent tests again
>> for 600 seconds.
>>
>> Directory here
>>
>> https://github.com/randomizedcoder/qdisc_results/tree/main/qdisc/2023-10-24T05%3A16%3A21
>>
>> Example direct links to the flent files:
>>
>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-24T05%3A16%3A21/pi4/cake20/flent/test/16_flent/rrul-2023-10-24T061153.944938.2023-10-24T05_16_21_pi4_cake20.flent.gz
>>
>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-24T05%3A16%3A21/jetson-nano/fq_codel/flent/test/16_flent/rrul-2023-10-24T081332.189258.2023-10-24T05_16_21_jetson-nano_fq_codel.flent.gz
>>
>> I'm not experienced at reading these flent reports, but it looks to me
>> like fq_codel has similar latency, but higher Mb/s than cake.
>>
>> Jetson nano fq_codel
>> [image: image.png]
>>
>>
>> Jetson nano cake20
>> [image: image.png]
>>
>> The new network card arrived, so I need to work out how to integrate the
>> latency injection. ( I kind of wish I had multiple switches, so I could do
>> q-in-q, because this could dramatically simply the linux latency injecting
>> bridge machine's config. )
>>
>> Kind regards,
>> Dave Seddon
>>
>>
>>
>> On Mon, Oct 23, 2023 at 1:31 PM dave seddon <dave.seddon.ca@gmail.com>
>> wrote:
>>
>>> G'day,
>>>
>>> Dave Taht and I have had a couple of phone conversations now, and he's
>>> convinced me that rather than inserting the netem delay on each laptop,
>>> that latency should be added by a seperate device.  To this end, I've got
>>> another little PC and a NIC coming, so that I can repeat all the tests with
>>> seperate latency injection.
>>>
>>> However, I've also completed the flent tests with the laptops adding
>>> latency at each end.
>>>
>>> Full test runs here:
>>>
>>> https://github.com/randomizedcoder/qdisc_results/tree/main/qdisc/2023-10-23T16%3A49%3A10
>>>
>>> You can find the actual rrul flent .tar.gz results for each test.
>>>
>>> e.g
>>> Pi4 fq is here:
>>>
>>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-23T16%3A49%3A10/pi4/fq/flent/test/16_flent/rrul-2023-10-23T170016.068273.2023-10-23T16_49_10_pi4_fq.flent.gz
>>>
>>> Lychee Pi Risv with cake qdisc:
>>>
>>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-23T16%3A49%3A10/lpi4a/cake20/flent/test/16_flent/rrul-2023-10-23T201354.818316.2023-10-23T16_49_10_lpi4a_cake20.flent.gz
>>>
>>> Just take these with a grain of salt until the new latency injection is
>>> in place.
>>>
>>> ... I'll see if I can script up the generation of all the pretty graphs
>>> soon
>>>
>>> Thanks,
>>> Dave Seddon
>>>
>>>
>>> On Sun, Oct 15, 2023 at 8:11 AM dave seddon <dave.seddon.ca@gmail.com>
>>> wrote:
>>>
>>>> G'day,
>>>>
>>>> I've put more work into a test framework around the qdisc tests, but
>>>> unfortunately flent doesn't work easily with Ubuntu LTS (
>>>> https://github.com/tohojo/flent/issues/232, which I think is an issue
>>>> with flent parsing the fping output ).
>>>>
>>>> Results and graphs in this sheet:
>>>>
>>>> https://docs.google.com/spreadsheets/d/1T59QwEdNwJFm4TgDFA_NY98gicOm8ABXKvDsSIMz9ag/edit#gid=1203641125
>>>>
>>>> Raw results of x2 test runs are here:
>>>>
>>>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/report.csv
>>>>
>>>> Each run:
>>>>
>>>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/report.csv
>>>>
>>>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-14T14%3A22%3A53/report.csv
>>>>
>>>> Full iperf outputs are available too, for example: https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/nanopi-r2s/fq_codel/iperf/test/16_iperf/stdout
>>>>
>>>>
>>>> Logs for each run are also available, for example:
>>>> https://github.com/randomizedcoder/qdisc_results/blob/main/qdisc/2023-10-13T18%3A45%3A45/log.json
>>>>
>>>> The code repo updated here: https://github.com/randomizedcoder/cake ,
>>>> with thehttps://github.com/randomizedcoder/cake/blob/main/README.md
>>>> which explains how the test work.
>>>> Updated google doc is started here:
>>>> https://docs.google.com/document/d/1fYKj3BS89aB9drg_DsSq289xSdVQhn1zUJYCj0WuCs0/edit?usp=sharing
>>>>
>>>> Based on the questions on this list earlier, there is a folder with
>>>> device information for each of the devices
>>>> https://github.com/randomizedcoder/cake/tree/main/device_info
>>>>
>>>> For example, the Pi4 and the Lichee Pi (risc-v) hardware layout is here:
>>>> - https://github.com/randomizedcoder/cake/blob/main/device_info/pi4/hwloc-ls-pi4.png
>>>>
>>>> -
>>>> https://github.com/randomizedcoder/cake/blob/main/device_info/lpi4a/hwloc-ls-lpi4a.png
>>>>
>>>> The switch has also been upgraded to a Cisco 3750x, which I think based
>>>> on the "show interface" output has a max queue size of 40 frames.  The test
>>>> process clears the counters before each test and gathers the "show
>>>> interface" output at the end.
>>>>
>>>> The Lichee Pi 4A doesn't look good (
>>>> https://wiki.sipeed.com/hardware/en/lichee/th1520/lp4a.html )
>>>>
>>>> [image: image.png]
>>>> I really wish the flent was working, so I'll probably see if I can work
>>>> out the parsing.
>>>>
>>>> Thanks,
>>>> Dave Seddon
>>>>
>>>> On Fri, Oct 13, 2023 at 10:25 AM dave seddon <dave.seddon.ca@gmail.com>
>>>> wrote:
>>>>
>>>>> My bad.  There's a bug for this.... Looks like I have to downgrade
>>>>> fping
>>>>>
>>>>> https://github.com/tohojo/flent/issues/232
>>>>> https://github.com/schweikert/fping/issues/203
>>>>>
>>>>> On Fri, Oct 13, 2023 at 8:59 AM dave seddon <dave.seddon.ca@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> G'day,
>>>>>>
>>>>>> I've been working away on automation of the tests.  Pretty close to
>>>>>> having much nicer tests with a lot more details.  I've also got the risc-v
>>>>>> device working.
>>>>>>
>>>>>> However, I've run into something funny with flent.  Flent is not
>>>>>> happy with fping or ping.
>>>>>>
>>>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ /usr/bin/sudo
>>>>>> /usr/sbin/ip netns exec network101 /usr/bin/flent rrul --output
>>>>>>  /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/flent_pi4_noqueue.png
>>>>>> --data-dir /tmp/qdisc/2023-10-13T15:53:21/pi4/noqueue/flent/test/15_flent/
>>>>>> --format summary --plot all_scaled --title-extra
>>>>>> 2023-10-13T15:53:21_pi4_noqueue --note 2023-10-13T15:53:21_pi4_noqueue
>>>>>> --extended-metadata --host 172.17.51.10 --length 60 --ipv4 --socket-stats
>>>>>> Starting Flent 2.0.1 using Python 3.10.12.
>>>>>> Starting rrul test. Expected run time: 70 seconds.
>>>>>> WARNING: Found fping, but couldn't parse its output. Not
>>>>>> using.              <---------------- ???
>>>>>> ERROR: Runner Ping (ms) ICMP failed check: Cannot parse output of the
>>>>>> system ping binary (/usr/bin/ping). Please install fping v3.5+.    <----- ??
>>>>>>
>>>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ dpkg --list | grep
>>>>>> ping
>>>>>> ii  fping                                 5.1-1
>>>>>>             amd64        sends ICMP ECHO_REQUEST packets to network hosts
>>>>>> ii  iputils-ping                          3:20211215-1
>>>>>>              amd64        Tools to test the reachability of network hosts
>>>>>> ii  kpartx                                0.8.8-1ubuntu1.22.04.1
>>>>>>              amd64        create device mappings for partitions
>>>>>> ii  libharfbuzz0b:amd64                   2.7.4-1ubuntu3.1
>>>>>>              amd64        OpenType text shaping engine (shared library)
>>>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ fping --version
>>>>>> fping: Version 5.1
>>>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ ping -V
>>>>>> ping from iputils 20211215
>>>>>>
>>>>>> das@3rd:~/Downloads/cake/cmd/run_qdiscs_tests$ cat /etc/lsb-release
>>>>>> DISTRIB_ID=Ubuntu
>>>>>> DISTRIB_RELEASE=22.04
>>>>>> DISTRIB_CODENAME=jammy
>>>>>> DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
>>>>>>
>>>>>> I did install via "apt install fping"
>>>>>>
>>>>>> Any thoughts please?
>>>>>>
>>>>>> Kind regards,
>>>>>> Dave
>>>>>>
>>>>>> On Thu, Sep 28, 2023 at 6:27 AM Sebastian Moeller via Cake <
>>>>>> cake@lists.bufferbloat.net> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> > On Sep 28, 2023, at 15:19, David Lang <david@lang.hm> wrote:
>>>>>>> >
>>>>>>> > On Thu, 28 Sep 2023, Sebastian Moeller via Cake wrote:
>>>>>>> >
>>>>>>> >> P.S.: I am tempted, but will likely wait until they are available
>>>>>>> in quantity and hope that the street price comes down a bit before getting
>>>>>>> one ;)
>>>>>>> >
>>>>>>> > They aren't available at all yet, and it's not clear when they
>>>>>>> will be available.
>>>>>>>
>>>>>>>         The announcement was end of October, but I think I could
>>>>>>> pre-order right now if I was feeling an urge. You are right though,
>>>>>>> announced != available or delivered.
>>>>>>>
>>>>>>> Regards
>>>>>>>         Sebastian
>>>>>>>
>>>>>>> P.S.: I have a pi400 in use as "desktop" for my oldest kid, this is
>>>>>>> close to be actually generally usable, I would guess that changing a
>>>>>>> potential p500 from the pi400's 4GB to 8 GB together with the other
>>>>>>> imprivements the 5 brings might push it over the threshold into the truly
>>>>>>> useful category. Which probably means that either a potential pi500 will
>>>>>>> come late and probably with only 4 GB, but let's see how this works out now
>>>>>>> that the supply situation is less problematic.
>>>>>>> And I understand that there are other capable ARM based SoCs for
>>>>>>> homerouter/desktop duty, I just happen ot have a soft spot for the
>>>>>>> raspberry project ;)
>>>>>>>
>>>>>>> >
>>>>>>> > David Lang
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Cake mailing list
>>>>>>> Cake@lists.bufferbloat.net
>>>>>>> https://lists.bufferbloat.net/listinfo/cake
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Dave Seddon
>>>>>> +1 415 857 5102
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Dave Seddon
>>>>> +1 415 857 5102
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Dave Seddon
>>>> +1 415 857 5102
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Dave Seddon
>>> +1 415 857 5102
>>>
>>
>>
>> --
>> Regards,
>> Dave Seddon
>> +1 415 857 5102
>>
>
>
> --
> Regards,
> Dave Seddon
> +1 415 857 5102
>


-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

[-- Attachment #1.2: Type: text/html, Size: 22065 bytes --]

[-- Attachment #2: image.png --]
[-- Type: image/png, Size: 59679 bytes --]

[-- Attachment #3: image.png --]
[-- Type: image/png, Size: 378056 bytes --]

[-- Attachment #4: image.png --]
[-- Type: image/png, Size: 336208 bytes --]