[Cake] Multiple Hardware Queues

[Jonathan Morton]

> On 15 Jul, 2018, at 10:41 am, dag dg <dagofthedofg at gmail.com> wrote:
> 
> In my box I have acting as a router I have an Intel i350-t2v2 nic that
> has two gigabit ports(uplink/local). This card and its corresponding
> driver supports multiple hardware-based transmit and receive queues
> depending on the number of cores the system has up to 8.

> qdisc cake 802c: dev enp2s0f0 root refcnt 9 bandwidth 23Mbit diffserv3 triple-isolate split-gso rtt 100.0ms raw overhead 0

> qdisc cake 802d: dev ifb4enp2s0f0 root refcnt 2 bandwidth 330Mbit besteffort triple-isolate wash split-gso rtt 100.0ms raw overhead 0

> Let me be clear that with cake and sqm I am seeing great results on
> the dslreports speed test(A+) so this inquiry is less about solving a
> problem and more along the lines of trying to take full advantage of
> my available hardware. Any insight would be appreciated, and thanks
> again for your contributions.

At these bandwidths, you are not stressing your hardware at all - and I don't even have to ask what CPU you have to know this.  The NIC's multiple queues would give you no benefit whatsoever.  An Intel Atom or an AMD E-450 can easily handle gigabit traffic through Cake, as long as the NIC is attached via a bus capable of carrying that much bandwidth (eg. PCIe).  These are some of the least powerful 64-bit x86 CPUs that ever reached the market.

In any case, the MQ qdisc simply sorts packets into hardware queues according to the CPU they were submitted from.  This is useful for something like a heavily loaded webserver, which has many worker processes distributed evenly across all available CPUs, since it avoids either passing data to a NIC-worker process on a fixed CPU, or contending for a single NIC lock.  But it's basically useless on a desktop PC, or on a machine acting primarily as a router, since the traffic is submitted from just one or two CPUs at a time, and usually most of the CPUs are idle anyway.  I have no idea what the hardware does to coalesce those packets into a single stream to be sent over the wire.

You can verify this for yourself by looking at your CPU load while running a full bidirectional speed test.  On any recent CPU, expect to see the most loaded core with at least 90% idle - unless your web browser happens to be occupying it.

 - Jonathan Morton