Re: [Cerowrt-devel] ingress rate limiting falling short

[Aaron Wood <woody77@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 2610 bytes --]

> > On the 3800, it never meets the rate, but it's only off by maybe 5%.
>
>         As Jonathan pointed out already this is in the range of the
> difference between raw rates and tcp good put, so nothing to write home
> about ;)
>

Yeah, I'm not too worried about that 5%, based on that explanation.


>
> > But on my new WRT1900AC, it's wildly off, even over the same performance
> range (I tested it from 80-220Mbps rates in 20Mbps jumps, and saw from
> 40-150Mbps.
>
>         So you started with the WRT1900AC where the wndr3800 dropped off?
> I wonder maybe the Belkin is also almost linear for the lower range?


Yeah, good point on a methodology fail.  I'll run another series of tests
walking up the same series of rate limits and see what I get.


> I also note we adjust the quantum based on the rates:
> from functions .sh:
> get_mtu() {
>
... snip

> }
>
> which we use in the htb invocations via this indirection:
> LQ="quantum `get_mtu $IFACE $CEIL`”
>
>
That is odd, and that's quite the aggressive curve on quantum, doubling
every 10-20Mbps.

I did some math, and plotted out the quantum vs. bandwidth based on that
snippet of code (and assuming a 1500-byte MTU):



​And then plotted out the corresponding time in ms that each quantum bytes
(it is bytes, right?) is on the wire:


​Which I think is a really interesting plot (and here are the points that
line up with the steps in the script):

kbps = quantum = time
20000 = 3000 = 1.2ms
30000 = 6000 = 1.6ms
40000 = 12000 = 2.4ms
50000 = 24000 = 3.84ms
60000 = 48000 = 6.4ms
80000 = 96000 = 9.6ms

So it appears that the goal of these values was to keep increases the
quantum as rates went up to provide more bytes per operation, but that's
going to risk adding latency as the time-per-quantum crosses the delay
target in fq_codel (if I'm understanding this correctly).

So one thing that I can do is play around with this, and see if I can keep
that quantum time at a linear level (ie, 10ms, which seems _awfully_ long),
or continue increasing it (which seems like a bad idea).  I'd love to hear
from whoever put this in as to what it's goal was (or was it just
empirically tuned?)

>
> > I have no idea where to start looking for the cause.  But for now, I'm
> just setting my ingress rate MUCH higher than I should, because it's
> working out to the right value as a result.
>
>         It would be great to understand why we need to massively
> under-shape in that situation to get decent shaping and decent latency
> under load.
>

Agreed.

-Aaron

[-- Attachment #1.2: Type: text/html, Size: 4186 bytes --]

[-- Attachment #2: quantum_per_kbps.png --]
[-- Type: image/png, Size: 5851 bytes --]

[-- Attachment #3: quantum_in_ms_per_kbps.png --]
[-- Type: image/png, Size: 8686 bytes --]