[Cerowrt-devel] ingress rate limiting falling short

[Cerowrt-devel] ingress rate limiting falling short

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 773 bytes --]

I wrote this up on my blog, where I can intersperse text and graphs a bit
better:

http://burntchrome.blogspot.com/2015/06/htb-rate-limiting-not-quite-lining-up.html

Basically, I ran a series of tcp_download tests, using increasing ingress
rates with sqm_scripts, and then used flent's box-plots to put the results
into a combined image for comparing.

On the 3800, it never meets the rate, but it's only off by maybe 5%.  But
on my new WRT1900AC, it's wildly off, even over the same performance range
(I tested it from 80-220Mbps rates in 20Mbps jumps, and saw from 40-150Mbps.

I have no idea where to start looking for the cause.  But for now, I'm just
setting my ingress rate MUCH higher than I should, because it's working out
to the right value as a result.

-Aaron

[-- Attachment #2: Type: text/html, Size: 1038 bytes --]

Re: [Cerowrt-devel] ingress rate limiting falling short

[Dave Taht]

On Tue, Jun 2, 2015 at 10:45 PM, Aaron Wood <woody77@gmail.com> wrote:
> I wrote this up on my blog, where I can intersperse text and graphs a bit
> better:
>
> http://burntchrome.blogspot.com/2015/06/htb-rate-limiting-not-quite-lining-up.html
>
> Basically, I ran a series of tcp_download tests, using increasing ingress
> rates with sqm_scripts, and then used flent's box-plots to put the results
> into a combined image for comparing.

Excellent approach, thank you.

> On the 3800, it never meets the rate, but it's only off by maybe 5%.  But on
> my new WRT1900AC, it's wildly off, even over the same performance range (I
> tested it from 80-220Mbps rates in 20Mbps jumps, and saw from 40-150Mbps.
>
> I have no idea where to start looking for the cause.  But for now, I'm just
> setting my ingress rate MUCH higher than I should, because it's working out
> to the right value as a result.

Me, having been at nanog all day, where I accidentally wiped out my
vpn servers (while having a deep conversation about quagga),
simultaneously terrified of losing the mail server and trying to fix
it..., and now (11PM PDT) having succeeded in uncrashing the email
server, and sorted through a zillion documents on how to do antispam,
and having also built 7 new virtual servers for flent, and restored my
vpn partially, and nuked 27,000 files that were clogging up the
works...

...am going to bed. (after driving 2 hrs home).

It would have been nice to have got the "fedora 22 uses fq_codel by
default!" email and had a little more time left over for dancing
before this email. ;)

Why do we do this?

For SCIENCE!

http://www.funnyjunk.com/funny_pictures/4673932/For+science/

gnight!

-- 
Dave Täht
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] ingress rate limiting falling short

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 257 bytes --]

> On the 3800, it never meets the rate, but it's only off by maybe 5%.

That's about right for Ethernet, IPv4 and TCP header overheads with 1500
MTU. The measured throughput is application level, while HTB controls at
the Ethernet level.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 323 bytes --]

Re: [Cerowrt-devel] ingress rate limiting falling short

[Sebastian Moeller]

[-- Attachment #1: Type: text/plain, Size: 2060 bytes --]

Hi Aaron,

about the 5% loss with the wndr, please remember that the shaper works typically on raw Ethernet rates, while flent reports TCP good put I believe. So roughly 2 to 6 percent difference can be explained with a combination of the following overheads: PTM/ATM, ethernet, VLAN(s), PPPoE, IPv4/IPv6, TCP, potential TCP options like timestamps... Now flent might report actual tcp-rates and I am out to lunch, but I have a hard time reconciling  how flent/netperf, can actually learn about all the additional overhead... As far as I can tell all netperf sees is the payload it sends and the payload it receives...
Tl;dr: Overhead unknown to flent effortlessly explains why the TCP good put as reported by flent falls short of the shaped rate as that rate contains all overheads.


Best Regards
        Sebastian

On June 3, 2015 7:45:47 AM GMT+02:00, Aaron Wood <woody77@gmail.com> wrote:
>I wrote this up on my blog, where I can intersperse text and graphs a
>bit
>better:
>
>http://burntchrome.blogspot.com/2015/06/htb-rate-limiting-not-quite-lining-up.html
>
>Basically, I ran a series of tcp_download tests, using increasing
>ingress
>rates with sqm_scripts, and then used flent's box-plots to put the
>results
>into a combined image for comparing.
>
>On the 3800, it never meets the rate, but it's only off by maybe 5%. 
>But
>on my new WRT1900AC, it's wildly off, even over the same performance
>range
>(I tested it from 80-220Mbps rates in 20Mbps jumps, and saw from
>40-150Mbps.
>
>I have no idea where to start looking for the cause.  But for now, I'm
>just
>setting my ingress rate MUCH higher than I should, because it's working
>out
>to the right value as a result.
>
>-Aaron
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Cerowrt-devel mailing list
>Cerowrt-devel@lists.bufferbloat.net
>https://lists.bufferbloat.net/listinfo/cerowrt-devel

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 2607 bytes --]

Re: [Cerowrt-devel] ingress rate limiting falling short

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 273 bytes --]

Remind me: does HTB have a divide in the fast path? ARMv6 and ARMv7-A CPUs
don't have a hardware integer divide, so that can really hurt.

This is fixed I think in ARMv8 and definitely in AArch64, but divides are
still expensive instructions on any CPU.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 332 bytes --]

Re: [Cerowrt-devel] ingress rate limiting falling short

[Sebastian Moeller]

HI Aaron,


On Jun 3, 2015, at 07:45 , Aaron Wood <woody77@gmail.com> wrote:

> I wrote this up on my blog, where I can intersperse text and graphs a bit better:
> 
> http://burntchrome.blogspot.com/2015/06/htb-rate-limiting-not-quite-lining-up.html
> 
> Basically, I ran a series of tcp_download tests, using increasing ingress rates with sqm_scripts, and then used flent's box-plots to put the results into a combined image for comparing.
> 
> On the 3800, it never meets the rate, but it's only off by maybe 5%.  

	As Jonathan pointed out already this is in the range of the difference between raw rates and tcp good put, so nothing to write home about ;)

> But on my new WRT1900AC, it's wildly off, even over the same performance range (I tested it from 80-220Mbps rates in 20Mbps jumps, and saw from 40-150Mbps.

	So you started with the WRT1900AC where the wndr3800 dropped off? I wonder maybe the Belkin is also almost linear for the lower range? I also note we adjust the quantum based on the rates:
from functions .sh:
get_mtu() {
        BW=$2
        F=`cat /sys/class/net/$1/mtu`
        if [ -z "$F" ]
        then
        F=1500
        fi
        if [ $BW -gt 20000 ]
        then
                F=$(($F * 2))
        fi
        if [ $BW -gt 30000 ]
        then
                F=$(($F * 2))
        fi
        if [ $BW -gt 40000 ]
        then
                F=$(($F * 2))
        fi
        if [ $BW -gt 50000 ]
        then
                F=$(($F * 2))
        fi
        if [ $BW -gt 60000 ]
        then
                F=$(($F * 2))
        fi
        if [ $BW -gt 80000 ]
        then
                F=$(($F * 2))
        fi
        echo $F
}

which we use in the htb invocations via this indirection:
LQ="quantum `get_mtu $IFACE $CEIL`”

$TC qdisc add dev $DEV root handle 1: `get_stab_string` htb default 12
$TC class add dev $DEV parent 1: classid 1:1 htb $LQ rate ${CEIL}kbit ceil ${CEIL}kbit `get_htb_adsll_string`
$TC class add dev $DEV parent 1:1 classid 1:10 htb $LQ rate ${CEIL}kbit ceil ${CEIL}kbit prio 0 `get_htb_adsll_string`
$TC class add dev $DEV parent 1:1 classid 1:11 htb $LQ rate 32kbit ceil ${PRIO_RATE}kbit prio 1 `get_htb_adsll_string`
$TC class add dev $DEV parent 1:1 classid 1:12 htb $LQ rate ${BE_RATE}kbit ceil ${BE_CEIL}kbit prio 2 `get_htb_adsll_string`
$TC class add dev $DEV parent 1:1 classid 1:13 htb $LQ rate ${BK_RATE}kbit ceil ${BE_CEIL}kbit prio 3 `get_htb_adsll_string`

So maybe we need to extend the quantum increases beyond 80Mbps to keep things linear?

I don’t know but it sure looks suspicious…



> 
> I have no idea where to start looking for the cause.  But for now, I'm just setting my ingress rate MUCH higher than I should, because it's working out to the right value as a result.

	It would be great to understand why we need to massively under-shape in that situation to get decent shaping and decent latency under load.

Best Regards
	Sebastian

> 
> -Aaron
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] ingress rate limiting falling short

[Aaron Wood]

[-- Attachment #1.1: Type: text/plain, Size: 2610 bytes --]

> > On the 3800, it never meets the rate, but it's only off by maybe 5%.
>
>         As Jonathan pointed out already this is in the range of the
> difference between raw rates and tcp good put, so nothing to write home
> about ;)
>

Yeah, I'm not too worried about that 5%, based on that explanation.


>
> > But on my new WRT1900AC, it's wildly off, even over the same performance
> range (I tested it from 80-220Mbps rates in 20Mbps jumps, and saw from
> 40-150Mbps.
>
>         So you started with the WRT1900AC where the wndr3800 dropped off?
> I wonder maybe the Belkin is also almost linear for the lower range?


Yeah, good point on a methodology fail.  I'll run another series of tests
walking up the same series of rate limits and see what I get.


> I also note we adjust the quantum based on the rates:
> from functions .sh:
> get_mtu() {
>
... snip

> }
>
> which we use in the htb invocations via this indirection:
> LQ="quantum `get_mtu $IFACE $CEIL`”
>
>
That is odd, and that's quite the aggressive curve on quantum, doubling
every 10-20Mbps.

I did some math, and plotted out the quantum vs. bandwidth based on that
snippet of code (and assuming a 1500-byte MTU):



​And then plotted out the corresponding time in ms that each quantum bytes
(it is bytes, right?) is on the wire:


​Which I think is a really interesting plot (and here are the points that
line up with the steps in the script):

kbps = quantum = time
20000 = 3000 = 1.2ms
30000 = 6000 = 1.6ms
40000 = 12000 = 2.4ms
50000 = 24000 = 3.84ms
60000 = 48000 = 6.4ms
80000 = 96000 = 9.6ms

So it appears that the goal of these values was to keep increases the
quantum as rates went up to provide more bytes per operation, but that's
going to risk adding latency as the time-per-quantum crosses the delay
target in fq_codel (if I'm understanding this correctly).

So one thing that I can do is play around with this, and see if I can keep
that quantum time at a linear level (ie, 10ms, which seems _awfully_ long),
or continue increasing it (which seems like a bad idea).  I'd love to hear
from whoever put this in as to what it's goal was (or was it just
empirically tuned?)

>
> > I have no idea where to start looking for the cause.  But for now, I'm
> just setting my ingress rate MUCH higher than I should, because it's
> working out to the right value as a result.
>
>         It would be great to understand why we need to massively
> under-shape in that situation to get decent shaping and decent latency
> under load.
>

Agreed.

-Aaron

[-- Attachment #1.2: Type: text/html, Size: 4186 bytes --]

[-- Attachment #2: quantum_per_kbps.png --]
[-- Type: image/png, Size: 5851 bytes --]

[-- Attachment #3: quantum_in_ms_per_kbps.png --]
[-- Type: image/png, Size: 8686 bytes --]

Re: [Cerowrt-devel] ingress rate limiting falling short

[Dave Taht]

[-- Attachment #1.1: Type: text/plain, Size: 3825 bytes --]

On Wed, Jun 3, 2015 at 3:16 PM, Aaron Wood <woody77@gmail.com> wrote:

>
> > On the 3800, it never meets the rate, but it's only off by maybe 5%.
>>
>>         As Jonathan pointed out already this is in the range of the
>> difference between raw rates and tcp good put, so nothing to write home
>> about ;)
>>
>
> Yeah, I'm not too worried about that 5%, based on that explanation.
>
>
>>
>> > But on my new WRT1900AC, it's wildly off, even over the same
>> performance range (I tested it from 80-220Mbps rates in 20Mbps jumps, and
>> saw from 40-150Mbps.
>>
>>         So you started with the WRT1900AC where the wndr3800 dropped off?
>> I wonder maybe the Belkin is also almost linear for the lower range?
>
>
> Yeah, good point on a methodology fail.  I'll run another series of tests
> walking up the same series of rate limits and see what I get.
>
>
>> I also note we adjust the quantum based on the rates:
>> from functions .sh:
>> get_mtu() {
>>
> ... snip
>
>> }
>>
>> which we use in the htb invocations via this indirection:
>> LQ="quantum `get_mtu $IFACE $CEIL`”
>>
>>
> That is odd, and that's quite the aggressive curve on quantum, doubling
> every 10-20Mbps.
>
> I did some math, and plotted out the quantum vs. bandwidth based on that
> snippet of code (and assuming a 1500-byte MTU):
>
>
>
> ​And then plotted out the corresponding time in ms that each quantum bytes
> (it is bytes, right?) is on the wire:
>
>
> ​Which I think is a really interesting plot (and here are the points that
> line up with the steps in the script):
>
> kbps = quantum = time
> 20000 = 3000 = 1.2ms
> 30000 = 6000 = 1.6ms
> 40000 = 12000 = 2.4ms
> 50000 = 24000 = 3.84ms
> 60000 = 48000 = 6.4ms
> 80000 = 96000 = 9.6ms
>


> So it appears that the goal of these values was to keep increases the
> quantum as rates went up to provide more bytes per operation, but that's
> going to risk adding latency as the time-per-quantum crosses the delay
> target in fq_codel (if I'm understanding this correctly).
>
> So one thing that I can do is play around with this, and see if I can keep
> that quantum time at a linear level (ie, 10ms, which seems _awfully_ long),
> or continue increasing it (which seems like a bad idea).  I'd love to hear
> from whoever put this in as to what it's goal was (or was it just
> empirically tuned?)
>

Empirical and tested only to about 60Mbits. I got back about 15% cpu to do
it this way at the time I did it on the wndr3800.

and WOW, thx for the analysis! I did not think much about this crossover
point at the time - because we'd maxed on cpu long beforehand.

I can certainly see this batching interacting with the codel target.

On the other hand, you gotta not be running out of cpu in the first place.
I am liking where cake is going.

One of my daydreams is that once we have writable custom ethernet hardware
that we can easily do hardware outbound rate limiting/shaping merely by
programming a register to return a completion interrupt at the set rate
rather than the actual rate.


> >
>> > I have no idea where to start looking for the cause.  But for now, I'm
>> just setting my ingress rate MUCH higher than I should, because it's
>> working out to the right value as a result.
>>
>>         It would be great to understand why we need to massively
>> under-shape in that situation to get decent shaping and decent latency
>> under load.
>>
>
> Agreed.
>
> -Aaron
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>


-- 
Dave Täht
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[-- Attachment #1.2: Type: text/html, Size: 6543 bytes --]

[-- Attachment #2: quantum_per_kbps.png --]
[-- Type: image/png, Size: 5851 bytes --]

[-- Attachment #3: quantum_in_ms_per_kbps.png --]
[-- Type: image/png, Size: 8686 bytes --]

Re: [Cerowrt-devel] ingress rate limiting falling short

[Dave Taht]

[-- Attachment #1.1: Type: text/plain, Size: 4500 bytes --]

On Wed, Jun 3, 2015 at 3:27 PM, Dave Taht <dave.taht@gmail.com> wrote:

>
>
> On Wed, Jun 3, 2015 at 3:16 PM, Aaron Wood <woody77@gmail.com> wrote:
>
>>
>> > On the 3800, it never meets the rate, but it's only off by maybe 5%.
>>>
>>>         As Jonathan pointed out already this is in the range of the
>>> difference between raw rates and tcp good put, so nothing to write home
>>> about ;)
>>>
>>
>> Yeah, I'm not too worried about that 5%, based on that explanation.
>>
>>
>>>
>>> > But on my new WRT1900AC, it's wildly off, even over the same
>>> performance range (I tested it from 80-220Mbps rates in 20Mbps jumps, and
>>> saw from 40-150Mbps.
>>>
>>>         So you started with the WRT1900AC where the wndr3800 dropped
>>> off? I wonder maybe the Belkin is also almost linear for the lower range?
>>
>>
>> Yeah, good point on a methodology fail.  I'll run another series of tests
>> walking up the same series of rate limits and see what I get.
>>
>>
>>> I also note we adjust the quantum based on the rates:
>>> from functions .sh:
>>> get_mtu() {
>>>
>> ... snip
>>
>>> }
>>>
>>> which we use in the htb invocations via this indirection:
>>> LQ="quantum `get_mtu $IFACE $CEIL`”
>>>
>>>
>> That is odd, and that's quite the aggressive curve on quantum, doubling
>> every 10-20Mbps.
>>
>> I did some math, and plotted out the quantum vs. bandwidth based on that
>> snippet of code (and assuming a 1500-byte MTU):
>>
>>
>>
>> ​And then plotted out the corresponding time in ms that each quantum
>> bytes (it is bytes, right?) is on the wire:
>>
>>
>> ​Which I think is a really interesting plot (and here are the points that
>> line up with the steps in the script):
>>
>> kbps = quantum = time
>> 20000 = 3000 = 1.2ms
>> 30000 = 6000 = 1.6ms
>> 40000 = 12000 = 2.4ms
>> 50000 = 24000 = 3.84ms
>> 60000 = 48000 = 6.4ms
>> 80000 = 96000 = 9.6ms
>>
>
>
>> So it appears that the goal of these values was to keep increases the
>> quantum as rates went up to provide more bytes per operation, but that's
>> going to risk adding latency as the time-per-quantum crosses the delay
>> target in fq_codel (if I'm understanding this correctly).
>>
>> So one thing that I can do is play around with this, and see if I can
>> keep that quantum time at a linear level (ie, 10ms, which seems _awfully_
>> long), or continue increasing it (which seems like a bad idea).  I'd love
>> to hear from whoever put this in as to what it's goal was (or was it just
>> empirically tuned?)
>>
>
> Empirical and tested only to about 60Mbits. I got back about 15% cpu to do
> it this way at the time I did it on the wndr3800.
>
> and WOW, thx for the analysis! I did not think much about this crossover
> point at the time - because we'd maxed on cpu long beforehand.
>

And most of my testing on x86 has been with this change to the htb quantum
entirely disabled and set to 1514.

the "production" sqm-scripts and my own hacked up version(s) have differed
in this respect for quite some time. (at least 6 months).

Great spot on this discrepancy.

:egg, otherwise, on face:



> I can certainly see this batching interacting with the codel target.
>
> On the other hand, you gotta not be running out of cpu in the first place.
> I am liking where cake is going.
>
> One of my daydreams is that once we have writable custom ethernet hardware
> that we can easily do hardware outbound rate limiting/shaping merely by
> programming a register to return a completion interrupt at the set rate
> rather than the actual rate.
>



>
>
>> >
>>> > I have no idea where to start looking for the cause.  But for now, I'm
>>> just setting my ingress rate MUCH higher than I should, because it's
>>> working out to the right value as a result.
>>>
>>>         It would be great to understand why we need to massively
>>> under-shape in that situation to get decent shaping and decent latency
>>> under load.
>>>
>>
>> Agreed.
>>
>> -Aaron
>>
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>
>>
>
>
> --
> Dave Täht
> What will it take to vastly improve wifi for everyone?
> https://plus.google.com/u/0/explore/makewififast
>



-- 
Dave Täht
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[-- Attachment #1.2: Type: text/html, Size: 8074 bytes --]

[-- Attachment #2: quantum_per_kbps.png --]
[-- Type: image/png, Size: 5851 bytes --]

[-- Attachment #3: quantum_in_ms_per_kbps.png --]
[-- Type: image/png, Size: 8686 bytes --]

Re: [Cerowrt-devel] ingress rate limiting falling short

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 2759 bytes --]

On Wed, Jun 3, 2015 at 3:27 PM, Dave Taht <dave.taht@gmail.com> wrote:

>
>
>> kbps = quantum = time
>> 20000 = 3000 = 1.2ms
>> 30000 = 6000 = 1.6ms
>> 40000 = 12000 = 2.4ms
>> 50000 = 24000 = 3.84ms
>> 60000 = 48000 = 6.4ms
>> 80000 = 96000 = 9.6ms
>>
>
>
>> So it appears that the goal of these values was to keep increases the
>> quantum as rates went up to provide more bytes per operation, but that's
>> going to risk adding latency as the time-per-quantum crosses the delay
>> target in fq_codel (if I'm understanding this correctly).
>>
>> So one thing that I can do is play around with this, and see if I can
>> keep that quantum time at a linear level (ie, 10ms, which seems _awfully_
>> long), or continue increasing it (which seems like a bad idea).  I'd love
>> to hear from whoever put this in as to what it's goal was (or was it just
>> empirically tuned?)
>>
>
> Empirical and tested only to about 60Mbits. I got back about 15% cpu to do
> it this way at the time I did it on the wndr3800.
>

Basically, increasing the quantums to get more cpu available...  So a
too-small quantum is going to be excessive cpu, and a too-large quantum is
going to be poor fairness?



> and WOW, thx for the analysis! I did not think much about this crossover
> point at the time - because we'd maxed on cpu long beforehand.
>

No problem, this is the sort of thing I _can_ help with, since I don't know
the kernel internals very well.


I can certainly see this batching interacting with the codel target.
>

Which may also explain your comments about poor fairness on my 3800 results
when up at 60-80Mbps, when htb's quantum has crossed over fq_codel's target?



> On the other hand, you gotta not be running out of cpu in the first place.
> I am liking where cake is going.
>

Yeah.  That's what I _also_ need to figure out.  Load seems "reasonable",
but load and cpu stats get reported oddly on multi-core (some things are
per-core, some are per-total available, etc).  I know I've seen the
"soft_irq" thread at 70% in top doing some tests (in the past).  I wouldn't
be surprised if this is a single-core-only bit of code?  (or can htb
processing and fq_codel processing be shoved to separate cores?)

One of my daydreams is that once we have writable custom ethernet hardware
> that we can easily do hardware outbound rate limiting/shaping merely by
> programming a register to return a completion interrupt at the set rate
> rather than the actual rate.
>

well, inbound is certainly more of an issue than outbound right now...

So, for my next rounds of tests, I can play around with different quantum
values/schemes, and also play with simple.qos vs. simplest.qos, and
instrument the whole thing to capture processor utilization vs. bandwidth.

-Aaron

[-- Attachment #2: Type: text/html, Size: 4844 bytes --]