[Cerowrt-devel] uplink_buffer_adjustment

[Cerowrt-devel] uplink_buffer_adjustment

[Oliver Niesner]

Hi list,

I use cerowrt (3.10.24-8) direct behind my main dsl Router.
SQM is set and performance is good.
When i used Netalyzr from my smartphone i've got good results.

> Network buffer measurements (?): Uplink 96 ms, Downlink is good 

But when i use my notebook i get this:

> Network buffer measurements (?): Uplink 1200 ms, Downlink is good 

I tried even wired connection and set ring buffer rx/tx with ethtool to 64, but
only minimal change in uplink buffer (1100ms).

Has anyone an idea, what i can try to get better uplink performance?

Regards,

Oliver

Re: [Cerowrt-devel] uplink_buffer_adjustment

[Sebastian Moeller]

Hi Oliver,


On Feb 25, 2014, at 12:14 , Oliver Niesner <oliver.niesner@gmail.com> wrote:

> Hi list,
> 
> I use cerowrt (3.10.24-8) direct behind my main dsl Router.
> SQM is set and performance is good.

	This is the most important part; you seem happy with the actual latency under load. 

> When i used Netalyzr from my smartphone i've got good results.
> 
>> Network buffer measurements (?): Uplink 96 ms, Downlink is good 

	Presumably the smart phone connects to the cerowrt router via wlan?

> 
> But when i use my notebook i get this:
> 
>> Network buffer measurements (?): Uplink 1200 ms, Downlink is good

	If you want to change that number neatly reports you will need to change the limit variable of the fq_codel instance(s) on your uplink device (ge00), smaller limit values will cause smaller reported buffering.
	BUT you should not be concerned about this report at all. Netalyzr tries to fill the buffers with (relative) high bandwidth inelastic unrelenting UDP traffic, one could say it represents a DOS attack better than real traffic. Real traffic be it TCP (by virtue of the protocol) and even UDP (by virtue of application writers implementing their own congestion control) will react to packet loss by reducing the transmit speed. SQM does a good job strategically dropping packets so that all flows can adapt their bandwidth to the available total bandwidth. The beauty of codel is that it starts out quite gentle in a way that is well matched to TCPs congestion avoidance. It will turn less gentle if the traffic does not respond and congestion stays high. Netalyzr now is this weird combination of short duration yet unrelenting traffic. The reason why SQM does not affect the netalyzr probe by much is that the netalyzr traffic stops before fq_codel has ramped up the dropping. So what happens is that the probe fills the largest buffer which is specified by the limit parameter of fq_codel, that defaults to 1000 packets on cerowrt (once the queue has limit packets all new packets get dropped, as it resorts to tail dropping, but that really is just an emergency behavior; under normal loads with a slow link the queue will never come close to 1000).

	TL:DR version, netalyzr probes a buffer depth that while existing has not much relevance on the latency behavior under load for cerowrt. 


>> 
> 
> I tried even wired connection and set ring buffer rx/tx with ethtool to 64, but
> only minimal change in uplink buffer (1100ms).

	Good idea but wrong tunable ;)

> 
> Has anyone an idea, what i can try to get better uplink performance?

	Unless you intend to run yur router under continuous DOS attacks, I would recommend to ignore netalyzr's buffer measurements (as they do not reflect cerowrt's behavior under a more realistic load well). 


Best Regards
	Sebastian

> 
> Regards,
> 
> Oliver
> 
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] uplink_buffer_adjustment

[Maciej Soltysiak]

On Tue, Feb 25, 2014 at 1:08 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
>         TL:DR version, netalyzr probes a buffer depth that while existing has not much relevance on the latency behavior under load for cerowrt.
Yes, it doesn't have much relevance for latency where fq_codel or
similar are employed.
But where it's not (or where fq_codel is defeated by DOS style
traffic), it seems this comes back as a meaningful metric. In the end
codel is workaround for overbuffering, which is not being removed very
hastily.

Best regards,
Maciej

Re: [Cerowrt-devel] uplink_buffer_adjustment

[Sebastian Moeller]

Hi Maciej,


On Feb 25, 2014, at 15:05 , Maciej Soltysiak <maciej@soltysiak.com> wrote:

> On Tue, Feb 25, 2014 at 1:08 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
>>        TL:DR version, netalyzr probes a buffer depth that while existing has not much relevance on the latency behavior under load for cerowrt.
> Yes, it doesn't have much relevance for latency where fq_codel or
> similar are employed.

	Exactly.

> But where it's not (or where fq_codel is defeated by DOS style
> traffic), it seems this comes back as a meaningful metric.

	That is what I thought as well initially,. But then (I think Eric D explained) if the system is under a DOS attack we have more severe problems at hand than ping latency ;) Think about it that way, if the flood saturates our uplink, fq_codel will sooner or later reign it in anyway, and we only see a temporary glitch in latency performance (during the time the queue stays in tail-drop or so). If the DOS attack is coming from the outside against our system we are hosed anyway, as a the upstream CMTS/DSLAM?BRAS buffers will fill up over which we have no control (as typically the ingress of a DSLAM is much greater than the egress to a specific line so there is going to be an normally temporary queue). I think during a DOS all hops with ingress-from-DOS > egress-to-destination will see buffers filing up...
	So my current thinking is it is nice to have some buffering against large swings in available bandwidth to smooth out the traffic a bit.
On the topic of the limit parameter I want to add, that I see the most relevant part of limit to keep cerowrt out of out-of-memory conditions which will cause the unit to reboot. So ideally the total amount of buffering in the system stays low enough so the box does not go belly-up. 

> In the end
> codel is workaround for overbuffering, which is not being removed very
> hastily.

	I beg to differ, once the buffer management gets smart buffers turn from liability to a boon ;)

Best Regards
	Sebastian

> 
> Best regards,
> Maciej

Re: [Cerowrt-devel] uplink_buffer_adjustment

[Jim Gettys]

[-- Attachment #1: Type: text/plain, Size: 1918 bytes --]

On Tue, Feb 25, 2014 at 6:14 AM, Oliver Niesner <oliver.niesner@gmail.com>wrote:

> Hi list,
>
> I use cerowrt (3.10.24-8) direct behind my main dsl Router.
> SQM is set and performance is good.
> When i used Netalyzr from my smartphone i've got good results.
>
> > Network buffer measurements (?): Uplink 96 ms, Downlink is good
>
> But when i use my notebook i get this:
>
> > Network buffer measurements (?): Uplink 1200 ms, Downlink is good
>
> I tried even wired connection and set ring buffer rx/tx with ethtool to
> 64, but
> only minimal change in uplink buffer (1100ms).
>
> Has anyone an idea, what i can try to get better uplink performance?
>

Netalyzr uses a UDP based test for "filling the buffers"; it is not
responsive to drops/marks at all, the way a TCP test would be.

So if you run it, the flows it generates are unresponsive, and indicate the
true size of the buffers at the bottleneck link, even though any normal TCP
would long since have responded and nothing like that amount of buffering
would have taken place.  Furthermore, the flow queuing of fq_codel isolates
those flows from other flows, and therefore you do not get the bad latency
you would otherwise get on those flows.

In short, (particularly since fq_codel is deployed in quantity millions by
a few ISP's already; it is no longer a fluke to find it only in hacker's
hands), Nick Weaver needs to improve netalyzr to detect flow queuing
algorithms and make some sense out of the situation.  It would be great to
monitor the spread of these algorithms around the Internet over the coming
years.

So it is arguably a "bug" in netalyzr.  It is certainly extremely
misleading.

Nick?

                                             - Jim


> Regards,
>
> Oliver
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>

[-- Attachment #2: Type: text/html, Size: 3379 bytes --]

Re: [Cerowrt-devel] uplink_buffer_adjustment

[Jim Gettys]

[-- Attachment #1: Type: text/plain, Size: 1013 bytes --]

On Tue, Feb 25, 2014 at 11:02 AM, Nicholas Weaver <nweaver@icsi.berkeley.edu
> wrote:

>
> On Feb 25, 2014, at 7:59 AM, Jim Gettys <jg@freedesktop.org> wrote:
> > So it is arguably a "bug" in netalyzr.  It is certainly extremely
> misleading.
> >
> > Nick?
>
> Rewriting it as a TCP-based stresser is definatly on our to-do list.
>

Good; though I'm not sure you'll be able to build a TCP one that fills the
buffers fast enough to determine some of the buffering out there (at least
without hacking the TCP implementation, anyway).

The other piece of this is detecting flow queuing being active; this makes
a bigger difference to actual latency than mark/drop algorithms do by
themselves.
                                  - Jim


>
>
> --
> Nicholas Weaver                  it is a tale, told by an idiot,
> nweaver@icsi.berkeley.edu                full of sound and fury,
> 510-666-2903                                 .signifying nothing
> PGP: http://www1.icsi.berkeley.edu/~nweaver/data/nweaver_pub.asc
>
>

[-- Attachment #2: Type: text/html, Size: 2079 bytes --]

Re: [Cerowrt-devel] uplink_buffer_adjustment

[dpreed]

I've measured buffer size with TCP, when there is no fq_codel or whatever doing drops.  After all, this is what caused me to get concerned.

And actually, since UDP packets are dropped by fq_codel the same as TCP packets, it's easy to see how big fq_codel lets the buffers get.

If the buffer gets to be 1200 msec. long with UDP, that's a problem with fq_codel - just think about it.  Someone's tuning fq_codel to allow excess buildup of queueing, if that's observed.

So I doubt this is a netalyzr bug at all.  Operator error more likely, in tuning fq_codel.




On Tuesday, February 25, 2014 11:46am, "Jim Gettys" <jg@freedesktop.org> said:

> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> On Tue, Feb 25, 2014 at 11:02 AM, Nicholas Weaver <nweaver@icsi.berkeley.edu
>> wrote:
> 
>>
>> On Feb 25, 2014, at 7:59 AM, Jim Gettys <jg@freedesktop.org> wrote:
>> > So it is arguably a "bug" in netalyzr.  It is certainly extremely
>> misleading.
>> >
>> > Nick?
>>
>> Rewriting it as a TCP-based stresser is definatly on our to-do list.
>>
> 
> Good; though I'm not sure you'll be able to build a TCP one that fills the
> buffers fast enough to determine some of the buffering out there (at least
> without hacking the TCP implementation, anyway).
> 
> The other piece of this is detecting flow queuing being active; this makes
> a bigger difference to actual latency than mark/drop algorithms do by
> themselves.
>                                   - Jim
> 
> 
>>
>>
>> --
>> Nicholas Weaver                  it is a tale, told by an idiot,
>> nweaver@icsi.berkeley.edu                full of sound and fury,
>> 510-666-2903                                 .signifying nothing
>> PGP: http://www1.icsi.berkeley.edu/~nweaver/data/nweaver_pub.asc
>>
>>
> 

Re: [Cerowrt-devel] uplink_buffer_adjustment

[Dave Taht]

On Tue, Feb 25, 2014 at 7:59 AM, Jim Gettys <jg@freedesktop.org> wrote:
>
>
>
> On Tue, Feb 25, 2014 at 6:14 AM, Oliver Niesner <oliver.niesner@gmail.com>
> wrote:
>>
>> Hi list,
>>
>> I use cerowrt (3.10.24-8) direct behind my main dsl Router.
>> SQM is set and performance is good.
>> When i used Netalyzr from my smartphone i've got good results.
>>
>> > Network buffer measurements (?): Uplink 96 ms, Downlink is good
>>
>> But when i use my notebook i get this:
>>
>> > Network buffer measurements (?): Uplink 1200 ms, Downlink is good
>>
>> I tried even wired connection and set ring buffer rx/tx with ethtool to
>> 64, but
>> only minimal change in uplink buffer (1100ms).
>>
>> Has anyone an idea, what i can try to get better uplink performance?
>
>
> Netalyzr uses a UDP based test for "filling the buffers"; it is not
> responsive to drops/marks at all, the way a TCP test would be.
>
> So if you run it, the flows it generates are unresponsive, and indicate the
> true size of the buffers at the bottleneck link, even though any normal TCP
> would long since have responded and nothing like that amount of buffering
> would have taken place.  Furthermore, the flow queuing of fq_codel isolates
> those flows from other flows, and therefore you do not get the bad latency
> you would otherwise get on those flows.
>
> In short, (particularly since fq_codel is deployed in quantity millions by a
> few ISP's already; it is no longer a fluke to find it only in hacker's
> hands), Nick Weaver needs to improve netalyzr to detect flow queuing
> algorithms and make some sense out of the situation.  It would be great to
> monitor the spread of these algorithms around the Internet over the coming
> years.
>
> So it is arguably a "bug" in netalyzr.  It is certainly extremely
> misleading.

A lot of evidence has accumulated that SFQ and SQF is widely deployed
by more than a few DSL providers, notably several in France have come
forward. There is some evidence it is in some firmwares, too.

SFQ is the default on free.fr's old boxes, and fq_codel was deployed
on their revolution V6 product august 2012 (an upgrade to linux 3.11
with better hashing support for ipv6 and 6rd is on its way).

They were kind enough to document their QoS/AQM/Scheduling system for
me (and point us at problems in setting the fq_codel target too low on
low bandwidth links which we've now fixed in cerowrt's sqm system (but
is not yet fixed in openwrt's qos-scripts or dd-wrts, sigh)), which I
have begun to encapsulate in this still very drafty internet draft:

http://snapon.lab.bufferbloat.net/~d/draft-taht-home-gateway-best-practices-00.html

So the naive single udp flow packet flooding test in netanalyzer in
speedtest and in netalyzer (and most other tools) gives misleading
results as to the effects of and size of bufferbloat in whole
countries, and I REALLY wish we could get good data on the total size
of the deployment of these technologies worldwide.

Measuring using a high rate flow, simultaneously with a low rate flow
on a different 5 tuple, will show more truth. Preferably
bidirectionally, at the same time....

> Nick?
>
>                                              - Jim
>
>>
>> Regards,
>>
>> Oliver
>>
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html