[Bloat] AQM and PPP on Linux

[Bloat] AQM and PPP on Linux

[Juliusz Chroboczek]

I'm currently away from home, and using a 3G modem for Internet access.
I've found out that both NetworkManager and wvdial/pppd setup the
interface to use pfifo_fast (with a qlen of a mere 3 packets!).  Setting
fq_codel manually appears to work fine, but needs to be redone every time
the modem has a hiccup.

Is that the expected behaviour?  Shouldn't we be pushing patches somewhere
to change the default?

-- Juliusz

Re: [Bloat] AQM and PPP on Linux

[Toke Høiland-Jørgensen]

Juliusz Chroboczek <jch@pps.univ-paris-diderot.fr> writes:

> I'm currently away from home, and using a 3G modem for Internet
> access. I've found out that both NetworkManager and wvdial/pppd setup
> the interface to use pfifo_fast (with a qlen of a mere 3 packets!).
> Setting fq_codel manually appears to work fine, but needs to be redone
> every time the modem has a hiccup.

I don't think NetworkManager does anything to set qdiscs. The kernel
simple sets pfifo_fast if not told otherwise (but see below).

> Is that the expected behaviour? Shouldn't we be pushing patches
> somewhere to change the default?

What distro are you on?

There's a sysctl to set the default qdisc: net.core.default_qdisc. Set
this to fq_codel and you should be good to go. This is becoming the
default in more and more distros (currently at least Arch and Fedora;
and OpenWrt has been patching out pfifo_fast entirely for a while).

There's a bug for Ubuntu here:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1436945 -- feel
free to bug them :)

-Toke

Re: [Bloat] AQM and PPP on Linux

[Juliusz Chroboczek]

> What distro are you on?

Debian.  (One machine stable, one testing.)

> There's a sysctl to set the default qdisc: net.core.default_qdisc.

    $ sysctl -a | grep default_qdisc
    net.core.default_qdisc = pfifo_fast

> Set this to fq_codel and you should be good to go.

Is that safe?  I'm thinking lo and virtual interfaces.

-- Juliusz

Re: [Bloat] AQM and PPP on Linux

[Toke Høiland-Jørgensen]

Juliusz Chroboczek <jch@pps.univ-paris-diderot.fr> writes:

>> What distro are you on?
>
> Debian.  (One machine stable, one testing.)

Right. Don't know if there's already an effort to convince the Debian
devs to switch the default...

>> Set this to fq_codel and you should be good to go.
>
> Is that safe?  I'm thinking lo and virtual interfaces.

Why wouldn't it be? The lo interface doesn't have a queue at all AFAIK.
And if you have congestion at your virtual interfaces wouldn't you want
it de-bloated? :)

-Toke

Re: [Bloat] AQM and PPP on Linux

[Simon Barber]

The main danger is the negative effects on performance of using Codel. You 
may experience low throughput on high RTT links.

Simon

Sent with AquaMail for Android
http://www.aqua-mail.com


On July 28, 2015 7:06:22 AM Toke Høiland-Jørgensen <toke@toke.dk> wrote:

> Juliusz Chroboczek <jch@pps.univ-paris-diderot.fr> writes:
>
> >> What distro are you on?
> >
> > Debian.  (One machine stable, one testing.)
>
> Right. Don't know if there's already an effort to convince the Debian
> devs to switch the default...
>
> >> Set this to fq_codel and you should be good to go.
> >
> > Is that safe?  I'm thinking lo and virtual interfaces.
>
> Why wouldn't it be? The lo interface doesn't have a queue at all AFAIK.
> And if you have congestion at your virtual interfaces wouldn't you want
> it de-bloated? :)
>
> -Toke
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] AQM and PPP on Linux

[Eric Dumazet]

On Tue, 2015-07-28 at 07:11 -0700, Simon Barber wrote:
> The main danger is the negative effects on performance of using Codel. You 
> may experience low throughput on high RTT links.

Really ? I've never seen this, unless you mess with codel qdisc
attributes maybe.

(Some guys seem to not really understand whole concept of a queue, and
set very low values for 'limit' attribute for example)

Re: [Bloat] AQM and PPP on Linux

[Simon Barber]

The issue is that Codel tries to keep the delay low, and will start 
dropping when sojourn time grows above 5ms for 100ms. For longer RTT links 
more delay is necessary to avoid underutilizing the link. This is due to 
the multiplicative decrease - it's worst with Reno, where the halving of 
cWind means that you need to have a full BDP of data in the buffer to avoid 
the link going idle when cWind is halved. With longer RTTs this means more 
delay than Codel allows is required to avoid a throughput hit. The worst 
case happens when a single flow is controlled, but that can be a common 
situation. My proposal is to sense and have the target value in Codel 
automatically adjust when this worst case scenario happens - which would 
mitigate most of the downside.

Simon

Sent with AquaMail for Android
http://www.aqua-mail.com


On July 28, 2015 7:21:56 AM Eric Dumazet <eric.dumazet@gmail.com> wrote:

> On Tue, 2015-07-28 at 07:11 -0700, Simon Barber wrote:
> > The main danger is the negative effects on performance of using Codel. You
> > may experience low throughput on high RTT links.
>
> Really ? I've never seen this, unless you mess with codel qdisc
> attributes maybe.
>
> (Some guys seem to not really understand whole concept of a queue, and
> set very low values for 'limit' attribute for example)
>
>
>
>

Re: [Bloat] AQM and PPP on Linux

[Jonathan Morton]

> On 28 Jul, 2015, at 17:31, Simon Barber <simon@superduper.net> wrote:
> 
> The issue is that Codel tries to keep the delay low, and will start dropping when sojourn time grows above 5ms for 100ms. For longer RTT links more delay is necessary to avoid underutilizing the link.

I’m pretty sure we’ve had this discussion before, several times.

Codel works pretty well, with an interval of 100ms, when the actual RTT is between about 10ms and 1000ms.  That accounts for 99.9% of the practical paths you’ll see on the Internet.

If you really are concerned about long-RTT paths, then you can increase the interval parameter to suit.  This will hurt Codel’s performance slightly on very short-RTT paths.

 - Jonathan Morton

Re: [Bloat] AQM and PPP on Linux

[Eric Dumazet]

On Tue, 2015-07-28 at 07:31 -0700, Simon Barber wrote:
> The issue is that Codel tries to keep the delay low, and will start 
> dropping when sojourn time grows above 5ms for 100ms. For longer RTT links 
> more delay is necessary to avoid underutilizing the link. This is due to 
> the multiplicative decrease - it's worst with Reno, where the halving of 
> cWind means that you need to have a full BDP of data in the buffer to avoid 
> the link going idle when cWind is halved. With longer RTTs this means more 
> delay than Codel allows is required to avoid a throughput hit. The worst 
> case happens when a single flow is controlled, but that can be a common 
> situation. My proposal is to sense and have the target value in Codel 
> automatically adjust when this worst case scenario happens - which would 
> mitigate most of the downside.

As I said, I've never seen what you describe.

100ms value is not a go/nogo threshold. It is a hint, based on real
world values. We are speaking of 100 ms sojourn time in the CoDel queue,
not sojourn time in the Internet !

You can still have flows with 500 ms or even 10 sec rtt and codel just
works fine.

Are you sure you understood how this was working ?

Re: [Bloat] AQM and PPP on Linux

[Eric Dumazet]

On Tue, 2015-07-28 at 16:49 +0200, Eric Dumazet wrote:
> On Tue, 2015-07-28 at 07:31 -0700, Simon Barber wrote:
> > The issue is that Codel tries to keep the delay low, and will start 
> > dropping when sojourn time grows above 5ms for 100ms. For longer RTT links 
> > more delay is necessary to avoid underutilizing the link. This is due to 
> > the multiplicative decrease - it's worst with Reno, where the halving of 
> > cWind means that you need to have a full BDP of data in the buffer to avoid 
> > the link going idle when cWind is halved. With longer RTTs this means more 
> > delay than Codel allows is required to avoid a throughput hit. The worst 
> > case happens when a single flow is controlled, but that can be a common 
> > situation. My proposal is to sense and have the target value in Codel 
> > automatically adjust when this worst case scenario happens - which would 
> > mitigate most of the downside.
> 
> As I said, I've never seen what you describe.
> 
> 100ms value is not a go/nogo threshold. It is a hint, based on real
> world values. We are speaking of 100 ms sojourn time in the CoDel queue,

typo : 5 ms ( codel target )

Re: [Bloat] AQM and PPP on Linux

[Alan Jenkins]

On 28/07/15 15:49, Eric Dumazet wrote:
> On Tue, 2015-07-28 at 07:31 -0700, Simon Barber wrote:
>> The issue is that Codel tries to keep the delay low, and will start
>> dropping when sojourn time grows above 5ms for 100ms. For longer RTT links
>> more delay is necessary to avoid underutilizing the link. This is due to
>> the multiplicative decrease - it's worst with Reno, where the halving of
>> cWind means that you need to have a full BDP of data in the buffer to avoid
>> the link going idle when cWind is halved. With longer RTTs this means more
>> delay than Codel allows is required to avoid a throughput hit. The worst
>> case happens when a single flow is controlled, but that can be a common
>> situation. My proposal is to sense and have the target value in Codel
>> automatically adjust when this worst case scenario happens - which would
>> mitigate most of the downside.
> As I said, I've never seen what you describe.
>
> 100ms value is not a go/nogo threshold. It is a hint, based on real
> world values. We are speaking of 100 ms sojourn time in the CoDel queue,
> not sojourn time in the Internet !
>
> You can still have flows with 500 ms or even 10 sec rtt and codel just
> works fine.
>
> Are you sure you understood how this was working ?
>

The codel paper indicates utilization dropoff as RTT increases above the 
'interval' (and that interval is set based on expected rtt). They 
presented an average of several loads, so this wasn't even the 
single-stream worst case.

200ms rtt was fine, 500ms was a slightly worse & varied a lot more.

If you wanted something that was "fine" at 10s (ow), you wouldn't want 
codel's defaults.  Tuning higher values in fq_codel would probably be 
enough though.

http://queue.acm.org/detail.cfm?id=2209336

Re: [Bloat] AQM and PPP on Linux

[Sebastian Moeller]

Hi Simon,

On Jul 28, 2015, at 16:31 , Simon Barber <simon@superduper.net> wrote:

> The issue is that Codel tries to keep the delay low, and will start dropping when sojourn time grows above 5ms for 100ms. For longer RTT links more delay is necessary to avoid underutilizing the link. This is due to the multiplicative decrease - it's worst with Reno, where the halving of cWind means that you need to have a full BDP of data in the buffer to avoid the link going idle when cWind is halved. With longer RTTs this means more delay than Codel allows is required to avoid a throughput hit. The worst case happens when a single flow is controlled, but that can be a common situation. My proposal is to sense and have the target value in Codel automatically adjust when this worst case scenario happens - which would mitigate most of the downside.

	According to theory you should adapt interval if at all and the set target between 5-10% of that interval. See https://datatracker.ietf.org/doc/draft-ietf-aqm-codel/?include_text=1 sections 4.3 and 4.4. Now that could all be off.The upshot is increasing target as a response for long RTTs will sacrifice latency again for bandwidth, pretty much that the avoidance of is codel’s claim to fame ;)

Best Regards
	Sebastian


> 
> Simon
> 
> Sent with AquaMail for Android
> http://www.aqua-mail.com
> 
> 
> On July 28, 2015 7:21:56 AM Eric Dumazet <eric.dumazet@gmail.com> wrote:
> 
>> On Tue, 2015-07-28 at 07:11 -0700, Simon Barber wrote:
>> > The main danger is the negative effects on performance of using Codel. You
>> > may experience low throughput on high RTT links.
>> 
>> Really ? I've never seen this, unless you mess with codel qdisc
>> attributes maybe.
>> 
>> (Some guys seem to not really understand whole concept of a queue, and
>> set very low values for 'limit' attribute for example)
>> 
>> 
>> 
>> 
> 
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] AQM and PPP on Linux

[Michael Welzl]

> On 28. jul. 2015, at 18.22, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> Hi Simon,
> 
> On Jul 28, 2015, at 16:31 , Simon Barber <simon@superduper.net> wrote:
> 
>> The issue is that Codel tries to keep the delay low, and will start dropping when sojourn time grows above 5ms for 100ms. For longer RTT links more delay is necessary to avoid underutilizing the link. This is due to the multiplicative decrease - it's worst with Reno, where the halving of cWind means that you need to have a full BDP of data in the buffer to avoid the link going idle when cWind is halved. With longer RTTs this means more delay than Codel allows is required to avoid a throughput hit. The worst case happens when a single flow is controlled, but that can be a common situation. My proposal is to sense and have the target value in Codel automatically adjust when this worst case scenario happens - which would mitigate most of the downside.
> 
> 	According to theory you should adapt interval if at all and the set target between 5-10% of that interval. See https://datatracker.ietf.org/doc/draft-ietf-aqm-codel/?include_text=1 sections 4.3 and 4.4. Now that could all be off.The upshot is increasing target as a response for long RTTs will sacrifice latency again for bandwidth, pretty much that the avoidance of is codel’s claim to fame ;)

The only way to fix this trade-off is to change TCP, to back off less *only when the queue was small*. We tried what we think is the simplest possible way to do this: only a sender-side change, using ECN as an indication that there is an AQM mechanism on the path, not a long queue (and hence backing off by less is ok). It works pretty well. See:
http://caia.swin.edu.au/reports/150710A/CAIA-TR-150710A.pdf
(a few of the graphs are simulations, but many were from real-life tests - appendix B says which).

...and the draft, which was presented to TCPM at the Prague IETF last week:
https://tools.ietf.org/html/draft-khademi-alternativebackoff-ecn

Cheers,
Michael

Re: [Bloat] AQM and PPP on Linux

[David Lang]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1322 bytes --]

If you have high RTT on loopback or virtual interfaces, Codel is going to be the 
least of your problems ;-)

David Lang

On Tue, 28 Jul 2015, Simon Barber wrote:

> The main danger is the negative effects on performance of using Codel. You 
> may experience low throughput on high RTT links.
>
> Simon
>
> Sent with AquaMail for Android
> http://www.aqua-mail.com
>
>
> On July 28, 2015 7:06:22 AM Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
>> Juliusz Chroboczek <jch@pps.univ-paris-diderot.fr> writes:
>>
>> >> What distro are you on?
>> >
>> > Debian.  (One machine stable, one testing.)
>>
>> Right. Don't know if there's already an effort to convince the Debian
>> devs to switch the default...
>>
>> >> Set this to fq_codel and you should be good to go.
>> >
>> > Is that safe?  I'm thinking lo and virtual interfaces.
>>
>> Why wouldn't it be? The lo interface doesn't have a queue at all AFAIK.
>> And if you have congestion at your virtual interfaces wouldn't you want
>> it de-bloated? :)
>>
>> -Toke
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>

Re: [Bloat] AQM and PPP on Linux

[Juliusz Chroboczek]

>> Debian.  (One machine stable, one testing.)

> Right. Don't know if there's already an effort to convince the Debian
> devs to switch the default...

Please file a bug against the procps package (which owns the sysctl.conf
file).  Just send a mail describing the issue and the proposed fix to
submit@bugs.debian.org, with the first lines of the *body* of the mail
having the following contents:

Package: procps
Version: 2:3.3.10-2
Severity: wishlist

In typical Debian style, expect this to cause a months long flamewar about
systemd ;-)

Re: [Bloat] AQM and PPP on Linux

[Alan Jenkins]

On 28/07/15 20:20, Juliusz Chroboczek wrote:
>>> Debian.  (One machine stable, one testing.)
>> Right. Don't know if there's already an effort to convince the Debian
>> devs to switch the default...
> Please file a bug against the procps package (which owns the sysctl.conf
> file).  Just send a mail describing the issue and the proposed fix to
> submit@bugs.debian.org, with the first lines of the *body* of the mail
> having the following contents:
>
> Package: procps
> Version: 2:3.3.10-2
> Severity: wishlist

Good point

>
> In typical Debian style, expect this to cause a months long flamewar about
> systemd ;-)

though it's funny you should mention that :).

http://cerowrt-devel.bufferbloat.narkive.com/0mzipESN/ding-dong-is-the-debian-drop-tail-queue-dead#post3

Re: [Bloat] AQM and PPP on Linux

[Eric Dumazet]

On Tue, 2015-07-28 at 15:50 +0200, Juliusz Chroboczek wrote:
> > What distro are you on?
> 
> Debian.  (One machine stable, one testing.)
> 
> > There's a sysctl to set the default qdisc: net.core.default_qdisc.
> 
>     $ sysctl -a | grep default_qdisc
>     net.core.default_qdisc = pfifo_fast
> 
> > Set this to fq_codel and you should be good to go.
> 
> Is that safe?  I'm thinking lo and virtual interfaces.

Unless some user space script change it, virtual interfaces have
txqueuelen = 0, so no qdisc is set up on them

$ ifconfig lo | grep txqueuelen
          collisions:0 txqueuelen:0

$ tc -s -d qdisc sh dev lo
<empty>

So yes, this is reasonably safe to change net.core.default_qdisc

Re: [Bloat] AQM and PPP on Linux

[Dave Taht]

On Tue, Jul 28, 2015 at 3:09 PM, Juliusz Chroboczek
<jch@pps.univ-paris-diderot.fr> wrote:
> I'm currently away from home, and using a 3G modem for Internet access.
> I've found out that both NetworkManager and wvdial/pppd setup the
> interface to use pfifo_fast (with a qlen of a mere 3 packets!).  Setting
> fq_codel manually appears to work fine, but needs to be redone every time
> the modem has a hiccup.
>
> Is that the expected behaviour?  Shouldn't we be pushing patches somewhere
> to change the default?

1) The ppp driver was debloated by david woodhouse way back when.

However, the netusb-related drivers, and the 3g devices themselves,
contain oft-huge buffers that reduce the effectiveness of any aqm or
fq system, sometimes to immeasurability.

I would be very interested in flent benchmarks of your 3g device with
the 3 packet txqueue and with fq_codel, for the tcp_upload, rrul, and
rrul_be tests.

2) Most advanced qdiscs ignore txqueuelen entirely.

3) Usually you can add a script in /etc/network/ifpreup.d or ifup.d to
do extra stuff you want.

4) setting the default qdisc to be fq_codel in the sysctl works on
most dynamically created devices, but does not work on devices that
come up at boot time before the sysctls have a chance to run and the
modprobes to complete. This is why fedora 22, for example, compiled in
fq_codel statically, rather than dynamically.


> -- Juliusz
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Bloat] AQM and PPP on Linux

[Eric Dumazet]

On Tue, 2015-07-28 at 16:44 +0200, Dave Taht wrote:

> 
> 4) setting the default qdisc to be fq_codel in the sysctl works on
> most dynamically created devices, but does not work on devices that
> come up at boot time before the sysctls have a chance to run and the
> modprobes to complete. This is why fedora 22, for example, compiled in
> fq_codel statically, rather than dynamically.

It does not matter fq_codel is static or a module.

The thing is : the sysctl itself must be set to fq_codel before any
network device is brought up.

Since this is a user space policy, nothing prevents the script changing
sysctl to first load fq_codel module, then change sysctl.

Re: [Bloat] AQM and PPP on Linux

[Juliusz Chroboczek]

> I would be very interested in flent benchmarks of your 3g device with
> the 3 packet txqueue and with fq_codel, for the tcp_upload, rrul, and
> rrul_be tests.

Please send me detailed instructions, as well as information about how
much traffic I'm going to have to pay for.

FWIW, the RTT of the (unloaded) link varies tremendously.  Right now
(around 21:20), I'm seeing 89ms to 823ms, with an average of 192ms and
a standard deviation of 146ms.  It's better at night, mid-morning it's
pretty much unusable.

-- Juliusz

Re: [Bloat] AQM and PPP on Linux

[Bill Ver Steeg (versteb)]

It sounds like (one or more ) interface upstream of your link is also subject to periodic congestion. What a shocker --- cascaded links subject to bloat.

Bvs


-----Original Message-----
From: bloat-bounces@lists.bufferbloat.net [mailto:bloat-bounces@lists.bufferbloat.net] On Behalf Of Juliusz Chroboczek
Sent: Tuesday, July 28, 2015 3:25 PM
To: Dave Taht
Cc: bloat
Subject: Re: [Bloat] AQM and PPP on Linux

> I would be very interested in flent benchmarks of your 3g device with 
> the 3 packet txqueue and with fq_codel, for the tcp_upload, rrul, and 
> rrul_be tests.

Please send me detailed instructions, as well as information about how much traffic I'm going to have to pay for.

FWIW, the RTT of the (unloaded) link varies tremendously.  Right now (around 21:20), I'm seeing 89ms to 823ms, with an average of 192ms and a standard deviation of 146ms.  It's better at night, mid-morning it's pretty much unusable.

-- Juliusz
_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] AQM and PPP on Linux

[Alan Jenkins]

On 28/07/15 20:24, Juliusz Chroboczek wrote:
>> I would be very interested in flent benchmarks of your 3g device with
>> the 3 packet txqueue and with fq_codel, for the tcp_upload, rrul, and
>> rrul_be tests.
> Please send me detailed instructions, as well as information about how
> much traffic I'm going to have to pay for.
>
> FWIW, the RTT of the (unloaded) link varies tremendously.  Right now
> (around 21:20), I'm seeing 89ms to 823ms, with an average of 192ms and
> a standard deviation of 146ms.  It's better at night, mid-morning it's
> pretty much unusable.
>
> -- Juliusz

It's a straightforward usage.  My one concern would be ensuring you have 
netperf (netperf.org) built as required.

I suggest building netperf from source.  (Cite: flent requirements[1], 
and experience of failures with the Debian package)

     ./configure --enable-demo=yes --enable-histogram
make
     sudo make install

Untested (on my part) but documented[2] install is by debs at

https://software.opensuse.org/download.html?project=home:tohojo:flent&package=flent

or "pip install flent".

flent defaults to a 60 second test (and will run at whatever 4x tcp 
streams can manage).  Given the variation & high rtt I wouldn't want to 
shorten that very much.  One available graph from the output file will 
be what I assume Dave means when he says "mountain plot". That is, 
bandwidth & latency v.s. time.

Assuming EU (server taken from 
https://github.com/richb-hanover/CeroWrtScripts):

flent -H netperf-eu.bufferbloat.net rrul -t 
my-little-3G-connection-on-fq_codel

rrul is the test.  The -t value provides a caption.  Move the output 
file it mentions from /tmp (huh?) to somewhere more useful.  View 
available graphs with flent --gui <file>.


Alan


[1] 
http://netoptimizer.blogspot.co.uk/2014/09/mini-tutorial-for-netperf-wrapper-setup.html
[2] https://github.com/tohojo/flent/blob/master/README.rst

additional ref, including using your own server for netperf instead:

https://flent.org/intro.html#quick-start

Re: [Bloat] AQM and PPP on Linux

[Alan Jenkins]

On 28/07/15 21:10, Alan Jenkins wrote:
> On 28/07/15 20:24, Juliusz Chroboczek wrote:
>>> I would be very interested in flent benchmarks of your 3g device with
>>> the 3 packet txqueue and with fq_codel, for the tcp_upload, rrul, and
>>> rrul_be tests.
>> Please send me detailed instructions, as well as information about how
>> much traffic I'm going to have to pay for.
>>
>> FWIW, the RTT of the (unloaded) link varies tremendously.  Right now
>> (around 21:20), I'm seeing 89ms to 823ms, with an average of 192ms and
>> a standard deviation of 146ms.  It's better at night, mid-morning it's
>> pretty much unusable.
>>
>> -- Juliusz
>
> It's a straightforward usage.  My one concern would be ensuring you 
> have netperf (netperf.org) built as required.
>
> I suggest building netperf from source.  (Cite: flent requirements[1], 
> and experience of failures with the Debian package)
>
>     ./configure --enable-demo=yes --enable-histogram
> make
>     sudo make install
>
> Untested (on my part) but documented[2] install is by debs at
>
> https://software.opensuse.org/download.html?project=home:tohojo:flent&package=flent 
>
>
> or "pip install flent".
>
> flent defaults to a 60 second test (and will run at whatever 4x tcp 
> streams can manage).  Given the variation & high rtt I wouldn't want 
> to shorten that very much.  One available graph from the output file 
> will be what I assume Dave means when he says "mountain plot". That 
> is, bandwidth & latency v.s. time.
>
> Assuming EU (server taken from 
> https://github.com/richb-hanover/CeroWrtScripts):
>
> flent -H netperf-eu.bufferbloat.net rrul -t 
> my-little-3G-connection-on-fq_codel
>
> rrul is the test.  The -t value provides a caption.  Move the output 
> file it mentions from /tmp (huh?) to somewhere more useful.  View 
> available graphs with flent --gui <file>.

* flent-gui may have additional dependencies which may or may not be 
findable, sorry

python-matplotlib
python-matplotlib-qt4 or maybe python-matplotlib-qt5

otherwise try

flent -i <result file> -p <plot> -o out.svg

where plot is in e.g. `flent --list-plots rrul`

The plot over time is "all_scaled".

>
>
> Alan
>
>
> [1] 
> http://netoptimizer.blogspot.co.uk/2014/09/mini-tutorial-for-netperf-wrapper-setup.html
> [2] https://github.com/tohojo/flent/blob/master/README.rst
>
> additional ref, including using your own server for netperf instead:
>
> https://flent.org/intro.html#quick-start

Re: [Bloat] AQM and PPP on Linux

[Stefan Alfredsson]

* Quoting Dave Taht <dave.taht@gmail.com> [28 Jul-15 16:44]:
> On Tue, Jul 28, 2015 at 3:09 PM, Juliusz Chroboczek
> <jch@pps.univ-paris-diderot.fr> wrote:
> > I'm currently away from home, and using a 3G modem for Internet access.
> > I've found out that both NetworkManager and wvdial/pppd setup the
> > interface to use pfifo_fast (with a qlen of a mere 3 packets!).  Setting
> > fq_codel manually appears to work fine, but needs to be redone every time
> > the modem has a hiccup.
> 
> However, the netusb-related drivers, and the 3g devices themselves,
> contain oft-huge buffers that reduce the effectiveness of any aqm or
> fq system, sometimes to immeasurability.
> 
> I would be very interested in flent benchmarks of your 3g device with
> the 3 packet txqueue and with fq_codel, for the tcp_upload, rrul, and
> rrul_be tests.


I have a station with mobile broadband USB modems (4xHuawei E392),
directly connected to a Linux box running kernel 3.16.3, with
a (non-public) netperf server (same kernel) hosted on the well-connected
Swedish university network.

I also have a 3 packet txqueue, and ran the flent tcp_upload, rrul, rrul_be
for 3.5G (HSPA+) and 4G (LTE) networks for one of the operators/modems.
For comparison, I ran both pfifo_fast and fq_codel.

The execution script, output and flent result files are available at
https://www.dropbox.com/s/n5tc1dhtu9jdplz/bloatlist-ppp-150729.zip

From flent --gui inspections, I'd say there is about a factor 10 increase
from base to load latency (from around 40-50 ms to around 400-500 ms),
for both pfifo_fast and fq_codel, for both link types,
with some transients above 1000 ms.


Juliusz: Regarding data usage, the 4G measurements consumed 1.4 Gbyte
download and 357 Mbyte upload - but it all depends on the network
throughput as the experiments are time bound (by default 60 seconds).
For 3.5G 100 Mbyte data was sent, and guesstimately 500 Mbyte received
(sorry, didn't capture the amount; but it can probably be found in
the trace files if necessary. However it all depends on your link 
throughput which will be different from mine anyway).


Cheers,
 Stefan

-- 
Stefan Alfredsson, PhD         Tel: +46 (0) 54 700 1668
Datavetenskap                  Kontor: 21E-414 (Hus Vanern)
Karlstads universitet          PGP 0xB19B4B16
SE-651 88 Karlstad             http://www.cs.kau.se/stefalfr/

Re: [Bloat] AQM and PPP on Linux

[Dave Taht]

On Wed, Jul 29, 2015 at 3:15 PM, Stefan Alfredsson
<Stefan.Alfredsson@kau.se> wrote:
> * Quoting Dave Taht <dave.taht@gmail.com> [28 Jul-15 16:44]:
>> On Tue, Jul 28, 2015 at 3:09 PM, Juliusz Chroboczek
>> <jch@pps.univ-paris-diderot.fr> wrote:
>> > I'm currently away from home, and using a 3G modem for Internet access.
>> > I've found out that both NetworkManager and wvdial/pppd setup the
>> > interface to use pfifo_fast (with a qlen of a mere 3 packets!).  Setting
>> > fq_codel manually appears to work fine, but needs to be redone every time
>> > the modem has a hiccup.
>>
>> However, the netusb-related drivers, and the 3g devices themselves,
>> contain oft-huge buffers that reduce the effectiveness of any aqm or
>> fq system, sometimes to immeasurability.
>>
>> I would be very interested in flent benchmarks of your 3g device with
>> the 3 packet txqueue and with fq_codel, for the tcp_upload, rrul, and
>> rrul_be tests.
>
>
> I have a station with mobile broadband USB modems (4xHuawei E392),
> directly connected to a Linux box running kernel 3.16.3, with
> a (non-public) netperf server (same kernel) hosted on the well-connected
> Swedish university network.
>
> I also have a 3 packet txqueue, and ran the flent tcp_upload, rrul, rrul_be
> for 3.5G (HSPA+) and 4G (LTE) networks for one of the operators/modems.
> For comparison, I ran both pfifo_fast and fq_codel.
>
> The execution script, output and flent result files are available at
> https://www.dropbox.com/s/n5tc1dhtu9jdplz/bloatlist-ppp-150729.zip

Thank you SO MUCH for the raw flent data. That made poking through this a snap!

Of course, the news is mostly bad - a tiny fifo qdisc on LTE looks to
be presently best than any amount of queuing/aqm/fq at the layers
above the device driver and device.

Something like BQL is needed for usbnet, and the usb-cellmodem
driver... and some means of tracking completion interrupts sanely to
change this picture. We did a bit of work on both these layers a while
back but we lost steam at the complex interactions of the usb stack.

as an aside - and on the wifi front, recently toke and I did a string
of overnight tests on the wifi test rig we are taking to battlemesh,
which includes the latest minstrel blues and minstrel variance
patches. Sadly, also, mostly of note is the wide variabilty of results
from run to run, in the same wifi environment. There were only a few
radios we could "hear" to interfere... and I'd done much better with
the exact same radios in an environment fully under my control.

A tarball of all these flent testruns and tests is at

http://snapon.lab.bufferbloat.net/~d/long-running.tgz

Anyway... the variance from run to run:

http://snapon.lab.bufferbloat.net/~d/long-running/10testscompared.png

Trying to pull apart the apparently bi-modal interactions with rate
control (and *why*) is going to take some *work*.

http://snapon.lab.bufferbloat.net/~d/long-running/themessthatiswifi.png

This plot, randomly picked out from the others

http://snapon.lab.bufferbloat.net/~d/long-running/wifiratecontrol.png

shows clearly the current interaction between the current rate of the
wifi and the latency. (but I have no idea *why* it stayed stuck at the
lower rate for 20+ seconds!). Still, I believe we have developed means
to hold the latency low at nearly any bandwidth, we just have to roll
up our sleeves and make per station queuing work and then start
glomming on new algorithms. Or so I hope.

...

I happened to rather like this plot, from an artistic perspective,
showing the bands that each 802.11e queue fits into. Sort of.

http://snapon.lab.bufferbloat.net/~d/long-running/vangogh.png

But maybe I am thinking of the wrong artist?

> From flent --gui inspections, I'd say there is about a factor 10 increase
> from base to load latency (from around 40-50 ms to around 400-500 ms),
> for both pfifo_fast and fq_codel, for both link types,
> with some transients above 1000 ms.
>
>
> Juliusz: Regarding data usage, the 4G measurements consumed 1.4 Gbyte
> download and 357 Mbyte upload - but it all depends on the network
> throughput as the experiments are time bound (by default 60 seconds).
> For 3.5G 100 Mbyte data was sent, and guesstimately 500 Mbyte received
> (sorry, didn't capture the amount; but it can probably be found in
> the trace files if necessary. However it all depends on your link
> throughput which will be different from mine anyway).

Yes. One reason why I have not focused much on fixing LTE, is I refuse
to have them charge me for attempting to fix their technology. It is
bad enough throwing good money away to Netgear to fix their routers,
but to get charged through the **** for LTE on a bandwidth basis - and
I could see, eating a lot of bandwidth with various tests - ENOBUDGET.

Now, if we could, like, get 20 free unlimited use sims for anywhere in
the world, that might be an incentive.... ;)

>
>
> Cheers,
>  Stefan
>
> --
> Stefan Alfredsson, PhD         Tel: +46 (0) 54 700 1668
> Datavetenskap                  Kontor: 21E-414 (Hus Vanern)
> Karlstads universitet          PGP 0xB19B4B16
> SE-651 88 Karlstad             http://www.cs.kau.se/stefalfr/
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast