[Codel] The challenge

[Codel] The challenge

[Dave Taht]

Both the acm queue article and jim's blog entry this morning were way
above mensa's standards.

Nobody has attempted to explain the elegant simplicity of the
algorithm itself in the inverse sqrt however!  I have a good grip on
it, and am trying, but can barely explain it to myself. Anyone else
care to dig through the codel code and try to put it into english?

Nice bit in ReadWrite News:

http://www.readwriteweb.com/enterprise/2012/05/good-news-for-solving-bufferbloat-codel-provides-no-knobs-solution.php

Bob Cringley lays down the challenge for us here on the bloat list,
and details the opportunity.

http://www.cringely.com/2012/05/beginning-of-the-end-for-bufferbloat/

He closes with:

"My advice to Cisco, Netgear, D-Link and others is that this could be
an important moment in their businesses if they choose to approach it
correctly. It’s a chance to get all of us to buy new routers, perhaps
new everything. Think of the music industry bonanza when we all
shifted our record libraries from vinyl to CDs. It could be the same
for networking equipment. But for that to happen the vendors have to
finally acknowledge bufferbloat and use their marketing dollars to
teach us all why we should upgrade ASAP. Everybody would win.

Take our money, please."

With the cerowrt project, at least, I've hoped to make that shift
possible, and to some extent... happen.

We have *working code*, and *proof of concept*. What's next? Where do
we go from here?

-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

Re: [Codel] [Bloat] The challenge

[Kevin Gross]

[-- Attachment #1: Type: text/plain, Size: 2560 bytes --]

The decision the algorithm is trying to make is when to drop packets. RED
drops packets when the maximum queue level rises. CoDel drops packets when
the minimum queue level rises. If the queue is never (nearly) empty, the
interface is oversubscribed and packet drops are in order. But because it
looks at minimum, not maximum, CoDel will not drop packets in bursts so
long as those bursts pass within the (5 ms) interval used to measure
minimum queue level.

1/sqrt bit controls how many packets get dropped as a function of time
and I gather it is control loop tuning used to strike a good balance
between responsiveness, stability and do no harm.

Kevin Gross

On Tue, May 8, 2012 at 7:04 PM, Dave Taht <dave.taht@gmail.com> wrote:

> Both the acm queue article and jim's blog entry this morning were way
> above mensa's standards.
>
> Nobody has attempted to explain the elegant simplicity of the
> algorithm itself in the inverse sqrt however!  I have a good grip on
> it, and am trying, but can barely explain it to myself. Anyone else
> care to dig through the codel code and try to put it into english?
>
> Nice bit in ReadWrite News:
>
>
> http://www.readwriteweb.com/enterprise/2012/05/good-news-for-solving-bufferbloat-codel-provides-no-knobs-solution.php
>
> Bob Cringley lays down the challenge for us here on the bloat list,
> and details the opportunity.
>
> http://www.cringely.com/2012/05/beginning-of-the-end-for-bufferbloat/
>
> He closes with:
>
> "My advice to Cisco, Netgear, D-Link and others is that this could be
> an important moment in their businesses if they choose to approach it
> correctly. It’s a chance to get all of us to buy new routers, perhaps
> new everything. Think of the music industry bonanza when we all
> shifted our record libraries from vinyl to CDs. It could be the same
> for networking equipment. But for that to happen the vendors have to
> finally acknowledge bufferbloat and use their marketing dollars to
> teach us all why we should upgrade ASAP. Everybody would win.
>
> Take our money, please."
>
> With the cerowrt project, at least, I've hoped to make that shift
> possible, and to some extent... happen.
>
> We have *working code*, and *proof of concept*. What's next? Where do
> we go from here?
>
> --
> Dave Täht
> SKYPE: davetaht
> US Tel: 1-239-829-5608
> http://www.bufferbloat.net
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>

[-- Attachment #2: Type: text/html, Size: 3520 bytes --]

Re: [Codel] The challenge

[Kathleen Nichols]

On 5/8/12 6:04 PM, Dave Taht wrote:
> Both the acm queue article and jim's blog entry this morning were way
> above mensa's standards.

But, omg, we are not saying this will solve everything.  And it's not
supposed
to be a scholarly paper, but a proposed solution to a real problem. Guess
Dave and Eric will let us know how real it is.

We are still tweaking to make sure the streamlined code does what the
old simulator code does. Should be a small update shortly.

> 
> Nobody has attempted to explain the elegant simplicity of the
> algorithm itself in the inverse sqrt however!  I have a good grip on
> it, and am trying, but can barely explain it to myself. Anyone else
> care to dig through the codel code and try to put it into english?

The reason I referenced our ancient unpublished work on this in additon to
Matt Mathis's paper is that Van had some nice pictures and explanations in
there.

> 
> Nice bit in ReadWrite News:
> 
> http://www.readwriteweb.com/enterprise/2012/05/good-news-for-solving-bufferbloat-codel-provides-no-knobs-solution.php
> 
> Bob Cringley lays down the challenge for us here on the bloat list,
> and details the opportunity.
> 
> http://www.cringely.com/2012/05/beginning-of-the-end-for-bufferbloat/
> 
> He closes with:
> 
> "My advice to Cisco, Netgear, D-Link and others is that this could be
> an important moment in their businesses if they choose to approach it
> correctly. It’s a chance to get all of us to buy new routers, perhaps
> new everything. Think of the music industry bonanza when we all
> shifted our record libraries from vinyl to CDs. It could be the same
> for networking equipment. But for that to happen the vendors have to
> finally acknowledge bufferbloat and use their marketing dollars to
> teach us all why we should upgrade ASAP. Everybody would win.
> 
> Take our money, please."
> 
> With the cerowrt project, at least, I've hoped to make that shift
> possible, and to some extent... happen.
> 
> We have *working code*, and *proof of concept*. What's next? Where do
> we go from here?
> 

Re: [Codel] The challenge

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 2006 bytes --]

On Tue, May 8, 2012 at 8:13 PM, Kathleen Nichols <nichols@pollere.com>wrote:

>
>
> On 5/8/12 6:04 PM, Dave Taht wrote:
> > Both the acm queue article and jim's blog entry this morning were way
> > above mensa's standards.
>
> But, omg, we are not saying this will solve everything.


Neither am I! Figuring out what it can solve and what it can't is on the
table.

That's going to take quite some time, I think. Title of email was

"The challenge". Next up is extensive real world tests and tweaking, more
simulation, varied workloads.



> And it's not
> supposed
> to be a scholarly paper,


thank ghu


> but a proposed solution to a real problem. Guess
> Dave and Eric will let us know how real it is.
>

And andrew is looking at the wobble right now in the ns3 sim. He has some
pretty graphs... I love to know if his model matches yours...

I've held off on doing much, I have an unrelated bug that's killing me in
reality and
I figure eric has another version soon


> We are still tweaking to make sure the streamlined code does what the
> old simulator code does. Should be a small update shortly.
>
>
Excellent.


> >
> > Nobody has attempted to explain the elegant simplicity of the
> > algorithm itself in the inverse sqrt however!  I have a good grip on
> > it, and am trying, but can barely explain it to myself. Anyone else
> > care to dig through the codel code and try to put it into english?
>
> The reason I referenced our ancient unpublished work on this in additon to
> Matt Mathis's paper is that Van had some nice pictures and explanations in
> there.
>

Ah. Lost in the footnotes. You mean redlight?

The link on the acm queue to:

http://www.cnaf.infn.it/~ferrari/papers/ispn/red_light_9_30.pdf is busted.

We have a link to a mirror of that here:

http://mirrors.bufferbloat.net/RelevantPapers/Red_in_a_different_light.pdf


-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

[-- Attachment #2: Type: text/html, Size: 3310 bytes --]

Re: [Codel] [Bloat] The challenge

[Simon Barber]

One question now remains - will codel AQM be sufficient on it's own in 
getting delays down to levels that users are happy with for the common 
latency sensitive interactive traffic - VoIP, gaming and Skype for 
example - or are the further reductions that can be had with traffic 
classification and smart queuing algorithms necessary? The nicest part 
about codel on it's own is that it works on opaque packets - it will 
handle VPNs and traffic within them nicely. It gets away from all the 
complexity required to classify traffic in a world where traffic is 
often trying to hide.

Simon

On Tue 08 May 2012 06:04:50 PM PDT, Dave Taht wrote:
> Both the acm queue article and jim's blog entry this morning were way
> above mensa's standards.
>
> Nobody has attempted to explain the elegant simplicity of the
> algorithm itself in the inverse sqrt however!  I have a good grip on
> it, and am trying, but can barely explain it to myself. Anyone else
> care to dig through the codel code and try to put it into english?
>
> Nice bit in ReadWrite News:
>
> http://www.readwriteweb.com/enterprise/2012/05/good-news-for-solving-bufferbloat-codel-provides-no-knobs-solution.php
>
> Bob Cringley lays down the challenge for us here on the bloat list,
> and details the opportunity.
>
> http://www.cringely.com/2012/05/beginning-of-the-end-for-bufferbloat/
>
> He closes with:
>
> "My advice to Cisco, Netgear, D-Link and others is that this could be
> an important moment in their businesses if they choose to approach it
> correctly. It’s a chance to get all of us to buy new routers, perhaps
> new everything. Think of the music industry bonanza when we all
> shifted our record libraries from vinyl to CDs. It could be the same
> for networking equipment. But for that to happen the vendors have to
> finally acknowledge bufferbloat and use their marketing dollars to
> teach us all why we should upgrade ASAP. Everybody would win.
>
> Take our money, please."
>
> With the cerowrt project, at least, I've hoped to make that shift
> possible, and to some extent... happen.
>
> We have *working code*, and *proof of concept*. What's next? Where do
> we go from here?
>

Re: [Codel] [Bloat] The challenge

[Eric Dumazet]

On Tue, 2012-05-08 at 22:16 -0700, Simon Barber wrote:
> One question now remains - will codel AQM be sufficient on it's own in 
> getting delays down to levels that users are happy with for the common 
> latency sensitive interactive traffic - VoIP, gaming and Skype for 
> example - or are the further reductions that can be had with traffic 
> classification and smart queuing algorithms necessary? The nicest part 
> about codel on it's own is that it works on opaque packets - it will 
> handle VPNs and traffic within them nicely. It gets away from all the 
> complexity required to classify traffic in a world where traffic is 
> often trying to hide.

It all depends on the requirements you have.

To me, CoDel is a RED replacement, because it provides something easier
to deploy (no knobs). So it wont solve by itself cases where you want
something that could not be done by a single RED queue.

I like to see Codel as a basic unit, to replace RED (or pfifo if some
fools still use this for whatever reasons) in a tree involving
classifiers and FQ.

Re: [Codel] [Bloat] The challenge

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 1289 bytes --]

On Tue, May 8, 2012 at 10:40 PM, Eric Dumazet <eric.dumazet@gmail.com>wrote:

> On Tue, 2012-05-08 at 22:16 -0700, Simon Barber wrote:
> > One question now remains - will codel AQM be sufficient on it's own in
> > getting delays down to levels that users are happy with for the common
> > latency sensitive interactive traffic - VoIP, gaming and Skype for
> > example - or are the further reductions that can be had with traffic
> > classification and smart queuing algorithms necessary? The nicest part
> > about codel on it's own is that it works on opaque packets - it will
> > handle VPNs and traffic within them nicely. It gets away from all the
> > complexity required to classify traffic in a world where traffic is
> > often trying to hide.
>
> It all depends on the requirements you have.
>
> To me, CoDel is a RED replacement, because it provides something easier
> to deploy (no knobs). So it wont solve by itself cases where you want
> something that could not be done by a single RED queue.
>
> I like to see Codel as a basic unit, to replace RED (or pfifo if some
> fools still use this for whatever reasons) in a tree involving
> classifiers and FQ.
>
>
+10



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

[-- Attachment #2: Type: text/html, Size: 1740 bytes --]

[Codel] [codel] some numbers on dual 10Gb links

[Eric Dumazet]

Some results on my setup (2 fibers at 10Gb), but my remote machine
is a slow quadcore (Q6600). I am waiting net-next compiles on this one
so that I can reverse the test (sender on the slow machine)

10 netperf on one fiber, 10 netperf on the second one.


TC Setup :
EST="est 1sec 4sec"

DEV=eth7
ethtool -K $DEV tso off gso off
tc qdisc del dev $DEV root 2>/dev/null

tc qdisc add dev $DEV root handle 1: $EST htb default 1
tc class add dev $DEV parent 1: classid 1:1 $EST \
	htb rate 4000Mbit mtu 1500 quantum 1514 
tc qdisc add dev $DEV parent 1:1 handle 10: $EST \
	codel ecn target 500us interval 10ms limit 2000

DEV=eth8
ethtool -K $DEV tso off gso off
tc qdisc del dev $DEV root 2>/dev/null

tc qdisc add dev $DEV root handle 1: $EST htb default 1
tc class add dev $DEV parent 1: classid 1:1 $EST \
	htb rate 4000Mbit mtu 1500 quantum 1514 
tc qdisc add dev $DEV parent 1:1 handle 10: $EST \
	codel ecn target 500us interval 10ms limit 2000


# tc -s -d qdisc
qdisc htb 1: dev eth7 root refcnt 33 r2q 10 default 1 direct_packets_stat 0 ver 3.17
 Sent 279883414602 bytes 184874085 pkt (dropped 0, overlimits 70072116 requeues 20193) 
 rate 3282Mbit 271033pps backlog 0b 368p requeues 20193 
qdisc codel 10: dev eth7 parent 1:1 ecn limit 2000p minbytes 1514 target 499us interval 10.0ms 
 Sent 279882995224 bytes 184873808 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 3282Mbit 271033pps backlog 557152b 368p requeues 0 
  count 3 delay 1.5ms ecn_mark 15130
  drop_overlimit 0 states 184873808 : 39403313 9921 5209

qdisc htb 1: dev eth8 root refcnt 33 r2q 10 default 1 direct_packets_stat 0 ver 3.17
 Sent 279835644380 bytes 184841333 pkt (dropped 0, overlimits 70249151 requeues 20188) 
 rate 3290Mbit 271687pps backlog 0b 266p requeues 20188 
qdisc codel 10: dev eth8 parent 1:1 ecn limit 2000p minbytes 1514 target 499us interval 10.0ms 
 Sent 279835421822 bytes 184841186 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 3290Mbit 271687pps backlog 405752b 268p requeues 0 
  count 3 delay 1.0ms ecn_mark 15068
  drop_overlimit 0 states 184841186 : 39470075 9793 5275

# ping 192.168.99.2
PING 192.168.99.2 (192.168.99.2) 56(84) bytes of data.
64 bytes from 192.168.99.2: icmp_req=1 ttl=64 time=0.072 ms
64 bytes from 192.168.99.2: icmp_req=2 ttl=64 time=0.087 ms
64 bytes from 192.168.99.2: icmp_req=3 ttl=64 time=0.338 ms
64 bytes from 192.168.99.2: icmp_req=4 ttl=64 time=0.810 ms
64 bytes from 192.168.99.2: icmp_req=5 ttl=64 time=0.827 ms
64 bytes from 192.168.99.2: icmp_req=6 ttl=64 time=1.22 ms
64 bytes from 192.168.99.2: icmp_req=7 ttl=64 time=0.840 ms
64 bytes from 192.168.99.2: icmp_req=8 ttl=64 time=0.898 ms
64 bytes from 192.168.99.2: icmp_req=9 ttl=64 time=1.66 ms
64 bytes from 192.168.99.2: icmp_req=10 ttl=64 time=0.886 ms
64 bytes from 192.168.99.2: icmp_req=11 ttl=64 time=0.571 ms
64 bytes from 192.168.99.2: icmp_req=12 ttl=64 time=0.422 ms
64 bytes from 192.168.99.2: icmp_req=13 ttl=64 time=0.259 ms
64 bytes from 192.168.99.2: icmp_req=14 ttl=64 time=1.47 ms
64 bytes from 192.168.99.2: icmp_req=15 ttl=64 time=1.32 ms
64 bytes from 192.168.99.2: icmp_req=16 ttl=64 time=0.765 ms
64 bytes from 192.168.99.2: icmp_req=17 ttl=64 time=1.23 ms
64 bytes from 192.168.99.2: icmp_req=18 ttl=64 time=1.36 ms
64 bytes from 192.168.99.2: icmp_req=19 ttl=64 time=0.151 ms
64 bytes from 192.168.99.2: icmp_req=20 ttl=64 time=2.14 ms
64 bytes from 192.168.99.2: icmp_req=21 ttl=64 time=1.12 ms
^C
--- 192.168.99.2 ping statistics ---
21 packets transmitted, 21 received, 0% packet loss, time 20008ms
rtt min/avg/max/mdev = 0.072/0.879/2.144/0.540 ms

Re: [Codel] [Bloat] The challenge

[Kevin Gross]

[-- Attachment #1: Type: text/plain, Size: 3080 bytes --]

From the paper (figure 7), you can see that CoDel still leaves spikes of
buffer occupancy when network conditions change. These will still
be disruptive to real-time traffic. Many networks that need QoS now will
still need QoS. Networks that do not have QoS will be much more usable with
CoDel.

Kevin Gross

On Tue, May 8, 2012 at 11:16 PM, Simon Barber <simon@superduper.net> wrote:

> One question now remains - will codel AQM be sufficient on it's own in
> getting delays down to levels that users are happy with for the common
> latency sensitive interactive traffic - VoIP, gaming and Skype for example
> - or are the further reductions that can be had with traffic classification
> and smart queuing algorithms necessary? The nicest part about codel on it's
> own is that it works on opaque packets - it will handle VPNs and traffic
> within them nicely. It gets away from all the complexity required to
> classify traffic in a world where traffic is often trying to hide.
>
> Simon
>
>
> On Tue 08 May 2012 06:04:50 PM PDT, Dave Taht wrote:
>
>> Both the acm queue article and jim's blog entry this morning were way
>> above mensa's standards.
>>
>> Nobody has attempted to explain the elegant simplicity of the
>> algorithm itself in the inverse sqrt however!  I have a good grip on
>> it, and am trying, but can barely explain it to myself. Anyone else
>> care to dig through the codel code and try to put it into english?
>>
>> Nice bit in ReadWrite News:
>>
>> http://www.readwriteweb.com/**enterprise/2012/05/good-news-**
>> for-solving-bufferbloat-codel-**provides-no-knobs-solution.php<http://www.readwriteweb.com/enterprise/2012/05/good-news-for-solving-bufferbloat-codel-provides-no-knobs-solution.php>
>>
>> Bob Cringley lays down the challenge for us here on the bloat list,
>> and details the opportunity.
>>
>> http://www.cringely.com/2012/**05/beginning-of-the-end-for-**bufferbloat/<http://www.cringely.com/2012/05/beginning-of-the-end-for-bufferbloat/>
>>
>> He closes with:
>>
>> "My advice to Cisco, Netgear, D-Link and others is that this could be
>> an important moment in their businesses if they choose to approach it
>> correctly. It’s a chance to get all of us to buy new routers, perhaps
>> new everything. Think of the music industry bonanza when we all
>> shifted our record libraries from vinyl to CDs. It could be the same
>> for networking equipment. But for that to happen the vendors have to
>> finally acknowledge bufferbloat and use their marketing dollars to
>> teach us all why we should upgrade ASAP. Everybody would win.
>>
>> Take our money, please."
>>
>> With the cerowrt project, at least, I've hoped to make that shift
>> possible, and to some extent... happen.
>>
>> We have *working code*, and *proof of concept*. What's next? Where do
>> we go from here?
>>
>>  ______________________________**_________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/**listinfo/bloat<https://lists.bufferbloat.net/listinfo/bloat>
>

[-- Attachment #2: Type: text/html, Size: 3884 bytes --]

Re: [Codel] [Bloat] The challenge

[Roger Jørgensen]

On Wed, May 9, 2012 at 7:16 AM, Simon Barber <simon@superduper.net> wrote:
> One question now remains - will codel AQM be sufficient on it's own in
> getting delays down to levels that users are happy with for the common
> latency sensitive interactive traffic - VoIP, gaming and Skype for example -
> or are the further reductions that can be had with traffic classification
> and smart queuing algorithms necessary? The nicest part about codel on it's
> own is that it works on opaque packets - it will handle VPNs and traffic
> within them nicely. It gets away from all the complexity required to
> classify traffic in a world where traffic is often trying to hide.

I'm more worried about how long cascades of codel code in equipment
will work together, add lots of traffic to the mix, just how will it
behave then?



-- 

Roger Jorgensen           | ROJO9-RIPE
rogerj@gmail.com          | - IPv6 is The Key!
http://www.jorgensen.no   | roger@jorgensen.no

Re: [Codel] [Bloat] The challenge

[Dave Taht]

On Wed, May 9, 2012 at 12:10 PM, Roger Jørgensen <rogerj@gmail.com> wrote:
> On Wed, May 9, 2012 at 7:16 AM, Simon Barber <simon@superduper.net> wrote:
>> One question now remains - will codel AQM be sufficient on it's own in
>> getting delays down to levels that users are happy with for the common
>> latency sensitive interactive traffic - VoIP, gaming and Skype for example -
>> or are the further reductions that can be had with traffic classification
>> and smart queuing algorithms necessary? The nicest part about codel on it's
>> own is that it works on opaque packets - it will handle VPNs and traffic
>> within them nicely. It gets away from all the complexity required to
>> classify traffic in a world where traffic is often trying to hide.
>
> I'm more worried about how long cascades of codel code in equipment
> will work together, add lots of traffic to the mix, just how will it
> behave then?

Dunno. I look forward to finding out.

I put up debian packages of the kernel for ubuntu 12.4 here:

http://huchra.bufferbloat.net/~d/codel/linux/

You'll need to build your own iproute2 from the current (v12) patch set.

Basic tests are throttling down your network card to like 100Mbit and
trying it, unless you have major cpu that can saturate gigE or higher.

I'm mostly using netperf from svn head.

>
>
> --
>
> Roger Jorgensen           | ROJO9-RIPE
> rogerj@gmail.com          | - IPv6 is The Key!
> http://www.jorgensen.no   | roger@jorgensen.no



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

Re: [Codel] [Bloat] The challenge

[Dave Taht]

On Wed, May 9, 2012 at 12:15 PM, Dave Taht <dave.taht@gmail.com> wrote:
> On Wed, May 9, 2012 at 12:10 PM, Roger Jørgensen <rogerj@gmail.com> wrote:
>> On Wed, May 9, 2012 at 7:16 AM, Simon Barber <simon@superduper.net> wrote:
>>> One question now remains - will codel AQM be sufficient on it's own in
>>> getting delays down to levels that users are happy with for the common
>>> latency sensitive interactive traffic - VoIP, gaming and Skype for example -
>>> or are the further reductions that can be had with traffic classification
>>> and smart queuing algorithms necessary? The nicest part about codel on it's
>>> own is that it works on opaque packets - it will handle VPNs and traffic
>>> within them nicely. It gets away from all the complexity required to
>>> classify traffic in a world where traffic is often trying to hide.
>>
>> I'm more worried about how long cascades of codel code in equipment
>> will work together, add lots of traffic to the mix, just how will it
>> behave then?
>
> Dunno. I look forward to finding out.
>
> I put up debian packages of the kernel for ubuntu 12.4 here:
>
> http://huchra.bufferbloat.net/~d/codel/linux/
>
> You'll need to build your own iproute2 from the current (v12) patch set.
>
> Basic tests are throttling down your network card to like 100Mbit and
> trying it, unless you have major cpu that can saturate gigE or higher.

Um, 'major' cpu is not required. (I spend too much time on relatively
lame embedded hardware.) Almost any box made in the last 5 years can
run at gigE+.

However, there are other reasons (like ease of packet captures, the
core targets of this being uplinks from consumer boxes running at
.1-20Mbit, etc., to run slower)

Seeing the scaling factors actually work is good, too. As a simple and
dramatic example, try running codel at 10Mbit, and compare it against
the default pfifo_fast.

ethtool -K eth0 TSO off
ethtool -K eth0 GSO off
ethtool -K eth0 UFO off
ethtool -s eth0 advertise 0x002 # 10Mbit. 0x008 100Mbit

some_bandwidth_test_or_tests

tc qdisc add dev eth0 root codel # optionally, add ecn

some_bandwidth_test_or_tests

>
> I'm mostly using netperf from svn head.
>
>>
>>
>> --
>>
>> Roger Jorgensen           | ROJO9-RIPE
>> rogerj@gmail.com          | - IPv6 is The Key!
>> http://www.jorgensen.no   | roger@jorgensen.no
>
>
>
> --
> Dave Täht
> SKYPE: davetaht
> US Tel: 1-239-829-5608
> http://www.bufferbloat.net



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

Re: [Codel] [Bloat] The challenge

[Simon Barber]

significant queues only form where there is a bottleneck, so there 
should not be long cascades of codel actually dropping.

Simon

On 05/09/2012 12:10 PM, Roger Jørgensen wrote:
> On Wed, May 9, 2012 at 7:16 AM, Simon Barber<simon@superduper.net>  wrote:
>> One question now remains - will codel AQM be sufficient on it's own in
>> getting delays down to levels that users are happy with for the common
>> latency sensitive interactive traffic - VoIP, gaming and Skype for example -
>> or are the further reductions that can be had with traffic classification
>> and smart queuing algorithms necessary? The nicest part about codel on it's
>> own is that it works on opaque packets - it will handle VPNs and traffic
>> within them nicely. It gets away from all the complexity required to
>> classify traffic in a world where traffic is often trying to hide.
>
> I'm more worried about how long cascades of codel code in equipment
> will work together, add lots of traffic to the mix, just how will it
> behave then?
>
>
>

Re: [Codel] [Bloat] The challenge

[Fred Baker]

On May 9, 2012, at 12:16 AM, Simon Barber wrote:

> One question now remains - will codel AQM be sufficient on it's own in getting delays down to levels that users are happy with for the common latency sensitive interactive traffic - VoIP, gaming and Skype for example - or are the further reductions that can be had with traffic classification and smart queuing algorithms necessary?

In my opinion, AQM is enough to get traffic into the "reasonable" range, but if you're looking for a specific SLA that might be applicable to gaming etc, you will need to do some engineering in terms of diffserv etc.

Re: [Codel] [Bloat] The challenge

[Jim Gettys]

On 05/09/2012 04:06 PM, Fred Baker wrote:
> On May 9, 2012, at 12:16 AM, Simon Barber wrote:
>
>> One question now remains - will codel AQM be sufficient on it's own in getting delays down to levels that users are happy with for the common latency sensitive interactive traffic - VoIP, gaming and Skype for example - or are the further reductions that can be had with traffic classification and smart queuing algorithms necessary?
> In my opinion, AQM is enough to get traffic into the "reasonable" range, but if you're looking for a specific SLA that might be applicable to gaming etc, you will need to do some engineering in terms of diffserv etc.
> _______________________________________________
>
I used to think that, before I understood:
    1) what the web was doing (again), by ignoring the 2 old TCP
connection rule, crossed by web site "sharding".  This is why I wrote
the IW10 considered Evil draft last fall.
    2) what the NIC offload engines were doing to generate line rate
packet trains and injecting them into the net, where they can land
"spat" at the customer end.
    3) Ledbat won't help once an effective AQM is in place; the delay
it's keying off of goes away, and then it competes like Reno with
regular flows.  100 BitTorrent flows competing with your traffic can
ruin your entire day.  We get to revisit this topic.
    4) a busy 802.11 net (or using one where the range to the AP drops
the bandwidth to low rate) means even single big packets has 12
milliseconds of latency @ 1Mbps (even ignoring other 802.11 effects,
just the bit transfer time).

Even on high speed (50Mbps broadband) I see transients of > 100ms just
browsing image heavy web sites.

So while CoDel will help (a lot, particularly policing TCP so that it
will respond quickly rather than suffering the quadratic responsiveness
problem), I now firmly believe we have to probably delve into "fair"
queuing and classification if we want a low latency edge to the Internet.

This doesn't mean I believe we'll have to do all this beyond the
broadband edge box (e.g. CMTS/DSLAM/FIOS) ; but I think these and the
home network have to actually be careful.  thankfully, at the edge, you
have way more cycles/packet you can *afford* to use on such schemes.

Having an AQM that actually works well enough to be "on" by default will
help most of the other instances of bloat I know of elsewhere, but the
edge, it's a different story....
                        - Jim

Re: [Codel] [Bloat] The challenge

[Eric Dumazet]

On Wed, 2012-05-09 at 17:47 -0400, Jim Gettys wrote:

>     2) what the NIC offload engines were doing to generate line rate
> packet trains and injecting them into the net, where they can land
> "spat" at the customer end.


If the server has 100Gb link, and the client a 1Mb link, TSO is hardly
making a difference in trains. Sending 10 consecutive 1500 bytes packets
or one 15000 bytes packets will have the same effect, unless the sender
is serving thousands of clients and a FairQueue interleaves packets.

One solution is at the producer level (TCP stack of the sender in this
case) to properly pace the transmits in MSS units given the RTT and
cwnd, supposedly done with TCP Hybla.

Re: [Codel] [Bloat] The challenge

[Jonathan Morton]

On 9 May, 2012, at 4:04 am, Dave Taht wrote:

> Both the acm queue article and jim's blog entry this morning were way
> above mensa's standards.

Does that mean we're allowed to feel extra smart if we understood it anyway?  :-)

> Nobody has attempted to explain the elegant simplicity of the
> algorithm itself in the inverse sqrt however!  I have a good grip on
> it, and am trying, but can barely explain it to myself. Anyone else
> care to dig through the codel code and try to put it into english?

Sounds like fun!

I have a suspicion myself of what the quantitative reasoning behind the inverse sqrt might be, but I also suspect that such details are much less important than some of the *qualitative* properties of the algorithm.  Most likely the people that most need to be convinced will respond better to that than to hard maths anyway.

I think however that everyone has missed the single most important property: the fact that drops occur on *dequeue* rather than enqueue, and the result that this keeps the "resonant frequency" of each flow near the physical RTT by minimising the delay between a packet drop and the TCP's detection of and response to it.

It is counter-intuitive to keep a queue filled with packets that you might drop later - as an engineer you tend to think it is more efficient to drop a packet *before* it starts to consume space in your precious queue.  This is a good deal of what needs to be solved in wireless drivers - there is a wrong assumption in there that dropping at the tail is all that is needed, and that leads to wrong design decisions.

Tail-drop and RED drop on enqueue (at tail), not dequeue (at head), so the missing packet cannot be detected by the TCP until it's "bubble" has passed through the queue.  This sets the resonant frequency lower than the true RTT and delays the TCP's response, so the queue remains overfilled for longer (triggering extra packet drops which require extra recovery) and recovery (through retransmission) also takes longer and is less reliable.

Another point that might be missed by people naively reading the paper - as opposed to regular readers of AQM literature - is that "drop" can actually mean "mark with ECN".  Looking at the code, ECN is indeed used when appropriate.  This is good because it signals the TCP without forcing a retransmission - and without "wasting" the space in the queue occupied by the packet.  This is a good reinforcement of the principle that tail-dropping is no longer the right choice.

Important property #3 is that the queue responds immediately when it's length decreases below the threshold, by stopping the marking/dropping process entirely.  This helps to maintain the high-frequency signal (of "too fast" versus "slow enough") to the TCPs.  The response to exceeding the threshold is slower, but only to the extent of avoiding over-reaction to a natural burst of traffic.

Now about the inverse sqrt: qualitatively, it is just a convenient method of gradually varying the mark/drop rate in terms of a time interval rather than a packet count interval.  The longer the queue remains overfull - controlled by the number of mark/drop events that have occurred - the higher the marking/dropping rate gets.  If the queue then empties (and stays empty-ish for a few RTTs), the rate is reset.  Meanwhile, if the queue regularly oscillates between "full" and "empty" states, the marking/dropping rate is remembered so that aggressive TCPs receive the correct magnitude signal.

Now as for the *quantitative* reason behind it, it is because as the interval between drops gets shorter, the number of increments per RTT goes up because there is one increment per drop.  If the interval is shortened linearly per increment, that means it will in practice shorten exponentially faster, such that the drop rate runs away faster than the TCP can reasonably be expected to respond.  This would result in excessive drop rates and oscillatory behaviour typical of an over-sensitive control system.  By basing the drop rate on the square root of the incremented variable, the runaway behaviour is curtailed since the drop interval now shortens linearly per RTT.

Maybe a diagram or animation would help to clarify that last paragraph.  I'm fairly sure I can draw one.

So overall we have an AQM that provides a low-latency signal of appropriate magnitude to TCPs when the link is genuinely congested, and gets out of the way when it isn't.  Combined with a fair queueing discipline (eg. SFQ or QFQ), I think this will turn out to be an excellent default setting for all sorts of equipment.  What would it take to get this into a DSL modem (at either end of the link)?

 - Jonathan Morton

Re: [Codel] [Bloat] The challenge

[Dave Taht]

On Wed, May 9, 2012 at 7:34 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
>
> On 9 May, 2012, at 4:04 am, Dave Taht wrote:
>

> So overall we have an AQM that provides a low-latency signal of appropriate magnitude to TCPs when the link is genuinely congested, and gets out of the way when it isn't.  Combined with a fair queueing discipline (eg. SFQ or QFQ), I think this will turn out to be an excellent default setting for all sorts of equipment.  What would it take to get this into a DSL modem (at either end of the link)?

David Woodhouse tossed a copy of the paper on his machine before
getting on a plane. He's been working on a ADSL device for quite some
time. I don't know if he and his box are together, or separated.

>
>  - Jonathan Morton
>



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

Re: [Codel] [Bloat] The challenge

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 1602 bytes --]

On Wed, 2012-05-09 at 19:37 -0700, Dave Taht wrote:
> On Wed, May 9, 2012 at 7:34 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
> >
> > On 9 May, 2012, at 4:04 am, Dave Taht wrote:
> >
> 
> > So overall we have an AQM that provides a low-latency signal of appropriate magnitude to TCPs when the link is genuinely congested, and gets out of the way when it isn't.  Combined with a fair queueing discipline (eg. SFQ or QFQ), I think this will turn out to be an excellent default setting for all sorts of equipment.  What would it take to get this into a DSL modem (at either end of the link)?
> 
> David Woodhouse tossed a copy of the paper on his machine before
> getting on a plane. He's been working on a ADSL device for quite some
> time. I don't know if he and his box are together, or separated.

Separated by about five thousand miles right now; I'll be home on
Monday; awake by Tuesday. OpenWrt supports a bunch of ADSL modems,
including the shiny dual-port Traverse Geos that I'm using at home. It's
running OpenWrt trunk, and it's simple enough for me to update the
kernel. The two lines are bonded using teql, but if you put your qdisc
on the teql0 interface I don't think that should make any difference at
all. I can give accounts on a machine at my end of the line, and you can
do what you like with it as long as you don't use too much *downstream*
bandwidth between the hours of 8am and 6pm (UK time) on weekdays.

As for the *ISP* end of the link, we could talk to the ISP about that.
Their LNS isn't running Linux though; it's their own code.

-- 
dwmw2

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 6171 bytes --]

Re: [Codel] [Bloat] The challenge

[Jonathan Morton]

On 10 May, 2012, at 9:35 am, David Woodhouse wrote:

>>> So overall we have an AQM that provides a low-latency signal of appropriate magnitude to TCPs when the link is genuinely congested, and gets out of the way when it isn't.  Combined with a fair queueing discipline (eg. SFQ or QFQ), I think this will turn out to be an excellent default setting for all sorts of equipment.  What would it take to get this into a DSL modem (at either end of the link)?
>> 
>> David Woodhouse tossed a copy of the paper on his machine before
>> getting on a plane. He's been working on a ADSL device for quite some
>> time. I don't know if he and his box are together, or separated.
> 
> Separated by about five thousand miles right now; I'll be home on
> Monday; awake by Tuesday. OpenWrt supports a bunch of ADSL modems,
> including the shiny dual-port Traverse Geos that I'm using at home. It's
> running OpenWrt trunk, and it's simple enough for me to update the
> kernel.

I want to buy a modem that I can put Linux on, and that supports ADSL2+ Annex M.  Where can I get one?

Vaguely acceptable solutions include CardBus or PCI/PCIe slot modems to put in a PowerBook or a PC.  I have enough spare hardware to make that work, and the lines here are underground and so not susceptible to lightning.

> As for the *ISP* end of the link, we could talk to the ISP about that.
> Their LNS isn't running Linux though; it's their own code.

I suppose the downstream side of ADSL is less vexing for now.  Eventually AQM and FQ needs to be in the DSLAM and cell tower by default, though.

I might have mentioned previously what my local cell tower is like - 30+ seconds (at 500Kbps) of tail-drop FIFO.  Yuck.

 - Jonathan Morton

Re: [Codel] [Bloat] The challenge

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 1421 bytes --]

On Thu, 2012-05-10 at 09:54 +0300, Jonathan Morton wrote:
> I want to buy a modem that I can put Linux on, and that supports ADSL2
> + Annex M.  Where can I get one?

http://wiki.openwrt.org/doc/hardware/soc/soc.lantiq
http://www.traverse.com.au/productview.php?product_id=117
http://www.traverse.com.au/productview.php?product_id=120

> Vaguely acceptable solutions include CardBus or PCI/PCIe slot modems
> to put in a PowerBook or a PC. 

http://www.traverse.com.au/productview.php?range_id=2

>  I have enough spare hardware to make that work, and the lines here
> are underground and so not susceptible to lightning.
> 
> > As for the *ISP* end of the link, we could talk to the ISP about
> that.
> > Their LNS isn't running Linux though; it's their own code.
> 
> I suppose the downstream side of ADSL is less vexing for now.
> Eventually AQM and FQ needs to be in the DSLAM and cell tower by
> default, though.

Probably not the DSLAM, but the ISP's LNS (where my PPP session is
terminated, over L2TP). That's what's currently limiting my downstream
traffic to 1Mb/s per line (or whatever my crappy pieces of wet string
are actually synced at today).

If we overload the DSLAM, it drops one ATM cell out of N, which nixes
fairly much *every* IP packet by dropping a tiny part of each one, and
results in almost zero throughput. We don't let the DSLAM do
buffering :)

-- 
dwmw2

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 6171 bytes --]

Re: [Codel] [Bloat] The challenge

[Justin McCann]

On Thu, May 10, 2012 at 3:02 AM, David Woodhouse <dwmw2@infradead.org> wrote:
>
> If we overload the DSLAM, it drops one ATM cell out of N, which nixes
> fairly much *every* IP packet by dropping a tiny part of each one, and
> results in almost zero throughput. We don't let the DSLAM do
> buffering :)

<tangent>
Can't they turn on Early Packet Discard or something similar? This
seems like a no-brainer.

Dynamics of TCP Traffic over ATM Networks (1994), Allyn Romanow , Sally Floyd
     http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.30.7647

"We investigate the performance of TCP connections over ATM networks
without ATM-level congestion control, and compare it to the
performance of TCP over packet-based networks. For simulations of
congested networks, the effective throughput of TCP over ATM can be
quite low when cells are dropped at the congested ATM switch. The low
throughput is due to wasted bandwidth as the congested link transmits
cells from `corrupted' packets, i.e., packets in which at least one
cell is dropped by the switch. We investigate two packet discard
strategies which alleviate the effects of fragmentation. Partial
Packet Discard, in which remaining cells are discarded after one cell
has been dropped from a packet, somewhat improves throughput. We
introduce Early Packet Discard, a strategy in which the switch drops
whole packets prior to buffer overflow. This mechanism prevents
fragmentation and restores throughput to maximal levels."

</tangent>

Re: [Codel] [Bloat] The challenge

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 470 bytes --]

On Thu, 2012-05-10 at 10:25 -0400, Justin McCann wrote:
> Can't they turn on Early Packet Discard or something similar? This
> seems like a no-brainer. 

This is British Telecom we're talking about.

No, we cannot expect them to do *anything* even half sane.

The ISP has clue, and the PPP session handled by BT's DSLAM is
terminated in an L2TP session on their LNS. For the foreseeable future,
that is where any queue management has to happen.

-- 
dwmw2

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 6171 bytes --]

Re: [Codel] [Bloat] The challenge

[Neil Davies]

Bit unfair to BT...

BT have architected their network so that cells loss doesn't happen. 

They pace the cell rate, on a per line basis,  in their BRASs so that 
individual DSLAM buffers are not overrun.

The PPP sessions are terminated in BT's BRAS - the ISPs LNS establishes
an L2TP tunnel to the BRAS. 

As for the sanity, we've measured other ISPs in the UK. They deliver consistently
more jitter and loss than BT's *wholesale* network.

Neil

On 10 May 2012, at 15:42, David Woodhouse wrote:

> On Thu, 2012-05-10 at 10:25 -0400, Justin McCann wrote:
>> Can't they turn on Early Packet Discard or something similar? This
>> seems like a no-brainer. 
> 
> This is British Telecom we're talking about.
> 
> No, we cannot expect them to do *anything* even half sane.
> 
> The ISP has clue, and the PPP session handled by BT's DSLAM is
> terminated in an L2TP session on their LNS. For the foreseeable future,
> that is where any queue management has to happen.
> 
> -- 
> dwmw2
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Codel] [Bloat] The challenge

[Jim Gettys]

Talk to Bob Brisco, who is on the bloat list.

Jim

On May 10, 2012, at 10:43 AM, David Woodhouse <dwmw2@infradead.org> wrote:

> On Thu, 2012-05-10 at 10:25 -0400, Justin McCann wrote:
>> Can't they turn on Early Packet Discard or something similar? This
>> seems like a no-brainer.
>
> This is British Telecom we're talking about.
>
> No, we cannot expect them to do *anything* even half sane.
>
> The ISP has clue, and the PPP session handled by BT's DSLAM is
> terminated in an L2TP session on their LNS. For the foreseeable future,
> that is where any queue management has to happen.
>
> --
> dwmw2
> _______________________________________________
> Codel mailing list
> Codel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/codel

Re: [Codel] [Bloat] The challenge

[Jonathan Morton]

On 10 May, 2012, at 10:02 am, David Woodhouse wrote:

> On Thu, 2012-05-10 at 09:54 +0300, Jonathan Morton wrote:
>> I want to buy a modem that I can put Linux on, and that supports ADSL2
>> + Annex M.  Where can I get one?
> 
> http://wiki.openwrt.org/doc/hardware/soc/soc.lantiq
> http://www.traverse.com.au/productview.php?product_id=117
> http://www.traverse.com.au/productview.php?product_id=120
> 
>> Vaguely acceptable solutions include CardBus or PCI/PCIe slot modems
>> to put in a PowerBook or a PC. 
> 
> http://www.traverse.com.au/productview.php?range_id=2

Interesting links, but just as frustrating as usual I'm afraid:

1) A long list of supported devices, without being able to filter for what's on the shelf in my local superhypermegaretailer, is not helpful.  I looked up the devices I already own, which are a reasonable selection from said shelf - no dice.  I broke open one (a Zyxel P660H-D1) that had a similar model number to a supported device - but it's totally different hardware, for which support is only "pending".

2) The Traverse links lead me to two types of devices - one of which is a router-type box which looks promising, but comes without documentation (see the next point below for why I want to read that *first*) and is not listed by *any* of the resellers linked to by the manufacturer - on any of the three continents listed.  So I can't buy it in any obvious way anyway.  (Yes, there is a chap in Belgium who says he will take requests.  I still want to read adequate documentation before imposing on him.)

3) The second type of device is a PCI card which appears to have the combination of an Ethernet NIC and a DSL modem *in series* on board.  This has no advantage, to my mind, over a standard consumer box such as I have several of already - the uplink bottleneck queue is still in a device that I can't control adequately, and which does not directly exert back-pressure on the Linux network stack.

So I repeat my question: where can I reliably go and buy a DSL modem that I can put Linux on?  It probably helps to mention that I'm in Finland, so a European (or at least British) reseller is preferable.


On the plus side, I did manage to get my PowerBook based router/firewall working again, so I've put my CPE back into bridge mode and suddenly the Internet works properly again.  Yay.  But I still have to manually configure HTB to match the link rate of the modem if I wan to control the queue.  That's the problem I really want to solve by putting Linux into the modem itself.

So I did some quick subjective tests using HTB/SFQ and HTB/CoDel - I'd like to try HTB/fq_codel too.  No packet classification going on here, traffic is traffic, let's make our modem devices as dumb as possible, but no dumber.  Bandwidth limits applied both upstream and down - I'm currently running on a simulated decade-old ADSL1 link at 1M/256K, but I can and have easily changed that to 128K/128K to simulate ISDN.

CoDel seems to cope quite well with everything I throw at it, keeping the link responsive.  However, so does SFQ - at least from a user perspective.  Looking at packet traces, it is immediately clear that SFQ still allows individual flows to run very unresponsive, with SACK flags maxed out for many seconds at a time.  CoDel achieves similar responsiveness (SSH remains essentially usable at 128K under multiple load, which is impressive) but does so by dropping several percent of all packets.

The fact that it is dropping packets rather than ECN-marking them suggests that I'm still missing something in my network setup, so I need to figure out what that is.  Even so, the flow recovers (by retransmission) within a dozen packets even over the real Internet, which is considerably faster than SFQ and gives the application smoother data delivery once the window size has grown beyond the buffer size.

With either CoDel or SFQ on the bottleneck queue, even 1M/256K feels fast, subjectively.  Objectively, I can still observe Steam taking 10x as long to download a game update than usual, or some YouTube videos no longer streaming in realtime (because they require 5Mbps for that), but those are expected.  Plain old Web browsing, e-mail, posting on forums (including reasonably sized photos) and software updating works smoothly and unobtrusively, even at a tenth of my usual bandwidth in either direction.

At ISDN speeds, a few problems start to appear with applications that have come to rely on always-on broadband characteristics.  Linux Mint has decided to download megabytes of software catalogue updates every hour by default, which consumes a large fraction of available bandwidth - I soon cut that back to every 12 hours.  No real problems emerged with SFQ or CoDel though.  Running with just HTB/FIFO, SSH was hilariously unusable as soon as any load was put on the link in either direction, but either option solved that almost entirely.

I really want to try combining CoDel and SFQ.

 - Jonathan Morton

Re: [Codel] [Bloat] The challenge

[Eric Dumazet]

On Mon, 2012-05-14 at 10:27 +0300, Jonathan Morton wrote:

> I really want to try combining CoDel and SFQ.

You could use fq_codel for that, available in net-next tree.

Re: [Codel] [Bloat] The challenge

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 1174 bytes --]

On Mon, 2012-05-14 at 10:27 +0300, Jonathan Morton wrote:
> 
> 2) The Traverse links lead me to two types of devices - one of which
> is a router-type box which looks promising, but comes without
> documentation 

linitx.com can get the Geos although it's not listed on their web site.
And the Solos PCI card *is* listed: http://linitx.com/product/12606

> 3) The second type of device is a PCI card which appears to have the
> combination of an Ethernet NIC and a DSL modem *in series* on board. 

You want the dual-port Solos, not the older Pulsar. It is an ATM device
(well, two ATM devices), properly supported by the upstream kernel via
drivers/atm/solos-pci.c

The Geos is just one (or two) of those on a Geode motherboard.

> So I repeat my question: where can I reliably go and buy a DSL modem
> that I can put Linux on?  It probably helps to mention that I'm in
> Finland, so a European (or at least British) reseller is preferable. 

If the Geos or Solos are too expensive, I'd be looking for other ADSL
devices supported by OpenWrt. Probably Lantiq-based. The Netgear
DGN3500B perhaps (see recent traffic on openwrt-devel).

-- 
dwmw2

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 6171 bytes --]

Re: [Codel] [Bloat] The challenge

[Jonathan Morton]

On 9 May, 2012, at 4:04 pm, Kevin Gross wrote:

> From the paper (figure 7), you can see that CoDel still leaves spikes of buffer occupancy when network conditions change. These will still be disruptive to real-time traffic. Many networks that need QoS now will still need QoS. Networks that do not have QoS will be much more usable with CoDel.

Combining AQM with FQ certainly seems like a good idea to me.  I haven't had a chance to try the implementation of fq_codel which already exists yet, but it's compiled and just waiting for me to get around to it.  If it works, then it should be an excellent default.

 - Jonathan Morton

Re: [Codel] [Bloat] The challenge

[Eric Dumazet]

On Wed, 2012-05-16 at 08:55 +0300, Jonathan Morton wrote:
> On 9 May, 2012, at 4:04 pm, Kevin Gross wrote:
> 
> > From the paper (figure 7), you can see that CoDel still leaves spikes of buffer occupancy when network conditions change. These will still be disruptive to real-time traffic. Many networks that need QoS now will still need QoS. Networks that do not have QoS will be much more usable with CoDel.
> 
> Combining AQM with FQ certainly seems like a good idea to me.  I haven't had a chance to try the implementation of fq_codel which already exists yet, but it's compiled and just waiting for me to get around to it.  If it works, then it should be an excellent default.


fq_codel has no priority concepts. It's only Fair. But what is
Fairness ?

In the presence of elephant flows, you get more packets drops from
elephant flows than from thin ones.

If the elephant flow is the 'high priority' traffic, dropping packets
from it is not what you wanted.

pfifo_fast is able to have strict priority. If you want Codel and strict
priority, you can use prio + codel  (or prio + fq_codel)

Re: [Codel] [Bloat] The challenge

[Dave Taht]

I

On Tue, May 15, 2012 at 10:55 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
>
> On 9 May, 2012, at 4:04 pm, Kevin Gross wrote:
>
>> From the paper (figure 7), you can see that CoDel still leaves spikes of buffer occupancy when network conditions change. These will still be disruptive to real-time traffic. Many networks that need QoS now will still need QoS. Networks that do not have QoS will be much more usable with CoDel.
>
> Combining AQM with FQ certainly seems like a good idea to me.  I haven't had a chance to try the implementation of fq_codel which already exists yet, but it's compiled and just waiting for me to get around to it.  If it works, then it should be an excellent default.

I would certainly like more folk to analyse fq_codel, particularly in
the context of TCP mice and what we call ANTs these days. It seems to
be really excellent, but I too haven't had much time to look at it,
yet. It certainly does an even better job than codel of allowing
sparse streams through in my own testing, and I run it by default on
my laptops, wireless gear, routers and desktop at present. Under heavy
load things like cerowrt and ssh and other stuff like mosh, remain
incredibly responsive.

With ponies that good, there has got to be some manure somewhere.

>  - Jonathan Morton
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

Re: [Codel] [Bloat] The challenge

[Eric Dumazet]

On Tue, 2012-05-15 at 23:09 -0700, Dave Taht wrote:

> I would certainly like more folk to analyse fq_codel, particularly in
> the context of TCP mice and what we call ANTs these days. It seems to
> be really excellent, but I too haven't had much time to look at it,
> yet. It certainly does an even better job than codel of allowing
> sparse streams through in my own testing, and I run it by default on
> my laptops, wireless gear, routers and desktop at present. Under heavy
> load things like cerowrt and ssh and other stuff like mosh, remain
> incredibly responsive.

fq_codel is like SFQ but has following new features :

1) More modern code :
  - Less hardcoded limits
    The only one remaining is the 65536 max flows limit, a linux qdisc
limit. (uses u32 splitted in 16:16 for class ids)
  - Easy to add new params in future (nested attributes)

2) head drops. Give to TCP chance to react faster to congestions.

3) Codel included (instead of pfifo)

4) New flows have priority against old ones : Their first quantum (or
first packet will be sent _before_ packets of old flows)

5) More efficient code (less cache misses in high end configurations)

But if traffic consist of transient flows (one packet per flow, think of
DDOS), your interactive traffic will suffer, there is no magic
inside ;) 

Definition of a new flow :

A) enqueue() time : Packet is classified and map to an empty flow X (a
flow that is not in a list (new or old)
   Flow added at tail of 'new_flows' list. Initialize flow X quantum to
1514 (or configured quantum for fq_codel)

B) dequeue() time. Packet dequeued. flow moved to tail of 'old flows'
list (if flow quantum exhausted)

C) if all flows present in 'old_flows' list are examined by dequeue(),
we notice empty flows and they are removed from list.

If a packet comes for flow X :
 - If flow X still in a old/new queue, we let X in the list, and only
add packet to the current list of packet for flow X.

[Codel] Exploring the potential of codel, fq_codel, and qfq

[dave taht]

On 05/15/2012 11:09 PM, Eric Dumazet wrote:
> On Wed, 2012-05-16 at 08:55 +0300, Jonathan Morton wrote:
>> On 9 May, 2012, at 4:04 pm, Kevin Gross wrote:
>>
>>>  From the paper (figure 7), you can see that CoDel still leaves spikes of buffer occupancy when network conditions change. These will still be disruptive to real-time traffic. Many networks that need QoS now will still need QoS. Networks that do not have QoS will be much more usable with CoDel.
>> Combining AQM with FQ certainly seems like a good idea to me.  I haven't had a chance to try the implementation of fq_codel which already exists yet, but it's compiled and just waiting for me to get around to it.  If it works, then it should be an excellent default.
>
> fq_codel has no priority concepts. It's only Fair. But what is
> Fairness ?
>
> In the presence of elephant flows, you get more packets drops from
> elephant flows than from thin ones.
The overall effects of fq_codel's implementation on top of codel's idea 
of good queue vs bad queue
has yet to be evaluated. I'd like to see people explore iw10 and TCP 
mice, for example on an apache benchmark emulating a typical web site, 
as well as typical accesses of a web site, as well as TCP_RR stats 
against saturating forward and reverse traffic, with and without ecn, at 
rates ranging from 1Mbit to 40GigE.

I'm hoping to get some of that said myself, but it's a lot to do, so I 
hope more people jump on it.

Then there's wireless.

All that said, I am loving what I see of fq_codel so far. I think in 
general fq is superior to anything a human fiddling with priorities can do.

Also I have hooked up qfq to codel in the debloat script, which might 
provide some useful comparisons to fq_codel, and also offers the 
possibility of using qfq to hash on device, and fq_codel to fq within 
flows to a given device, thus yielding 1/d behavior.

But I am throwing ENOTIME, ENOBRAINCELLS, and ENOENERGY errors a lot 
personally, right now. I'd planned to be on vacation last week and 
really need some time in a yurt far away from anything electronic.

I am looking forward very much towards published results, analysis, and 
benchmarks from everyone else. Certainly I intend to get to where I can 
do CDF plots of what I reference below over large
sample periods, etc, at some point... but for your low latency pleasure:

50 netperf -l 120 -t TCP_STREAM, then 1 netperf -l 30 -t TCP_RR, ecn on, 
100Mbit ethernet, 3 hops
1 machine has codel, the other does not (so many kernels, so little time)

I note that QFQ is WAY more cpu/memory overhead and I haven't captured 
or analysed the streams.

                                                  TCP_RR    PING

PFIFO_FAST on both, no load:      2440.42   .26 to .59
PFIFO_FAST to PFIFO_FAST:              8.13     121ms
CODEL to PFIFO_FAST:                    27.56     Initial impulse of 
117ms, settles down to about 25ms
FQ_CODEL to PFIFO_FAST:             272.76    1.2ms, no spike
QFQ+CODEL to PFIFO_FAST:        1002.39     .7ish, no spike
QFQ+FQ_CODEL to PFIFO_FAST  1020.62     .95ms no spike (we are down at 
statistical noise levels)

I haven't tried codel to codel or fq_codel to fq_codel much yet.

I note also that I get best results when also applying fq_codel as an 
ingress qdisc. With that
on it's hard to get any latencies in excess of 5ms in any direction. 
Without it, traffic in the incoming direction (via TCP_MAERTS on 
netperf) cracks 30ms worth of delay and 110 TCP_RR vs pfifo_fast.

   in debloat is script that does that using a three tier pfifo_fast 
superset. Using ifb is rocket science, as is tc, however. It would be 
very interesting to come up with a simpler way of using codel or 
fq_codel on ingress, perhaps with tbf.

I just checked in the latest revision of debloat and the simple_qos and ingres scripts to the deBloat repository
on github.


  >If the elephant flow is the 'high priority' traffic, dropping packets
>from it is not what you wanted.

>pfifo_fast is able to have strict priority.


Which can lead to starvation of all flows, in pfifo_fast's case. Two 
possible compromises are
using a 3 tier htb bucket to emulate pfifo_fasts behavior with the 
addition of codel for sane
packet drop, or to wedge some interesting behavior into the extra 15 
bits in the codel vars.


> If you want Codel and strict
> priority, you can use prio + codel  (or prio + fq_codel)

I'm not fond of strict prioritization.

I am unsure of how useful mq_prio is at this point.

>
>
> _______________________________________________
> Codel mailing list
> Codel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/codel

Re: [Codel] Exploring the potential of codel, fq_codel, and qfq

[Dave Taht]

After running those numbers I tried pure codel with ecn and with noecn
just to verify results, against the 50 streams.

of note: I was unable to duplicate the initial 120ms
spike I saw. Definately more tests and more rigorous testing is needed.

All tests were against v13 of the code.

codel ecn off, you get an initial spike of about 30ms, then it settles
down in this range.

64 bytes from 149.20.63.18: icmp_req=19 ttl=64 time=4.62 ms
64 bytes from 149.20.63.18: icmp_req=20 ttl=64 time=2.06 ms
64 bytes from 149.20.63.18: icmp_req=21 ttl=64 time=4.28 ms
64 bytes from 149.20.63.18: icmp_req=22 ttl=64 time=1.03 ms
64 bytes from 149.20.63.18: icmp_req=23 ttl=64 time=8.11 ms
64 bytes from 149.20.63.18: icmp_req=24 ttl=64 time=5.10 ms

TCP_RR is: 112.69

With ecn:

64 bytes from 149.20.63.18: icmp_req=46 ttl=64 time=10.6 ms
64 bytes from 149.20.63.18: icmp_req=47 ttl=64 time=5.66 ms
64 bytes from 149.20.63.18: icmp_req=48 ttl=64 time=11.8 ms
64 bytes from 149.20.63.18: icmp_req=49 ttl=64 time=3.68 ms
64 bytes from 149.20.63.18: icmp_req=50 ttl=64 time=10.2 ms
64 bytes from 149.20.63.18: icmp_req=51 ttl=64 time=12.8 ms
64 bytes from 149.20.63.18: icmp_req=52 ttl=64 time=2.62 ms
64 bytes from 149.20.63.18: icmp_req=53 ttl=64 time=7.86 ms

TCP_RR: 102

All of these sets of results need more rigor attached.




-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

Re: [Codel] [Bloat] Exploring the potential of codel, fq_codel, and qfq

[Eric Dumazet]

On Wed, 2012-05-16 at 00:20 -0700, Dave Taht wrote:
> After running those numbers I tried pure codel with ecn and with noecn
> just to verify results, against the 50 streams.
> 
> of note: I was unable to duplicate the initial 120ms
> spike I saw. Definately more tests and more rigorous testing is needed.
> 
> All tests were against v13 of the code.
> 
> codel ecn off, you get an initial spike of about 30ms, then it settles
> down in this range.

Spike is probably because of ARP resolution.

You should first populate ARP cache if you want to remoe ARP from the
equation (it adds another RTT for first ICMP packet)

Re: [Codel] [Bloat] Exploring the potential of codel, fq_codel, and qfq

[Eric Dumazet]

On Wed, 2012-05-16 at 00:20 -0700, Dave Taht wrote:

> With ecn:
> 
> 64 bytes from 149.20.63.18: icmp_req=46 ttl=64 time=10.6 ms
> 64 bytes from 149.20.63.18: icmp_req=47 ttl=64 time=5.66 ms
> 64 bytes from 149.20.63.18: icmp_req=48 ttl=64 time=11.8 ms
> 64 bytes from 149.20.63.18: icmp_req=49 ttl=64 time=3.68 ms
> 64 bytes from 149.20.63.18: icmp_req=50 ttl=64 time=10.2 ms
> 64 bytes from 149.20.63.18: icmp_req=51 ttl=64 time=12.8 ms
> 64 bytes from 149.20.63.18: icmp_req=52 ttl=64 time=2.62 ms
> 64 bytes from 149.20.63.18: icmp_req=53 ttl=64 time=7.86 ms
> 
> TCP_RR: 102
> 
> All of these sets of results need more rigor attached.

On TCP_RR pure workload, you have one packet in flight per flow.

ECN adds nothing in this case, only that no 'drops' occurs at all.

It might be good to change fq_codel to perform ECN mark only if flow
queue has more packets.

If not, plain drop.

Re: [Codel] [Bloat] The challenge

[Jonathan Morton]

I managed to run some quick tests this morning. Biggest factor was figuring out that ECN wasn't enabled on my Mac. This makes a significant difference to being able to observe it. 

So far all of the nice properties of SFQ seem to be intact (so at 128K I can still easily use SSH over a heavily loaded link), and furthermore I got to see ECN marking being performed and echoed on connections to real Internet servers. I think that led to zero packet loss but I need to inspect more before concluding that. 

So far, then, it seems to be working fine with no knobs set. That has to be a first. 

The key to knowledge is not to rely on others to teach you it. 

On 16 May 2012, at 09:31, Eric Dumazet <eric.dumazet@gmail.com> wrote:

> On Tue, 2012-05-15 at 23:09 -0700, Dave Taht wrote:
> 
>> I would certainly like more folk to analyse fq_codel, particularly in
>> the context of TCP mice and what we call ANTs these days. It seems to
>> be really excellent, but I too haven't had much time to look at it,
>> yet. It certainly does an even better job than codel of allowing
>> sparse streams through in my own testing, and I run it by default on
>> my laptops, wireless gear, routers and desktop at present. Under heavy
>> load things like cerowrt and ssh and other stuff like mosh, remain
>> incredibly responsive.
> 
> fq_codel is like SFQ but has following new features :
> 
> 1) More modern code :
>  - Less hardcoded limits
>    The only one remaining is the 65536 max flows limit, a linux qdisc
> limit. (uses u32 splitted in 16:16 for class ids)
>  - Easy to add new params in future (nested attributes)
> 
> 2) head drops. Give to TCP chance to react faster to congestions.
> 
> 3) Codel included (instead of pfifo)
> 
> 4) New flows have priority against old ones : Their first quantum (or
> first packet will be sent _before_ packets of old flows)
> 
> 5) More efficient code (less cache misses in high end configurations)
> 
> But if traffic consist of transient flows (one packet per flow, think of
> DDOS), your interactive traffic will suffer, there is no magic
> inside ;) 
> 
> Definition of a new flow :
> 
> A) enqueue() time : Packet is classified and map to an empty flow X (a
> flow that is not in a list (new or old)
>   Flow added at tail of 'new_flows' list. Initialize flow X quantum to
> 1514 (or configured quantum for fq_codel)
> 
> B) dequeue() time. Packet dequeued. flow moved to tail of 'old flows'
> list (if flow quantum exhausted)
> 
> C) if all flows present in 'old_flows' list are examined by dequeue(),
> we notice empty flows and they are removed from list.
> 
> If a packet comes for flow X :
> - If flow X still in a old/new queue, we let X in the list, and only
> add packet to the current list of packet for flow X.
> 
> 
> 
> 

Re: [Codel] [Bloat] Exploring the potential of codel, fq_codel, and qfq

[Eric Dumazet]

On Wed, 2012-05-16 at 09:47 +0200, Eric Dumazet wrote:

> On TCP_RR pure workload, you have one packet in flight per flow.
> 
> ECN adds nothing in this case, only that no 'drops' occurs at all.
> 
> It might be good to change fq_codel to perform ECN mark only if flow
> queue has more packets.
> 
> If not, plain drop.

And for codel, if all packets are ECN enabled, there is no 'drop' on a
pure TCP_RR workload, you endup codel downgraded to pfifo

An idea would be following :

If sojourn_time is above a new limit (say ecn_threshold), drop packet no
matter it is ECN capable or not.

This still gives incentive to ECN flows, but allow drops in case all
flows are ECN.

target = 5ms ;
ecn_threshold = target * 2;
interval = 100ms;

Re: [Codel] [Bloat] Exploring the potential of codel, fq_codel, and qfq

[Jonathan Morton]

With FQ, I don't see what that would buy you. 

The key to knowledge is not to rely on others to teach you it. 

On 16 May 2012, at 11:17, Eric Dumazet <eric.dumazet@gmail.com> wrote:

> On Wed, 2012-05-16 at 09:47 +0200, Eric Dumazet wrote:
> 
>> On TCP_RR pure workload, you have one packet in flight per flow.
>> 
>> ECN adds nothing in this case, only that no 'drops' occurs at all.
>> 
>> It might be good to change fq_codel to perform ECN mark only if flow
>> queue has more packets.
>> 
>> If not, plain drop.
> 
> And for codel, if all packets are ECN enabled, there is no 'drop' on a
> pure TCP_RR workload, you endup codel downgraded to pfifo
> 
> An idea would be following :
> 
> If sojourn_time is above a new limit (say ecn_threshold), drop packet no
> matter it is ECN capable or not.
> 
> This still gives incentive to ECN flows, but allow drops in case all
> flows are ECN.
> 
> target = 5ms ;
> ecn_threshold = target * 2;
> interval = 100ms;
> 
> 
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Codel] [Bloat] Exploring the potential of codel, fq_codel, and qfq

[Eric Dumazet]

On Wed, 2012-05-16 at 12:02 +0300, Jonathan Morton wrote:
> With FQ, I don't see what that would buy you. 

Sorry I dont understand your point.

Re: [Codel] [Bloat] Exploring the potential of codel, fq_codel, and qfq

[Jonathan Morton]

There are two goals here. One: provide real feedback to TCPs so that they know when the link is full and thus don't also fill up the buffer constantly. Two: prevent flows from unduly interfering with each other, so they don't have to fill the buffer just to be sure of good throughput. 

What you seem to be saying is that you have a queue full of unresponsive flows that aren't being dropped because they have ECN support and are being marked instead. With FQ, that doesn't matter because other flows can still get through with low latency, and in fq_codel they are treated separately for mark/drop decision purposes. And if the queue really does fill up physically, codel already drops packets at head regardless of ECN capability. 

The key to knowledge is not to rely on others to teach you it. 

On 16 May 2012, at 12:14, Eric Dumazet <eric.dumazet@gmail.com> wrote:

> On Wed, 2012-05-16 at 12:02 +0300, Jonathan Morton wrote:
>> With FQ, I don't see what that would buy you. 
> 
> Sorry I dont understand your point.
> 

Re: [Codel] [Bloat] Exploring the potential of codel, fq_codel, and qfq

[Eric Dumazet]

On Wed, 2012-05-16 at 12:31 +0300, Jonathan Morton wrote:
> There are two goals here. One: provide real feedback to TCPs so that
> they know when the link is full and thus don't also fill up the buffer
> constantly. Two: prevent flows from unduly interfering with each
> other, so they don't have to fill the buffer just to be sure of good
> throughput. 
> 
> What you seem to be saying is that you have a queue full of
> unresponsive flows that aren't being dropped because they have ECN
> support and are being marked instead. With FQ, that doesn't matter
> because other flows can still get through with low latency, and in
> fq_codel they are treated separately for mark/drop decision purposes.
> And if the queue really does fill up physically, codel already drops
> packets at head regardless of ECN capability. 

It matters because :

1) We setup high limit for codel so "head drop" at enqueue time is a
last resort thing.

2) If you have ECN blackhole _after_ you, flow can be non responsive not
because it doesnt want to, only because of the blackhole. Detecting a
too long delay in this flow queue and reverting to drops instead of
marks can help to recover from ECN blackhole.

Re: [Codel] [Bloat] Exploring the potential of codel, fq_codel, and qfq

[Jonathan Morton]

That's a pretty specific type of blackhole - one that can pass enough ECN information to permit negotiation of an ECN flow, but which then squashes Congestion Experienced codepoints. How common are those?

If fq_codel drops from the head of the longest flow when the queue is physically full, rather than the next one to be serviced, all should be well even in that case. 

The key to knowledge is not to rely on others to teach you it. 

On 16 May 2012, at 12:37, Eric Dumazet <eric.dumazet@gmail.com> wrote:

> On Wed, 2012-05-16 at 12:31 +0300, Jonathan Morton wrote:
>> There are two goals here. One: provide real feedback to TCPs so that
>> they know when the link is full and thus don't also fill up the buffer
>> constantly. Two: prevent flows from unduly interfering with each
>> other, so they don't have to fill the buffer just to be sure of good
>> throughput. 
>> 
>> What you seem to be saying is that you have a queue full of
>> unresponsive flows that aren't being dropped because they have ECN
>> support and are being marked instead. With FQ, that doesn't matter
>> because other flows can still get through with low latency, and in
>> fq_codel they are treated separately for mark/drop decision purposes.
>> And if the queue really does fill up physically, codel already drops
>> packets at head regardless of ECN capability. 
> 
> It matters because :
> 
> 1) We setup high limit for codel so "head drop" at enqueue time is a
> last resort thing.
> 
> 2) If you have ECN blackhole _after_ you, flow can be non responsive not
> because it doesnt want to, only because of the blackhole. Detecting a
> too long delay in this flow queue and reverting to drops instead of
> marks can help to recover from ECN blackhole.
> 
> 
> 

Re: [Codel] [Bloat] Exploring the potential of codel, fq_codel, and qfq

[Eric Dumazet]

On Wed, 2012-05-16 at 12:59 +0300, Jonathan Morton wrote:
> That's a pretty specific type of blackhole - one that can pass enough
> ECN information to permit negotiation of an ECN flow, but which then
> squashes Congestion Experienced codepoints. How common are those?
> 

Very common since TCP ECN negociation doesnt use ECN on IP header.

(SYN, SYNACK, and ACK messages dont use ECT)

So a network playing games with TOS field can be a ECN blackhole, still
allowing TCP to negociate ECN enabled sessions.

> If fq_codel drops from the head of the longest flow when the queue is
> physically full, rather than the next one to be serviced, all should
> be well even in that case. 

Not really. Unless you afford 3 second delay in queue.

Re: [Codel] [Bloat] Exploring the potential of codel, fq_codel, and qfq

[Eric Dumazet]

On Wed, 2012-05-16 at 10:17 +0200, Eric Dumazet wrote:
> On Wed, 2012-05-16 at 09:47 +0200, Eric Dumazet wrote:
> 
> > On TCP_RR pure workload, you have one packet in flight per flow.
> > 
> > ECN adds nothing in this case, only that no 'drops' occurs at all.
> > 
> > It might be good to change fq_codel to perform ECN mark only if flow
> > queue has more packets.
> > 
> > If not, plain drop.
> 
> And for codel, if all packets are ECN enabled, there is no 'drop' on a
> pure TCP_RR workload, you endup codel downgraded to pfifo

Actually there is a bug in fq_codel :

The backlog used in codel_should_drop() should not be the flow backlog,
but qdisc backlog, since its more in the codel spirit (allowing a packet
to pass the congestion point regardless of its sojourn_time if the
current queue size is very small)

I'll send a patch to fix this.

Re: [Codel] Exploring the potential of codel, fq_codel, and qfq

[Rick Jones]

On 05/16/2012 12:20 AM, Dave Taht wrote:
> After running those numbers I tried pure codel with ecn and with noecn
> just to verify results, against the 50 streams.
>
> of note: I was unable to duplicate the initial 120ms
> spike I saw. Definately more tests and more rigorous testing is needed.
>
> All tests were against v13 of the code.
>
> codel ecn off, you get an initial spike of about 30ms, then it settles
> down in this range.
>
> 64 bytes from 149.20.63.18: icmp_req=19 ttl=64 time=4.62 ms
> 64 bytes from 149.20.63.18: icmp_req=20 ttl=64 time=2.06 ms
> 64 bytes from 149.20.63.18: icmp_req=21 ttl=64 time=4.28 ms
> 64 bytes from 149.20.63.18: icmp_req=22 ttl=64 time=1.03 ms
> 64 bytes from 149.20.63.18: icmp_req=23 ttl=64 time=8.11 ms
> 64 bytes from 149.20.63.18: icmp_req=24 ttl=64 time=5.10 ms
>
> TCP_RR is: 112.69
>
> With ecn:
>
> 64 bytes from 149.20.63.18: icmp_req=46 ttl=64 time=10.6 ms
> 64 bytes from 149.20.63.18: icmp_req=47 ttl=64 time=5.66 ms
> 64 bytes from 149.20.63.18: icmp_req=48 ttl=64 time=11.8 ms
> 64 bytes from 149.20.63.18: icmp_req=49 ttl=64 time=3.68 ms
> 64 bytes from 149.20.63.18: icmp_req=50 ttl=64 time=10.2 ms
> 64 bytes from 149.20.63.18: icmp_req=51 ttl=64 time=12.8 ms
> 64 bytes from 149.20.63.18: icmp_req=52 ttl=64 time=2.62 ms
> 64 bytes from 149.20.63.18: icmp_req=53 ttl=64 time=7.86 ms
>
> TCP_RR: 102
>
> All of these sets of results need more rigor attached.

FWIW, netperf can report the number of retransmissions it saw on the 
data connection (when running on Linux).  That is enabled via the omni 
output selection and the local_transport_retrans and 
remote_transport_retrans output selectors.

rick jones

Re: [Codel] [Bloat] Exploring the potential of codel, fq_codel, and qfq

[Rick Jones]

On 05/16/2012 12:46 AM, Eric Dumazet wrote:
> On Wed, 2012-05-16 at 00:20 -0700, Dave Taht wrote:
>
>> With ecn:
>>
>> 64 bytes from 149.20.63.18: icmp_req=46 ttl=64 time=10.6 ms
>> 64 bytes from 149.20.63.18: icmp_req=47 ttl=64 time=5.66 ms
>> 64 bytes from 149.20.63.18: icmp_req=48 ttl=64 time=11.8 ms
>> 64 bytes from 149.20.63.18: icmp_req=49 ttl=64 time=3.68 ms
>> 64 bytes from 149.20.63.18: icmp_req=50 ttl=64 time=10.2 ms
>> 64 bytes from 149.20.63.18: icmp_req=51 ttl=64 time=12.8 ms
>> 64 bytes from 149.20.63.18: icmp_req=52 ttl=64 time=2.62 ms
>> 64 bytes from 149.20.63.18: icmp_req=53 ttl=64 time=7.86 ms
>>
>> TCP_RR: 102
>>
>> All of these sets of results need more rigor attached.
>
> On TCP_RR pure workload, you have one packet in flight per flow.
>
> ECN adds nothing in this case, only that no 'drops' occurs at all.
>
> It might be good to change fq_codel to perform ECN mark only if flow
> queue has more packets.
>
> If not, plain drop.

I like netperf as much as anyone :)  but keep in mind that the TCP_RR 
test has only one segment outstanding at a time only when the 
request/response size is < MSS and/or one has not enabled burst mode to 
have multiple transactions in flight at one time.

Are we really likely to see a situation where all the flows are one 
packet at a time?   If all the flows are either that way naturally, or 
have gotten there thanks to a one segment cwnd are we not already in a 
very pathological situation?

For the impossible to define "fair" question, is it fair to drop a 
flow's only packet if there are other, multiple-packet flows around?

rick jones

Re: [Codel] [Bloat] Exploring the potential of codel, fq_codel, and qfq

[Eric Dumazet]

On Wed, 2012-05-16 at 10:33 -0700, Rick Jones wrote:

> FWIW, netperf can report the number of retransmissions it saw on the 
> data connection (when running on Linux).  That is enabled via the omni 
> output selection and the local_transport_retrans and 
> remote_transport_retrans output selectors.

Yes, and unfortunately linux lacks SNMP counters for ECN frames...

Re: [Codel] [Bloat] Exploring the potential of codel, fq_codel, and qfq

[Eric Dumazet]

On Wed, 2012-05-16 at 10:40 -0700, Rick Jones wrote:

> 
> I like netperf as much as anyone :)  but keep in mind that the TCP_RR 
> test has only one segment outstanding at a time only when the 
> request/response size is < MSS and/or one has not enabled burst mode to 
> have multiple transactions in flight at one time.
> 
> Are we really likely to see a situation where all the flows are one 
> packet at a time?   If all the flows are either that way naturally, or 
> have gotten there thanks to a one segment cwnd are we not already in a 
> very pathological situation?
> 
> For the impossible to define "fair" question, is it fair to drop a 
> flow's only packet if there are other, multiple-packet flows around?

Please take a look at the patch I sent some hours ago.

In fact the real problem was that thin flows were bypassing codel algo
because we considered the flow backlog (one packet) and decided to _not_
drop packet regardless of its sojourn_time.

With the fix, there is no more problem : fq_codel downgrades to a codel
qdisc in case of a DDOS.

Re: [Codel] [Bloat] Linux-able modems

[Jonathan Morton]

On 14 May, 2012, at 4:55 pm, David Woodhouse wrote:

>> So I repeat my question: where can I reliably go and buy a DSL modem
>> that I can put Linux on?  It probably helps to mention that I'm in
>> Finland, so a European (or at least British) reseller is preferable. 
> 
> If the Geos or Solos are too expensive, I'd be looking for other ADSL
> devices supported by OpenWrt. Probably Lantiq-based. The Netgear
> DGN3500B perhaps (see recent traffic on openwrt-devel).

The 3700 is back-ordered by weeks, and the 3800 doesn't have a modem, and the 3500 isn't even in the local catalogues.  Netgear doesn't seem to be well regarded over here, even by the hacker types - and you *know* we have hacker types aplenty.

I double-checked all my hardware again, and while I can technically get OpenWRT on my D-Link modem, it can't actually drive the *modem* part because Broadcom is mean and stingy.

After biting the bullet and going through the local retailers' catalogues with a fine-toothed comb, I eventually found this - a fine specimen as far as hardware specs goes:

http://www.jimms.fi/tuote/WBMR-HP-G300H-EU

It happens to be Lantiq based, and so OpenWRT shows full support for it:

http://wiki.openwrt.org/toh/buffalo/wbmr-hp-g300h

Jimm's is transitioning to a purely mail order system and is based in a different city, but it turned out that my local megahypersuperretailer had just got a batch in stock, so I got one from there.

Well, this should be fun.

 - Jonathan Morton

Re: [Codel] [Bloat] Linux-able modems

[Dave Taht]

On Fri, May 18, 2012 at 1:56 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
>
> On 14 May, 2012, at 4:55 pm, David Woodhouse wrote:
>
>>> So I repeat my question: where can I reliably go and buy a DSL modem
>>> that I can put Linux on?  It probably helps to mention that I'm in
>>> Finland, so a European (or at least British) reseller is preferable.
>>
>> If the Geos or Solos are too expensive, I'd be looking for other ADSL
>> devices supported by OpenWrt. Probably Lantiq-based. The Netgear
>> DGN3500B perhaps (see recent traffic on openwrt-devel).
>
> The 3700 is back-ordered by weeks, and the 3800 doesn't have a modem, and the 3500 isn't even in the local catalogues.  Netgear doesn't seem to be well regarded over here, even by the hacker types - and you *know* we have hacker types aplenty.
>
> I double-checked all my hardware again, and while I can technically get OpenWRT on my D-Link modem, it can't actually drive the *modem* part because Broadcom is mean and stingy.
>
> After biting the bullet and going through the local retailers' catalogues with a fine-toothed comb, I eventually found this - a fine specimen as far as hardware specs goes:
>
> http://www.jimms.fi/tuote/WBMR-HP-G300H-EU
>
> It happens to be Lantiq based, and so OpenWRT shows full support for it:
>
> http://wiki.openwrt.org/toh/buffalo/wbmr-hp-g300h
>
> Jimm's is transitioning to a purely mail order system and is based in a different city, but it turned out that my local megahypersuperretailer had just got a batch in stock, so I got one from there.
>
> Well, this should be fun.

As noted elsewhere, fq_codel is now the default qos system in openwrt
as of build 31761

http://buildbot.openwrt.org:8010/one_line_per_build

37 arches, 150 platforms, no waiting.

There are still a few patches left to land, and it doesn't turn it on
on the wireless or ethernet by default, although it should definitely
do so, in the case of the latter, and well, it helps a bit on
wireless, too, although major surgery on the stack remains to get it
right.

And codel + QFQ remains interesting.

I look forward to hearing about your results.

>  - Jonathan Morton
>



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

Re: [Codel] [Bloat] Linux-able modems

[Dave Taht]

And felix tells me that the fq_codel fix for better backlog handling
is in openwrt 31813.

This should improve performance in the case of various forms of flooding,
and probably, bittorrent.

I've heard a rumor that codel (I don't know about fq_codel, don't have
a ns3 simulation
yet) is slightly more controlling of TCP reno, than TCP cubic. This is
not a disadvantage,
in my mind! as the only thing these days that uses a reno-like system is uTP.

Would love some data. I'd intended to setup bittorrent seeder of
things like the gutenberg
library to take a look at these characteristics, but I'll do that next
week, unless someone
beats me to it.

The openwrt buildbot system is running a little slow these days, while
I've donated
a machine to the effort, and am hoping to add a second, more would be
nice to have.

Me, I'm taking the first long weekend off I've had in a year, and
going to the Maker
Faire, in San Mateo. Should be some pretty nice toys to see, there.

But hopefully, builds should be available on Sunday for all platforms.
Still a few
patches left to come...

On Fri, May 18, 2012 at 2:36 PM, Dave Taht <dave.taht@gmail.com> wrote:
> On Fri, May 18, 2012 at 1:56 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
>>
>> On 14 May, 2012, at 4:55 pm, David Woodhouse wrote:
>>
>>>> So I repeat my question: where can I reliably go and buy a DSL modem
>>>> that I can put Linux on?  It probably helps to mention that I'm in
>>>> Finland, so a European (or at least British) reseller is preferable.
>>>
>>> If the Geos or Solos are too expensive, I'd be looking for other ADSL
>>> devices supported by OpenWrt. Probably Lantiq-based. The Netgear
>>> DGN3500B perhaps (see recent traffic on openwrt-devel).
>>
>> The 3700 is back-ordered by weeks, and the 3800 doesn't have a modem, and the 3500 isn't even in the local catalogues.  Netgear doesn't seem to be well regarded over here, even by the hacker types - and you *know* we have hacker types aplenty.
>>
>> I double-checked all my hardware again, and while I can technically get OpenWRT on my D-Link modem, it can't actually drive the *modem* part because Broadcom is mean and stingy.
>>
>> After biting the bullet and going through the local retailers' catalogues with a fine-toothed comb, I eventually found this - a fine specimen as far as hardware specs goes:
>>
>> http://www.jimms.fi/tuote/WBMR-HP-G300H-EU
>>
>> It happens to be Lantiq based, and so OpenWRT shows full support for it:
>>
>> http://wiki.openwrt.org/toh/buffalo/wbmr-hp-g300h
>>
>> Jimm's is transitioning to a purely mail order system and is based in a different city, but it turned out that my local megahypersuperretailer had just got a batch in stock, so I got one from there.
>>
>> Well, this should be fun.
>
> As noted elsewhere, fq_codel is now the default qos system in openwrt
> as of build 31761
>
> http://buildbot.openwrt.org:8010/one_line_per_build
>
> 37 arches, 150 platforms, no waiting.
>
> There are still a few patches left to land, and it doesn't turn it on
> on the wireless or ethernet by default, although it should definitely
> do so, in the case of the latter, and well, it helps a bit on
> wireless, too, although major surgery on the stack remains to get it
> right.
>
> And codel + QFQ remains interesting.
>
> I look forward to hearing about your results.
>
>>  - Jonathan Morton
>>
>
>
>
> --
> Dave Täht
> SKYPE: davetaht
> US Tel: 1-239-829-5608
> http://www.bufferbloat.net



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net