[Codel] New to codel-- how to run codel on a linux box?

[Codel] New to codel-- how to run codel on a linux box?

[Dong Mo]

[-- Attachment #1: Type: text/plain, Size: 644 bytes --]

Dear List,

I am new to codel and I would like to run some experiments with codel on a
linux box acting as a router. I see on the wiki that running codel on linux
requires BQL support from NIC driver. I would like to know is this an
essential requirement to run codel on linux box or it is optional?

I don't require line rate performance, something like 100Mbps throughput
will be enough for the experiment. However, It is also not clear to me that
how Codel is going to interact with some software traffic shaping say
bandwidth limiting?

Or more in general, what is the recommended way to run and test out codel
easily?


Thank you
-Mo Dong

[-- Attachment #2: Type: text/html, Size: 811 bytes --]

Re: [Codel] New to codel-- how to run codel on a linux box?

[Jonathan Morton]

On 23 Nov, 2013, at 7:04 am, Dong Mo wrote:

> I am new to codel and I would like to run some experiments with codel on a linux box acting as a router. I see on the wiki that running codel on linux requires BQL support from NIC driver. I would like to know is this an essential requirement to run codel on linux box or it is optional?

It's not a hard requirement - but large hardware queues (which BQL tries to solve) tend to interfere with line-rate operation.

> I don't require line rate performance, something like 100Mbps throughput will be enough for the experiment. However, It is also not clear to me that how Codel is going to interact with some software traffic shaping say bandwidth limiting?

tc qdisc add dev $IFACE root handle 1: htb default 1
tc class add dev $IFACE parent 1: classid 1:1 htb rate $RATE burst 15k
tc qdisc add dev $IFACE parent 1:1 handle 10: fq_codel

That works perfectly well if $RATE is less than line rate, even without BQL.

 - Jonathan Morton

Re: [Codel] New to codel-- how to run codel on a linux box?

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 1807 bytes --]

Why burst 15k? Quantum 1514 works for me.

While htb will give the best results it is also the most CPU intensive.

I also note that running it on ingress can be useful especially with ecn.

IF you have a simple network driver with no ring buffer... Or you have
working BQL... And  you want to run at line rate at 100 Mbit... You can
tell your Ethernet driver to switch to 100mbit via

ethtool -s your device advertise 0x008

See also caveats on tso etc:

http://www.bufferbloat.net/projects/codel/wiki/Best_Practices_for_Benchmarking_CoDel_and_FQ_CoDel
 On Nov 23, 2013 2:26 AM, "Jonathan Morton" <chromatix99@gmail.com> wrote:

>
> On 23 Nov, 2013, at 7:04 am, Dong Mo wrote:
>
> > I am new to codel and I would like to run some experiments with codel on
> a linux box acting as a router. I see on the wiki that running codel on
> linux requires BQL support from NIC driver. I would like to know is this an
> essential requirement to run codel on linux box or it is optional?
>
> It's not a hard requirement - but large hardware queues (which BQL tries
> to solve) tend to interfere with line-rate operation.
>
> > I don't require line rate performance, something like 100Mbps throughput
> will be enough for the experiment. However, It is also not clear to me that
> how Codel is going to interact with some software traffic shaping say
> bandwidth limiting?
>
> tc qdisc add dev $IFACE root handle 1: htb default 1
> tc class add dev $IFACE parent 1: classid 1:1 htb rate $RATE burst 15k
> tc qdisc add dev $IFACE parent 1:1 handle 10: fq_codel
>
> That works perfectly well if $RATE is less than line rate, even without
> BQL.
>
>  - Jonathan Morton
>
> _______________________________________________
> Codel mailing list
> Codel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/codel
>

[-- Attachment #2: Type: text/html, Size: 2472 bytes --]

Re: [Codel] New to codel-- how to run codel on a linux box?

[Dong Mo]

[-- Attachment #1: Type: text/plain, Size: 2230 bytes --]

Thanks for the reply and I think I have successfully set up codel.
Unfortunately I am also new to qdisc.

I am trying to compare fq_codel and sfq's performance. I was wondering how
to setup a software rate limiting using htb and at the same time have sfq
on this bottleneck link using qdisc.

Thank you so much!
-Mo


2013/11/23 Dave Taht <dave.taht@gmail.com>

> Why burst 15k? Quantum 1514 works for me.
>
> While htb will give the best results it is also the most CPU intensive.
>
> I also note that running it on ingress can be useful especially with ecn.
>
> IF you have a simple network driver with no ring buffer... Or you have
> working BQL... And  you want to run at line rate at 100 Mbit... You can
> tell your Ethernet driver to switch to 100mbit via
>
> ethtool -s your device advertise 0x008
>
> See also caveats on tso etc:
>
>
> http://www.bufferbloat.net/projects/codel/wiki/Best_Practices_for_Benchmarking_CoDel_and_FQ_CoDel
>  On Nov 23, 2013 2:26 AM, "Jonathan Morton" <chromatix99@gmail.com> wrote:
>
>>
>> On 23 Nov, 2013, at 7:04 am, Dong Mo wrote:
>>
>> > I am new to codel and I would like to run some experiments with codel
>> on a linux box acting as a router. I see on the wiki that running codel on
>> linux requires BQL support from NIC driver. I would like to know is this an
>> essential requirement to run codel on linux box or it is optional?
>>
>> It's not a hard requirement - but large hardware queues (which BQL tries
>> to solve) tend to interfere with line-rate operation.
>>
>> > I don't require line rate performance, something like 100Mbps
>> throughput will be enough for the experiment. However, It is also not clear
>> to me that how Codel is going to interact with some software traffic
>> shaping say bandwidth limiting?
>>
>> tc qdisc add dev $IFACE root handle 1: htb default 1
>> tc class add dev $IFACE parent 1: classid 1:1 htb rate $RATE burst 15k
>> tc qdisc add dev $IFACE parent 1:1 handle 10: fq_codel
>>
>> That works perfectly well if $RATE is less than line rate, even without
>> BQL.
>>
>>  - Jonathan Morton
>>
>> _______________________________________________
>> Codel mailing list
>> Codel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/codel
>>
>

[-- Attachment #2: Type: text/html, Size: 3319 bytes --]

Re: [Codel] New to codel-- how to run codel on a linux box?

[Dong Mo]

[-- Attachment #1: Type: text/plain, Size: 2996 bytes --]

The reason I am confused is that when I use this combination of commands:

tc qdisc add dev $IFACE root handle 1: htb default 1
tc class add dev $IFACE parent 1: classid 1:1 htb rate 40mbit
tc qdisc del dev eth4 parent 1:1 handle 10: sfq perturb 10

And sending packet from the sender, where qdisc is set, to the receiver,
the queuing delay is quite small and is approximately the same as when
fq_codel enabled.

However, if I am not using sfq, and send only one flow over this link, the
delay will explode again.

So is sfq here cut off the queue length somehow? How should I make a
bufferbloat fair queue on linux box using tc qdisc?

Thanks
-Mo


2013/12/5 Dong Mo <montedong@gmail.com>

> Thanks for the reply and I think I have successfully set up codel.
> Unfortunately I am also new to qdisc.
>
> I am trying to compare fq_codel and sfq's performance. I was wondering how
> to setup a software rate limiting using htb and at the same time have sfq
> on this bottleneck link using qdisc.
>
> Thank you so much!
> -Mo
>
>
> 2013/11/23 Dave Taht <dave.taht@gmail.com>
>
>> Why burst 15k? Quantum 1514 works for me.
>>
>> While htb will give the best results it is also the most CPU intensive.
>>
>> I also note that running it on ingress can be useful especially with ecn.
>>
>> IF you have a simple network driver with no ring buffer... Or you have
>> working BQL... And  you want to run at line rate at 100 Mbit... You can
>> tell your Ethernet driver to switch to 100mbit via
>>
>> ethtool -s your device advertise 0x008
>>
>> See also caveats on tso etc:
>>
>>
>> http://www.bufferbloat.net/projects/codel/wiki/Best_Practices_for_Benchmarking_CoDel_and_FQ_CoDel
>>  On Nov 23, 2013 2:26 AM, "Jonathan Morton" <chromatix99@gmail.com>
>> wrote:
>>
>>>
>>> On 23 Nov, 2013, at 7:04 am, Dong Mo wrote:
>>>
>>> > I am new to codel and I would like to run some experiments with codel
>>> on a linux box acting as a router. I see on the wiki that running codel on
>>> linux requires BQL support from NIC driver. I would like to know is this an
>>> essential requirement to run codel on linux box or it is optional?
>>>
>>> It's not a hard requirement - but large hardware queues (which BQL tries
>>> to solve) tend to interfere with line-rate operation.
>>>
>>> > I don't require line rate performance, something like 100Mbps
>>> throughput will be enough for the experiment. However, It is also not clear
>>> to me that how Codel is going to interact with some software traffic
>>> shaping say bandwidth limiting?
>>>
>>> tc qdisc add dev $IFACE root handle 1: htb default 1
>>> tc class add dev $IFACE parent 1: classid 1:1 htb rate $RATE burst 15k
>>> tc qdisc add dev $IFACE parent 1:1 handle 10: fq_codel
>>>
>>> That works perfectly well if $RATE is less than line rate, even without
>>> BQL.
>>>
>>>  - Jonathan Morton
>>>
>>> _______________________________________________
>>> Codel mailing list
>>> Codel@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/codel
>>>
>>
>

[-- Attachment #2: Type: text/html, Size: 4788 bytes --]

Re: [Codel] New to codel-- how to run codel on a linux box?

[Jonathan Morton]

On 5 Dec, 2013, at 9:49 pm, Dong Mo wrote:

> And sending packet from the sender, where qdisc is set, to the receiver, the queuing delay is quite small and is approximately the same as when fq_codel enabled.
> 
> However, if I am not using sfq, and send only one flow over this link, the delay will explode again.
> 
> So is sfq here cut off the queue length somehow? How should I make a bufferbloat fair queue on linux box using tc qdisc?

How are you measuring the delay?  Ping?

This is relevant because ICMP (ping) occupies a different "flow" than the TCP stream(s) you are loading the link with.  SFQ and fq_codel both service flows fairly, ensuring that each gets some traffic through continuously.  So your pings have low latency even if SFQ's queue is full.

The difference is that fq_codel keeps the queue of each flow short, while SFQ allows each queue to grow up to the limit.  The latter behaviour is adequate in some situations, but has some bad consequences such as slow recovery from packet loss on each flow.  And because SFQ can *only* keep the queue within limits by dropping packets, there *will* be packet loss on a regular basis unless the receive window of the TCP flow is exhausted first.  By contrast fq_codel is ECN-aware, so on ECN-enabled flows it can avoid dropping packets altogether while still keeping latency low enough for rapid recovery if packet loss occurs anyway.

It also often happens that packets dropped due to a full queue happen to be at the end of a TCP connection, for which detection and recovery of the problem is much slower (and usually on very human-visible timescales) than in the middle of a connection.  In this case SFQ is no help, because it also drops from teh tail of the queue, where the last packets of a connection arrive.  By contrast fq_codel drops from the head of the queue when required (ie. when ECN is not available), so unless two connections share a flow (rare but not impossible), the lost packet will always be from the middle of a connection and will be retransmitted quickly.

That is the sort of behaviour you should test for when comparing fq_codel and SFQ.  A simple ping test under load is satisfied by both qdiscs.

 - Jonathan Morton

Re: [Codel] New to codel-- how to run codel on a linux box?

[Dong Mo]

[-- Attachment #1: Type: text/plain, Size: 2806 bytes --]

That makes a lot of sense.

I missed the point the fq separate ICMP and tcp flows.

My current setting is

Ubuntu sender A with codel (htb and fq and fq_codel happens here)
---1G---freebsd ipfw machine inject link delay of 20ms------1G-----Ubuntu
receiver B machine.

What I am trying to do is measure the delay of packets of the same flow
experienced in A under different queuing policies (sfq or fq_codel).

Is there a simple way to do this?

Thanks
-Mo



2013/12/5 Jonathan Morton <chromatix99@gmail.com>

>
> On 5 Dec, 2013, at 9:49 pm, Dong Mo wrote:
>
> > And sending packet from the sender, where qdisc is set, to the receiver,
> the queuing delay is quite small and is approximately the same as when
> fq_codel enabled.
> >
> > However, if I am not using sfq, and send only one flow over this link,
> the delay will explode again.
> >
> > So is sfq here cut off the queue length somehow? How should I make a
> bufferbloat fair queue on linux box using tc qdisc?
>
> How are you measuring the delay?  Ping?
>
> This is relevant because ICMP (ping) occupies a different "flow" than the
> TCP stream(s) you are loading the link with.  SFQ and fq_codel both service
> flows fairly, ensuring that each gets some traffic through continuously.
>  So your pings have low latency even if SFQ's queue is full.
>
> The difference is that fq_codel keeps the queue of each flow short, while
> SFQ allows each queue to grow up to the limit.  The latter behaviour is
> adequate in some situations, but has some bad consequences such as slow
> recovery from packet loss on each flow.  And because SFQ can *only* keep
> the queue within limits by dropping packets, there *will* be packet loss on
> a regular basis unless the receive window of the TCP flow is exhausted
> first.  By contrast fq_codel is ECN-aware, so on ECN-enabled flows it can
> avoid dropping packets altogether while still keeping latency low enough
> for rapid recovery if packet loss occurs anyway.
>
> It also often happens that packets dropped due to a full queue happen to
> be at the end of a TCP connection, for which detection and recovery of the
> problem is much slower (and usually on very human-visible timescales) than
> in the middle of a connection.  In this case SFQ is no help, because it
> also drops from teh tail of the queue, where the last packets of a
> connection arrive.  By contrast fq_codel drops from the head of the queue
> when required (ie. when ECN is not available), so unless two connections
> share a flow (rare but not impossible), the lost packet will always be from
> the middle of a connection and will be retransmitted quickly.
>
> That is the sort of behaviour you should test for when comparing fq_codel
> and SFQ.  A simple ping test under load is satisfied by both qdiscs.
>
>  - Jonathan Morton
>
>

[-- Attachment #2: Type: text/html, Size: 3421 bytes --]

Re: [Codel] New to codel-- how to run codel on a linux box?

[Toke Høiland-Jørgensen]

[-- Attachment #1: Type: text/plain, Size: 811 bytes --]

Dong Mo <montedong@gmail.com> writes:

> What I am trying to do is measure the delay of packets of the same
> flow experienced in A under different queuing policies (sfq or
> fq_codel).
>
> Is there a simple way to do this?

I'm currently in the early planning stages for having something like
this incorporated into the netperf-wrapper [1] testing tool. In the
meantime, you could look at either thrulay [2] (old and apparently
abandoned, but may work in one of its variants) as a measurement tool,
or tcpprobe [3] to output statistics that can (probably) tell you.

-Toke

[1] https://github.com/tohojo/netperf-wrapper

[2] http://e2epi.internet2.edu/thrulay/
http://thrulay-hd.sourceforge.net/
http://thrulay-ng.sourceforge.net/

[3] http://www.linuxfoundation.org/collaborate/workgroups/networking/tcpprobe

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 489 bytes --]

Re: [Codel] New to codel-- how to run codel on a linux box?

[Dong Mo]

[-- Attachment #1: Type: text/plain, Size: 1199 bytes --]

I also found this to sample tcp latency performance:
https://code.google.com/p/paping/

<https://code.google.com/p/paping/>Does fq_codel's flow classification
supports external classifier filter? Say I only want to hash on dst port in
my case.

Thanks
Mo


2013/12/5 Toke Høiland-Jørgensen <toke@toke.dk>

> Dong Mo <montedong@gmail.com> writes:
>
> > What I am trying to do is measure the delay of packets of the same
> > flow experienced in A under different queuing policies (sfq or
> > fq_codel).
> >
> > Is there a simple way to do this?
>
> I'm currently in the early planning stages for having something like
> this incorporated into the netperf-wrapper [1] testing tool. In the
> meantime, you could look at either thrulay [2] (old and apparently
> abandoned, but may work in one of its variants) as a measurement tool,
> or tcpprobe [3] to output statistics that can (probably) tell you.
>
> -Toke
>
> [1] https://github.com/tohojo/netperf-wrapper
>
> [2] http://e2epi.internet2.edu/thrulay/
> http://thrulay-hd.sourceforge.net/
> http://thrulay-ng.sourceforge.net/
>
> [3]
> http://www.linuxfoundation.org/collaborate/workgroups/networking/tcpprobe
>

[-- Attachment #2: Type: text/html, Size: 2109 bytes --]

Re: [Codel] New to codel-- how to run codel on a linux box?

[Toke Høiland-Jørgensen]

[-- Attachment #1: Type: text/plain, Size: 673 bytes --]

Dong Mo <montedong@gmail.com> writes:

> I also found this to sample tcp latency performance:
> https://code.google.com/p/paping/

Right; note that that measures time to establish a new connection, not
the delay of packets within an already established connection. My guess
would be that each established connection would have a new source port,
thus hashing into a new bin.

> Does fq_codel's flow classification supports external classifier
> filter? Say I only want to hash on dst port in my case.

I'm pretty sure it supports the normal classifier (tc filter). That
overrides the built-in hashing completely, but IIRC you can do your own
hashing in tc filter :)

-Toke

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 489 bytes --]

Re: [Codel] New to codel-- how to run codel on a linux box?

[Dong Mo]

[-- Attachment #1: Type: text/plain, Size: 1058 bytes --]

Thanks. My problem is solved.
It seems to me paping can sample for the RTT over this particular port, not
the whole connection setup time. Maybe it only record for the sync time but
I am not fully sure. From the latency result, it is quite close to RTT of
the link.

-Mo


2013/12/5 Toke Høiland-Jørgensen <toke@toke.dk>

> Dong Mo <montedong@gmail.com> writes:
>
> > I also found this to sample tcp latency performance:
> > https://code.google.com/p/paping/
>
> Right; note that that measures time to establish a new connection, not
> the delay of packets within an already established connection. My guess
> would be that each established connection would have a new source port,
> thus hashing into a new bin.
>
> > Does fq_codel's flow classification supports external classifier
> > filter? Say I only want to hash on dst port in my case.
>
> I'm pretty sure it supports the normal classifier (tc filter). That
> overrides the built-in hashing completely, but IIRC you can do your own
> hashing in tc filter :)
>
> -Toke
>

[-- Attachment #2: Type: text/html, Size: 1645 bytes --]

Re: [Codel] New to codel-- how to run codel on a linux box?

[Toke Høiland-Jørgensen]

[-- Attachment #1: Type: text/plain, Size: 926 bytes --]

Dong Mo <montedong@gmail.com> writes:

> Thanks. My problem is solved.

Excellent. Do share your results :)

> It seems to me paping can sample for the RTT over this particular port, not the
> whole connection setup time. Maybe it only record for the sync time but I am not
> fully sure. From the latency result, it is quite close to RTT of the
> link.

Well, looking at the source code it seems to do a new connection each
time:
https://code.google.com/p/paping/source/browse/trunk/src/main.cpp#80

and

https://code.google.com/p/paping/source/browse/trunk/src/socket.cpp#106

It seems to call connect(), then measure the time until select() returns
with the socket ready to either read or write. Not sure when Linux
considers a TCP socket to be ready, but I guess it's probably after
receiving the SYN+ACK of connection setup, which would be one RTT after
calling connect(), and would fit with what you're seeing. :)

-Toke

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 489 bytes --]