[Codel] Floating an Idea. ip_fq_codel

[Codel] Floating an Idea. ip_fq_codel

[Richard Edmands]

[-- Attachment #1: Type: text/plain, Size: 946 bytes --]

In my environment we've got a fair chunk of torrent usage happening (+
gaming) and with fq_codel giving the advantage to whichever individual
could open up as many connections as possible the entire situation imploded
very quickly.
So to balance this out I used htb to implement the IP part of this
(actually not really, i made groups of ip's which belonged to individuals)
and stuck fq_codel on top of the divided setup.
With this system what now happens is each IP now gets equal utilization of
the link (actually, i'm a lazy hack. I only implemented the uplink section)
which prevents the advantage of opening up as many connections as possible.
Now when an individual decides to go nuts, they're limited to what is
available to them without harming everyone else, without compromising
maximum possible speed.

I have had this running in my environment for the past month and WOW.

See pastebin'd implementation.

http://pastebin.com/hXtzFL9f

[-- Attachment #2: Type: text/html, Size: 1141 bytes --]

Re: [Codel] Floating an Idea. ip_fq_codel

[Andrew McGregor]

[-- Attachment #1: Type: text/plain, Size: 1440 bytes --]

That's about what constitutes a flow.  fq_codel as implemented in linux
works per (source ip, dest ip, protocol, source port, dest port) 5-tuple.
 Linux should probably support multiple flow hashing algorithms in the
kernel.


On Tue, May 20, 2014 at 7:15 PM, Richard Edmands <thesirdmz@gmail.com>wrote:

> In my environment we've got a fair chunk of torrent usage happening (+
> gaming) and with fq_codel giving the advantage to whichever individual
> could open up as many connections as possible the entire situation imploded
> very quickly.
> So to balance this out I used htb to implement the IP part of this
> (actually not really, i made groups of ip's which belonged to individuals)
> and stuck fq_codel on top of the divided setup.
> With this system what now happens is each IP now gets equal utilization of
> the link (actually, i'm a lazy hack. I only implemented the uplink section)
> which prevents the advantage of opening up as many connections as possible.
> Now when an individual decides to go nuts, they're limited to what is
> available to them without harming everyone else, without compromising
> maximum possible speed.
>
> I have had this running in my environment for the past month and WOW.
>
> See pastebin'd implementation.
>
> http://pastebin.com/hXtzFL9f
>
> _______________________________________________
> Codel mailing list
> Codel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/codel
>
>

[-- Attachment #2: Type: text/html, Size: 2068 bytes --]

Re: [Codel] Floating an Idea. ip_fq_codel

[Dave Taht]

What I'd done for torrent was to rely on either a l7 classifier or the
user to mark packets as CS1, (background), and have a 3 level shaper
like yours that distinguished between diffserv classes.
Obviously you can't rely on users marking their traffic
appropriately... and yes, per dest fairness
is often a good idea. I've asked brahm to make CS1 the default rather
than optional.

Secondly, at higher rates, I generally found torrent and web traffic
co-existed pretty well, stuff in
slow start mode (web) blasting stuff in congestion avoidance out of
the way. However ledbat behaves
like reno with low delays on the link, and gets reprioritized to have
the same priority as other flows

This is universal - aqm (red) - or FQ (sfq) do this to torrent, also.

http://perso.telecom-paristech.fr/~drossi/paper/rossi13tma-b.pdf

(a mitigating fact is latencies stay low for everybody, even with 25
torrent flows going
full blast at 20mbit I hardly notice)

Some comments on your script.

0) Yes, getting the ppoe and atm framing calculation correct for htb
is a PITA. (see tons
of traffic on it on the cerowrt-devel list)

1) a problem in your script is that it only applies to ipv4 rather
than ipv6 traffic. You should have
at least a filter that sends that to a good bucket (htb default 13 is
sending ipv6 to the last bucket)

2) A feature of the native fq_codel hash is that it is unique per
instance, by applying a random
number to the derived hash. I'd like to see syntax added to tc get the
same random number
from custom hashes (and also hashing on mac address) .

eg, dst-mac,random

2a) An error, if you were using this filter, is the divisor has to be
equal to or less than the number of
flows defined in the qdisc.

#$tc filter add dev $ext prio 1 protocol ip parent 1: handle 100 flow
hash keys nfct-src,dst divisor 2048 baseclass 1:11

3) Running at slow rates is a PITA, as is framing compensation.

$tc class add dev $ext parent 1:1 classid 1:11 htb rate 300kbit ceil
$ext_up overhead 40 mtu 1492 mpu 53 linklayer atm


4) Running any qdisc at a very low rate is problematic. Here, I don't
think this is very correct.

$tc qdisc add dev $ext parent 1:11 handle 11: fq_codel noecn target
25ms interval 75ms quantum 512 flows 512 limit 300

limit 300 # Unless you are running with very low memory or at very low
rates it's best to leave this higher. You ARE running at a very low
rate.

quantum 512 # generally I've settled on quantum 300 as a good value
for low rates.

flows 512 # to maximize  aqm behavior, flows 16 works pretty good. to
minimize the birthday problem, 1024
works pretty good. And in either case it needs to match the custom
filter you are using's divisor.

target 25ms, interval 75ms. # The overall recomendation in the draft
is that target be set to 5-10% of interval, and that interval
correspond to the typical path length. If you are on dsl, that initial
path length
can be quite large....

In practice, with htb, the path length seems to get much larger at low
rates (htb & codel each buffers up a packet), so I THINK (but this
would need some testing) a more correct value for 300kbit interval is
150ms,
and target should account for a MTU's worth of packets, or 40ms or so....

BUT, as we are now combining multiple qdiscs in parallel in this
script, it's feasible your target is correct *compared to your overall
900kbit bandwidth*

we definately need to do more work at really low speeds.


On Tue, May 20, 2014 at 3:16 AM, Andrew McGregor <andrewmcgr@gmail.com> wrote:
> That's about what constitutes a flow.  fq_codel as implemented in linux
> works per (source ip, dest ip, protocol, source port, dest port) 5-tuple.
> Linux should probably support multiple flow hashing algorithms in the
> kernel.
>
>
> On Tue, May 20, 2014 at 7:15 PM, Richard Edmands <thesirdmz@gmail.com>
> wrote:
>>
>> In my environment we've got a fair chunk of torrent usage happening (+
>> gaming) and with fq_codel giving the advantage to whichever individual could
>> open up as many connections as possible the entire situation imploded very
>> quickly.
>> So to balance this out I used htb to implement the IP part of this
>> (actually not really, i made groups of ip's which belonged to individuals)
>> and stuck fq_codel on top of the divided setup.
>> With this system what now happens is each IP now gets equal utilization of
>> the link (actually, i'm a lazy hack. I only implemented the uplink section)
>> which prevents the advantage of opening up as many connections as possible.
>> Now when an individual decides to go nuts, they're limited to what is
>> available to them without harming everyone else, without compromising
>> maximum possible speed.
>>
>> I have had this running in my environment for the past month and WOW.
>>
>> See pastebin'd implementation.
>>
>> http://pastebin.com/hXtzFL9f
>>
>> _______________________________________________
>> Codel mailing list
>> Codel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/codel
>>
>
>
> _______________________________________________
> Codel mailing list
> Codel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/codel
>



-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article

Re: [Codel] Floating an Idea. ip_fq_codel

[Eric Dumazet]

On Tue, 2014-05-20 at 20:16 +1000, Andrew McGregor wrote:
> That's about what constitutes a flow.  fq_codel as implemented in
> linux works per (source ip, dest ip, protocol, source port, dest port)
> 5-tuple.  Linux should probably support multiple flow hashing
> algorithms in the kernel.
> 

Right. fq_codel uses the same trick than other qdisc, like SFQ

By _default_ it uses a 5-tuple hash. Thats convenient but not a
requirement.

You can classify packets using a filter and for example hash only on the
part you need. For example destination IP, or source IP, depending if
you use fq_codel on egress or ingress sides.

tc qdisc add dev eth0 root handle 1: fq_codel flows 8192
tc filter add dev eth0 protocol ip parent 1: handle 1 \
	flow hash keys dst divisor 8192

Simple as that.

> On Tue, May 20, 2014 at 7:15 PM, Richard Edmands <thesirdmz@gmail.com>
> wrote:
>         In my environment we've got a fair chunk of torrent usage
>         happening (+ gaming) and with fq_codel giving the advantage to
>         whichever individual could open up as many connections as
>         possible the entire situation imploded very quickly.
>         
>         So to balance this out I used htb to implement the IP part of
>         this (actually not really, i made groups of ip's which
>         belonged to individuals) and stuck fq_codel on top of the
>         divided setup.
>         
>         With this system what now happens is each IP now gets equal
>         utilization of the link (actually, i'm a lazy hack. I only
>         implemented the uplink section) which prevents the advantage
>         of opening up as many connections as possible. 
>         Now when an individual decides to go nuts, they're limited to
>         what is available to them without harming everyone else,
>         without compromising maximum possible speed.
>         
>         
>         I have had this running in my environment for the past month
>         and WOW.
>         
>         
>         See pastebin'd implementation.
>         
>         
>         http://pastebin.com/hXtzFL9f
>         
>         
>         _______________________________________________
>         Codel mailing list
>         Codel@lists.bufferbloat.net
>         https://lists.bufferbloat.net/listinfo/codel
>         
> 
> 
> _______________________________________________
> Codel mailing list
> Codel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/codel

Re: [Codel] Floating an Idea. ip_fq_codel

[Eric Dumazet]

On Tue, 2014-05-20 at 08:31 -0700, Eric Dumazet wrote:
> On Tue, 2014-05-20 at 20:16 +1000, Andrew McGregor wrote:
> > That's about what constitutes a flow.  fq_codel as implemented in
> > linux works per (source ip, dest ip, protocol, source port, dest port)
> > 5-tuple.  Linux should probably support multiple flow hashing
> > algorithms in the kernel.
> > 
> 
> Right. fq_codel uses the same trick than other qdisc, like SFQ
> 
> By _default_ it uses a 5-tuple hash. Thats convenient but not a
> requirement.
> 
> You can classify packets using a filter and for example hash only on the
> part you need. For example destination IP, or source IP, depending if
> you use fq_codel on egress or ingress sides.
> 
> tc qdisc add dev eth0 root handle 1: fq_codel flows 8192
> tc filter add dev eth0 protocol ip parent 1: handle 1 \
> 	flow hash keys dst divisor 8192

Btw, while fq_codel can have 'flows xxx'  with any value for xxx,
the filter part requires 'divisor power_of_two', because it uses a AND
operation, not a MODULO.

So I doubt the filter at line 63 of http://pastebin.com/hXtzFL9f 
has any chance working. It probably is not installed.

Re: [Codel] Floating an Idea. ip_fq_codel

[Andrew McGregor]

[-- Attachment #1: Type: text/plain, Size: 2649 bytes --]

Ah, so it DOES support multiple flow hashes... I didn't know that.  Nice.


On Wed, May 21, 2014 at 1:31 AM, Eric Dumazet <eric.dumazet@gmail.com>wrote:

>
> On Tue, 2014-05-20 at 20:16 +1000, Andrew McGregor wrote:
> > That's about what constitutes a flow.  fq_codel as implemented in
> > linux works per (source ip, dest ip, protocol, source port, dest port)
> > 5-tuple.  Linux should probably support multiple flow hashing
> > algorithms in the kernel.
> >
>
> Right. fq_codel uses the same trick than other qdisc, like SFQ
>
> By _default_ it uses a 5-tuple hash. Thats convenient but not a
> requirement.
>
> You can classify packets using a filter and for example hash only on the
> part you need. For example destination IP, or source IP, depending if
> you use fq_codel on egress or ingress sides.
>
> tc qdisc add dev eth0 root handle 1: fq_codel flows 8192
> tc filter add dev eth0 protocol ip parent 1: handle 1 \
>         flow hash keys dst divisor 8192
>
> Simple as that.
>
> > On Tue, May 20, 2014 at 7:15 PM, Richard Edmands <thesirdmz@gmail.com>
> > wrote:
> >         In my environment we've got a fair chunk of torrent usage
> >         happening (+ gaming) and with fq_codel giving the advantage to
> >         whichever individual could open up as many connections as
> >         possible the entire situation imploded very quickly.
> >
> >         So to balance this out I used htb to implement the IP part of
> >         this (actually not really, i made groups of ip's which
> >         belonged to individuals) and stuck fq_codel on top of the
> >         divided setup.
> >
> >         With this system what now happens is each IP now gets equal
> >         utilization of the link (actually, i'm a lazy hack. I only
> >         implemented the uplink section) which prevents the advantage
> >         of opening up as many connections as possible.
> >         Now when an individual decides to go nuts, they're limited to
> >         what is available to them without harming everyone else,
> >         without compromising maximum possible speed.
> >
> >
> >         I have had this running in my environment for the past month
> >         and WOW.
> >
> >
> >         See pastebin'd implementation.
> >
> >
> >         http://pastebin.com/hXtzFL9f
> >
> >
> >         _______________________________________________
> >         Codel mailing list
> >         Codel@lists.bufferbloat.net
> >         https://lists.bufferbloat.net/listinfo/codel
> >
> >
> >
> > _______________________________________________
> > Codel mailing list
> > Codel@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/codel
>
>
>

[-- Attachment #2: Type: text/html, Size: 3850 bytes --]

Re: [Codel] Floating an Idea. ip_fq_codel

[Dan Siemon]

[-- Attachment #1: Type: text/plain, Size: 1290 bytes --]

On Tue, 2014-05-20 at 19:15 +1000, Richard Edmands wrote:
> In my environment we've got a fair chunk of torrent usage happening (+
> gaming) and with fq_codel giving the advantage to whichever individual
> could open up as many connections as possible the entire situation
> imploded very quickly.
> 
> So to balance this out I used htb to implement the IP part of this
> (actually not really, i made groups of ip's which belonged to
> individuals) and stuck fq_codel on top of the divided setup.
> 
> With this system what now happens is each IP now gets equal
> utilization of the link (actually, i'm a lazy hack. I only implemented
> the uplink section) which prevents the advantage of opening up as many
> connections as possible. 
> Now when an individual decides to go nuts, they're limited to what is
> available to them without harming everyone else, without compromising
> maximum possible speed.
> 
> 
> I have had this running in my environment for the past month and WOW.

I did something similar as well. You can find the script and results
below. Let me know if you find any bugs.

http://git.coverfire.com/?p=linux-qos-scripts.git;a=blob;f=src-3tos.sh;hb=HEAD

http://www.coverfire.com/archives/2013/01/01/improving-my-home-internet-performance/


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [Codel] Floating an Idea. ip_fq_codel

[Richard Edmands]

[-- Attachment #1: Type: text/plain, Size: 1750 bytes --]

With the tc filter flow hash keys i've been having problems verifying if
they've been installed. I've searched the man pages and haven't found the
solution. It "should" be tc filter show dev *insert dev*
but that doesn't list them :(
any ideas what i've missed.



tc filter add dev eth0 protocol ip parent 1: handle 1 \
        flow hash keys dst divisor 8192


On Sat, Jun 7, 2014 at 11:56 AM, Dan Siemon <dan@coverfire.com> wrote:

> On Tue, 2014-05-20 at 19:15 +1000, Richard Edmands wrote:
> > In my environment we've got a fair chunk of torrent usage happening (+
> > gaming) and with fq_codel giving the advantage to whichever individual
> > could open up as many connections as possible the entire situation
> > imploded very quickly.
> >
> > So to balance this out I used htb to implement the IP part of this
> > (actually not really, i made groups of ip's which belonged to
> > individuals) and stuck fq_codel on top of the divided setup.
> >
> > With this system what now happens is each IP now gets equal
> > utilization of the link (actually, i'm a lazy hack. I only implemented
> > the uplink section) which prevents the advantage of opening up as many
> > connections as possible.
> > Now when an individual decides to go nuts, they're limited to what is
> > available to them without harming everyone else, without compromising
> > maximum possible speed.
> >
> >
> > I have had this running in my environment for the past month and WOW.
>
> I did something similar as well. You can find the script and results
> below. Let me know if you find any bugs.
>
>
> http://git.coverfire.com/?p=linux-qos-scripts.git;a=blob;f=src-3tos.sh;hb=HEAD
>
>
> http://www.coverfire.com/archives/2013/01/01/improving-my-home-internet-performance/
>
>

[-- Attachment #2: Type: text/html, Size: 3496 bytes --]

Re: [Codel] Floating an Idea. ip_fq_codel

[Dave Taht]

On Fri, Jun 20, 2014 at 2:35 AM, Richard Edmands <thesirdmz@gmail.com> wrote:
> With the tc filter flow hash keys i've been having problems verifying if
> they've been installed. I've searched the man pages and haven't found the
> solution. It "should" be tc filter show dev *insert dev*
> but that doesn't list them :(
> any ideas what i've missed.
>
>
>
> tc filter add dev eth0 protocol ip parent 1: handle 1 \
>         flow hash keys dst divisor 8192

modprobe cls_flow # or, insmod
tc qdisc add dev eth0 root handle 1 fq_codel flows 8182
tc filter add dev eth0 protocol ip parent 1:0 handle 1 prio 1 flow
hash keys dst divisor 8192

If you don't do this, you will get back errors from the kernel. tc is
quite unforgiving. the flows
parameter and the divisor need to be a power of two and match the
fq_codel flows parameter.

And you probably need to match more than just "ip" filters for a working config.

netperf -H 172.21.0.1 -l 60 & # run a couple times to fill up the queue -

root@nuc-client:~# tc -s class show dev eth0
class fq_codel 1:1962 parent 1:
 (dropped 0, overlimits 0 requeues 0)
 backlog 6056b 4p requeues 0
  deficit 0 count 1 lastcount 1 ldelay 3.5ms

tc -s filter show doesn't anything useful, but does show these filters
installed.

filter parent 1: protocol ip pref 1 flow
filter parent 1: protocol ip pref 1 flow handle 0x1 hash keys dst
divisor 8192 baseclass 1:1

Running netperf to 2 different destinations, I get, 2 queues in use.

root@nuc-client:~# tc -s class show dev eth0
class fq_codel 1:b7c parent 1:
 (dropped 0, overlimits 0 requeues 0)
 backlog 7570b 5p requeues 0
  deficit 1514 count 2 lastcount 1 ldelay 5.2ms
class fq_codel 1:1962 parent 1:
 (dropped 2, overlimits 0 requeues 0)
 backlog 6056b 4p requeues 0
  deficit 0 count 1 lastcount 1 ldelay 7.2ms


fiddling with wifi is harder as you have 4 hw queues. This appears to
be working correctly, but
I can seem to print any of the filter rules. (?) Something I had been
experimenting with as
a ¨good¨ value to improve aggregation behavior on an wifi access point was:

cat debloat_wifi_ap.sh

#!/bin/sh
IFACE=wlan0
QDISC=fq_codel
FLOWS=1024
TARGET="target 20ms interval 300ms" # wifi buffering and scheduling is a PITA

wifi() {
        tc qdisc add dev $IFACE handle 1 root mq
        tc qdisc add dev $IFACE parent 1:1 handle 10 $QDISC flows
$FLOWS noecn quantum 1514
        tc qdisc add dev $IFACE parent 1:2 handle 20 $QDISC flows
$FLOWS quantum 4542
        tc qdisc add dev $IFACE parent 1:3 handle 30 $QDISC flows
$FLOWS $TARGET quantum 4542
        tc qdisc add dev $IFACE parent 1:4 handle 40 $QDISC flows
$FLOWS $TARGET noecn quantum 4542

        tc filter add dev wlan0 protocol ip parent 10:0 handle 1 prio
1 flow hash keys dst divisor $FLOWS
        tc filter add dev wlan0 protocol ip parent 20:0 handle 1 prio
1 flow hash keys dst divisor $FLOWS
        tc filter add dev wlan0 protocol ip parent 30:0 handle 1 prio
1 flow hash keys dst divisor $FLOWS
        tc filter add dev wlan0 protocol ip parent 40:0 handle 1 prio
1 flow hash keys dst divisor $FLOWS

# pretty sure you need to do icmp, ipv6, arp, and other protocols like ESP.
}

wifi

# I need to get around to benchmarking this sort of stuff again. Or
someone does.

>
>
> On Sat, Jun 7, 2014 at 11:56 AM, Dan Siemon <dan@coverfire.com> wrote:
>>
>> On Tue, 2014-05-20 at 19:15 +1000, Richard Edmands wrote:
>> > In my environment we've got a fair chunk of torrent usage happening (+
>> > gaming) and with fq_codel giving the advantage to whichever individual
>> > could open up as many connections as possible the entire situation
>> > imploded very quickly.
>> >
>> > So to balance this out I used htb to implement the IP part of this
>> > (actually not really, i made groups of ip's which belonged to
>> > individuals) and stuck fq_codel on top of the divided setup.
>> >
>> > With this system what now happens is each IP now gets equal
>> > utilization of the link (actually, i'm a lazy hack. I only implemented
>> > the uplink section) which prevents the advantage of opening up as many
>> > connections as possible.
>> > Now when an individual decides to go nuts, they're limited to what is
>> > available to them without harming everyone else, without compromising
>> > maximum possible speed.
>> >
>> >
>> > I have had this running in my environment for the past month and WOW.
>>
>> I did something similar as well. You can find the script and results
>> below. Let me know if you find any bugs.
>>
>>
>> http://git.coverfire.com/?p=linux-qos-scripts.git;a=blob;f=src-3tos.sh;hb=HEAD
>>
>>
>> http://www.coverfire.com/archives/2013/01/01/improving-my-home-internet-performance/
>>
>
>
> _______________________________________________
> Codel mailing list
> Codel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/codel
>



-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article