[Bloat] Cake "out of tree" on Debian

[Bloat] Cake "out of tree" on Debian

[Kristian Kielhofner]

Hello All,

  Having enjoyed great success with codel in various places I'm
finally getting around to dipping my toes in the water with Cake.

  Router in this case is a Debian 8.7 machine running the provided
3.16.0-4-amd64 kernel. Following the instructions here:

https://www.bufferbloat.net/projects/codel/wiki/Cake/#installing-cake-out-of-tree-on-linux

  I wrote myself a quick and dirty script:

#!/bin/bash

WAN="eth2"
DOWN="117mbit"
UP="22mbit"

if [ -n "$1" ]; then
  echo "DOWN:"
  tc -s qdisc show dev "$WAN"
  echo
  echo "UP:"
  tc -s qdisc show dev ifb4"$WAN"
  exit
fi

tc qdisc del dev "$WAN" root
tc qdisc del dev ifb4"$WAN" root

modprobe sch_cake
modprobe act_mirred
modprobe ifb
tc qdisc add dev "$WAN" root cake bandwidth "$DOWN"

ip link add name ifb4"$WAN" type ifb
tc qdisc del dev "$WAN" ingress
tc qdisc add dev "$WAN" handle ffff: ingress
tc qdisc del dev ifb4"$WAN" root
tc qdisc add dev ifb4"$WAN" root cake bandwidth "$UP" besteffort
ifconfig ifb4"$WAN" up

tc filter add dev "$WAN" parent ffff: protocol all prio 10 u32 match
u32 0 0 flowid 1:1 action mirred egress redirect dev ifb4"$WAN"

status shows:

DOWN:
qdisc cake 8003: root refcnt 9 bandwidth 117Mbit (?diffserv?)
triple-isolate rtt 100.0ms noatm overhead 14
 Sent 6280 bytes 27 pkt (dropped 0, overlimits 4 requeues 0)
 backlog 0b 0p requeues 0
memory used: 4736b of 5850000b
capacity estimate: 117Mbit
             Tin 0       Tin 1       Tin 2
  thresh    7312Kbit     117Mbit   29250Kbit
  target       5.0ms       5.0ms       5.0ms
interval     100.0ms     100.0ms      10.0ms
Pk-delay         0us        86us         0us
Av-delay         0us         1us         0us
Sp-delay         0us         0us         0us
  pkts             0          27           0
  bytes            0        6280           0
way-inds           0           0           0
way-miss           0           5           0
way-cols           0           0           0
  drops            0           0           0
  marks            0           0           0
Sp-flows           0           1           0
Bk-flows           0           1           0
last-len           0           0           0
max-len            0        1412           0

qdisc ingress ffff: parent ffff:fff1 ----------------
 Sent 31023 bytes 32 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

UP:
qdisc cake 8004: root refcnt 2 bandwidth 22Mbit besteffort
triple-isolate rtt 100.0ms noatm overhead 14
 Sent 29861 bytes 30 pkt (dropped 1, overlimits 42 requeues 0)
 backlog 0b 0p requeues 0
memory used: 53888b of 4Mb
capacity estimate: 22Mbit
             Tin 0
  thresh      22Mbit
  target       5.0ms
interval     100.0ms
Pk-delay       6.9ms
Av-delay       349us
Sp-delay       198us
  pkts            31
  bytes        31273
way-inds           0
way-miss           4
way-cols           0
  drops            1
  marks            0
Sp-flows           1
Bk-flows           1
last-len           0
max-len         1412

  From what I've tested to far everything works exactly as expected
(upstream throughput, downstream throughput, etc) until the last line.
When the egress traffic is redirected through the IFB device test
downstream drops to the same rate specified by upstream.

  Hardware is Rangeley C2758, eth2 is an Intel I354 using Debian
provided igb. What am I missing here?

  Thanks!

-- 
Kristian Kielhofner

Re: [Bloat] Cake "out of tree" on Debian

[Y]

Hi,all.

Did you install cake qdisc to Debian 8.7 as usual?
I can not at follow debian. 

Linux debian 3.16.0-4-686-pae #1 SMP Debian 3.16.39-1 (2016-12-30) i686
GNU/Linux

Yuta.

2017-01-28 (土) の 21:48 -0500 に Kristian Kielhofner さんは書きました:
> Hello All,
> 
>   Having enjoyed great success with codel in various places I'm
> finally getting around to dipping my toes in the water with Cake.
> 
>   Router in this case is a Debian 8.7 machine running the provided
> 3.16.0-4-amd64 kernel. Following the instructions here:
> 
> https://www.bufferbloat.net/projects/codel/wiki/Cake/#installing-cake-out-of-tree-on-linux
> 
>   I wrote myself a quick and dirty script:
> 
> #!/bin/bash
> 
> WAN="eth2"
> DOWN="117mbit"
> UP="22mbit"
> 
> if [ -n "$1" ]; then
>   echo "DOWN:"
>   tc -s qdisc show dev "$WAN"
>   echo
>   echo "UP:"
>   tc -s qdisc show dev ifb4"$WAN"
>   exit
> fi
> 
> tc qdisc del dev "$WAN" root
> tc qdisc del dev ifb4"$WAN" root
> 
> modprobe sch_cake
> modprobe act_mirred
> modprobe ifb
> tc qdisc add dev "$WAN" root cake bandwidth "$DOWN"
> 
> ip link add name ifb4"$WAN" type ifb
> tc qdisc del dev "$WAN" ingress
> tc qdisc add dev "$WAN" handle ffff: ingress
> tc qdisc del dev ifb4"$WAN" root
> tc qdisc add dev ifb4"$WAN" root cake bandwidth "$UP" besteffort
> ifconfig ifb4"$WAN" up
> 
> tc filter add dev "$WAN" parent ffff: protocol all prio 10 u32 match
> u32 0 0 flowid 1:1 action mirred egress redirect dev ifb4"$WAN"
> 
> status shows:
> 
> DOWN:
> qdisc cake 8003: root refcnt 9 bandwidth 117Mbit (?diffserv?)
> triple-isolate rtt 100.0ms noatm overhead 14
>  Sent 6280 bytes 27 pkt (dropped 0, overlimits 4 requeues 0)
>  backlog 0b 0p requeues 0
> memory used: 4736b of 5850000b
> capacity estimate: 117Mbit
>              Tin 0       Tin 1       Tin 2
>   thresh    7312Kbit     117Mbit   29250Kbit
>   target       5.0ms       5.0ms       5.0ms
> interval     100.0ms     100.0ms      10.0ms
> Pk-delay         0us        86us         0us
> Av-delay         0us         1us         0us
> Sp-delay         0us         0us         0us
>   pkts             0          27           0
>   bytes            0        6280           0
> way-inds           0           0           0
> way-miss           0           5           0
> way-cols           0           0           0
>   drops            0           0           0
>   marks            0           0           0
> Sp-flows           0           1           0
> Bk-flows           0           1           0
> last-len           0           0           0
> max-len            0        1412           0
> 
> qdisc ingress ffff: parent ffff:fff1 ----------------
>  Sent 31023 bytes 32 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
> 
> UP:
> qdisc cake 8004: root refcnt 2 bandwidth 22Mbit besteffort
> triple-isolate rtt 100.0ms noatm overhead 14
>  Sent 29861 bytes 30 pkt (dropped 1, overlimits 42 requeues 0)
>  backlog 0b 0p requeues 0
> memory used: 53888b of 4Mb
> capacity estimate: 22Mbit
>              Tin 0
>   thresh      22Mbit
>   target       5.0ms
> interval     100.0ms
> Pk-delay       6.9ms
> Av-delay       349us
> Sp-delay       198us
>   pkts            31
>   bytes        31273
> way-inds           0
> way-miss           4
> way-cols           0
>   drops            1
>   marks            0
> Sp-flows           1
> Bk-flows           1
> last-len           0
> max-len         1412
> 
>   From what I've tested to far everything works exactly as expected
> (upstream throughput, downstream throughput, etc) until the last line.
> When the egress traffic is redirected through the IFB device test
> downstream drops to the same rate specified by upstream.
> 
>   Hardware is Rangeley C2758, eth2 is an Intel I354 using Debian
> provided igb. What am I missing here?
> 
>   Thanks!
> 

Re: [Bloat] Cake "out of tree" on Debian

[Y]

hi ,all
I found Debian 8.7 PAE kernel is having problem with installing cake.
Normal kernel is good with installing cake  . 

2017-01-28 (土) の 21:48 -0500 に Kristian Kielhofner さんは書きました:
> Hello All,
> 
>   Having enjoyed great success with codel in various places I'm
> finally getting around to dipping my toes in the water with Cake.
> 
>   Router in this case is a Debian 8.7 machine running the provided
> 3.16.0-4-amd64 kernel. Following the instructions here:
> 
> https://www.bufferbloat.net/projects/codel/wiki/Cake/#installing-cake-out-of-tree-on-linux
> 
>   I wrote myself a quick and dirty script:
> 
> #!/bin/bash
> 
> WAN="eth2"
> DOWN="117mbit"
> UP="22mbit"
> 
> if [ -n "$1" ]; then
>   echo "DOWN:"
>   tc -s qdisc show dev "$WAN"
>   echo
>   echo "UP:"
>   tc -s qdisc show dev ifb4"$WAN"
>   exit
> fi
> 
> tc qdisc del dev "$WAN" root
> tc qdisc del dev ifb4"$WAN" root
> 
> modprobe sch_cake
> modprobe act_mirred
> modprobe ifb
> tc qdisc add dev "$WAN" root cake bandwidth "$DOWN"
> 
> ip link add name ifb4"$WAN" type ifb
> tc qdisc del dev "$WAN" ingress
> tc qdisc add dev "$WAN" handle ffff: ingress
> tc qdisc del dev ifb4"$WAN" root
> tc qdisc add dev ifb4"$WAN" root cake bandwidth "$UP" besteffort
> ifconfig ifb4"$WAN" up
> 
> tc filter add dev "$WAN" parent ffff: protocol all prio 10 u32 match
> u32 0 0 flowid 1:1 action mirred egress redirect dev ifb4"$WAN"
> 
> status shows:
> 
> DOWN:
> qdisc cake 8003: root refcnt 9 bandwidth 117Mbit (?diffserv?)
> triple-isolate rtt 100.0ms noatm overhead 14
>  Sent 6280 bytes 27 pkt (dropped 0, overlimits 4 requeues 0)
>  backlog 0b 0p requeues 0
> memory used: 4736b of 5850000b
> capacity estimate: 117Mbit
>              Tin 0       Tin 1       Tin 2
>   thresh    7312Kbit     117Mbit   29250Kbit
>   target       5.0ms       5.0ms       5.0ms
> interval     100.0ms     100.0ms      10.0ms
> Pk-delay         0us        86us         0us
> Av-delay         0us         1us         0us
> Sp-delay         0us         0us         0us
>   pkts             0          27           0
>   bytes            0        6280           0
> way-inds           0           0           0
> way-miss           0           5           0
> way-cols           0           0           0
>   drops            0           0           0
>   marks            0           0           0
> Sp-flows           0           1           0
> Bk-flows           0           1           0
> last-len           0           0           0
> max-len            0        1412           0
> 
> qdisc ingress ffff: parent ffff:fff1 ----------------
>  Sent 31023 bytes 32 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
> 
> UP:
> qdisc cake 8004: root refcnt 2 bandwidth 22Mbit besteffort
> triple-isolate rtt 100.0ms noatm overhead 14
>  Sent 29861 bytes 30 pkt (dropped 1, overlimits 42 requeues 0)
>  backlog 0b 0p requeues 0
> memory used: 53888b of 4Mb
> capacity estimate: 22Mbit
>              Tin 0
>   thresh      22Mbit
>   target       5.0ms
> interval     100.0ms
> Pk-delay       6.9ms
> Av-delay       349us
> Sp-delay       198us
>   pkts            31
>   bytes        31273
> way-inds           0
> way-miss           4
> way-cols           0
>   drops            1
>   marks            0
> Sp-flows           1
> Bk-flows           1
> last-len           0
> max-len         1412
> 
>   From what I've tested to far everything works exactly as expected
> (upstream throughput, downstream throughput, etc) until the last line.
> When the egress traffic is redirected through the IFB device test
> downstream drops to the same rate specified by upstream.
> 
>   Hardware is Rangeley C2758, eth2 is an Intel I354 using Debian
> provided igb. What am I missing here?
> 
>   Thanks!
> 

Re: [Bloat] Cake "out of tree" on Debian

[Jonathan Morton]

> On 29 Jan, 2017, at 04:48, Kristian Kielhofner <kris@kriskinc.com> wrote:
> 
> When the egress traffic is redirected through the IFB device test
> downstream drops to the same rate specified by upstream.

Looks like your definitions are backwards.  “Down” is the *ingress* direction, which should be attached to the IFB device, while “Up” is the *egress* direction, which should be attached directly to the physical device.

 - Jonathan Morton

Re: [Bloat] Cake "out of tree" on Debian

[Kristian Kielhofner]

On Sun, Jan 29, 2017 at 5:50 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>
> Looks like your definitions are backwards.  “Down” is the *ingress* direction, which should be attached to the IFB device, while “Up” is the *egress* direction, which should be attached directly to the physical device.
>
>  - Jonathan Morton
>

Thanks for pointing that out! I blew away this entire configuration
(moved to Ubuntu Server 16.04) and started from scratch; everything is
fantastic so far.

Thanks again.

-- 
Kristian Kielhofner

Re: [Bloat] Cake "out of tree" on Debian

[Dave Taht]

tc should not be showing ?diffserv?

did you pull your tc from here? https://github.com/dtaht/tc-adv -
fixed this yesterday, or so I thought.

Also: I just refreshed iproute to head and rebased - (I needed working
ra expires support, and it's long past time we updated - regrettably
lede is on 4.4, can't change that, there). It's generally safe to be
running a later iproute2 than the kernel you have.

https://github.com/dtaht/iproute2-head-cake

On Sun, Jan 29, 2017 at 9:36 AM, Kristian Kielhofner <kris@kriskinc.com> wrote:
> On Sun, Jan 29, 2017 at 5:50 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>>
>> Looks like your definitions are backwards.  “Down” is the *ingress* direction, which should be attached to the IFB device, while “Up” is the *egress* direction, which should be attached directly to the physical device.
>>
>>  - Jonathan Morton
>>
>
> Thanks for pointing that out! I blew away this entire configuration
> (moved to Ubuntu Server 16.04) and started from scratch; everything is
> fantastic so far.
>
> Thanks again.
>
> --
> Kristian Kielhofner
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Bloat] Cake "out of tree" on Debian

[Kristian Kielhofner]

On Sun, Jan 29, 2017 at 12:42 PM, Dave Taht <dave.taht@gmail.com> wrote:
> tc should not be showing ?diffserv?
>
> did you pull your tc from here? https://github.com/dtaht/tc-adv -
> fixed this yesterday, or so I thought.
>
> Also: I just refreshed iproute to head and rebased - (I needed working
> ra expires support, and it's long past time we updated - regrettably
> lede is on 4.4, can't change that, there). It's generally safe to be
> running a later iproute2 than the kernel you have.
>
> https://github.com/dtaht/iproute2-head-cake
>

Now:

UP:
qdisc cake 8003: root refcnt 9 bandwidth 22Mbit diffserv3
triple-isolate rtt 100.0ms raw
 Sent 12868692 bytes 8521 pkt (dropped 0, overlimits 10941 requeues 0)
 backlog 7570b 5p requeues 0
 memory used: 44320b of 4Mb
 capacity estimate: 22Mbit
                 Bulk   Best Effort      Voice
  thresh      1375Kbit      22Mbit    5500Kbit
  target        13.2ms       5.0ms       5.0ms
  interval     108.2ms     100.0ms      10.0ms
  pk_delay         0us       7.0ms         0us
  av_delay         0us       2.7ms         0us
  sp_delay         0us       465us         0us
  pkts               0        8530           1
  bytes              0    12883790          42
  way_inds           0           0           0
  way_miss           0          10           1
  way_cols           0           0           0
  drops              0           0           0
  marks              0           0           0
  sp_flows           0           0           0
  bk_flows           0           1           0
  un_flows           0           0           0
  max_len            0       10598          42

qdisc ingress ffff: parent ffff:fff1 ----------------
 Sent 264453 bytes 4795 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

DOWN:
qdisc cake 8004: root refcnt 2 bandwidth 130Mbit besteffort
triple-isolate rtt 100.0ms raw
 Sent 331715 bytes 4797 pkt (dropped 0, overlimits 610 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 1792b of 6500000b
 capacity estimate: 130Mbit
                 Tin 0
  thresh       130Mbit
  target         5.0ms
  interval     100.0ms
  pk_delay        13us
  av_delay         1us
  sp_delay         0us
  pkts            4797
  bytes         331715
  way_inds           0
  way_miss           9
  way_cols           0
  drops              0
  marks              0
  sp_flows           0
  bk_flows           1
  un_flows           0
  max_len          198

-- 
Kristian Kielhofner

Re: [Bloat] Cake "out of tree" on Debian

[Dave Taht]

On Sun, Jan 29, 2017 at 9:48 AM, Kristian Kielhofner <kris@kriskinc.com> wrote:
> On Sun, Jan 29, 2017 at 12:42 PM, Dave Taht <dave.taht@gmail.com> wrote:
>> tc should not be showing ?diffserv?
>>
>> did you pull your tc from here? https://github.com/dtaht/tc-adv -
>> fixed this yesterday, or so I thought.
>>
>> Also: I just refreshed iproute to head and rebased - (I needed working
>> ra expires support, and it's long past time we updated - regrettably
>> lede is on 4.4, can't change that, there). It's generally safe to be
>> running a later iproute2 than the kernel you have.
>>
>> https://github.com/dtaht/iproute2-head-cake
>>
>
> Now:
>
> UP:
> qdisc cake 8003: root refcnt 9 bandwidth 22Mbit diffserv3
> triple-isolate rtt 100.0ms raw
>  Sent 12868692 bytes 8521 pkt (dropped 0, overlimits 10941 requeues 0)
>  backlog 7570b 5p requeues 0
>  memory used: 44320b of 4Mb
>  capacity estimate: 22Mbit
>                  Bulk   Best Effort      Voice
>   thresh      1375Kbit      22Mbit    5500Kbit
>   target        13.2ms       5.0ms       5.0ms
>   interval     108.2ms     100.0ms      10.0ms
>   pk_delay         0us       7.0ms         0us
>   av_delay         0us       2.7ms         0us
>   sp_delay         0us       465us         0us
>   pkts               0        8530           1
>   bytes              0    12883790          42
>   way_inds           0           0           0
>   way_miss           0          10           1
>   way_cols           0           0           0
>   drops              0           0           0
>   marks              0           0           0
>   sp_flows           0           0           0
>   bk_flows           0           1           0
>   un_flows           0           0           0
>   max_len            0       10598          42
>
> qdisc ingress ffff: parent ffff:fff1 ----------------
>  Sent 264453 bytes 4795 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0


Groovy. I thought you were "Mr. VOIP"?

While the default fq scheme works really well in cake, if you test
marking packets as voip (EF, CS4,CS6,CS7,VA), it will end up in the
diffserv3 voice queue.

asterisk used to have an encapsulating protocol called iax2, which
generated a single flow as backhaul - is that still deployed?

We have a tool in flent based on dit-itg to test this. It's a bit
painful to setup the first time. I've longed to have a full
asterisk/freeswitch/jitsy test setup to look harder at voip/video
characteristics.

There are also now several as yet underdocumented options in cake:

"nat" will look at flows before they are natted so as to isolate them better.
When combined with the dual-dsthost or dual-srchost option (depending
on traffic direction) that gives you per host fq, along with per-flow
fq.

(nat triple-isolate should also do this but we're still sorting out a
bug on that:
https://github.com/dtaht/sch_cake/issues/46
)

wash: washes out dscp markings. Helpful when your provider (:cough:
comcast) remarks nearly all traffic to CS1.

I just found that appear.in is using the new "goog" marker, which
marks all videoconferencing traffic as AF41, which is more or less
appropriately handled in the "diffserv4" model.

In general I have always had good results with the simplest
(besteffort or diffserv3) settings.

There's also new support for a docsis mode using the new "mpu" idea.

https://github.com/dtaht/sch_cake/pull/45

>
> DOWN:
> qdisc cake 8004: root refcnt 2 bandwidth 130Mbit besteffort
> triple-isolate rtt 100.0ms raw
>  Sent 331715 bytes 4797 pkt (dropped 0, overlimits 610 requeues 0)
>  backlog 0b 0p requeues 0
>  memory used: 1792b of 6500000b
>  capacity estimate: 130Mbit
>                  Tin 0
>   thresh       130Mbit
>   target         5.0ms
>   interval     100.0ms
>   pk_delay        13us
>   av_delay         1us
>   sp_delay         0us
>   pkts            4797
>   bytes         331715
>   way_inds           0
>   way_miss           9
>   way_cols           0
>   drops              0
>   marks              0
>   sp_flows           0
>   bk_flows           1
>   un_flows           0
>   max_len          198

Blow it up. Tell us what happens. :)

>
> --
> Kristian Kielhofner



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Bloat] Cake "out of tree" on Debian

[Kristian Kielhofner]

On Sun, Jan 29, 2017 at 1:06 PM, Dave Taht <dave.taht@gmail.com> wrote:
>
>
> Groovy. I thought you were "Mr. VOIP"?
>
> While the default fq scheme works really well in cake, if you test
> marking packets as voip (EF, CS4,CS6,CS7,VA), it will end up in the
> diffserv3 voice queue.
>
> asterisk used to have an encapsulating protocol called iax2, which
> generated a single flow as backhaul - is that still deployed?
>
> We have a tool in flent based on dit-itg to test this. It's a bit
> painful to setup the first time. I've longed to have a full
> asterisk/freeswitch/jitsy test setup to look harder at voip/video
> characteristics.
>
> There are also now several as yet underdocumented options in cake:
>
> "nat" will look at flows before they are natted so as to isolate them better.
> When combined with the dual-dsthost or dual-srchost option (depending
> on traffic direction) that gives you per host fq, along with per-flow
> fq.
>
> (nat triple-isolate should also do this but we're still sorting out a
> bug on that:
> https://github.com/dtaht/sch_cake/issues/46
> )
>
> wash: washes out dscp markings. Helpful when your provider (:cough:
> comcast) remarks nearly all traffic to CS1.
>
> I just found that appear.in is using the new "goog" marker, which
> marks all videoconferencing traffic as AF41, which is more or less
> appropriately handled in the "diffserv4" model.
>
> In general I have always had good results with the simplest
> (besteffort or diffserv3) settings.
>
> There's also new support for a docsis mode using the new "mpu" idea.
>
> https://github.com/dtaht/sch_cake/pull/45
>

Thank you but I'm not sure I was ever "Mr. VOIP"!

Regarding what I would broadly call "QoS" with other priorities and
projects my personal experience looks something like this:

WonderShaper ----------> CoDel/SQM in OpenWRT "just works" ---> "Cake
looks really cool" (rabbit hole...)

Asterisk still has iax2 although I haven't used it in at least 12
years. If I were to guess iax2 probably represents a tiny fraction of
what I'd call "realtime/VoIP traffic" in networks. However, it's
probably overly represented in smaller deployments, hobbyists, and
home users.

In 2017 I'd suggest looking closer at a FreeSWITCH test setup:

- WebRTC (as well as "traditional" SIP/RTP, of course)
- One of the most robust OPUS implementations I'm aware of; with
dynamic and adaptive FEC (forward error correction) and PLC (packet
loss concealment)
- Relatively rich (in the open source ecosystem) video support
(codecs, "MCU functionality", etc):

https://freeswitch.org/confluence/display/FREESWITCH/FreeSWITCH+1.6+Video

-- 
Kristian Kielhofner