[Cake] Cake on elements of a bridge

[Cake] Cake on elements of a bridge

[Georgios Amanakis]

[-- Attachment #1: Type: text/plain, Size: 429 bytes --]

Dear All,

I was giving a transparent firewall a try, and wondered whether cake can be
applied on the interfaces of a bridge. I want to put an extra router
in-line between clients and the ISP-modem-router. It will have two
interfaces (eth0 facing wan, eth1 facing lan), bridged together as br0.

Can I fearlessly apply cake on eth0 and eth1? Would this be compatible with
features like ingress, ack-filter or even nat?

Georgios

[-- Attachment #2: Type: text/html, Size: 592 bytes --]

Re: [Cake] Cake on elements of a bridge

[Toke Høiland-Jørgensen]

Georgios Amanakis <gamanakis@gmail.com> writes:

> Dear All,
>
> I was giving a transparent firewall a try, and wondered whether cake
> can be applied on the interfaces of a bridge. I want to put an extra
> router in-line between clients and the ISP-modem-router. It will have
> two interfaces (eth0 facing wan, eth1 facing lan), bridged together as
> br0.
>
> Can I fearlessly apply cake on eth0 and eth1? Would this be compatible
> with features like ingress, ack-filter or even nat?

Well, you wouldn't get much benefit from the nat feature, as the machine
running CAKE would not be the one doing the nat'ing. But other than
that, it should work fine :)

-Toke

Re: [Cake] Cake on elements of a bridge

[Georgios Amanakis]

[-- Attachment #1: Type: text/plain, Size: 1050 bytes --]

Awesome, thanks to both of you!
I am aware of the uselessness of nat (in terms of cake) in this setup. It's
good to know what Sebastian pointed out. I ran it for a couple of hours and
it seems to be working fine. I am going to finalize the setup and will get
back to you.

Georgios

On 4 Sep 2018 1:31 pm, "Toke Høiland-Jørgensen" <toke@toke.dk> wrote:

Georgios Amanakis <gamanakis@gmail.com> writes:

> Dear All,
>
> I was giving a transparent firewall a try, and wondered whether cake
> can be applied on the interfaces of a bridge. I want to put an extra
> router in-line between clients and the ISP-modem-router. It will have
> two interfaces (eth0 facing wan, eth1 facing lan), bridged together as
> br0.
>
> Can I fearlessly apply cake on eth0 and eth1? Would this be compatible
> with features like ingress, ack-filter or even nat?

Well, you wouldn't get much benefit from the nat feature, as the machine
running CAKE would not be the one doing the nat'ing. But other than
that, it should work fine :)


-Toke

[-- Attachment #2: Type: text/html, Size: 1830 bytes --]

Re: [Cake] Cake on elements of a bridge

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 2937 bytes --]

I happen to also be working on a bridge setup, but it’s different. For one, I used fq_codel on a transparent bridge for a couple years in production and it worked well, so I trust it also would for cake.

But now, my neighbor will access the Internet through my CPE device, but they must have a separate IP obtained through DHCP (i.e. a separate MAC address as well), and I want to use cake to manage the queue for both of us. I could do this with two routers and a transparent bridge, but I want to see if I can make it work with as few devices as possible, preferably just one EdgeRouter-X. I had two failures thus far:

Fail #1: Do routing for the neighbors on their NS5AC Loco, and use the ER-X’s internal switch to bridge the neighbor’s and my WAN interfaces to the CPE. Doing cake on switch0 results in my WAN traffic going through the qdisc, but unsurprisingly, the neighbor’s traffic passes through the switch without going through the qdisc layer.

Fail #2: Use the ER-X’s pseudo-ethernet functionality to add a second virtual Ethernet interface to the ER-X’s WAN interface. I could use IFB if I got two WAN interfaces working on the same box. This looks promising and I can pick up two DHCP addresses on one physical interface, but the ER-X doesn’t handle the routing situation where two interfaces have the same default router IP. (Using policy-based routing, what does it do when next-hop is the same for two different LAN subnets?)

There will be a solution here, I just haven’t found it yet. I’m now thinking of a setup with a smart switch / VLANs and a transparent bridge through two physical interfaces of the ER-X (which only has 5 ports total), but I’ll figure it out… :)

> On Sep 4, 2018, at 2:01 PM, Georgios Amanakis <gamanakis@gmail.com> wrote:
> 
> Awesome, thanks to both of you! 
> I am aware of the uselessness of nat (in terms of cake) in this setup. It's good to know what Sebastian pointed out. I ran it for a couple of hours and it seems to be working fine. I am going to finalize the setup and will get back to you.
> 
> Georgios
> 
> On 4 Sep 2018 1:31 pm, "Toke Høiland-Jørgensen" <toke@toke.dk <mailto:toke@toke.dk>> wrote:
> Georgios Amanakis <gamanakis@gmail.com <mailto:gamanakis@gmail.com>> writes:
> 
> > Dear All,
> >
> > I was giving a transparent firewall a try, and wondered whether cake
> > can be applied on the interfaces of a bridge. I want to put an extra
> > router in-line between clients and the ISP-modem-router. It will have
> > two interfaces (eth0 facing wan, eth1 facing lan), bridged together as
> > br0.
> >
> > Can I fearlessly apply cake on eth0 and eth1? Would this be compatible
> > with features like ingress, ack-filter or even nat?
> 
> Well, you wouldn't get much benefit from the nat feature, as the machine
> running CAKE would not be the one doing the nat'ing. But other than
> that, it should work fine :)


[-- Attachment #2: Type: text/html, Size: 4372 bytes --]

Re: [Cake] Cake on elements of a bridge

[Toke Høiland-Jørgensen]

Pete Heist <pete@heistp.net> writes:

> I happen to also be working on a bridge setup, but it’s different. For
> one, I used fq_codel on a transparent bridge for a couple years in
> production and it worked well, so I trust it also would for cake.
>
> But now, my neighbor will access the Internet through my CPE device,
> but they must have a separate IP obtained through DHCP (i.e. a
> separate MAC address as well), and I want to use cake to manage the
> queue for both of us. I could do this with two routers and a
> transparent bridge, but I want to see if I can make it work with as
> few devices as possible, preferably just one EdgeRouter-X. I had two
> failures thus far:
>
> Fail #1: Do routing for the neighbors on their NS5AC Loco, and use the
> ER-X’s internal switch to bridge the neighbor’s and my WAN interfaces
> to the CPE. Doing cake on switch0 results in my WAN traffic going
> through the qdisc, but unsurprisingly, the neighbor’s traffic passes
> through the switch without going through the qdisc layer.
>
> Fail #2: Use the ER-X’s pseudo-ethernet functionality to add a second
> virtual Ethernet interface to the ER-X’s WAN interface. I could use
> IFB if I got two WAN interfaces working on the same box. This looks
> promising and I can pick up two DHCP addresses on one physical
> interface, but the ER-X doesn’t handle the routing situation where two
> interfaces have the same default router IP. (Using policy-based
> routing, what does it do when next-hop is the same for two different
> LAN subnets?)
>
> There will be a solution here, I just haven’t found it yet. I’m now
> thinking of a setup with a smart switch / VLANs and a transparent
> bridge through two physical interfaces of the ER-X (which only has 5
> ports total), but I’ll figure it out… :)

DHCP relay and normal routing? Or bridging with a kernel software bridge
rather than the hardware switch?

-Toke

Re: [Cake] Cake on elements of a bridge

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 1022 bytes --]


> On Sep 6, 2018, at 8:04 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Pete Heist <pete@heistp.net <mailto:pete@heistp.net>> writes:
> 
>> But now, my neighbor will access the Internet through my CPE device,
>> but they must have a separate IP obtained through DHCP (i.e. a
>> separate MAC address as well), and I want to use cake to manage the
>> queue for both of us. I could do this with two routers and a
>> transparent bridge, but I want to see if I can make it work with as
>> few devices as possible, preferably just one EdgeRouter-X. I had two
>> failures thus far:
> 
> DHCP relay and normal routing? Or bridging with a kernel software bridge
> rather than the hardware switch?

I bet a regular software bridge would work. I’ll try it.

It looks like I’ll also need to do stateful firewalling for the neighbors. I was able to get my transparent bridge to do this with net.bridge.bridge-nf-call-iptables=1, I believe, so this should also theoretically work fine, somehow… :)


[-- Attachment #2: Type: text/html, Size: 5534 bytes --]

Re: [Cake] Cake on elements of a bridge

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 2327 bytes --]


> On Sep 6, 2018, at 8:51 PM, Pete Heist <pete@heistp.net> wrote:
> 
>> On Sep 6, 2018, at 8:04 PM, Toke Høiland-Jørgensen <toke@toke.dk <mailto:toke@toke.dk>> wrote:
>> 
>> Pete Heist <pete@heistp.net <mailto:pete@heistp.net>> writes:
>> 
>>> But now, my neighbor will access the Internet through my CPE device,
>>> but they must have a separate IP obtained through DHCP (i.e. a
>>> separate MAC address as well), and I want to use cake to manage the
>>> queue for both of us. I could do this with two routers and a
>>> transparent bridge, but I want to see if I can make it work with as
>>> few devices as possible, preferably just one EdgeRouter-X. I had two
>>> failures thus far:
>> 
>> DHCP relay and normal routing? Or bridging with a kernel software bridge
>> rather than the hardware switch?
> 
> I bet a regular software bridge would work. I’ll try it.
> 
> It looks like I’ll also need to do stateful firewalling for the neighbors. I was able to get my transparent bridge to do this with net.bridge.bridge-nf-call-iptables=1, I believe, so this should also theoretically work fine, somehow… :)

For anyone who followed this, yes, the regular soft bridge (i.e. set interfaces bridge br0) works fine on the ER-X, as I suspect it would on most any Linux. A few notes about it:

- Your qdisc must be added to the physical interface (e.g. eth4), not the bridge interface
- Unlike the hardware bridge which has its own MAC, the soft bridge seems to take the MAC of the lowest (or first listed?) interface port
- On ER-X, bridge-nf-call-iptables=1 is the default so nothing needs to be changed there for firewalling
- When firewalling the bridged WAN interface, ‘in’ corresponds to bridged traffic and ‘local’ to routed traffic, which is different from the semantics for ordinary routed traffic
- I can do stateful firewalling for bridged hosts with “accept established and related”, but have to explicitly allow DHCP (UDP source/dest port 67-68) in the WAN interface’s ‘in’ rules for DHCP traffic to pass through the bridge

Performance:

Using Cake with this setup, the fun ends at around 110 Mbit with ksoftirqd thrashing. Unsurprisingly, there’s probably some overhead here with the soft bridge. For my purposes though (50 Mbit), it’s enough, barely…


[-- Attachment #2: Type: text/html, Size: 7314 bytes --]

Re: [Cake] Cake on elements of a bridge

[Dave Taht]

On Mon, Sep 10, 2018 at 12:29 PM Pete Heist <pete@heistp.net> wrote:
>
>
> On Sep 6, 2018, at 8:51 PM, Pete Heist <pete@heistp.net> wrote:
>
> On Sep 6, 2018, at 8:04 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> Pete Heist <pete@heistp.net> writes:
>
> But now, my neighbor will access the Internet through my CPE device,
> but they must have a separate IP obtained through DHCP (i.e. a
> separate MAC address as well), and I want to use cake to manage the
> queue for both of us. I could do this with two routers and a
> transparent bridge, but I want to see if I can make it work with as
> few devices as possible, preferably just one EdgeRouter-X. I had two
> failures thus far:
>
>
> DHCP relay and normal routing? Or bridging with a kernel software bridge
> rather than the hardware switch?
>
>
> I bet a regular software bridge would work. I’ll try it.
>
> It looks like I’ll also need to do stateful firewalling for the neighbors. I was able to get my transparent bridge to do this with net.bridge.bridge-nf-call-iptables=1, I believe, so this should also theoretically work fine, somehow… :)
>
>
> For anyone who followed this, yes, the regular soft bridge (i.e. set interfaces bridge br0) works fine on the ER-X, as I suspect it would on most any Linux. A few notes about it:
>
> - Your qdisc must be added to the physical interface (e.g. eth4), not the bridge interface
> - Unlike the hardware bridge which has its own MAC, the soft bridge seems to take the MAC of the lowest (or first listed?) interface port
> - On ER-X, bridge-nf-call-iptables=1 is the default so nothing needs to be changed there for firewalling
> - When firewalling the bridged WAN interface, ‘in’ corresponds to bridged traffic and ‘local’ to routed traffic, which is different from the semantics for ordinary routed traffic
> - I can do stateful firewalling for bridged hosts with “accept established and related”, but have to explicitly allow DHCP (UDP source/dest port 67-68) in the WAN interface’s ‘in’ rules for DHCP traffic to pass through the bridge
>
> Performance:
>
> Using Cake with this setup, the fun ends at around 110 Mbit with ksoftirqd thrashing. Unsurprisingly, there’s probably some overhead here with the soft bridge. For my purposes though (50 Mbit), it’s enough, barely…

Can I encourage you to give regular ole htb+fq_codel sqm a shot with a
bigger burst and cburst size for htb? Fiddling with the htb quantum
isn't helping much,
but try this, from: https://github.com/tohojo/sqm-scripts/issues/71

(I am thinking burst and cburst should be about 1.1ms of buffering in size)

root@apu2:/home/d/git/sqm-scripts/src# git diff .
diff --git a/src/functions.sh b/src/functions.sh
index 226a6c5..8ad4f38 100644
--- a/src/functions.sh
+++ b/src/functions.sh
@@ -364,7 +364,9 @@ htb_quantum_linear() {

     sqm_debug "HTB_QUANTUM (linear): ${HTB_QUANTUM}, BANDWIDTH: ${BANDWIDTH}"

-    echo $HTB_QUANTUM
+    echo $HTB_QUANTUM >> /tmp/taht.log
+    echo 32000
+#$HTB_QUANTUM
 }

 # Fixed step scaling
@@ -438,7 +440,7 @@ get_htb_burst() {
     if [ -n "${HTB_MTU}" -a "${SHAPER_BURST}" -eq "1" ] ; then
         BURST=$( get_burst $HTB_MTU $BANDWIDTH )
         if [ -n "$BURST" ]; then
-            echo burst $BURST cburst $BURST
+            echo burst 96000 cburst 96000
         else
             sqm_debug "Default Burst, HTB will use MTU plus shipping
and handling"
         fi

>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

[Cake] Cake vs fq_codel and c/burst on an ER-X bridge

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 4650 bytes --]

Subject changed from “Cake on elements of a bridge”...

On Sep 10, 2018, at 9:55 PM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> On Mon, Sep 10, 2018 at 12:29 PM Pete Heist <pete@heistp.net> wrote:
>> 
>> For anyone who followed this, yes, the regular soft bridge (i.e. set interfaces bridge br0) works fine on the ER-X, as I suspect it would on most any Linux. A few notes about it:
>> 
>> - Your qdisc must be added to the physical interface (e.g. eth4), not the bridge interface
>> - Unlike the hardware bridge which has its own MAC, the soft bridge seems to take the MAC of the lowest (or first listed?) interface port
>> - On ER-X, bridge-nf-call-iptables=1 is the default so nothing needs to be changed there for firewalling
>> - When firewalling the bridged WAN interface, ‘in’ corresponds to bridged traffic and ‘local’ to routed traffic, which is different from the semantics for ordinary routed traffic
>> - I can do stateful firewalling for bridged hosts with “accept established and related”, but have to explicitly allow DHCP (UDP source/dest port 67-68) in the WAN interface’s ‘in’ rules for DHCP traffic to pass through the bridge
>> 
>> Performance:
>> 
>> Using Cake with this setup, the fun ends at around 110 Mbit with ksoftirqd thrashing. Unsurprisingly, there’s probably some overhead here with the soft bridge. For my purposes though (50 Mbit), it’s enough, barely…
> 
> Can I encourage you to give regular ole htb+fq_codel sqm a shot with a
> bigger burst and cburst size for htb? Fiddling with the htb quantum
> isn't helping much,
> but try this, from: https://github.com/tohojo/sqm-scripts/issues/71
> 
> (I am thinking burst and cburst should be about 1.1ms of buffering in size)

So this has turned info an interesting exercise that produced a result counter to what the common wisdom has been (that fq_codel is “faster” than cake). Because of that, I’m open to criticism of my methodology and different criteria for a successful bitrate for the shaper.

First, note that these tests still through a bridge as above, but are for a more typical setup with separate qdisc instances on egress and ingress, as opposed to my “110 Mbit” result from above, which was for egress and ingress through a common IFB.

It occurs to me that what I really want to know is the maximum set bitrate for the shaper where it still appears to be behaving properly and consistently, meaning, the actual measured TCP throughput is held steady, and at the expected percentage less than the set bitrate. I first find this out by setting a “comfortable” rate of 100Mbit and checking TCP throughput with iperf3, which is typically around 5% less than the set bitrate. Then I increase the shaper’s bitrate 5Mbit at a time and re-run the test until I find the last bitrate at which iperf3 reports a stable (within a few percent) and correct rate over 10 seconds for several runs in a row. See the attached iperf3 results for sample runs around the threshold rates.

qdisc: egress Mbit / ingress Mbit

cake nat dual-srchost / cake nat dual-dsthost ingress: 135 / 145
htb+fq_codel: 125 / 125
htb+fq_codel with burst/cburst=96000: 155 / 155

So with this testing criteria, I’m actually seeing cake “win” (with the exception of setting htb's burst/cburst to 96000, which shows a clear improvement, probably at the expense of something). I also see that the ingress rate for cake can be held steady to a bit higher of a bitrate than egress. I am using the ‘ingress’ keyword on ingress. I have to be careful here because from run to run there can be slight variations in behavior, but having repeated it several times at each bitrate around the threshold, I’m fairly certain about the results.

In the ER-X manual (https://dl.ubnt.com/guides/edgemax/EdgeOS_UG.pdf), they give a guideline of 100-250Mbps on the “expected Smart Queue shaping performance” (which means fq_codel) for the ER-X. In reality, 100Mbps is comfortable, and 250Mbps seems impossible. You might be able to get that rate by setting fq_codel to 300+Mbit (and you can’t, through a bridge anyway), but is the queue really controlled? I think I’m applying at least a little more consistent criteria for “success" here at a given bitrate than we have before.

I suppose I should repeat this test with different hardware to be surer of the claim, but I’m not sure when I’ll have the time. I will say that Cake’s shaper overall produces more satisfyingly consistent rates, and given its NAT support and host fairness, that’s why I’m likely to continue to use it when I can.


[-- Attachment #2: fq_codel_125.txt --]
[-- Type: text/plain, Size: 1173 bytes --]

hostname:~/.ssh:% iperf3 -c a.b.c.230
Connecting to host a.b.c.230, port 5201
[  5] local 192.168.1.20 port 51310 connected to a.b.c.230 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  14.5 MBytes   122 Mbits/sec                  
[  5]   1.00-2.00   sec  14.2 MBytes   119 Mbits/sec                  
[  5]   2.00-3.00   sec  14.3 MBytes   120 Mbits/sec                  
[  5]   3.00-4.00   sec  14.3 MBytes   120 Mbits/sec                  
[  5]   4.00-5.00   sec  14.3 MBytes   120 Mbits/sec                  
[  5]   5.00-6.00   sec  14.2 MBytes   119 Mbits/sec                  
[  5]   6.00-7.00   sec  14.3 MBytes   120 Mbits/sec                  
[  5]   7.00-8.00   sec  14.3 MBytes   120 Mbits/sec                  
[  5]   8.00-9.00   sec  14.2 MBytes   119 Mbits/sec                  
[  5]   9.00-10.00  sec  14.3 MBytes   120 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   143 MBytes   120 Mbits/sec                  sender
[  5]   0.00-10.00  sec   143 MBytes   120 Mbits/sec                  receiver

[-- Attachment #3: fq_codel_130.txt --]
[-- Type: text/plain, Size: 1178 bytes --]

hostname:~/Downloads:% iperf3 -c a.b.c.230
Connecting to host a.b.c.230, port 5201
[  5] local 192.168.1.20 port 51324 connected to a.b.c.230 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  13.3 MBytes   111 Mbits/sec                  
[  5]   1.00-2.00   sec  14.0 MBytes   117 Mbits/sec                  
[  5]   2.00-3.00   sec  13.9 MBytes   117 Mbits/sec                  
[  5]   3.00-4.00   sec  13.9 MBytes   116 Mbits/sec                  
[  5]   4.00-5.00   sec  13.6 MBytes   114 Mbits/sec                  
[  5]   5.00-6.00   sec  13.9 MBytes   116 Mbits/sec                  
[  5]   6.00-7.00   sec  14.0 MBytes   117 Mbits/sec                  
[  5]   7.00-8.00   sec  13.9 MBytes   116 Mbits/sec                  
[  5]   8.00-9.00   sec  13.8 MBytes   116 Mbits/sec                  
[  5]   9.00-10.00  sec  13.8 MBytes   116 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   138 MBytes   116 Mbits/sec                  sender
[  5]   0.00-10.00  sec   138 MBytes   116 Mbits/sec                  receiver

[-- Attachment #4: cake_135.txt --]
[-- Type: text/plain, Size: 1183 bytes --]

hostname:~/Downloads:% iperf3 -c a.b.c.230     
Connecting to host a.b.c.230, port 5201
[  5] local 192.168.1.20 port 51328 connected to a.b.c.230 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  15.5 MBytes   130 Mbits/sec                  
[  5]   1.00-2.00   sec  15.4 MBytes   129 Mbits/sec                  
[  5]   2.00-3.00   sec  15.4 MBytes   129 Mbits/sec                  
[  5]   3.00-4.00   sec  15.2 MBytes   128 Mbits/sec                  
[  5]   4.00-5.00   sec  15.4 MBytes   129 Mbits/sec                  
[  5]   5.00-6.00   sec  15.4 MBytes   129 Mbits/sec                  
[  5]   6.00-7.00   sec  15.4 MBytes   129 Mbits/sec                  
[  5]   7.00-8.00   sec  15.4 MBytes   129 Mbits/sec                  
[  5]   8.00-9.00   sec  15.3 MBytes   128 Mbits/sec                  
[  5]   9.00-10.00  sec  14.8 MBytes   124 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   153 MBytes   129 Mbits/sec                  sender
[  5]   0.00-10.00  sec   153 MBytes   128 Mbits/sec                  receiver

[-- Attachment #5: cake_140.txt --]
[-- Type: text/plain, Size: 1178 bytes --]

hostname:~/Downloads:% iperf3 -c a.b.c.230
Connecting to host a.b.c.230, port 5201
[  5] local 192.168.1.20 port 51332 connected to a.b.c.230 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  16.3 MBytes   136 Mbits/sec                  
[  5]   1.00-2.00   sec  16.3 MBytes   137 Mbits/sec                  
[  5]   2.00-3.00   sec  15.6 MBytes   131 Mbits/sec                  
[  5]   3.00-4.00   sec  15.6 MBytes   131 Mbits/sec                  
[  5]   4.00-5.00   sec  15.3 MBytes   128 Mbits/sec                  
[  5]   5.00-6.00   sec  16.5 MBytes   139 Mbits/sec                  
[  5]   6.00-7.00   sec  15.5 MBytes   130 Mbits/sec                  
[  5]   7.00-8.00   sec  15.3 MBytes   129 Mbits/sec                  
[  5]   8.00-9.00   sec  16.6 MBytes   139 Mbits/sec                  
[  5]   9.00-10.00  sec  15.3 MBytes   128 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   158 MBytes   133 Mbits/sec                  sender
[  5]   0.00-10.00  sec   158 MBytes   133 Mbits/sec                  receiver

[-- Attachment #6: Type: text/plain, Size: 2 bytes --]

Re: [Cake] Cake vs fq_codel and c/burst on an ER-X bridge

[Sebastian Moeller]

[-- Attachment #1: Type: text/plain, Size: 7549 bytes --]

Hi Pete,


> On Sep 11, 2018, at 00:40, Pete Heist <pete@heistp.net> wrote:
> 
> Subject changed from “Cake on elements of a bridge”...
> 
> On Sep 10, 2018, at 9:55 PM, Dave Taht <dave.taht@gmail.com> wrote:
>> 
>> On Mon, Sep 10, 2018 at 12:29 PM Pete Heist <pete@heistp.net> wrote:
>>> 
>>> For anyone who followed this, yes, the regular soft bridge (i.e. set interfaces bridge br0) works fine on the ER-X, as I suspect it would on most any Linux. A few notes about it:
>>> 
>>> - Your qdisc must be added to the physical interface (e.g. eth4), not the bridge interface
>>> - Unlike the hardware bridge which has its own MAC, the soft bridge seems to take the MAC of the lowest (or first listed?) interface port
>>> - On ER-X, bridge-nf-call-iptables=1 is the default so nothing needs to be changed there for firewalling
>>> - When firewalling the bridged WAN interface, ‘in’ corresponds to bridged traffic and ‘local’ to routed traffic, which is different from the semantics for ordinary routed traffic
>>> - I can do stateful firewalling for bridged hosts with “accept established and related”, but have to explicitly allow DHCP (UDP source/dest port 67-68) in the WAN interface’s ‘in’ rules for DHCP traffic to pass through the bridge
>>> 
>>> Performance:
>>> 
>>> Using Cake with this setup, the fun ends at around 110 Mbit with ksoftirqd thrashing. Unsurprisingly, there’s probably some overhead here with the soft bridge. For my purposes though (50 Mbit), it’s enough, barely…
>> 
>> Can I encourage you to give regular ole htb+fq_codel sqm a shot with a
>> bigger burst and cburst size for htb? Fiddling with the htb quantum
>> isn't helping much,
>> but try this, from: https://github.com/tohojo/sqm-scripts/issues/71
>> 
>> (I am thinking burst and cburst should be about 1.1ms of buffering in size)
> 
> So this has turned info an interesting exercise that produced a result counter to what the common wisdom has been (that fq_codel is “faster” than cake

	I believe the argument is more about htb+fq_codel versus cake instead of fq_codel versus cake, as it seems to be the shaper functionality that incurs the highest cost.

> ). Because of that, I’m open to criticism of my methodology and different criteria for a successful bitrate for the shaper.
> 
> First, note that these tests still through a bridge as above, but are for a more typical setup with separate qdisc instances on egress and ingress, as opposed to my “110 Mbit” result from above, which was for egress and ingress through a common IFB.
> 
> It occurs to me that what I really want to know is the maximum set bitrate for the shaper where it still appears to be behaving properly and consistently, meaning, the actual measured TCP throughput is held steady, and at the expected percentage less than the set bitrate. I first find this out by setting a “comfortable” rate of 100Mbit and checking TCP throughput with iperf3, which is typically around 5% less than the set bitrate.

	So the expected values somewhat depend on the exact configuration, but over all the expected TCP/IPv4 goodput is calculated as follows (I assume you are well aware of that, but I believe this worth repeating to calibrate the expectancy):
Expected overhead percentage: 100 - 100 * ((MTU - IP-Overhead - TCP-Overhead) / (MTU + Framing-Overhead))
assuming MTU 1500, IPv4, no TCP options, and ethernet framing (of which the kernel only accounts for 14 bytes) we get
100 - 100 * ((1500 - 20 - 20) / (1500 + 14)) = 3.57 % so the observed difference between set gross-rate and measured net-tcp payload rate matches the theory reasonably well.
with tcp timestamps which you might/should have enabled you get:
100 - 100 * ((1500 - 20 - 20 - 12) / (1500 + 14)) = 4.39 %
and with proper accounting for an ethernet carrier:
100 - 100 * ((1500 - 20 - 20 - 12) / (1500 + 38)) = 5.85175552666 %
all of these are close enough to 5% to make the 5% rule a reasonable threshold to compare against, at least to me.

> Then I increase the shaper’s bitrate 5Mbit at a time and re-run the test until I find the last bitrate at which iperf3 reports a stable (within a few percent) and correct rate over 10 seconds for several runs in a row. See the attached iperf3 results for sample runs around the threshold rates.

	Except for the 10 seconds this sounds reasonable, I would aim for at least 30, even tough this will be more important once you also monitor the latency under load concurrently to the bandwidth-probing flows...

> 
> qdisc: egress Mbit / ingress Mbit
> 
> cake nat dual-srchost / cake nat dual-dsthost ingress: 135 / 145

	On your box is there actual NAT masquerading happening?

> htb+fq_codel: 125 / 125
> htb+fq_codel with burst/cburst=96000: 155 / 155

	The last time we discussed the bust issue, I could not manage to see any difference with or without a specified burst, but I strongly believe I simply did not properly test. Btw, this is unidirectional shaping or with bidirectional saturation? 

> 
> So with this testing criteria, I’m actually seeing cake “win” (with the exception of setting htb's burst/cburst to 96000, which shows a clear improvement, probably at the expense of something).

	I could be wrong but the cost of burst/cburst is basically potentially more delay and jitter with the delay bound by the time required to empty the burst bucket, so at 96000 or 96kb and a set rate of 155Mbps I expect an additional delay of 1000 * 96/(155*1000) = 0.62 milliseconds, which fits as I believe that the 96k were targeted for 1ms @ 100Mbps. Also it will make the output of the shaper be less smooth and more choppy as sending is not paced ideally any more.
	My (too simplistic) mental model is that burst allows the shaper to work better in sirq-constrained conditions, as the issue basically seems to be that the shaper does not run often enough but without any overcommit permission will continue putting less data to the NIC than can be sent (at the desired shaper rate) in the (longer than expected/desired) interval between shaper executions. Te bust basically allow the shaper to dump in a batch of packets, hopefully getting allowing the interface to keep sending on the average desired sending rate.


> I also see that the ingress rate for cake can be held steady to a bit higher of a bitrate than egress. I am using the ‘ingress’ keyword on ingress. I have to be careful here because from run to run there can be slight variations in behavior, but having repeated it several times at each bitrate around the threshold, I’m fairly certain about the results.
> 
> In the ER-X manual (https://dl.ubnt.com/guides/edgemax/EdgeOS_UG.pdf), they give a guideline of 100-250Mbps on the “expected Smart Queue shaping performance” (which means fq_codel) for the ER-X. In reality, 100Mbps is comfortable, and 250Mbps seems impossible. You might be able to get that rate by setting fq_codel to 300+Mbit (and you can’t, through a bridge anyway), but is the queue really controlled? I think I’m applying at least a little more consistent criteria for “success" here at a given bitrate than we have before.
> 
> I suppose I should repeat this test with different hardware to be surer of the claim, but I’m not sure when I’ll have the time. I will say that Cake’s shaper overall produces more satisfyingly consistent rates, and given its NAT support and host fairness, that’s why I’m likely to continue to use it when I can.
> 

[-- Attachment #2: fq_codel_125.txt --]
[-- Type: text/plain, Size: 0 bytes --]



[-- Attachment #3: fq_codel_130.txt --]
[-- Type: text/plain, Size: 0 bytes --]



[-- Attachment #4: cake_135.txt --]
[-- Type: text/plain, Size: 0 bytes --]



[-- Attachment #5: cake_140.txt --]
[-- Type: text/plain, Size: 0 bytes --]



[-- Attachment #6: Type: text/plain, Size: 283 bytes --]

> 

	I am quite curious about these files, but I seem incapable of downloading/opening them...

Best Regards
	Sebastian

> 
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Cake vs fq_codel and c/burst on an ER-X bridge

[Dave Taht]

What I "fixed" was on the apu2 with the burst/cburst change, I went
from completely bottlenecked on one softirq to having 3 eat cpu, and
from 400mbps to 900mbps. Now, that's a quad core and the e1000 (?)
driver. The edgerouter X is a dual core, and you did see a small
improvement in throughput, but I'd hoped for more.

Re: [Cake] Cake vs fq_codel and c/burst on an ER-X bridge

[Sebastian Moeller]

Hi Dave,

> On Sep 11, 2018, at 10:20, Dave Taht <dave.taht@gmail.com> wrote:
> 
> What I "fixed" was on the apu2 with the burst/cburst change, I went
> from completely bottlenecked on one softirq to having 3 eat cpu, and
> from 400mbps to 900mbps. Now, that's a quad core and the e1000 (?)
> driver. The edgerouter X is a dual core, and you did see a small
> improvement in throughput, but I'd hoped for more.

Well, assuming my intuitions about how burst/cburst ameliorate the issue, we might simply have to say accept an additional 5ms delay/jitter to make it perform better. It is basically the same batching approach that always helps throughput of a cyclic process can not be repeated often enough, simply do more per iteration...

Best Regards
	Sebastian

Re: [Cake] Cake vs fq_codel and c/burst on an ER-X bridge

[Dave Taht]

On Tue, Sep 11, 2018 at 1:20 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>
> Hi Dave,
>
> > On Sep 11, 2018, at 10:20, Dave Taht <dave.taht@gmail.com> wrote:
> >
> > What I "fixed" was on the apu2 with the burst/cburst change, I went
> > from completely bottlenecked on one softirq to having 3 eat cpu, and
> > from 400mbps to 900mbps. Now, that's a quad core and the e1000 (?)
> > driver. The edgerouter X is a dual core, and you did see a small
> > improvement in throughput, but I'd hoped for more.
>
> Well, assuming my intuitions about how burst/cburst ameliorate the issue, we might simply have to say accept an additional 5ms delay/jitter to make it perform better. It is basically the same batching approach that always helps throughput of a cyclic process can not be repeated often enough, simply do more per iteration...
>
> Best Regards
>         Sebastian
>

I buy what you are saying. I just wished it was a magic bullet for
other than the apu2!
-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] Cake vs fq_codel and c/burst on an ER-X bridge

[Sebastian Moeller]

Hi Dave,


> On Sep 11, 2018, at 10:30, Dave Taht <dave.taht@gmail.com> wrote:
> 
> On Tue, Sep 11, 2018 at 1:20 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> Hi Dave,
>> 
>>> On Sep 11, 2018, at 10:20, Dave Taht <dave.taht@gmail.com> wrote:
>>> 
>>> What I "fixed" was on the apu2 with the burst/cburst change, I went
>>> from completely bottlenecked on one softirq to having 3 eat cpu, and
>>> from 400mbps to 900mbps. Now, that's a quad core and the e1000 (?)
>>> driver. The edgerouter X is a dual core, and you did see a small
>>> improvement in throughput, but I'd hoped for more.
>> 
>> Well, assuming my intuitions about how burst/cburst ameliorate the issue, we might simply have to say accept an additional 5ms delay/jitter to make it perform better. It is basically the same batching approach that always helps throughput of a cyclic process can not be repeated often enough, simply do more per iteration...
>> 
>> Best Regards
>>        Sebastian
>> 
> 
> I buy what you are saying. I just wished it was a magic bullet for
> other than the apu2!

I guess I will have a look at exposing burst in the gui/config file to allow easier testing. (I aim for defaulting to the automatic mode we currently use, but will also look at potentially changing the calculations).
I wonder to what degree htb's quantum is at play again after that. If I understand correctly quantum defines the granularity of dequeuing the different priority tiers, so the higher quantum the higher the choppyness/lumpiness of packets of different tiers, no? If this is true, shouldn't quantum also be defined in milliseconds instead of size, so that we have a better handle on the potential latency cost of doing so?

Best Regards
	Sebastian




> -- 
> 
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619

Re: [Cake] Cake vs fq_codel and c/burst on an ER-X bridge

[Pete Heist]

> On Sep 11, 2018, at 10:20 AM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> What I "fixed" was on the apu2 with the burst/cburst change, I went
> from completely bottlenecked on one softirq to having 3 eat cpu, and
> from 400mbps to 900mbps. Now, that's a quad core and the e1000 (?)
> driver. The edgerouter X is a dual core, and you did see a small
> improvement in throughput, but I'd hoped for more.


That’s rather dramatic. When you do your tests, do you set the shaper to a high number and see how high actual throughput goes, or do you do it some other way? Just in case, I tried setting the ER-X to 350Mbit. htb+fq_codel topped out at 149Mbit, and with c/burst 96000 186 Mbit, so in both types of tests it was about a 25% increase, not small, but unfortunately not 2x!

Pete

Re: [Cake] Cake vs fq_codel and c/burst on an ER-X bridge

[Dave Taht]

that was me shaping at 900.
On Tue, Sep 11, 2018 at 11:27 AM Pete Heist <pete@heistp.net> wrote:
>
>
> > On Sep 11, 2018, at 10:20 AM, Dave Taht <dave.taht@gmail.com> wrote:
> >
> > What I "fixed" was on the apu2 with the burst/cburst change, I went
> > from completely bottlenecked on one softirq to having 3 eat cpu, and
> > from 400mbps to 900mbps. Now, that's a quad core and the e1000 (?)
> > driver. The edgerouter X is a dual core, and you did see a small
> > improvement in throughput, but I'd hoped for more.
>
>
> That’s rather dramatic. When you do your tests, do you set the shaper to a high number and see how high actual throughput goes, or do you do it some other way? Just in case, I tried setting the ER-X to 350Mbit. htb+fq_codel topped out at 149Mbit, and with c/burst 96000 186 Mbit, so in both types of tests it was about a 25% increase, not small, but unfortunately not 2x!
>
> Pete
>


-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] Cake vs fq_codel and c/burst on an ER-X bridge

[Dave Taht]

Next time I have sufficient spare brain cells I would like to try
shaping across cores. I don't know how to
share a bit of data between qdiscs as yet, so my plan to prove it out,
is to use a static variable for time_next_packet (has to be 32 bits
also), and atomically update it between the ksoftirq threads on each
hardware queue.

https://github.com/dtaht/fq_codel_fast/

Don't know if we can round effectively to ns at 32 bits

That would do less work per core with larger intervals between
reschedules with sufficient multiplexing.

In addition to that problem, shaping inbound across cores is currently
serialized to one core (I think) by the sch_ingress qdisc. There was
some new work *way* earlier in the path
https://lwn.net/Articles/763056/ recently - and to try and find some
why to bypass the ingress and mirred portions of the path, and just
pull directly from that would help.

On Tue, Sep 11, 2018 at 11:29 AM Dave Taht <dave.taht@gmail.com> wrote:
>
> that was me shaping at 900.
> On Tue, Sep 11, 2018 at 11:27 AM Pete Heist <pete@heistp.net> wrote:
> >
> >
> > > On Sep 11, 2018, at 10:20 AM, Dave Taht <dave.taht@gmail.com> wrote:
> > >
> > > What I "fixed" was on the apu2 with the burst/cburst change, I went
> > > from completely bottlenecked on one softirq to having 3 eat cpu, and
> > > from 400mbps to 900mbps. Now, that's a quad core and the e1000 (?)
> > > driver. The edgerouter X is a dual core, and you did see a small
> > > improvement in throughput, but I'd hoped for more.
> >
> >
> > That’s rather dramatic. When you do your tests, do you set the shaper to a high number and see how high actual throughput goes, or do you do it some other way? Just in case, I tried setting the ER-X to 350Mbit. htb+fq_codel topped out at 149Mbit, and with c/burst 96000 186 Mbit, so in both types of tests it was about a 25% increase, not small, but unfortunately not 2x!
> >
> > Pete
> >
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] Cake vs fq_codel and c/burst on an ER-X bridge

[Sebastian Moeller]

Hi Dave, hi All,

so I tried to run with this idea and prototyped something into the burst_by_duration branch at the sqm repository. Would be interested to hear whether that does "the right thing" with your APU or on other's devices? The key variable is TARGET_BURST_DUR_MS in defaults.sh. Let me know how this performs for you (or whether there are bugs). As Toke proposed elsewhere I will try to streamline that branch to only configure burst size by duration so expect some changes in organization, but it should keep functional.

Best Regards
	Sebastian

> On Sep 11, 2018, at 10:20, Dave Taht <dave.taht@gmail.com> wrote:
> 
> What I "fixed" was on the apu2 with the burst/cburst change, I went
> from completely bottlenecked on one softirq to having 3 eat cpu, and
> from 400mbps to 900mbps. Now, that's a quad core and the e1000 (?)
> driver. The edgerouter X is a dual core, and you did see a small
> improvement in throughput, but I'd hoped for more.

Re: [Cake] Cake vs fq_codel and c/burst on an ER-X bridge

[Dave Taht]

thx!
On Wed, Sep 19, 2018 at 6:28 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>
> Hi Dave, hi All,
>
> so I tried to run with this idea and prototyped something into the burst_by_duration branch at the sqm repository. Would be interested to hear whether that does "the right thing" with your APU or on other's devices? The key variable is TARGET_BURST_DUR_MS in defaults.sh. Let me know how this performs for you (or whether there are bugs). As Toke proposed elsewhere I will try to streamline that branch to only configure burst size by duration so expect some changes in organization, but it should keep functional.
>
> Best Regards
>         Sebastian
>
> > On Sep 11, 2018, at 10:20, Dave Taht <dave.taht@gmail.com> wrote:
> >
> > What I "fixed" was on the apu2 with the burst/cburst change, I went
> > from completely bottlenecked on one softirq to having 3 eat cpu, and
> > from 400mbps to 900mbps. Now, that's a quad core and the e1000 (?)
> > driver. The edgerouter X is a dual core, and you did see a small
> > improvement in throughput, but I'd hoped for more.
>


-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] Cake vs fq_codel and c/burst on an ER-X bridge

[Sebastian Moeller]

Hi Dave,



> On Sep 19, 2018, at 19:02, Dave Taht <dave.taht@gmail.com> wrote:
> 
> thx!

	You are welcome, but does this actually work for you? I want to clean up things by removing the old burst scaling mode, but want/need confirmation that the new duration based method actually works as intended. (My next step then will be putting the axe on the quantum scaling, where I plan to use exactly the same size as for burst, on the theory that worst case we can only push bust bytes to the NIC and should try to service the other priority tiers at latest on the next HTB execution iteration).

Best Regards
	Sebastian



> On Wed, Sep 19, 2018 at 6:28 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> Hi Dave, hi All,
>> 
>> so I tried to run with this idea and prototyped something into the burst_by_duration branch at the sqm repository. Would be interested to hear whether that does "the right thing" with your APU or on other's devices? The key variable is TARGET_BURST_DUR_MS in defaults.sh. Let me know how this performs for you (or whether there are bugs). As Toke proposed elsewhere I will try to streamline that branch to only configure burst size by duration so expect some changes in organization, but it should keep functional.
>> 
>> Best Regards
>>        Sebastian
>> 
>>> On Sep 11, 2018, at 10:20, Dave Taht <dave.taht@gmail.com> wrote:
>>> 
>>> What I "fixed" was on the apu2 with the burst/cburst change, I went
>>> from completely bottlenecked on one softirq to having 3 eat cpu, and
>>> from 400mbps to 900mbps. Now, that's a quad core and the e1000 (?)
>>> driver. The edgerouter X is a dual core, and you did see a small
>>> improvement in throughput, but I'd hoped for more.
>> 
> 
> 
> -- 
> 
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619

Re: [Cake] Cake vs fq_codel and c/burst on an ER-X bridge

[Dave Taht]

On Thu, Sep 20, 2018 at 3:34 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>
> Hi Dave,
>
>
>
> > On Sep 19, 2018, at 19:02, Dave Taht <dave.taht@gmail.com> wrote:
> >
> > thx!
>
>         You are welcome, but does this actually work for you? I want to clean up things by removing the old burst scaling mode, but want/need confirmation that the new duration based method actually works as intended. (My next step then will be putting the axe on the quantum scaling, where I plan to use exactly the same size as for burst, on the theory that worst case we can only push bust bytes to the NIC and should try to service the other priority tiers at latest on the next HTB execution iteration).
>
> Best Regards
>         Sebastian

I meant to get time to fiddle with it yesterday and I didn't. It's
sort of stacked in with the ecn-babel test

>
>
>
> > On Wed, Sep 19, 2018 at 6:28 AM Sebastian Moeller <moeller0@gmx.de> wrote:
> >>
> >> Hi Dave, hi All,
> >>
> >> so I tried to run with this idea and prototyped something into the burst_by_duration branch at the sqm repository. Would be interested to hear whether that does "the right thing" with your APU or on other's devices? The key variable is TARGET_BURST_DUR_MS in defaults.sh. Let me know how this performs for you (or whether there are bugs). As Toke proposed elsewhere I will try to streamline that branch to only configure burst size by duration so expect some changes in organization, but it should keep functional.
> >>
> >> Best Regards
> >>        Sebastian
> >>
> >>> On Sep 11, 2018, at 10:20, Dave Taht <dave.taht@gmail.com> wrote:
> >>>
> >>> What I "fixed" was on the apu2 with the burst/cburst change, I went
> >>> from completely bottlenecked on one softirq to having 3 eat cpu, and
> >>> from 400mbps to 900mbps. Now, that's a quad core and the e1000 (?)
> >>> driver. The edgerouter X is a dual core, and you did see a small
> >>> improvement in throughput, but I'd hoped for more.
> >>
> >
> >
> > --
> >
> > Dave Täht
> > CEO, TekLibre, LLC
> > http://www.teklibre.com
> > Tel: 1-669-226-2619
>


-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] Cake vs fq_codel and c/burst on an ER-X bridge

[Sebastian Moeller]

Hi Dave,

please excuse my apparent impatience, all I wanted to convey is that only after proper testing should we roll-out the new burst configuration mode to all users ;) . One thing we should have by then is a decent idea of what to select as default burst duration. But since this is not urgent, there is no objective need to expedite this testing...

Best Regards
	Sebastian


> On Sep 20, 2018, at 19:05, Dave Taht <dave.taht@gmail.com> wrote:
> 
> On Thu, Sep 20, 2018 at 3:34 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> Hi Dave,
>> 
>> 
>> 
>>> On Sep 19, 2018, at 19:02, Dave Taht <dave.taht@gmail.com> wrote:
>>> 
>>> thx!
>> 
>>        You are welcome, but does this actually work for you? I want to clean up things by removing the old burst scaling mode, but want/need confirmation that the new duration based method actually works as intended. (My next step then will be putting the axe on the quantum scaling, where I plan to use exactly the same size as for burst, on the theory that worst case we can only push bust bytes to the NIC and should try to service the other priority tiers at latest on the next HTB execution iteration).
>> 
>> Best Regards
>>        Sebastian
> 
> I meant to get time to fiddle with it yesterday and I didn't. It's
> sort of stacked in with the ecn-babel test
> 
>> 
>> 
>> 
>>> On Wed, Sep 19, 2018 at 6:28 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>>>> 
>>>> Hi Dave, hi All,
>>>> 
>>>> so I tried to run with this idea and prototyped something into the burst_by_duration branch at the sqm repository. Would be interested to hear whether that does "the right thing" with your APU or on other's devices? The key variable is TARGET_BURST_DUR_MS in defaults.sh. Let me know how this performs for you (or whether there are bugs). As Toke proposed elsewhere I will try to streamline that branch to only configure burst size by duration so expect some changes in organization, but it should keep functional.
>>>> 
>>>> Best Regards
>>>>       Sebastian
>>>> 
>>>>> On Sep 11, 2018, at 10:20, Dave Taht <dave.taht@gmail.com> wrote:
>>>>> 
>>>>> What I "fixed" was on the apu2 with the burst/cburst change, I went
>>>>> from completely bottlenecked on one softirq to having 3 eat cpu, and
>>>>> from 400mbps to 900mbps. Now, that's a quad core and the e1000 (?)
>>>>> driver. The edgerouter X is a dual core, and you did see a small
>>>>> improvement in throughput, but I'd hoped for more.
>>>> 
>>> 
>>> 
>>> --
>>> 
>>> Dave Täht
>>> CEO, TekLibre, LLC
>>> http://www.teklibre.com
>>> Tel: 1-669-226-2619
>> 
> 
> 
> -- 
> 
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619

Re: [Cake] Cake vs fq_codel and c/burst on an ER-X bridge

[Dave Taht]

On Thu, Sep 20, 2018 at 11:19 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>
> Hi Dave,
>
> please excuse my apparent impatience, all I wanted to convey is that only after proper testing should we roll-out the new burst configuration mode to all users ;) . One thing we should have by then is a decent idea of what to select as default burst duration. But since this is not urgent, there is no objective need to expedite this testing...

I'm delighted you jumped on it. I am just slow these days.

>
> Best Regards
>         Sebastian
>
>
> > On Sep 20, 2018, at 19:05, Dave Taht <dave.taht@gmail.com> wrote:
> >
> > On Thu, Sep 20, 2018 at 3:34 AM Sebastian Moeller <moeller0@gmx.de> wrote:
> >>
> >> Hi Dave,
> >>
> >>
> >>
> >>> On Sep 19, 2018, at 19:02, Dave Taht <dave.taht@gmail.com> wrote:
> >>>
> >>> thx!
> >>
> >>        You are welcome, but does this actually work for you? I want to clean up things by removing the old burst scaling mode, but want/need confirmation that the new duration based method actually works as intended. (My next step then will be putting the axe on the quantum scaling, where I plan to use exactly the same size as for burst, on the theory that worst case we can only push bust bytes to the NIC and should try to service the other priority tiers at latest on the next HTB execution iteration).
> >>
> >> Best Regards
> >>        Sebastian
> >
> > I meant to get time to fiddle with it yesterday and I didn't. It's
> > sort of stacked in with the ecn-babel test
> >
> >>
> >>
> >>
> >>> On Wed, Sep 19, 2018 at 6:28 AM Sebastian Moeller <moeller0@gmx.de> wrote:
> >>>>
> >>>> Hi Dave, hi All,
> >>>>
> >>>> so I tried to run with this idea and prototyped something into the burst_by_duration branch at the sqm repository. Would be interested to hear whether that does "the right thing" with your APU or on other's devices? The key variable is TARGET_BURST_DUR_MS in defaults.sh. Let me know how this performs for you (or whether there are bugs). As Toke proposed elsewhere I will try to streamline that branch to only configure burst size by duration so expect some changes in organization, but it should keep functional.
> >>>>
> >>>> Best Regards
> >>>>       Sebastian
> >>>>
> >>>>> On Sep 11, 2018, at 10:20, Dave Taht <dave.taht@gmail.com> wrote:
> >>>>>
> >>>>> What I "fixed" was on the apu2 with the burst/cburst change, I went
> >>>>> from completely bottlenecked on one softirq to having 3 eat cpu, and
> >>>>> from 400mbps to 900mbps. Now, that's a quad core and the e1000 (?)
> >>>>> driver. The edgerouter X is a dual core, and you did see a small
> >>>>> improvement in throughput, but I'd hoped for more.
> >>>>
> >>>
> >>>
> >>> --
> >>>
> >>> Dave Täht
> >>> CEO, TekLibre, LLC
> >>> http://www.teklibre.com
> >>> Tel: 1-669-226-2619
> >>
> >
> >
> > --
> >
> > Dave Täht
> > CEO, TekLibre, LLC
> > http://www.teklibre.com
> > Tel: 1-669-226-2619
>


-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] Cake vs fq_codel and c/burst on an ER-X bridge

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 3222 bytes --]


> On Sep 11, 2018, at 9:54 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> So this has turned info an interesting exercise that produced a result counter to what the common wisdom has been (that fq_codel is “faster” than cake
> 
> 	I believe the argument is more about htb+fq_codel versus cake instead of fq_codel versus cake, as it seems to be the shaper functionality that incurs the highest cost.

I sometimes loosely use fq_codel to mean htb+fq_codel.

>> It occurs to me that what I really want to know is the maximum set bitrate for the shaper where it still appears to be behaving properly and consistently, meaning, the actual measured TCP throughput is held steady, and at the expected percentage less than the set bitrate. I first find this out by setting a “comfortable” rate of 100Mbit and checking TCP throughput with iperf3, which is typically around 5% less than the set bitrate.
> 
> 	So the expected values somewhat depend on the exact configuration, but over all the expected TCP/IPv4 goodput is calculated as follows (I assume you are well aware of that, but I believe this worth repeating to calibrate the expectancy):

Yes, it’s pretty much right on the money.

>> Then I increase the shaper’s bitrate 5Mbit at a time and re-run the test until I find the last bitrate at which iperf3 reports a stable (within a few percent) and correct rate over 10 seconds for several runs in a row. See the attached iperf3 results for sample runs around the threshold rates.
> 
> 	Except for the 10 seconds this sounds reasonable, I would aim for at least 30, even tough this will be more important once you also monitor the latency under load concurrently to the bandwidth-probing flows…

I agree, so far I was just trying not to spend an all-nighter on it last night. :) I was actually running 3-5 trials of ten seconds, also because sometimes with fq_codel once it’s slightly above its limit, results would vary from run to run.

>> cake nat dual-srchost / cake nat dual-dsthost ingress: 135 / 145
> 
> 	On your box is there actual NAT masquerading happening?

Yeah, good point, I left nat there because I had one port configured for routing and the other for the bridge and was sometimes swapping between the two. I realize now I actually sent the numbers for routing, not bridging. Bridging without ‘nat’ looks a bit higher (155 Mbit for cake instead of 135 Mbit). I would re-do all these tests for completeness but I’m out of time now.

> 	The last time we discussed the bust issue, I could not manage to see any difference with or without a specified burst, but I strongly believe I simply did not properly test. Btw, this is unidirectional shaping or with bidirectional saturation?

Unidirectional. I definitely see a difference, but I wonder what criteria we (and I) used for “out of CPU’ in the past.

> <fq_codel_125.txt><fq_codel_130.txt><cake_135.txt><cake_140.txt>
> 	I am quite curious about these files, but I seem incapable of downloading/opening them...


Ok, no more sending attachments to the list I see. If this link doesn’t work I might be replacing a disk at the time… :)
https://www.heistp.net/downloads/erx-sqm.tgz


[-- Attachment #2: Type: text/html, Size: 13657 bytes --]

Re: [Cake] Cake vs fq_codel and c/burst on an ER-X bridge

[Sebastian Moeller]

Hi Pete,


> On Sep 11, 2018, at 20:09, Pete Heist <pete@heistp.net> wrote:
> 
> 
>> On Sep 11, 2018, at 9:54 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
>>> 
>>> So this has turned info an interesting exercise that produced a result counter to what the common wisdom has been (that fq_codel is “faster” than cake
>> 
>> 	I believe the argument is more about htb+fq_codel versus cake instead of fq_codel versus cake, as it seems to be the shaper functionality that incurs the highest cost.
> 
> I sometimes loosely use fq_codel to mean htb+fq_codel.

	Sorry for being a smart aleck, unfortunately it is my second nature.

> 
>>> It occurs to me that what I really want to know is the maximum set bitrate for the shaper where it still appears to be behaving properly and consistently, meaning, the actual measured TCP throughput is held steady, and at the expected percentage less than the set bitrate. I first find this out by setting a “comfortable” rate of 100Mbit and checking TCP throughput with iperf3, which is typically around 5% less than the set bitrate.
>> 
>> 	So the expected values somewhat depend on the exact configuration, but over all the expected TCP/IPv4 goodput is calculated as follows (I assume you are well aware of that, but I believe this worth repeating to calibrate the expectancy):
> 
> Yes, it’s pretty much right on the money.
> 
>>> Then I increase the shaper’s bitrate 5Mbit at a time and re-run the test until I find the last bitrate at which iperf3 reports a stable (within a few percent) and correct rate over 10 seconds for several runs in a row. See the attached iperf3 results for sample runs around the threshold rates.
>> 
>> 	Except for the 10 seconds this sounds reasonable, I would aim for at least 30, even tough this will be more important once you also monitor the latency under load concurrently to the bandwidth-probing flows…
> 
> I agree, so far I was just trying not to spend an all-nighter on it last night. :) I was actually running 3-5 trials of ten seconds, also because sometimes with fq_codel once it’s slightly above its limit, results would vary from run to run.

	Ah, why is it so often reality that crosses the most diligent plans ;) ?

> 
>>> cake nat dual-srchost / cake nat dual-dsthost ingress: 135 / 145
>> 
>> 	On your box is there actual NAT masquerading happening?
> 
> Yeah, good point, I left nat there because I had one port configured for routing and the other for the bridge and was sometimes swapping between the two. I realize now I actually sent the numbers for routing, not bridging. Bridging without ‘nat’ looks a bit higher (155 Mbit for cake instead of 135 Mbit). I would re-do all these tests for completeness but I’m out of time now.

	Ouch, a ten percent bandwidth cost for the nat feature certainly answeers the question whether nat should be the default...

> 
>> 	The last time we discussed the bust issue, I could not manage to see any difference with or without a specified burst, but I strongly believe I simply did not properly test. Btw, this is unidirectional shaping or with bidirectional saturation?
> 
> Unidirectional. I definitely see a difference, but I wonder what criteria we (and I) used for “out of CPU’ in the past.

	So totally unscientifically me yardstick was as long as throughput increases more or less linearly with configured shaper bandwidth things are fine, and then at the candidate bandwidths I ran "top -d 1" and monitored both idle% ad sirq% with idle falling below 5% being a strong indicator of bottlenecking on cpu cycles. Dlakelan over at github (https://github.com/dlakelan/routerperf) is working on a small side project that aims for tighter multi-core aware logging of cpu usage on a router, but that has not left the early prototype stage.

> 
>> <fq_codel_125.txt><fq_codel_130.txt><cake_135.txt><cake_140.txt>
>> 	I am quite curious about these files, but I seem incapable of downloading/opening them...
> 
> Ok, no more sending attachments to the list I see. If this link doesn’t work I might be replacing a disk at the time… :)
> https://www.heistp.net/downloads/erx-sqm.tgz

Great, that worked; but now I realize that I need to get acquinted with iperf output ;)

Best Regards
	Sebastian

> 

Re: [Cake] Cake vs fq_codel and c/burst on an ER-X bridge

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 2168 bytes --]


> On Sep 11, 2018, at 8:28 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> Yeah, good point, I left nat there because I had one port configured for routing and the other for the bridge and was sometimes swapping between the two. I realize now I actually sent the numbers for routing, not bridging. Bridging without ‘nat’ looks a bit higher (155 Mbit for cake instead of 135 Mbit). I would re-do all these tests for completeness but I’m out of time now.
> 
> 	Ouch, a ten percent bandwidth cost for the nat feature certainly answeers the question whether nat should be the default…

That probably has a lot to do with routing vs bridging though also. If I turn QoS off, the ER-X does about 250Mbit when routing and 280Mbit with the soft bridge, so that’s probably most of that difference. I’m not seeing a throughput difference above random noise between ‘nat’ and ‘nonat’. When I benchmarked it before I saw an ~1.5% CPU difference, not nothing.

>>> 	The last time we discussed the bust issue, I could not manage to see any difference with or without a specified burst, but I strongly believe I simply did not properly test. Btw, this is unidirectional shaping or with bidirectional saturation?
>> 
>> Unidirectional. I definitely see a difference, but I wonder what criteria we (and I) used for “out of CPU’ in the past.
> 
> 	So totally unscientifically me yardstick was as long as throughput increases more or less linearly with configured shaper bandwidth things are fine, and then at the candidate bandwidths I ran "top -d 1" and monitored both idle% ad sirq% with idle falling below 5% being a strong indicator of bottlenecking on cpu cycles. Dlakelan over at github (https://github.com/dlakelan/routerperf <https://github.com/dlakelan/routerperf>) is working on a small side project that aims for tighter multi-core aware logging of cpu usage on a router, but that has not left the early prototype stage.

Ok, my frustration with the testing has also been variable results from run to run. My inner self is saying, yes, do some testing, but don’t spend too much time on it when it has this stochastic side to it.

[-- Attachment #2: Type: text/html, Size: 6827 bytes --]

Re: [Cake] Cake vs fq_codel and c/burst on an ER-X bridge

[Dave Taht]

On Tue, Sep 11, 2018 at 11:45 AM Pete Heist <pete@heistp.net> wrote:
>
>
> On Sep 11, 2018, at 8:28 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
>
>
> Yeah, good point, I left nat there because I had one port configured for routing and the other for the bridge and was sometimes swapping between the two. I realize now I actually sent the numbers for routing, not bridging. Bridging without ‘nat’ looks a bit higher (155 Mbit for cake instead of 135 Mbit). I would re-do all these tests for completeness but I’m out of time now.
>
>
> Ouch, a ten percent bandwidth cost for the nat feature certainly answeers the question whether nat should be the default…
>
>
> That probably has a lot to do with routing vs bridging though also. If I turn QoS off, the ER-X does about 250Mbit when routing and 280Mbit with the soft bridge, so that’s probably most of that difference. I’m not seeing a throughput difference above random noise between ‘nat’ and ‘nonat’. When I benchmarked it before I saw an ~1.5% CPU difference, not nothing.
>
> The last time we discussed the bust issue, I could not manage to see any difference with or without a specified burst, but I strongly believe I simply did not properly test. Btw, this is unidirectional shaping or with bidirectional saturation?
>
>
> Unidirectional. I definitely see a difference, but I wonder what criteria we (and I) used for “out of CPU’ in the past.
>
>
> So totally unscientifically me yardstick was as long as throughput increases more or less linearly with configured shaper bandwidth things are fine, and then at the candidate bandwidths I ran "top -d 1" and monitored both idle% ad sirq% with idle falling below 5% being a strong indicator of bottlenecking on cpu cycles. Dlakelan over at github (https://github.com/dlakelan/routerperf) is working on a small side project that aims for tighter multi-core aware logging of cpu usage on a router, but that has not left the early prototype stage.
>
>
> Ok, my frustration with the testing has also been variable results from run to run. My inner self is saying, yes, do some testing, but don’t spend too much time on it when it has this stochastic side to it.
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

My inner self is saying... "go out and enjoy the beautiful fall
weather, while it lasts."

-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619