[Cake] Getting Cake to work better with Steam and similar applications

[Cake] Getting Cake to work better with Steam and similar applications

[Dendari Marini]

[-- Attachment #1: Type: text/plain, Size: 1037 bytes --]

Hello,

Not sure if this is the right place, just recently discovered about the
bufferbloat issue, so I'm still learning new things every day.

I bought an Ubiquiti EdgeRouter X just for this and for the most part the
built-in Smart Queue (which uses fq_codel + HTB) works wonderfully but
there are some applications which still give me issues.

In particular I'm currently trying to resolve these issues with Steam,
which doesn't seem to be affected much by the Smart Queue. Someone
suggested to use Cake Cobalt, which I'm currently using, but I can't seem
to get any better result.

From my understanding I should at least get fairer per-host bandwidth, but
when one PC is downloading with Steam the other will only receive ~15% of
the total bandwidth.

My connection isn't really fast (just 16/0.9 Mbitps ADSL) and I have only
two PCs connected via ethernet to the EdgeRouter X plus a WiFi access point
which is mainly used by smartphones. I haven't done any particular change
to the ER-X so for the most is kinda stock.

Any advice?

[-- Attachment #2: Type: text/html, Size: 1237 bytes --]

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Sebastian Moeller]

Hi,

> On Apr 20, 2017, at 15:39, Dendari Marini <dendari92@gmail.com> wrote:
> 
> Hello,
> 
> Not sure if this is the right place, just recently discovered about the bufferbloat issue, so I'm still learning new things every day.
> 
> I bought an Ubiquiti EdgeRouter X just for this and for the most part the built-in Smart Queue (which uses fq_codel + HTB) works wonderfully but there are some applications which still give me issues.
> 
> In particular I'm currently trying to resolve these issues with Steam, which doesn't seem to be affected much by the Smart Queue. Someone suggested to use Cake Cobalt, which I'm currently using, but I can't seem to get any better result.
> 
> From my understanding I should at least get fairer per-host bandwidth, but when one PC is downloading with Steam the other will only receive ~15% of the total bandwidth. 

	Could you post the output of calling “tc -s qdisc” here on the list please? That should allow to figure out what you actually told cake to do ;0


> 
> My connection isn't really fast (just 16/0.9 Mbitps ADSL)

	That uplink is going to be painful no matter what.

Best Regards

> and I have only two PCs connected via ethernet to the EdgeRouter X plus a WiFi access point which is mainly used by smartphones. I haven't done any particular change to the ER-X so for the most is kinda stock.
> 
> Any advice?
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Dendari Marini]

[-- Attachment #1: Type: text/plain, Size: 5873 bytes --]

> Could you post the output of calling “tc -s qdisc” here on the list
please? That should allow to figure out what you actually told cake to do ;0


> /home/ubnt$ sudo tc -s qdisc
> qdisc pfifo_fast 0: dev switch0 root refcnt 2 bands 3 priomap  1 2 2 2 1 2
> 0 0 1 1 1 1 1 1 1 1
>  Sent 133387235 bytes 191612 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: dev imq0 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0
> 0 1 1 1 1 1 1 1 1
>  Sent 288 bytes 9 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
> qdisc cake 8001: dev eth0 root refcnt 2 bandwidth 900Kbit diffserv3
> dual-srchost nat rtt 100.0ms raw
>  Sent 183589 bytes 1316 pkt (dropped 6, overlimits 627 requeues 0)
>  backlog 0b 0p requeues 0
>  memory used: 91392b of 4Mb
>  capacity estimate: 900Kbit
>                  Bulk   Best Effort      Voice
>   thresh      56248bit     900Kbit     225Kbit
>   target       323.0ms      20.2ms      80.7ms
>   interval     646.0ms     115.2ms     161.5ms
>   pk_delay         0us       7.8ms         4us
>   av_delay         0us       695us         0us
>   sp_delay         0us         5us         0us
>   pkts               0        1314           8
>   bytes              0      188234         336
>   way_inds           0           0           0
>   way_miss           0          98           1
>   way_cols           0           0           0
>   drops              0           6           0
>   marks              0           0           0
>   sp_flows           0           1           0
>   bk_flows           0           1           0
>   un_flows           0           0           0
>   max_len            0        1514          42
> qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
>  Sent 1396868 bytes 1615 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: dev eth1 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0
> 0 1 1 1 1 1 1 1 1
>  Sent 822 bytes 13 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: dev eth2 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0
> 0 1 1 1 1 1 1 1 1
>  Sent 822 bytes 13 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: dev eth3 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0
> 0 1 1 1 1 1 1 1 1
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: dev eth4 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0
> 0 1 1 1 1 1 1 1 1
>  Sent 822 bytes 13 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: dev pppoe0 root refcnt 2 bands 3 priomap  1 2 2 2 1 2
> 0 0 1 1 1 1 1 1 1 1
>  Sent 23509667 bytes 164606 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
> qdisc cake 8002: dev ifb4eth0 root refcnt 2 bandwidth 16Mbit diffserv3
> dual-dsthost nat ingress rtt 100.0ms raw
>  Sent 1394936 bytes 1483 pkt (dropped 2, overlimits 1380 requeues 0)
>  backlog 0b 0p requeues 0
>  memory used: 47872b of 4Mb
>  capacity estimate: 16Mbit
>                  Bulk   Best Effort      Voice
>   thresh         1Mbit      16Mbit       4Mbit
>   target        18.2ms       5.0ms       5.0ms
>   interval     113.2ms     100.0ms      10.0ms
>   pk_delay         0us       979us         8us
>   av_delay         0us       277us         0us
>   sp_delay         0us         4us         0us
>   pkts               0        1479           6
>   bytes              0     1396354         360
>   way_inds           0           0           0
>   way_miss           0          76           1
>   way_cols           0           0           0
>   drops              0           2           0
>   marks              0           0           0
>   sp_flows           0           2           0
>   bk_flows           0           1           0
>   un_flows           0           0           0
>   max_len            0        1514          60


Sorry, the formatting isn't that great :\

And yeah I know my upload isn't that great either

On 20 April 2017 at 15:43, Sebastian Moeller <moeller0@gmx.de> wrote:

> Hi,
>
> > On Apr 20, 2017, at 15:39, Dendari Marini <dendari92@gmail.com> wrote:
> >
> > Hello,
> >
> > Not sure if this is the right place, just recently discovered about the
> bufferbloat issue, so I'm still learning new things every day.
> >
> > I bought an Ubiquiti EdgeRouter X just for this and for the most part
> the built-in Smart Queue (which uses fq_codel + HTB) works wonderfully but
> there are some applications which still give me issues.
> >
> > In particular I'm currently trying to resolve these issues with Steam,
> which doesn't seem to be affected much by the Smart Queue. Someone
> suggested to use Cake Cobalt, which I'm currently using, but I can't seem
> to get any better result.
> >
> > From my understanding I should at least get fairer per-host bandwidth,
> but when one PC is downloading with Steam the other will only receive ~15%
> of the total bandwidth.
>
>         Could you post the output of calling “tc -s qdisc” here on the
> list please? That should allow to figure out what you actually told cake to
> do ;0
>
>
> >
> > My connection isn't really fast (just 16/0.9 Mbitps ADSL)
>
>         That uplink is going to be painful no matter what.
>
> Best Regards
>
> > and I have only two PCs connected via ethernet to the EdgeRouter X plus
> a WiFi access point which is mainly used by smartphones. I haven't done any
> particular change to the ER-X so for the most is kinda stock.
> >
> > Any advice?
> > _______________________________________________
> > Cake mailing list
> > Cake@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cake
>
>

[-- Attachment #2: Type: text/html, Size: 7442 bytes --]

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Jonathan Morton]

> On 20 Apr, 2017, at 18:23, Dendari Marini <dendari92@gmail.com> wrote:
> 
> > Could you post the output of calling “tc -s qdisc” here on the list please? That should allow to figure out what you actually told cake to do ;0

> qdisc cake 8001: dev eth0 root refcnt 2 bandwidth 900Kbit diffserv3 dual-srchost nat rtt 100.0ms raw 

> qdisc cake 8002: dev ifb4eth0 root refcnt 2 bandwidth 16Mbit diffserv3 dual-dsthost nat ingress rtt 100.0ms raw 

Looks like most of your options are okay, including the correct “dual” modes and “ingress” mode in the right place.  However, I think you need to adjust your bandwidth and overhead settings, otherwise Cake isn’t reliably in control of the bottleneck queues.  Try these to begin with:

… bandwidth 850Kbit conservative dual-srchost nat

… bandwidth 15Mbit conservative dual-dsthost nat ingress

That should give you correct operation, and you can fine-tune from there.

 - Jonathan Morton

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Dendari Marini]

[-- Attachment #1: Type: text/plain, Size: 1944 bytes --]

Hello, thanks for your reply.

Looks like most of your options are okay, including the correct “dual”
> modes and “ingress” mode in the right place.  However, I think you need to
> adjust your bandwidth and overhead settings, otherwise Cake isn’t reliably
> in control of the bottleneck queues.  Try these to begin with:
>
> … bandwidth 850Kbit conservative dual-srchost nat
>
> … bandwidth 15Mbit conservative dual-dsthost nat ingress
>
> That should give you correct operation, and you can fine-tune from there.


Just did quick test with your settings. First thing I noticed is my final
download bandwidth is about 12Mbps, Steam on PC1 downloads at 1.4-1.5MB/s
while downloading a file on PC2 seems to max out at ~250KB/s. From my
understanding I should see each PC download at ~700KB/s, or am I mistaken?

On 20 April 2017 at 17:32, Jonathan Morton <chromatix99@gmail.com> wrote:

>
> > On 20 Apr, 2017, at 18:23, Dendari Marini <dendari92@gmail.com> wrote:
> >
> > > Could you post the output of calling “tc -s qdisc” here on the list
> please? That should allow to figure out what you actually told cake to do ;0
>
> > qdisc cake 8001: dev eth0 root refcnt 2 bandwidth 900Kbit diffserv3
> dual-srchost nat rtt 100.0ms raw
>
> > qdisc cake 8002: dev ifb4eth0 root refcnt 2 bandwidth 16Mbit diffserv3
> dual-dsthost nat ingress rtt 100.0ms raw
>
> Looks like most of your options are okay, including the correct “dual”
> modes and “ingress” mode in the right place.  However, I think you need to
> adjust your bandwidth and overhead settings, otherwise Cake isn’t reliably
> in control of the bottleneck queues.  Try these to begin with:
>
> … bandwidth 850Kbit conservative dual-srchost nat
>
> … bandwidth 15Mbit conservative dual-dsthost nat ingress
>
> That should give you correct operation, and you can fine-tune from there.
>
>  - Jonathan Morton
>
>
>

[-- Attachment #2: Type: text/html, Size: 2916 bytes --]

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Andy Furniss]

Dendari Marini wrote:
> Hello, thanks for your reply.
> 
> Looks like most of your options are okay, including the correct “dual”
>> modes and “ingress” mode in the right place.  However, I think you need to
>> adjust your bandwidth and overhead settings, otherwise Cake isn’t reliably
>> in control of the bottleneck queues.  Try these to begin with:
>>
>> … bandwidth 850Kbit conservative dual-srchost nat
>>
>> … bandwidth 15Mbit conservative dual-dsthost nat ingress
>>
>> That should give you correct operation, and you can fine-tune from there.
> 
> 
> Just did quick test with your settings. First thing I noticed is my final
> download bandwidth is about 12Mbps, Steam on PC1 downloads at 1.4-1.5MB/s
> while downloading a file on PC2 seems to max out at ~250KB/s. From my
> understanding I should see each PC download at ~700KB/s, or am I mistaken?

What is dev pppoe0 on your setup?

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Jonathan Morton]

> On 20 Apr, 2017, at 19:05, Dendari Marini <dendari92@gmail.com> wrote:
> 
> Just did quick test with your settings. First thing I noticed is my final download bandwidth is about 12Mbps, Steam on PC1 downloads at 1.4-1.5MB/s while downloading a file on PC2 seems to max out at ~250KB/s. From my understanding I should see each PC download at ~700KB/s, or am I mistaken?

It should be possible to get equal bandwidth to each PC, yes.  It could be that with such a strongly asymmetric connection, Cake still isn’t actually the bottleneck in the download direction.  Try setting it to 10Mbit and see if that brings it under better control.

 - Jonathan Morton

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Sebastian Moeller]

> On Apr 20, 2017, at 18:05, Dendari Marini <dendari92@gmail.com> wrote:
> 
> Hello, thanks for your reply.
> 
> Looks like most of your options are okay, including the correct “dual” modes and “ingress” mode in the right place.  However, I think you need to adjust your bandwidth and overhead settings, otherwise Cake isn’t reliably in control of the bottleneck queues.  Try these to begin with:
> 
> … bandwidth 850Kbit conservative dual-srchost nat
> 
> … bandwidth 15Mbit conservative dual-dsthost nat ingress
> 
> That should give you correct operation, and you can fine-tune from there.
> 
> Just did quick test with your settings. First thing I noticed is my final download bandwidth is about 12Mbps, Steam on PC1 downloads at 1.4-1.5MB/s while downloading a file on PC2 seems to max out at ~250KB/s. From my understanding I should see each PC download at ~700KB/s, or am I mistaken?

Assuming you measured good put in [M|K]iBytes this adds up to  1.5+0.25 = 1.75 * 1024^2 * 8 = 14680064 Bits or (1.4+0.25) * 8 *1024^2 / 1000^2 = 13.84 Mbps which seems a bit high for a 16Mbps ADSL link. I would ecpext something like 16 * (48/53)  * ((1500 - 8 - 20 -20) / (1500 + 32)) = 13.73 Mbps TCP/IPv4 goodput… so you seem to be running close to theoretical maximum of your link (assuming I am not totally off with the overhead (estimated ADSL overhead on top of MTU: 6 destination MAC + 6 source MAC + 2 ethertype + 3 ATM LLC + 5 ATM SNAP + 2 ATM pad + 8 ATM AAL5 SAR 32 bytes). But with your shaper set at 15Mbps without the atm option you will actually accept up to 15 * (53/48) = 16.5625 Mbps on the wire, which probably is above your link bandwidth. This fits well with the really low number of drops in your cake stats, you simply never have cake feel that shaping is needed?

Best Regards




> 
> On 20 April 2017 at 17:32, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
> > On 20 Apr, 2017, at 18:23, Dendari Marini <dendari92@gmail.com> wrote:
> >
> > > Could you post the output of calling “tc -s qdisc” here on the list please? That should allow to figure out what you actually told cake to do ;0
> 
> > qdisc cake 8001: dev eth0 root refcnt 2 bandwidth 900Kbit diffserv3 dual-srchost nat rtt 100.0ms raw
> 
> > qdisc cake 8002: dev ifb4eth0 root refcnt 2 bandwidth 16Mbit diffserv3 dual-dsthost nat ingress rtt 100.0ms raw
> 
> Looks like most of your options are okay, including the correct “dual” modes and “ingress” mode in the right place.  However, I think you need to adjust your bandwidth and overhead settings, otherwise Cake isn’t reliably in control of the bottleneck queues.  Try these to begin with:
> 
> … bandwidth 850Kbit conservative dual-srchost nat
> 
> … bandwidth 15Mbit conservative dual-dsthost nat ingress
> 
> That should give you correct operation, and you can fine-tune from there.
> 
>  - Jonathan Morton
> 
> 
> 

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Sebastian Moeller]

> On Apr 20, 2017, at 18:05, Dendari Marini <dendari92@gmail.com> wrote:
> 
> Hello, thanks for your reply.
> 
> Looks like most of your options are okay, including the correct “dual” modes and “ingress” mode in the right place.  However, I think you need to adjust your bandwidth and overhead settings, otherwise Cake isn’t reliably in control of the bottleneck queues.  Try these to begin with:
> 
> … bandwidth 850Kbit conservative dual-srchost nat
> 
> … bandwidth 15Mbit conservative dual-dsthost nat ingress
> 
> That should give you correct operation, and you can fine-tune from there.
> 
> Just did quick test with your settings. First thing I noticed is my final download bandwidth is about 12Mbps, Steam on PC1 downloads at 1.4-1.5MB/s while downloading a file on PC2 seems to max out at ~250KB/s. From my understanding I should see each PC download at ~700KB/s, or am I mistaken?

Assuming you measured good put in [M|K]iBytes this adds up to  1.5+0.25 = 1.75 * 1024^2 * 8 = 14680064 Bits or (1.4+0.25) * 8 *1024^2 / 1000^2 = 13.84 Mbps which seems a bit high for a 16Mbps ADSL link. I would ecpext something like 16 * (48/53)  * ((1500 - 8 - 20 -20) / (1500 + 32)) = 13.73 Mbps TCP/IPv4 goodput… so you seem to be running close to theoretical maximum of your link (assuming I am not totally off with the overhead (estimated ADSL overhead on top of MTU: 6 destination MAC + 6 source MAC + 2 ethertype + 3 ATM LLC + 5 ATM SNAP + 2 ATM pad + 8 ATM AAL5 SAR 32 bytes). But with your shaper set at 15Mbps without the atm option you will actually accept up to 15 * (53/48) = 16.5625 Mbps on the wire, which probably is above your link bandwidth. This fits well with the really low number of drops in your cake stats, you simply never have cake feel that shaping is needed?

Best Regards




> 
> On 20 April 2017 at 17:32, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> On 20 Apr, 2017, at 18:23, Dendari Marini <dendari92@gmail.com> wrote:
>> 
>>> Could you post the output of calling “tc -s qdisc” here on the list please? That should allow to figure out what you actually told cake to do ;0
> 
>> qdisc cake 8001: dev eth0 root refcnt 2 bandwidth 900Kbit diffserv3 dual-srchost nat rtt 100.0ms raw
> 
>> qdisc cake 8002: dev ifb4eth0 root refcnt 2 bandwidth 16Mbit diffserv3 dual-dsthost nat ingress rtt 100.0ms raw
> 
> Looks like most of your options are okay, including the correct “dual” modes and “ingress” mode in the right place.  However, I think you need to adjust your bandwidth and overhead settings, otherwise Cake isn’t reliably in control of the bottleneck queues.  Try these to begin with:
> 
> … bandwidth 850Kbit conservative dual-srchost nat
> 
> … bandwidth 15Mbit conservative dual-dsthost nat ingress
> 
> That should give you correct operation, and you can fine-tune from there.
> 
> - Jonathan Morton
> 
> 
> 

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Dendari Marini]

[-- Attachment #1: Type: text/plain, Size: 4534 bytes --]

Hello, thanks for all of your replies.

First of all, my connection encapsulation should be ATM LLC and it can
actually reach up to 17.5/1 Mbps, but that's kinda best case scenario which
is why I wanted to play it safe with just 16/.9 (which I should reach more
consistently).

Back to the Steam issue. Unfortunately I can't seem to get really
consistent results, mainly because sometimes it's downloading the game from
just a few connections and other times it's downloading from dozens and
dozens connections. The latter is the one giving me more issues both in
terms of latency/packet loss and in terms of evenly splitting the bandwidth
across the hosts.

One thing that seems to give better results is changing the interface where
Cake is used from eth0 to pppoe0. When I used fq_codel it seemed to give
better results when using eth0 and so I went ahead and did the same with
Cake.

Anyway more testing needed, will report if I notice any consistent result.

By the way this is the thread I opened on the Ubiquiti forums talking about
this issue (not sure if it can give you some more info):
https://community.ubnt.com/t5/EdgeMAX/Smart-Queue-seemingly-not-working-for-Steam-downloads/td-p/1890405
Also the thread where I got Cake for the ER-X from:
https://community.ubnt.com/t5/EdgeMAX/Cake-and-FQ-PIE-compiled-for-the-EdgeRouter-devices/td-p/1679844

On 20 April 2017 at 20:36, Sebastian Moeller <moeller0@gmx.de> wrote:

>
> > On Apr 20, 2017, at 18:05, Dendari Marini <dendari92@gmail.com> wrote:
> >
> > Hello, thanks for your reply.
> >
> > Looks like most of your options are okay, including the correct “dual”
> modes and “ingress” mode in the right place.  However, I think you need to
> adjust your bandwidth and overhead settings, otherwise Cake isn’t reliably
> in control of the bottleneck queues.  Try these to begin with:
> >
> > … bandwidth 850Kbit conservative dual-srchost nat
> >
> > … bandwidth 15Mbit conservative dual-dsthost nat ingress
> >
> > That should give you correct operation, and you can fine-tune from there.
> >
> > Just did quick test with your settings. First thing I noticed is my
> final download bandwidth is about 12Mbps, Steam on PC1 downloads at
> 1.4-1.5MB/s while downloading a file on PC2 seems to max out at ~250KB/s.
> From my understanding I should see each PC download at ~700KB/s, or am I
> mistaken?
>
> Assuming you measured good put in [M|K]iBytes this adds up to  1.5+0.25 =
> 1.75 * 1024^2 * 8 = 14680064 Bits or (1.4+0.25) * 8 *1024^2 / 1000^2 =
> 13.84 Mbps which seems a bit high for a 16Mbps ADSL link. I would ecpext
> something like 16 * (48/53)  * ((1500 - 8 - 20 -20) / (1500 + 32)) = 13.73
> Mbps TCP/IPv4 goodput… so you seem to be running close to theoretical
> maximum of your link (assuming I am not totally off with the overhead
> (estimated ADSL overhead on top of MTU: 6 destination MAC + 6 source MAC +
> 2 ethertype + 3 ATM LLC + 5 ATM SNAP + 2 ATM pad + 8 ATM AAL5 SAR 32
> bytes). But with your shaper set at 15Mbps without the atm option you will
> actually accept up to 15 * (53/48) = 16.5625 Mbps on the wire, which
> probably is above your link bandwidth. This fits well with the really low
> number of drops in your cake stats, you simply never have cake feel that
> shaping is needed?
>
> Best Regards
>
>
>
>
> >
> > On 20 April 2017 at 17:32, Jonathan Morton <chromatix99@gmail.com>
> wrote:
> >
> >> On 20 Apr, 2017, at 18:23, Dendari Marini <dendari92@gmail.com> wrote:
> >>
> >>> Could you post the output of calling “tc -s qdisc” here on the list
> please? That should allow to figure out what you actually told cake to do ;0
> >
> >> qdisc cake 8001: dev eth0 root refcnt 2 bandwidth 900Kbit diffserv3
> dual-srchost nat rtt 100.0ms raw
> >
> >> qdisc cake 8002: dev ifb4eth0 root refcnt 2 bandwidth 16Mbit diffserv3
> dual-dsthost nat ingress rtt 100.0ms raw
> >
> > Looks like most of your options are okay, including the correct “dual”
> modes and “ingress” mode in the right place.  However, I think you need to
> adjust your bandwidth and overhead settings, otherwise Cake isn’t reliably
> in control of the bottleneck queues.  Try these to begin with:
> >
> > … bandwidth 850Kbit conservative dual-srchost nat
> >
> > … bandwidth 15Mbit conservative dual-dsthost nat ingress
> >
> > That should give you correct operation, and you can fine-tune from there.
> >
> > - Jonathan Morton
> >
> >
> >
>
>

[-- Attachment #2: Type: text/html, Size: 5695 bytes --]

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Sebastian Moeller]

Hi Dendari,


> On Apr 21, 2017, at 10:34, Dendari Marini <dendari92@gmail.com> wrote:
> 
> Hello, thanks for all of your replies.
> 
> First of all, my connection encapsulation should be ATM LLC and it can actually reach up to 17.5/1 Mbps, but that's kinda best case scenario which is why I wanted to play it safe with just 16/.9 (which I should reach more consistently).

	Okay, I read ATM LLC together with the fact that you have a choice of et0 and pppoe0 that you hace 32 bytes of overhead (well including the 8 byte pppoe header it would be 40):

	case 40
		disp('Connection: PPPoE, LLC/SNAP RFC-2684');
		disp('Protocol (bytes): PPP (2), PPPoE (6), Ethernet Header (14), ATM LLC (3), ATM SNAP (5), ATM pad (2), ATM AAL5 SAR (8) : Total 40');
		overhead_bytes_around_MTU = 32;
		overhead_bytes_in_MTU = 8;

So please add “atm overhead 32" to cake on eth0 or “atm overhead 40” to cake instances on pppoe (these packets do not have the PPPoE header added yet and hence appear 8 bytes to small).

With that information you can expect the following maximum TCP/IPv4 goodput:
17.5 * (48/53) * ((1500 - 8 - 20 -20)/(1500 + 32)) = 15.02 Mbps 
or 
(17.5 * (48/53) * ((1500 - 8 - 20 -20)/(1500 + 32))) * 1000^2/1024^2 = 14.32  Mibps or 1.79 MBps, still pretty close to the 1.75MBps you seem to measure. Please note that if you do not specify the “atm” keyword your shaper will not account for the static 100-100*48/53 = 9.43 % rate increase going from ethernet frames to atm’s AAL5, so to account for that part alone you would need to set your shaper to 17.5*0.9 = 15.75 Mbps, but that completely ignores the other overhead (and the interger atm cell per ethernet frame rule that AAL5 insists upon)


> Back to the Steam issue. Unfortunately I can't seem to get really consistent results, mainly because sometimes it's downloading the game from just a few connections and other times it's downloading from dozens and dozens connections. The latter is the one giving me more issues both in terms of latency/packet loss and in terms of evenly splitting the bandwidth across the hosts.

	Question: if you set the shaper’s to 50% of line rate (8.75/0.5?) do you still see that unfairness? And if you add “atm overhead 40” to cake on pppoe0 and set the shaper to 90% of line rates (15.75/0.9) how does the Steam affect per-host fairness? Also how transient are these connections team uses?

> 
> One thing that seems to give better results is changing the interface where Cake is used from eth0 to pppoe0. When I used fq_codel it seemed to give better results when using eth0 and so I went ahead and did the same with Cake.

	As far as I can tell cake can drill down to the required IP/TCP/UDP fields independent of whether there are VLAN tags or PPPoE headers so cake should not care (except for the different overhead specifications you need to add as stated above). BUT if instantiated on eth0 cake will see pppoe LCP packets and might decide to drop them, which can take down the link, so out of caution I would still instantiate on pppoe in your case.


> 
> Anyway more testing needed, will report if I notice any consistent result.
> 
> By the way this is the thread I opened on the Ubiquiti forums talking about this issue (not sure if it can give you some more info): https://community.ubnt.com/t5/EdgeMAX/Smart-Queue-seemingly-not-working-for-Steam-downloads/td-p/1890405
> Also the thread where I got Cake for the ER-X from: https://community.ubnt.com/t5/EdgeMAX/Cake-and-FQ-PIE-compiled-for-the-EdgeRouter-devices/td-p/1679844
> 
> On 20 April 2017 at 20:36, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> > On Apr 20, 2017, at 18:05, Dendari Marini <dendari92@gmail.com> wrote:
> >
> > Hello, thanks for your reply.
> >
> > Looks like most of your options are okay, including the correct “dual” modes and “ingress” mode in the right place.  However, I think you need to adjust your bandwidth and overhead settings, otherwise Cake isn’t reliably in control of the bottleneck queues.  Try these to begin with:
> >
> > … bandwidth 850Kbit conservative dual-srchost nat
> >
> > … bandwidth 15Mbit conservative dual-dsthost nat ingress
> >
> > That should give you correct operation, and you can fine-tune from there.
> >
> > Just did quick test with your settings. First thing I noticed is my final download bandwidth is about 12Mbps, Steam on PC1 downloads at 1.4-1.5MB/s while downloading a file on PC2 seems to max out at ~250KB/s. From my understanding I should see each PC download at ~700KB/s, or am I mistaken?
> 
> Assuming you measured good put in [M|K]iBytes this adds up to  1.5+0.25 = 1.75 * 1024^2 * 8 = 14680064 Bits or (1.4+0.25) * 8 *1024^2 / 1000^2 = 13.84 Mbps which seems a bit high for a 16Mbps ADSL link. I would ecpext something like 16 * (48/53)  * ((1500 - 8 - 20 -20) / (1500 + 32)) = 13.73 Mbps TCP/IPv4 goodput… so you seem to be running close to theoretical maximum of your link (assuming I am not totally off with the overhead (estimated ADSL overhead on top of MTU: 6 destination MAC + 6 source MAC + 2 ethertype + 3 ATM LLC + 5 ATM SNAP + 2 ATM pad + 8 ATM AAL5 SAR 32 bytes). But with your shaper set at 15Mbps without the atm option you will actually accept up to 15 * (53/48) = 16.5625 Mbps on the wire, which probably is above your link bandwidth. This fits well with the really low number of drops in your cake stats, you simply never have cake feel that shaping is needed?
> 
> Best Regards
> 
> 
> 
> 
> >
> > On 20 April 2017 at 17:32, Jonathan Morton <chromatix99@gmail.com> wrote:
> >
> >> On 20 Apr, 2017, at 18:23, Dendari Marini <dendari92@gmail.com> wrote:
> >>
> >>> Could you post the output of calling “tc -s qdisc” here on the list please? That should allow to figure out what you actually told cake to do ;0
> >
> >> qdisc cake 8001: dev eth0 root refcnt 2 bandwidth 900Kbit diffserv3 dual-srchost nat rtt 100.0ms raw
> >
> >> qdisc cake 8002: dev ifb4eth0 root refcnt 2 bandwidth 16Mbit diffserv3 dual-dsthost nat ingress rtt 100.0ms raw
> >
> > Looks like most of your options are okay, including the correct “dual” modes and “ingress” mode in the right place.  However, I think you need to adjust your bandwidth and overhead settings, otherwise Cake isn’t reliably in control of the bottleneck queues.  Try these to begin with:
> >
> > … bandwidth 850Kbit conservative dual-srchost nat
> >
> > … bandwidth 15Mbit conservative dual-dsthost nat ingress
> >
> > That should give you correct operation, and you can fine-tune from there.
> >
> > - Jonathan Morton
> >
> >
> >
> 
> 

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Dendari Marini]

[-- Attachment #1: Type: text/plain, Size: 5460 bytes --]

Hello, I have an update.

The download test (PC1 downloading Steam game, PC2 downloading a file)
seems to work fine now, even while using higher bandwidth value. So that
seems resolved, probably the interface change was the fix.

Now about the gaming test (PC1 downloading Steam game, PC2 online gaming),
this is the one I'm mostly looking forward to fix and the main reason I
bought the ER-X. Unfortunately Cake doesn't seem to do much in this case,
specifically I'm still getting high latency spikes and packet loss on PC2.
One weird thing: starting a download on PC2 will decrease the
lantecy/p.loss issues, not sure if it's because of the Steam speed being
halved on PC1 or something else.

On 21 April 2017 at 10:34, Dendari Marini <dendari92@gmail.com> wrote:

> Hello, thanks for all of your replies.
>
> First of all, my connection encapsulation should be ATM LLC and it can
> actually reach up to 17.5/1 Mbps, but that's kinda best case scenario which
> is why I wanted to play it safe with just 16/.9 (which I should reach more
> consistently).
>
> Back to the Steam issue. Unfortunately I can't seem to get really
> consistent results, mainly because sometimes it's downloading the game from
> just a few connections and other times it's downloading from dozens and
> dozens connections. The latter is the one giving me more issues both in
> terms of latency/packet loss and in terms of evenly splitting the bandwidth
> across the hosts.
>
> One thing that seems to give better results is changing the interface
> where Cake is used from eth0 to pppoe0. When I used fq_codel it seemed to
> give better results when using eth0 and so I went ahead and did the same
> with Cake.
>
> Anyway more testing needed, will report if I notice any consistent result.
>
> By the way this is the thread I opened on the Ubiquiti forums talking
> about this issue (not sure if it can give you some more info):
> https://community.ubnt.com/t5/EdgeMAX/Smart-Queue-seemingly-
> not-working-for-Steam-downloads/td-p/1890405
> Also the thread where I got Cake for the ER-X from:
> https://community.ubnt.com/t5/EdgeMAX/Cake-and-FQ-PIE-compiled-for-the-
> EdgeRouter-devices/td-p/1679844
>
> On 20 April 2017 at 20:36, Sebastian Moeller <moeller0@gmx.de> wrote:
>
>>
>> > On Apr 20, 2017, at 18:05, Dendari Marini <dendari92@gmail.com> wrote:
>> >
>> > Hello, thanks for your reply.
>> >
>> > Looks like most of your options are okay, including the correct “dual”
>> modes and “ingress” mode in the right place.  However, I think you need to
>> adjust your bandwidth and overhead settings, otherwise Cake isn’t reliably
>> in control of the bottleneck queues.  Try these to begin with:
>> >
>> > … bandwidth 850Kbit conservative dual-srchost nat
>> >
>> > … bandwidth 15Mbit conservative dual-dsthost nat ingress
>> >
>> > That should give you correct operation, and you can fine-tune from
>> there.
>> >
>> > Just did quick test with your settings. First thing I noticed is my
>> final download bandwidth is about 12Mbps, Steam on PC1 downloads at
>> 1.4-1.5MB/s while downloading a file on PC2 seems to max out at ~250KB/s.
>> From my understanding I should see each PC download at ~700KB/s, or am I
>> mistaken?
>>
>> Assuming you measured good put in [M|K]iBytes this adds up to  1.5+0.25 =
>> 1.75 * 1024^2 * 8 = 14680064 Bits or (1.4+0.25) * 8 *1024^2 / 1000^2 =
>> 13.84 Mbps which seems a bit high for a 16Mbps ADSL link. I would ecpext
>> something like 16 * (48/53)  * ((1500 - 8 - 20 -20) / (1500 + 32)) = 13.73
>> Mbps TCP/IPv4 goodput… so you seem to be running close to theoretical
>> maximum of your link (assuming I am not totally off with the overhead
>> (estimated ADSL overhead on top of MTU: 6 destination MAC + 6 source MAC +
>> 2 ethertype + 3 ATM LLC + 5 ATM SNAP + 2 ATM pad + 8 ATM AAL5 SAR 32
>> bytes). But with your shaper set at 15Mbps without the atm option you will
>> actually accept up to 15 * (53/48) = 16.5625 Mbps on the wire, which
>> probably is above your link bandwidth. This fits well with the really low
>> number of drops in your cake stats, you simply never have cake feel that
>> shaping is needed?
>>
>> Best Regards
>>
>>
>>
>>
>> >
>> > On 20 April 2017 at 17:32, Jonathan Morton <chromatix99@gmail.com>
>> wrote:
>> >
>> >> On 20 Apr, 2017, at 18:23, Dendari Marini <dendari92@gmail.com> wrote:
>> >>
>> >>> Could you post the output of calling “tc -s qdisc” here on the list
>> please? That should allow to figure out what you actually told cake to do ;0
>> >
>> >> qdisc cake 8001: dev eth0 root refcnt 2 bandwidth 900Kbit diffserv3
>> dual-srchost nat rtt 100.0ms raw
>> >
>> >> qdisc cake 8002: dev ifb4eth0 root refcnt 2 bandwidth 16Mbit diffserv3
>> dual-dsthost nat ingress rtt 100.0ms raw
>> >
>> > Looks like most of your options are okay, including the correct “dual”
>> modes and “ingress” mode in the right place.  However, I think you need to
>> adjust your bandwidth and overhead settings, otherwise Cake isn’t reliably
>> in control of the bottleneck queues.  Try these to begin with:
>> >
>> > … bandwidth 850Kbit conservative dual-srchost nat
>> >
>> > … bandwidth 15Mbit conservative dual-dsthost nat ingress
>> >
>> > That should give you correct operation, and you can fine-tune from
>> there.
>> >
>> > - Jonathan Morton
>> >
>> >
>> >
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 7159 bytes --]

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Dendari Marini]

[-- Attachment #1: Type: text/plain, Size: 7956 bytes --]

Hi Sebastian,

Only now I noticed your message, it seems we were writing at the same time.

So please add “atm overhead 32" to cake on eth0 or “atm overhead 40” to
> cake instances on pppoe (these packets do not have the PPPoE header added
> yet and hence appear 8 bytes to small).


Thanks for your help, will definitely use them. Just wondering if I use
"pppoe-vcmux/bridged-llcsnap" on eth0 or "pppoe-llcsnap" on pppoe0 would
have the same effect? Or are there some other "under-the-hood" changes when
using them?

Question: if you set the shaper’s to 50% of line rate (8.75/0.5?) do you
> still see that unfairness? And if you add “atm overhead 40” to cake on
> pppoe0 and set the shaper to 90% of line rates (15.75/0.9) how does the
> Steam affect per-host fairness? Also how transient are these connections
> team uses?


Actually did more testing about this and it seems that as far I have set
the bandwidth to ~15Mbps (so ~15% less of my max speed) and use the "nat"
parameter, the per-host fairness works even without the "dual-host" and
"overhead" parameters. I definitely find this very interesting, is this
behaviour caused by the way Steam downloads games?

As far as I can tell cake can drill down to the required IP/TCP/UDP fields
> independent of whether there are VLAN tags or PPPoE headers so cake should
> not care (except for the different overhead specifications you need to add
> as stated above). BUT if instantiated on eth0 cake will see pppoe LCP
> packets and might decide to drop them, which can take down the link, so out
> of caution I would still instantiate on pppoe in your case.


Yeah, with further testing it seems the interface wasn't the culprit but
I'll still do all my testing on pppoe0 just to be safe.

Anyway I was wondering if there's some kind of manual for Cake and the
various parameters, I'm looking to set it up best way possible but there
are some parameters which I'm not sure what they do (one of them being
"ingress"). Also while reading on the bufferbloat.net Cake page I noticed a
possible "fix" for BitTorrent (by setting it as "background",
https://www.bufferbloat.net/projects/codel/wiki/Cake/#diffserv-support),
I'm wondering if this can be done with Steam too?

On 21 April 2017 at 15:27, Dendari Marini <dendari92@gmail.com> wrote:

> Hello, I have an update.
>
> The download test (PC1 downloading Steam game, PC2 downloading a file)
> seems to work fine now, even while using higher bandwidth value. So that
> seems resolved, probably the interface change was the fix.
>
> Now about the gaming test (PC1 downloading Steam game, PC2 online gaming),
> this is the one I'm mostly looking forward to fix and the main reason I
> bought the ER-X. Unfortunately Cake doesn't seem to do much in this case,
> specifically I'm still getting high latency spikes and packet loss on PC2.
> One weird thing: starting a download on PC2 will decrease the
> lantecy/p.loss issues, not sure if it's because of the Steam speed being
> halved on PC1 or something else.
>
> On 21 April 2017 at 10:34, Dendari Marini <dendari92@gmail.com> wrote:
>
>> Hello, thanks for all of your replies.
>>
>> First of all, my connection encapsulation should be ATM LLC and it can
>> actually reach up to 17.5/1 Mbps, but that's kinda best case scenario which
>> is why I wanted to play it safe with just 16/.9 (which I should reach more
>> consistently).
>>
>> Back to the Steam issue. Unfortunately I can't seem to get really
>> consistent results, mainly because sometimes it's downloading the game from
>> just a few connections and other times it's downloading from dozens and
>> dozens connections. The latter is the one giving me more issues both in
>> terms of latency/packet loss and in terms of evenly splitting the bandwidth
>> across the hosts.
>>
>> One thing that seems to give better results is changing the interface
>> where Cake is used from eth0 to pppoe0. When I used fq_codel it seemed to
>> give better results when using eth0 and so I went ahead and did the same
>> with Cake.
>>
>> Anyway more testing needed, will report if I notice any consistent result.
>>
>> By the way this is the thread I opened on the Ubiquiti forums talking
>> about this issue (not sure if it can give you some more info):
>> https://community.ubnt.com/t5/EdgeMAX/Smart-Queue-seemingly-
>> not-working-for-Steam-downloads/td-p/1890405
>> Also the thread where I got Cake for the ER-X from:
>> https://community.ubnt.com/t5/EdgeMAX/Cake-and-FQ-PIE-
>> compiled-for-the-EdgeRouter-devices/td-p/1679844
>>
>> On 20 April 2017 at 20:36, Sebastian Moeller <moeller0@gmx.de> wrote:
>>
>>>
>>> > On Apr 20, 2017, at 18:05, Dendari Marini <dendari92@gmail.com> wrote:
>>> >
>>> > Hello, thanks for your reply.
>>> >
>>> > Looks like most of your options are okay, including the correct “dual”
>>> modes and “ingress” mode in the right place.  However, I think you need to
>>> adjust your bandwidth and overhead settings, otherwise Cake isn’t reliably
>>> in control of the bottleneck queues.  Try these to begin with:
>>> >
>>> > … bandwidth 850Kbit conservative dual-srchost nat
>>> >
>>> > … bandwidth 15Mbit conservative dual-dsthost nat ingress
>>> >
>>> > That should give you correct operation, and you can fine-tune from
>>> there.
>>> >
>>> > Just did quick test with your settings. First thing I noticed is my
>>> final download bandwidth is about 12Mbps, Steam on PC1 downloads at
>>> 1.4-1.5MB/s while downloading a file on PC2 seems to max out at ~250KB/s.
>>> From my understanding I should see each PC download at ~700KB/s, or am I
>>> mistaken?
>>>
>>> Assuming you measured good put in [M|K]iBytes this adds up to  1.5+0.25
>>> = 1.75 * 1024^2 * 8 = 14680064 Bits or (1.4+0.25) * 8 *1024^2 / 1000^2 =
>>> 13.84 Mbps which seems a bit high for a 16Mbps ADSL link. I would ecpext
>>> something like 16 * (48/53)  * ((1500 - 8 - 20 -20) / (1500 + 32)) = 13.73
>>> Mbps TCP/IPv4 goodput… so you seem to be running close to theoretical
>>> maximum of your link (assuming I am not totally off with the overhead
>>> (estimated ADSL overhead on top of MTU: 6 destination MAC + 6 source MAC +
>>> 2 ethertype + 3 ATM LLC + 5 ATM SNAP + 2 ATM pad + 8 ATM AAL5 SAR 32
>>> bytes). But with your shaper set at 15Mbps without the atm option you will
>>> actually accept up to 15 * (53/48) = 16.5625 Mbps on the wire, which
>>> probably is above your link bandwidth. This fits well with the really low
>>> number of drops in your cake stats, you simply never have cake feel that
>>> shaping is needed?
>>>
>>> Best Regards
>>>
>>>
>>>
>>>
>>> >
>>> > On 20 April 2017 at 17:32, Jonathan Morton <chromatix99@gmail.com>
>>> wrote:
>>> >
>>> >> On 20 Apr, 2017, at 18:23, Dendari Marini <dendari92@gmail.com>
>>> wrote:
>>> >>
>>> >>> Could you post the output of calling “tc -s qdisc” here on the list
>>> please? That should allow to figure out what you actually told cake to do ;0
>>> >
>>> >> qdisc cake 8001: dev eth0 root refcnt 2 bandwidth 900Kbit diffserv3
>>> dual-srchost nat rtt 100.0ms raw
>>> >
>>> >> qdisc cake 8002: dev ifb4eth0 root refcnt 2 bandwidth 16Mbit
>>> diffserv3 dual-dsthost nat ingress rtt 100.0ms raw
>>> >
>>> > Looks like most of your options are okay, including the correct “dual”
>>> modes and “ingress” mode in the right place.  However, I think you need to
>>> adjust your bandwidth and overhead settings, otherwise Cake isn’t reliably
>>> in control of the bottleneck queues.  Try these to begin with:
>>> >
>>> > … bandwidth 850Kbit conservative dual-srchost nat
>>> >
>>> > … bandwidth 15Mbit conservative dual-dsthost nat ingress
>>> >
>>> > That should give you correct operation, and you can fine-tune from
>>> there.
>>> >
>>> > - Jonathan Morton
>>> >
>>> >
>>> >
>>>
>>>
>>
>

[-- Attachment #2: Type: text/html, Size: 10913 bytes --]

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Jonathan Morton]

>> So please add “atm overhead 32" to cake on eth0 or “atm overhead 40” to cake instances on pppoe (these packets do not have the PPPoE header added yet and hence appear 8 bytes to small).
> 
> Thanks for your help, will definitely use them. Just wondering if I use "pppoe-vcmux/bridged-llcsnap" on eth0 or "pppoe-llcsnap" on pppoe0 would have the same effect? Or are there some other "under-the-hood" changes when using them?

On the pppoe interface, use pppoe-vcmux if your modem is set to use VC-MUX, or pppoe-llcsnap if it’s set to use LLC-SNAP (they might be described using slightly different terms, but should still be recognisable as one or the other).  This probably depends on your ISP, and may further vary regionally within the same ISP.

I really prefer to use the self-explanatory keywords (which is why I added them in the first place) instead of opaque magic numbers.  This is a point on which Sebastian has long disagreed with me.

>> Question: if you set the shaper’s to 50% of line rate (8.75/0.5?) do you still see that unfairness? And if you add “atm overhead 40” to cake on pppoe0 and set the shaper to 90% of line rates (15.75/0.9) how does the Steam affect per-host fairness? Also how transient are these connections team uses?
> 
> Actually did more testing about this and it seems that as far I have set the bandwidth to ~15Mbps (so ~15% less of my max speed) and use the "nat" parameter, the per-host fairness works even without the "dual-host" and "overhead" parameters. I definitely find this very interesting, is this behaviour caused by the way Steam downloads games?

By default, Cake uses triple-isolate mode, which uses information about both source and destination hosts to perform per-host isolation; this usually works well regardless of which side of the connection has the LAN hosts.  The “dual” modes let you specify that fact explicitly, making it a little more robust and predictable.

Without overhead compensation, Cake will actually use more of the physical link than it thinks it does - by default it only accounts for raw IP or Ethernet packets, depending on the type of interface it’s attached to.  With full-size packets as in a bulk download, the difference is relatively small, so the 15% margin is just about sufficient to make things work.  But with small packets mixed in, the difference grows, such that Cake might no longer control the bottleneck with some traffic mixes.

The “conservative” keyword I recommended earlier (which is exactly equivalent to Sebastian’s recommendation of “atm overhead 48”) reverses that situation; Cake will then always end up using *less* of the physical link than it accounts for, which is safe for troubleshooting with.  The keyword is there specifically so that we do’t have to figure out the precise overhead profile before tackling more substantive issues.

At any rate, it has nothing to do with Steam specifically.

>> As far as I can tell cake can drill down to the required IP/TCP/UDP fields independent of whether there are VLAN tags or PPPoE headers so cake should not care (except for the different overhead specifications you need to add as stated above). BUT if instantiated on eth0 cake will see pppoe LCP packets and might decide to drop them, which can take down the link, so out of caution I would still instantiate on pppoe in your case.
> 
> Yeah, with further testing it seems the interface wasn't the culprit but I'll still do all my testing on pppoe0 just to be safe.
> 
> Anyway I was wondering if there's some kind of manual for Cake and the various parameters, I'm looking to set it up best way possible but there are some parameters which I'm not sure what they do (one of them being "ingress”).

With the correct version of iproute2 installed, just issue “man tc-cake”.  That’s the official documentation.

Currently it doesn’t have the ingress keyword yet.  That’ll be fixed soon.

> Also while reading on the bufferbloat.net Cake page I noticed a possible "fix" for BitTorrent (by setting it as "background", https://www.bufferbloat.net/projects/codel/wiki/Cake/#diffserv-support), I'm wondering if this can be done with Steam too?

It’s possible, if you can figure out which traffic is Steam in the first place, and write filters to match on it.  This is complicated by the fact that Valve runs a sophisticated CDN to handle their rather impressive bandwidth load.

 - Jonathan Morton

Re: [Cake] Getting Cake to work better with Steam and similar applications

[xnoreq]

[-- Attachment #1: Type: text/plain, Size: 5484 bytes --]

FYI, when I looked at Steam traffic a few years ago it was very bursty,
meaning that there is nothing transmitted for a short period and then
there's a burst that uses up all link bandwidth for a short while.
Internet was unusable without limiting Steam downloads quite a lot.

Now I don't have that problem anymore. Cake's ingress mode works almost as
well as I have expected (still need to set bw about 1Mbps lower on a 20Mbps
link - but that's ok).
Maybe the Steam CDN now also uses a saner network scheduler (like fq with
pacing). I'd guess so anyway.


On 22 Apr 2017 11:36, "Jonathan Morton" <chromatix99@gmail.com> wrote:

> >> So please add “atm overhead 32" to cake on eth0 or “atm overhead 40” to
> cake instances on pppoe (these packets do not have the PPPoE header added
> yet and hence appear 8 bytes to small).
> >
> > Thanks for your help, will definitely use them. Just wondering if I use
> "pppoe-vcmux/bridged-llcsnap" on eth0 or "pppoe-llcsnap" on pppoe0 would
> have the same effect? Or are there some other "under-the-hood" changes when
> using them?
>
> On the pppoe interface, use pppoe-vcmux if your modem is set to use
> VC-MUX, or pppoe-llcsnap if it’s set to use LLC-SNAP (they might be
> described using slightly different terms, but should still be recognisable
> as one or the other).  This probably depends on your ISP, and may further
> vary regionally within the same ISP.
>
> I really prefer to use the self-explanatory keywords (which is why I added
> them in the first place) instead of opaque magic numbers.  This is a point
> on which Sebastian has long disagreed with me.
>
> >> Question: if you set the shaper’s to 50% of line rate (8.75/0.5?) do
> you still see that unfairness? And if you add “atm overhead 40” to cake on
> pppoe0 and set the shaper to 90% of line rates (15.75/0.9) how does the
> Steam affect per-host fairness? Also how transient are these connections
> team uses?
> >
> > Actually did more testing about this and it seems that as far I have set
> the bandwidth to ~15Mbps (so ~15% less of my max speed) and use the "nat"
> parameter, the per-host fairness works even without the "dual-host" and
> "overhead" parameters. I definitely find this very interesting, is this
> behaviour caused by the way Steam downloads games?
>
> By default, Cake uses triple-isolate mode, which uses information about
> both source and destination hosts to perform per-host isolation; this
> usually works well regardless of which side of the connection has the LAN
> hosts.  The “dual” modes let you specify that fact explicitly, making it a
> little more robust and predictable.
>
> Without overhead compensation, Cake will actually use more of the physical
> link than it thinks it does - by default it only accounts for raw IP or
> Ethernet packets, depending on the type of interface it’s attached to.
> With full-size packets as in a bulk download, the difference is relatively
> small, so the 15% margin is just about sufficient to make things work.  But
> with small packets mixed in, the difference grows, such that Cake might no
> longer control the bottleneck with some traffic mixes.
>
> The “conservative” keyword I recommended earlier (which is exactly
> equivalent to Sebastian’s recommendation of “atm overhead 48”) reverses
> that situation; Cake will then always end up using *less* of the physical
> link than it accounts for, which is safe for troubleshooting with.  The
> keyword is there specifically so that we do’t have to figure out the
> precise overhead profile before tackling more substantive issues.
>
> At any rate, it has nothing to do with Steam specifically.
>
> >> As far as I can tell cake can drill down to the required IP/TCP/UDP
> fields independent of whether there are VLAN tags or PPPoE headers so cake
> should not care (except for the different overhead specifications you need
> to add as stated above). BUT if instantiated on eth0 cake will see pppoe
> LCP packets and might decide to drop them, which can take down the link, so
> out of caution I would still instantiate on pppoe in your case.
> >
> > Yeah, with further testing it seems the interface wasn't the culprit but
> I'll still do all my testing on pppoe0 just to be safe.
> >
> > Anyway I was wondering if there's some kind of manual for Cake and the
> various parameters, I'm looking to set it up best way possible but there
> are some parameters which I'm not sure what they do (one of them being
> "ingress”).
>
> With the correct version of iproute2 installed, just issue “man tc-cake”.
> That’s the official documentation.
>
> Currently it doesn’t have the ingress keyword yet.  That’ll be fixed soon.
>
> > Also while reading on the bufferbloat.net Cake page I noticed a
> possible "fix" for BitTorrent (by setting it as "background",
> https://www.bufferbloat.net/projects/codel/wiki/Cake/#diffserv-support),
> I'm wondering if this can be done with Steam too?
>
> It’s possible, if you can figure out which traffic is Steam in the first
> place, and write filters to match on it.  This is complicated by the fact
> that Valve runs a sophisticated CDN to handle their rather impressive
> bandwidth load.
>
>  - Jonathan Morton
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>

[-- Attachment #2: Type: text/html, Size: 6371 bytes --]

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Tristan Seligmann]

[-- Attachment #1: Type: text/plain, Size: 1645 bytes --]

On Sat, 22 Apr 2017 at 11:36 Jonathan Morton <chromatix99@gmail.com> wrote:

> With the correct version of iproute2 installed, just issue “man tc-cake”.
> That’s the official documentation.
>

FWIW, since I gather the OP is using Lochnair's builds on an ER-X:
unfortunately EdgeMAX does not ship the man utility, but there is an HTML
rendered version of the man page here:
https://dl.lochnair.net/Bufferbloat/Cake%20(Cobalt)/tc-cake.8.html

It’s possible, if you can figure out which traffic is Steam in the first
> place, and write filters to match on it.  This is complicated by the fact
> that Valve runs a sophisticated CDN to handle their rather impressive
> bandwidth load.
>

I haven't succeeded at this before, as Steam's content server network is
large and complicated making it difficult to identify the servers by IP
address, and the actual traffic itself is TCP over standard HTTP(S) ports.

The behaviour I've seen in the past is that the Steam client will maintain
10 connections (each to a different content server), starting with the
region configured in settings, but cycling through various servers to find
the fastest ones. (The number of connections may depend on your detected
connection speed, I haven't tried this on very fast or very slow
connections) This tends to produce pretty nice results for your download
performance, but of course per-flow fairness alone means it'll tend to
drown out single flows. cake's dualhost/triple-isolate fairness works
pretty well for me, though, at least in terms of shielding other hosts on
the network from the host doing the Steam downloading.

[-- Attachment #2: Type: text/html, Size: 2163 bytes --]

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Andy Furniss]

Jonathan Morton wrote:
>>> So please add “atm overhead 32" to cake on eth0 or “atm overhead
>>>  40” to cake instances on pppoe (these packets do not have the 
>>> PPPoE header added yet and hence appear 8 bytes to small).
>> 
>> Thanks for your help, will definitely use them. Just wondering if I
>> use "pppoe-vcmux/bridged-llcsnap" on eth0 or "pppoe-llcsnap" on 
>> pppoe0 would have the same effect? Or are there some other 
>> "under-the-hood" changes when using them?
> 
> On the pppoe interface, use pppoe-vcmux if your modem is set to use 
> VC-MUX, or pppoe-llcsnap if it’s set to use LLC-SNAP (they might be 
> described using slightly different terms, but should still be 
> recognisable as one or the other).  This probably depends on your 
> ISP, and may further vary regionally within the same ISP.
> 
> I really prefer to use the self-explanatory keywords (which is why I
>  added them in the first place) instead of opaque magic numbers. This
> is a point on which Sebastian has long disagreed with me.

Either way (or maybe not!), what about the observation that attaching
cake on pppoe, for me at least, required the use of the raw param due to
the "auto compensation" mechanism seeing pppoe as +8 when the actual
packets are just ip len so not +8.

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Andy Furniss]

Andy Furniss wrote:
> Jonathan Morton wrote:
>>>> So please add “atm overhead 32" to cake on eth0 or “atm
>>>> overhead 40” to cake instances on pppoe (these packets do not
>>>> have the PPPoE header added yet and hence appear 8 bytes to
>>>> small).
>>> 
>>> Thanks for your help, will definitely use them. Just wondering if
>>> I use "pppoe-vcmux/bridged-llcsnap" on eth0 or "pppoe-llcsnap" on
>>>  pppoe0 would have the same effect? Or are there some other 
>>> "under-the-hood" changes when using them?
>> 
>> On the pppoe interface, use pppoe-vcmux if your modem is set to use
>>  VC-MUX, or pppoe-llcsnap if it’s set to use LLC-SNAP (they might
>> be described using slightly different terms, but should still be 
>> recognisable as one or the other).  This probably depends on your
>> ISP, and may further vary regionally within the same ISP.
>> 
>> I really prefer to use the self-explanatory keywords (which is why
>> I added them in the first place) instead of opaque magic numbers.
>> This is a point on which Sebastian has long disagreed with me.
> 
> Either way (or maybe not!), what about the observation that
> attaching cake on pppoe, for me at least, required the use of the raw
> param due to the "auto compensation" mechanism seeing pppoe as +8
> when the actual packets are just ip len so not +8.

My case is not atm though, perhaps the atm param cancels out all
other auto overhead compensation?

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Andy Furniss]

Andy Furniss wrote:
> Andy Furniss wrote:
>> Jonathan Morton wrote:
>>>>> So please add “atm overhead 32" to cake on eth0 or “atm 
>>>>> overhead 40” to cake instances on pppoe (these packets do not
>>>>> have the PPPoE header added yet and hence appear 8 bytes to
>>>>> small).
>>>> 
>>>> Thanks for your help, will definitely use them. Just wondering 
>>>> if I use "pppoe-vcmux/bridged-llcsnap" on eth0 or 
>>>> "pppoe-llcsnap" on pppoe0 would have the same effect? Or are 
>>>> there some other "under-the-hood" changes when using them?
>>> 
>>> On the pppoe interface, use pppoe-vcmux if your modem is set to 
>>> use VC-MUX, or pppoe-llcsnap if it’s set to use LLC-SNAP (they 
>>> might be described using slightly different terms, but should 
>>> still be recognisable as one or the other).  This probably 
>>> depends on your ISP, and may further vary regionally within the 
>>> same ISP.
>>> 
>>> I really prefer to use the self-explanatory keywords (which is 
>>> why I added them in the first place) instead of opaque magic 
>>> numbers. This is a point on which Sebastian has long disagreed 
>>> with me.
>> 
>> Either way (or maybe not!), what about the observation that 
>> attaching cake on pppoe, for me at least, required the use of the 
>> raw param due to the "auto compensation" mechanism seeing pppoe as 
>> +8 when the actual packets are just ip len so not +8.
> 
> My case is not atm though, perhaps the atm param cancels out all 
> other auto overhead compensation?

Though that wouldn't really make sense for other use cases like shaping
on a real eth for some remote atm link.

Checking my not atm pppoe it seems without the raw param I would
actually be 14 + 8 bytes wrong.

tc qdisc add dev ppp0 handle 1:0 root cake bandwidth 19690kbit raw
overhead 34 diffserv4 dual-srchost nat rtt 200ms

results in -

qdisc cake 1: root refcnt 2 bandwidth 19690Kbit diffserv4 dual-srchost
nat rtt 200.0ms noatm overhead 56 via-ethernet


Random rambling on dual-srchost vs default triple - if I were to shape
ingress, which I don't because my ISP does it again now, I think
dual-srchost would be better in practice for home use. This being
because the CDNs of youtube or TV streamers - well blippers :-) like the
BBC netflix etc. may well mean that users that should be separate get
somehow lumped together - basically any assertion that it's rare for
different users to be downloading from the same remote server doesn't
really apply to streaming services.

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Dave Taht]

My hope would be that somehow someone would get through to steam and
encourage them to not have 10 flows open at a time.

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Tristan Seligmann]

[-- Attachment #1: Type: text/plain, Size: 365 bytes --]

On Sat, 22 Apr 2017 at 18:45 Dave Taht <dave.taht@gmail.com> wrote:

> My hope would be that somehow someone would get through to steam and
> encourage them to not have 10 flows open at a time.
>

For various reasons (that I'm sure members of this list are acquainted
with), this is probably necessary to get good download performance for most
of their userbase :(

[-- Attachment #2: Type: text/html, Size: 634 bytes --]

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Andy Furniss]

Andy Furniss wrote:

>>>>>> So please add “atm overhead 32" to cake on eth0 or “atm 
>>>>>> overhead 40” to cake instances on pppoe (these packets do 
>>>>>> not have the PPPoE header added yet and hence appear 8 
>>>>>> bytes to small).

Ugh, so I totally missed that Dendari was already using the raw
parameter and will still need to when adding "atm" (on pppoe, but not on
eth0) AFAICT.

He must be using different cake/tc from me that doesn't print out the
overhead XX via-ethernet info.

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Sebastian Moeller]

> On Apr 22, 2017, at 11:36, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>>> So please add “atm overhead 32" to cake on eth0 or “atm overhead 40” to cake instances on pppoe (these packets do not have the PPPoE header added yet and hence appear 8 bytes to small).
>> 
>> Thanks for your help, will definitely use them. Just wondering if I use "pppoe-vcmux/bridged-llcsnap" on eth0 or "pppoe-llcsnap" on pppoe0 would have the same effect? Or are there some other "under-the-hood" changes when using them?
> 
> On the pppoe interface, use pppoe-vcmux if your modem is set to use VC-MUX, or pppoe-llcsnap if it’s set to use LLC-SNAP (they might be described using slightly different terms, but should still be recognisable as one or the other).  This probably depends on your ISP, and may further vary regionally within the same ISP.

	In my experience it is rather bothersome trying to get that information from one’s ISP, this is why I would recommend to follow the instructions on https://github.com/moeller0/ATM_overhead_detector and empirically measure the actual overhead. In that case one ends up with the numeric overhead, hence my inclination to use that number directly instead of looking into a table to translate that back into a symbolic keyword… especially since say for an overhead of 32 (and 36) there are two different encapsulation schees that add up to that number:

case 32
		disp('Connection: Bridged, LLC/SNAP RFC-1483/2684');
		disp('Protocol (bytes): Ethernet Header (14), ATM LLC (3), ATM SNAP (5), ATM pad (2), ATM AAL5 SAR (8) : Total 32');

		disp('Connection: PPPoE, VC/Mux RFC-2684');
		disp('Protocol (bytes): PPP (2), PPPoE (6), Ethernet Header (14), ATM pad (2), ATM AAL5 SAR (8) : Total 32’);

good luck divining which of those is in use if all you know is the numeric overhead...



> 
> I really prefer to use the self-explanatory keywords (which is why I added them in the first place) instead of opaque magic numbers.  This is a point on which Sebastian has long disagreed with me.

	True, but I am not going to re-hash that here again ;)

> 
>>> Question: if you set the shaper’s to 50% of line rate (8.75/0.5?) do you still see that unfairness? And if you add “atm overhead 40” to cake on pppoe0 and set the shaper to 90% of line rates (15.75/0.9) how does the Steam affect per-host fairness? Also how transient are these connections team uses?
>> 
>> Actually did more testing about this and it seems that as far I have set the bandwidth to ~15Mbps (so ~15% less of my max speed) and use the "nat" parameter, the per-host fairness works even without the "dual-host" and "overhead" parameters. I definitely find this very interesting, is this behaviour caused by the way Steam downloads games?
> 
> By default, Cake uses triple-isolate mode, which uses information about both source and destination hosts to perform per-host isolation; this usually works well regardless of which side of the connection has the LAN hosts.  The “dual” modes let you specify that fact explicitly, making it a little more robust and predictable.
> 
> Without overhead compensation, Cake will actually use more of the physical link than it thinks it does - by default it only accounts for raw IP or Ethernet packets, depending on the type of interface it’s attached to.  With full-size packets as in a bulk download, the difference is relatively small, so the 15% margin is just about sufficient to make things work.  But with small packets mixed in, the difference grows, such that Cake might no longer control the bottleneck with some traffic mixes.

	All true, to elaborate a bit on the ATM specific issue, due to AAL5’s insistence that each ethernet frame is packaged into an integer number of ATM cells (where the unused octets are simply padded out) the worst case is something like 100%, if a hypothetical packet would only require 49 Bytes, it will still require two ATM cells of 53 bytes...

> 
> The “conservative” keyword I recommended earlier (which is exactly equivalent to Sebastian’s recommendation of “atm overhead 48”) reverses that situation; Cake will then always end up using *less* of the physical link than it accounts for, which is safe for troubleshooting with.  The keyword is there specifically so that we do’t have to figure out the precise overhead profile before tackling more substantive issues.

	Due to the boundary observation above, one other option is to start with the shaper set to 50% of link rate, that should have sufficient wiggle room for all realistic overheads… (but honestly on a known ATM link I would always run the ATM_overhead_detector to get the precise number).

> 
> At any rate, it has nothing to do with Steam specifically.
> 
>>> As far as I can tell cake can drill down to the required IP/TCP/UDP fields independent of whether there are VLAN tags or PPPoE headers so cake should not care (except for the different overhead specifications you need to add as stated above). BUT if instantiated on eth0 cake will see pppoe LCP packets and might decide to drop them, which can take down the link, so out of caution I would still instantiate on pppoe in your case.
>> 
>> Yeah, with further testing it seems the interface wasn't the culprit but I'll still do all my testing on pppoe0 just to be safe.
>> 
>> Anyway I was wondering if there's some kind of manual for Cake and the various parameters, I'm looking to set it up best way possible but there are some parameters which I'm not sure what they do (one of them being "ingress”).
> 
> With the correct version of iproute2 installed, just issue “man tc-cake”.  That’s the official documentation.
> 
> Currently it doesn’t have the ingress keyword yet.  That’ll be fixed soon.
> 
>> Also while reading on the bufferbloat.net Cake page I noticed a possible "fix" for BitTorrent (by setting it as "background", https://www.bufferbloat.net/projects/codel/wiki/Cake/#diffserv-support), I'm wondering if this can be done with Steam too?
> 
> It’s possible, if you can figure out which traffic is Steam in the first place, and write filters to match on it.  This is complicated by the fact that Valve runs a sophisticated CDN to handle their rather impressive bandwidth load.
> 
> - Jonathan Morton
> 

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Dendari Marini]

[-- Attachment #1: Type: text/plain, Size: 7441 bytes --]

Hello, thank you all for your replies!

For the overhead I'm gonna use "pppoe-llcsnap" (or "overhead 40 atm), as I
believe it's the one which should work best for my connection.

About the per-host fairness download issue: while it's kinda resolved I
still feel like it's mainly related to Steam, as normally downloading files
from PC1 and PC2 halved the speed as expected even at full bandwidth (so no
overhead, no -15%).

Anyway back to Steam: assuming the IP addresses aren't a big issue, what
steps should I follow to start filtering its traffic so it's considered
background? Would the DPI from the ER-X be any helpful?

On 22 April 2017 at 18:47, Sebastian Moeller <moeller0@gmx.de> wrote:

>
> > On Apr 22, 2017, at 11:36, Jonathan Morton <chromatix99@gmail.com>
> wrote:
> >
> >>> So please add “atm overhead 32" to cake on eth0 or “atm overhead 40”
> to cake instances on pppoe (these packets do not have the PPPoE header
> added yet and hence appear 8 bytes to small).
> >>
> >> Thanks for your help, will definitely use them. Just wondering if I use
> "pppoe-vcmux/bridged-llcsnap" on eth0 or "pppoe-llcsnap" on pppoe0 would
> have the same effect? Or are there some other "under-the-hood" changes when
> using them?
> >
> > On the pppoe interface, use pppoe-vcmux if your modem is set to use
> VC-MUX, or pppoe-llcsnap if it’s set to use LLC-SNAP (they might be
> described using slightly different terms, but should still be recognisable
> as one or the other).  This probably depends on your ISP, and may further
> vary regionally within the same ISP.
>
>         In my experience it is rather bothersome trying to get that
> information from one’s ISP, this is why I would recommend to follow the
> instructions on https://github.com/moeller0/ATM_overhead_detector and
> empirically measure the actual overhead. In that case one ends up with the
> numeric overhead, hence my inclination to use that number directly instead
> of looking into a table to translate that back into a symbolic keyword…
> especially since say for an overhead of 32 (and 36) there are two different
> encapsulation schees that add up to that number:
>
> case 32
>                 disp('Connection: Bridged, LLC/SNAP RFC-1483/2684');
>                 disp('Protocol (bytes): Ethernet Header (14), ATM LLC (3),
> ATM SNAP (5), ATM pad (2), ATM AAL5 SAR (8) : Total 32');
>
>                 disp('Connection: PPPoE, VC/Mux RFC-2684');
>                 disp('Protocol (bytes): PPP (2), PPPoE (6), Ethernet
> Header (14), ATM pad (2), ATM AAL5 SAR (8) : Total 32’);
>
> good luck divining which of those is in use if all you know is the numeric
> overhead...
>
>
>
> >
> > I really prefer to use the self-explanatory keywords (which is why I
> added them in the first place) instead of opaque magic numbers.  This is a
> point on which Sebastian has long disagreed with me.
>
>         True, but I am not going to re-hash that here again ;)
>
> >
> >>> Question: if you set the shaper’s to 50% of line rate (8.75/0.5?) do
> you still see that unfairness? And if you add “atm overhead 40” to cake on
> pppoe0 and set the shaper to 90% of line rates (15.75/0.9) how does the
> Steam affect per-host fairness? Also how transient are these connections
> team uses?
> >>
> >> Actually did more testing about this and it seems that as far I have
> set the bandwidth to ~15Mbps (so ~15% less of my max speed) and use the
> "nat" parameter, the per-host fairness works even without the "dual-host"
> and "overhead" parameters. I definitely find this very interesting, is this
> behaviour caused by the way Steam downloads games?
> >
> > By default, Cake uses triple-isolate mode, which uses information about
> both source and destination hosts to perform per-host isolation; this
> usually works well regardless of which side of the connection has the LAN
> hosts.  The “dual” modes let you specify that fact explicitly, making it a
> little more robust and predictable.
> >
> > Without overhead compensation, Cake will actually use more of the
> physical link than it thinks it does - by default it only accounts for raw
> IP or Ethernet packets, depending on the type of interface it’s attached
> to.  With full-size packets as in a bulk download, the difference is
> relatively small, so the 15% margin is just about sufficient to make things
> work.  But with small packets mixed in, the difference grows, such that
> Cake might no longer control the bottleneck with some traffic mixes.
>
>         All true, to elaborate a bit on the ATM specific issue, due to
> AAL5’s insistence that each ethernet frame is packaged into an integer
> number of ATM cells (where the unused octets are simply padded out) the
> worst case is something like 100%, if a hypothetical packet would only
> require 49 Bytes, it will still require two ATM cells of 53 bytes...
>
> >
> > The “conservative” keyword I recommended earlier (which is exactly
> equivalent to Sebastian’s recommendation of “atm overhead 48”) reverses
> that situation; Cake will then always end up using *less* of the physical
> link than it accounts for, which is safe for troubleshooting with.  The
> keyword is there specifically so that we do’t have to figure out the
> precise overhead profile before tackling more substantive issues.
>
>         Due to the boundary observation above, one other option is to
> start with the shaper set to 50% of link rate, that should have sufficient
> wiggle room for all realistic overheads… (but honestly on a known ATM link
> I would always run the ATM_overhead_detector to get the precise number).
>
> >
> > At any rate, it has nothing to do with Steam specifically.
> >
> >>> As far as I can tell cake can drill down to the required IP/TCP/UDP
> fields independent of whether there are VLAN tags or PPPoE headers so cake
> should not care (except for the different overhead specifications you need
> to add as stated above). BUT if instantiated on eth0 cake will see pppoe
> LCP packets and might decide to drop them, which can take down the link, so
> out of caution I would still instantiate on pppoe in your case.
> >>
> >> Yeah, with further testing it seems the interface wasn't the culprit
> but I'll still do all my testing on pppoe0 just to be safe.
> >>
> >> Anyway I was wondering if there's some kind of manual for Cake and the
> various parameters, I'm looking to set it up best way possible but there
> are some parameters which I'm not sure what they do (one of them being
> "ingress”).
> >
> > With the correct version of iproute2 installed, just issue “man
> tc-cake”.  That’s the official documentation.
> >
> > Currently it doesn’t have the ingress keyword yet.  That’ll be fixed
> soon.
> >
> >> Also while reading on the bufferbloat.net Cake page I noticed a
> possible "fix" for BitTorrent (by setting it as "background",
> https://www.bufferbloat.net/projects/codel/wiki/Cake/#diffserv-support),
> I'm wondering if this can be done with Steam too?
> >
> > It’s possible, if you can figure out which traffic is Steam in the first
> place, and write filters to match on it.  This is complicated by the fact
> that Valve runs a sophisticated CDN to handle their rather impressive
> bandwidth load.
> >
> > - Jonathan Morton
> >
>
>

[-- Attachment #2: Type: text/html, Size: 8600 bytes --]

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Sebastian Moeller]

Hi Dendari,


> On Apr 22, 2017, at 23:56, Dendari Marini <dendari92@gmail.com> wrote:
> 
> Hello, thank you all for your replies!
> 
> For the overhead I'm gonna use "pppoe-llcsnap" (or "overhead 40 atm), as I believe it's the one which should work best for my connection.

	Probably correct, but you do not have to resort to believing, you can actually try to measure that ;) In case I have been too subtle before, have a look at https://github.com/moeller0/ATM_overhead_detector and follow the instructions there...


> 
> About the per-host fairness download issue: while it's kinda resolved I still feel like it's mainly related to Steam, as normally downloading files from PC1 and PC2 halved the speed as expected even at full bandwidth (so no overhead, no -15%).

	This might be true, but for cake to meaningfully resolve bufferbloat you absolutely _must_ take care to account for encapsulation and overhead one way or another.

> 
> Anyway back to Steam: assuming the IP addresses aren't a big issue, what steps should I follow to start filtering its traffic so it's considered background? Would the DPI from the ER-X be any helpful?

	Sorry, no direct experience so no practically applicable recommendations.

Best Regards


> 
> On 22 April 2017 at 18:47, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> > On Apr 22, 2017, at 11:36, Jonathan Morton <chromatix99@gmail.com> wrote:
> >
> >>> So please add “atm overhead 32" to cake on eth0 or “atm overhead 40” to cake instances on pppoe (these packets do not have the PPPoE header added yet and hence appear 8 bytes to small).
> >>
> >> Thanks for your help, will definitely use them. Just wondering if I use "pppoe-vcmux/bridged-llcsnap" on eth0 or "pppoe-llcsnap" on pppoe0 would have the same effect? Or are there some other "under-the-hood" changes when using them?
> >
> > On the pppoe interface, use pppoe-vcmux if your modem is set to use VC-MUX, or pppoe-llcsnap if it’s set to use LLC-SNAP (they might be described using slightly different terms, but should still be recognisable as one or the other).  This probably depends on your ISP, and may further vary regionally within the same ISP.
> 
>         In my experience it is rather bothersome trying to get that information from one’s ISP, this is why I would recommend to follow the instructions on https://github.com/moeller0/ATM_overhead_detector and empirically measure the actual overhead. In that case one ends up with the numeric overhead, hence my inclination to use that number directly instead of looking into a table to translate that back into a symbolic keyword… especially since say for an overhead of 32 (and 36) there are two different encapsulation schees that add up to that number:
> 
> case 32
>                 disp('Connection: Bridged, LLC/SNAP RFC-1483/2684');
>                 disp('Protocol (bytes): Ethernet Header (14), ATM LLC (3), ATM SNAP (5), ATM pad (2), ATM AAL5 SAR (8) : Total 32');
> 
>                 disp('Connection: PPPoE, VC/Mux RFC-2684');
>                 disp('Protocol (bytes): PPP (2), PPPoE (6), Ethernet Header (14), ATM pad (2), ATM AAL5 SAR (8) : Total 32’);
> 
> good luck divining which of those is in use if all you know is the numeric overhead...
> 
> 
> 
> >
> > I really prefer to use the self-explanatory keywords (which is why I added them in the first place) instead of opaque magic numbers.  This is a point on which Sebastian has long disagreed with me.
> 
>         True, but I am not going to re-hash that here again ;)
> 
> >
> >>> Question: if you set the shaper’s to 50% of line rate (8.75/0.5?) do you still see that unfairness? And if you add “atm overhead 40” to cake on pppoe0 and set the shaper to 90% of line rates (15.75/0.9) how does the Steam affect per-host fairness? Also how transient are these connections team uses?
> >>
> >> Actually did more testing about this and it seems that as far I have set the bandwidth to ~15Mbps (so ~15% less of my max speed) and use the "nat" parameter, the per-host fairness works even without the "dual-host" and "overhead" parameters. I definitely find this very interesting, is this behaviour caused by the way Steam downloads games?
> >
> > By default, Cake uses triple-isolate mode, which uses information about both source and destination hosts to perform per-host isolation; this usually works well regardless of which side of the connection has the LAN hosts.  The “dual” modes let you specify that fact explicitly, making it a little more robust and predictable.
> >
> > Without overhead compensation, Cake will actually use more of the physical link than it thinks it does - by default it only accounts for raw IP or Ethernet packets, depending on the type of interface it’s attached to.  With full-size packets as in a bulk download, the difference is relatively small, so the 15% margin is just about sufficient to make things work.  But with small packets mixed in, the difference grows, such that Cake might no longer control the bottleneck with some traffic mixes.
> 
>         All true, to elaborate a bit on the ATM specific issue, due to AAL5’s insistence that each ethernet frame is packaged into an integer number of ATM cells (where the unused octets are simply padded out) the worst case is something like 100%, if a hypothetical packet would only require 49 Bytes, it will still require two ATM cells of 53 bytes...
> 
> >
> > The “conservative” keyword I recommended earlier (which is exactly equivalent to Sebastian’s recommendation of “atm overhead 48”) reverses that situation; Cake will then always end up using *less* of the physical link than it accounts for, which is safe for troubleshooting with.  The keyword is there specifically so that we do’t have to figure out the precise overhead profile before tackling more substantive issues.
> 
>         Due to the boundary observation above, one other option is to start with the shaper set to 50% of link rate, that should have sufficient wiggle room for all realistic overheads… (but honestly on a known ATM link I would always run the ATM_overhead_detector to get the precise number).
> 
> >
> > At any rate, it has nothing to do with Steam specifically.
> >
> >>> As far as I can tell cake can drill down to the required IP/TCP/UDP fields independent of whether there are VLAN tags or PPPoE headers so cake should not care (except for the different overhead specifications you need to add as stated above). BUT if instantiated on eth0 cake will see pppoe LCP packets and might decide to drop them, which can take down the link, so out of caution I would still instantiate on pppoe in your case.
> >>
> >> Yeah, with further testing it seems the interface wasn't the culprit but I'll still do all my testing on pppoe0 just to be safe.
> >>
> >> Anyway I was wondering if there's some kind of manual for Cake and the various parameters, I'm looking to set it up best way possible but there are some parameters which I'm not sure what they do (one of them being "ingress”).
> >
> > With the correct version of iproute2 installed, just issue “man tc-cake”.  That’s the official documentation.
> >
> > Currently it doesn’t have the ingress keyword yet.  That’ll be fixed soon.
> >
> >> Also while reading on the bufferbloat.net Cake page I noticed a possible "fix" for BitTorrent (by setting it as "background", https://www.bufferbloat.net/projects/codel/wiki/Cake/#diffserv-support), I'm wondering if this can be done with Steam too?
> >
> > It’s possible, if you can figure out which traffic is Steam in the first place, and write filters to match on it.  This is complicated by the fact that Valve runs a sophisticated CDN to handle their rather impressive bandwidth load.
> >
> > - Jonathan Morton
> >
> 
> 

Re: [Cake] Getting Cake to work better with Steam and similar applications

[David Lang]

On Sun, 23 Apr 2017, Sebastian Moeller wrote:

>> About the per-host fairness download issue: while it's kinda resolved I still feel like it's mainly related to Steam, as normally downloading files from PC1 and PC2 halved the speed as expected even at full bandwidth (so no overhead, no -15%).
>
> 	This might be true, but for cake to meaningfully resolve bufferbloat you absolutely _must_ take care to account for encapsulation and overhead one way or another.

well, one way to account for this overhead is to set the allowed bandwidth low 
enough. Being precise on this overhead lets you get closer to the actual line 
rate, but if you have enough bandwidth, it may not really matter (i.e. if you 
have a 100Mb connection and only get 70Mb out of it, you probably won't notice 
unless you go looking)

David Lang

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Sebastian Moeller]

Hi David,

> On Apr 23, 2017, at 14:32, David Lang <david@lang.hm> wrote:
> 
> On Sun, 23 Apr 2017, Sebastian Moeller wrote:
> 
>>> About the per-host fairness download issue: while it's kinda resolved I still feel like it's mainly related to Steam, as normally downloading files from PC1 and PC2 halved the speed as expected even at full bandwidth (so no overhead, no -15%).
>> 
>> 	This might be true, but for cake to meaningfully resolve bufferbloat you absolutely _must_ take care to account for encapsulation and overhead one way or another.
> 
> well, one way to account for this overhead is to set the allowed bandwidth low enough. Being precise on this overhead lets you get closer to the actual line rate, but if you have enough bandwidth, it may not really matter (i.e. if you have a 100Mb connection and only get 70Mb out of it, you probably won't notice unless you go looking)

	Violent agreement. But note that with AAL5’s rule to always use an integer number of ATM cells per user packet the required bandwidth sacrifice to statically cover the worst case gets ludicrous (theoretical worst case: requiring 2 53 byte ATM cells for on 49 Byte data packet: 100 * 49 / (53 * 2) = 46.2% and this is on top of any potential unaccounted overhead inside the 49 Byte packet). Luckily the ATM padding issue is not as severe for bigger packets… but still to statically fully solve modem/dslam bufferbloat the required bandwidth sacrifice seems excessive… But again you are right, there might be users who do not mind to go to this length. For this reason I occasionally recommend to start the bandwidth at 50% to certainly rule out overhead/encapsulation accounting issues (mind you take 50% as starting point from which to ramp up…)

Best Regards
	Sebastian.


> 
> David Lang

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Dendari Marini]

[-- Attachment #1: Type: text/plain, Size: 3437 bytes --]

Hello,

Probably correct, but you do not have to resort to believing, you can
> actually try to measure that ;) In case I have been too subtle before, have
> a look at https://github.com/moeller0/ATM_overhead_detector and follow
> the instructions there...


I just used your script and it estimated an overhead of 20 bytes, so should
I use "overhead 20 atm" or am I missing something? In the last few days
I've been using "pppoe-llcsnap" ("overhead 40 atm") without any evident
issue, should I change it?

FWIW here's a quick example on ingress ppp that I tested using connmark
> the connmarks (1 or 2 or unmarked) being set by iptables rules on outbound
> connections/traffic classes.


Unfortunately I'm really not sure how to apply those settings to my case,
it's something I've never done so some hand-holding is probably needed,
sorry. At the moment I've limited the Steam bandwidth using the built-in
Basic Queue and DPI features from the ER-X. They're easy to set up but
aren't really ideal, would rather prefer Cake would take care about it more
dynamically.

Anyway about the Steam IP addresses I've noticed, in the almost three weeks
of testing, they're almost always the same IP blocks (most of which can be
found on the Steam Support website,
https://support.steampowered.com/kb_article.php?ref=8571-GLVN-8711). I
believe it would be a good starting point for limiting Steam, what do you
think?

On 24 April 2017 at 09:55, Sebastian Moeller <moeller0@gmx.de> wrote:

> Hi David,
>
> > On Apr 23, 2017, at 14:32, David Lang <david@lang.hm> wrote:
> >
> > On Sun, 23 Apr 2017, Sebastian Moeller wrote:
> >
> >>> About the per-host fairness download issue: while it's kinda resolved
> I still feel like it's mainly related to Steam, as normally downloading
> files from PC1 and PC2 halved the speed as expected even at full bandwidth
> (so no overhead, no -15%).
> >>
> >>      This might be true, but for cake to meaningfully resolve
> bufferbloat you absolutely _must_ take care to account for encapsulation
> and overhead one way or another.
> >
> > well, one way to account for this overhead is to set the allowed
> bandwidth low enough. Being precise on this overhead lets you get closer to
> the actual line rate, but if you have enough bandwidth, it may not really
> matter (i.e. if you have a 100Mb connection and only get 70Mb out of it,
> you probably won't notice unless you go looking)
>
>         Violent agreement. But note that with AAL5’s rule to always use an
> integer number of ATM cells per user packet the required bandwidth
> sacrifice to statically cover the worst case gets ludicrous (theoretical
> worst case: requiring 2 53 byte ATM cells for on 49 Byte data packet: 100 *
> 49 / (53 * 2) = 46.2% and this is on top of any potential unaccounted
> overhead inside the 49 Byte packet). Luckily the ATM padding issue is not
> as severe for bigger packets… but still to statically fully solve
> modem/dslam bufferbloat the required bandwidth sacrifice seems excessive…
> But again you are right, there might be users who do not mind to go to this
> length. For this reason I occasionally recommend to start the bandwidth at
> 50% to certainly rule out overhead/encapsulation accounting issues (mind
> you take 50% as starting point from which to ramp up…)
>
> Best Regards
>         Sebastian.
>
>
> >
> > David Lang
>
>

[-- Attachment #2: Type: text/html, Size: 4645 bytes --]

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Sebastian Moeller]

Hello,


> On Apr 24, 2017, at 10:41, Dendari Marini <dendari92@gmail.com> wrote:
> 
> Hello,
> 
> Probably correct, but you do not have to resort to believing, you can actually try to measure that ;) In case I have been too subtle before, have a look at https://github.com/moeller0/ATM_overhead_detector and follow the instructions there...
> 
> I just used your script and it estimated an overhead of 20 bytes, so should I use "overhead 20 atm" or am I missing something? In the last few days I've been using "pppoe-llcsnap" ("overhead 40 atm") without any evident issue, should I change it?

	Hmm, 20 seems rather interesting and something I never saw before. Could you share the two output plots somewhere, so I can have a look at those? (Also I might want tto ask for the text file that actually was generated by the ping collector script, just so I can run and confirm/de-bug things my self). I am not saying 20 is impossible, just that it is improbable enough to require more scrutiny.


Best Regards
	Sebastian


> 
> FWIW here's a quick example on ingress ppp that I tested using connmark
> the connmarks (1 or 2 or unmarked) being set by iptables rules on outbound
> connections/traffic classes.
> 
> Unfortunately I'm really not sure how to apply those settings to my case, it's something I've never done so some hand-holding is probably needed, sorry. At the moment I've limited the Steam bandwidth using the built-in Basic Queue and DPI features from the ER-X. They're easy to set up but aren't really ideal, would rather prefer Cake would take care about it more dynamically.
> 
> Anyway about the Steam IP addresses I've noticed, in the almost three weeks of testing, they're almost always the same IP blocks (most of which can be found on the Steam Support website, https://support.steampowered.com/kb_article.php?ref=8571-GLVN-8711). I believe it would be a good starting point for limiting Steam, what do you think?
> 
> On 24 April 2017 at 09:55, Sebastian Moeller <moeller0@gmx.de> wrote:
> Hi David,
> 
> > On Apr 23, 2017, at 14:32, David Lang <david@lang.hm> wrote:
> >
> > On Sun, 23 Apr 2017, Sebastian Moeller wrote:
> >
> >>> About the per-host fairness download issue: while it's kinda resolved I still feel like it's mainly related to Steam, as normally downloading files from PC1 and PC2 halved the speed as expected even at full bandwidth (so no overhead, no -15%).
> >>
> >>      This might be true, but for cake to meaningfully resolve bufferbloat you absolutely _must_ take care to account for encapsulation and overhead one way or another.
> >
> > well, one way to account for this overhead is to set the allowed bandwidth low enough. Being precise on this overhead lets you get closer to the actual line rate, but if you have enough bandwidth, it may not really matter (i.e. if you have a 100Mb connection and only get 70Mb out of it, you probably won't notice unless you go looking)
> 
>         Violent agreement. But note that with AAL5’s rule to always use an integer number of ATM cells per user packet the required bandwidth sacrifice to statically cover the worst case gets ludicrous (theoretical worst case: requiring 2 53 byte ATM cells for on 49 Byte data packet: 100 * 49 / (53 * 2) = 46.2% and this is on top of any potential unaccounted overhead inside the 49 Byte packet). Luckily the ATM padding issue is not as severe for bigger packets… but still to statically fully solve modem/dslam bufferbloat the required bandwidth sacrifice seems excessive… But again you are right, there might be users who do not mind to go to this length. For this reason I occasionally recommend to start the bandwidth at 50% to certainly rule out overhead/encapsulation accounting issues (mind you take 50% as starting point from which to ramp up…)
> 
> Best Regards
>         Sebastian.
> 
> 
> >
> > David Lang
> 
> 

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Dendari Marini]

[-- Attachment #1: Type: text/plain, Size: 4609 bytes --]

Hello,

Could you share the two output plots somewhere, so I can have a look at
> those? (Also I might want tto ask for the text file that actually was
> generated by the ping collector script, just so I can run and
> confirm/de-bug things my self).


Sure thing. The plot images: http://imgur.com/a/qDtA0 And the output text
file: https://drive.google.com/open?id=0B7vEuplJWEIkc1ozbUZRSGstajQ

On 24 April 2017 at 13:34, Sebastian Moeller <moeller0@gmx.de> wrote:

> Hello,
>
>
> > On Apr 24, 2017, at 10:41, Dendari Marini <dendari92@gmail.com> wrote:
> >
> > Hello,
> >
> > Probably correct, but you do not have to resort to believing, you can
> actually try to measure that ;) In case I have been too subtle before, have
> a look at https://github.com/moeller0/ATM_overhead_detector and follow
> the instructions there...
> >
> > I just used your script and it estimated an overhead of 20 bytes, so
> should I use "overhead 20 atm" or am I missing something? In the last few
> days I've been using "pppoe-llcsnap" ("overhead 40 atm") without any
> evident issue, should I change it?
>
>         Hmm, 20 seems rather interesting and something I never saw before.
> Could you share the two output plots somewhere, so I can have a look at
> those? (Also I might want tto ask for the text file that actually was
> generated by the ping collector script, just so I can run and
> confirm/de-bug things my self). I am not saying 20 is impossible, just that
> it is improbable enough to require more scrutiny.
>
>
> Best Regards
>         Sebastian
>
>
> >
> > FWIW here's a quick example on ingress ppp that I tested using connmark
> > the connmarks (1 or 2 or unmarked) being set by iptables rules on
> outbound
> > connections/traffic classes.
> >
> > Unfortunately I'm really not sure how to apply those settings to my
> case, it's something I've never done so some hand-holding is probably
> needed, sorry. At the moment I've limited the Steam bandwidth using the
> built-in Basic Queue and DPI features from the ER-X. They're easy to set up
> but aren't really ideal, would rather prefer Cake would take care about it
> more dynamically.
> >
> > Anyway about the Steam IP addresses I've noticed, in the almost three
> weeks of testing, they're almost always the same IP blocks (most of which
> can be found on the Steam Support website, https://support.steampowered.
> com/kb_article.php?ref=8571-GLVN-8711). I believe it would be a good
> starting point for limiting Steam, what do you think?
> >
> > On 24 April 2017 at 09:55, Sebastian Moeller <moeller0@gmx.de> wrote:
> > Hi David,
> >
> > > On Apr 23, 2017, at 14:32, David Lang <david@lang.hm> wrote:
> > >
> > > On Sun, 23 Apr 2017, Sebastian Moeller wrote:
> > >
> > >>> About the per-host fairness download issue: while it's kinda
> resolved I still feel like it's mainly related to Steam, as normally
> downloading files from PC1 and PC2 halved the speed as expected even at
> full bandwidth (so no overhead, no -15%).
> > >>
> > >>      This might be true, but for cake to meaningfully resolve
> bufferbloat you absolutely _must_ take care to account for encapsulation
> and overhead one way or another.
> > >
> > > well, one way to account for this overhead is to set the allowed
> bandwidth low enough. Being precise on this overhead lets you get closer to
> the actual line rate, but if you have enough bandwidth, it may not really
> matter (i.e. if you have a 100Mb connection and only get 70Mb out of it,
> you probably won't notice unless you go looking)
> >
> >         Violent agreement. But note that with AAL5’s rule to always use
> an integer number of ATM cells per user packet the required bandwidth
> sacrifice to statically cover the worst case gets ludicrous (theoretical
> worst case: requiring 2 53 byte ATM cells for on 49 Byte data packet: 100 *
> 49 / (53 * 2) = 46.2% and this is on top of any potential unaccounted
> overhead inside the 49 Byte packet). Luckily the ATM padding issue is not
> as severe for bigger packets… but still to statically fully solve
> modem/dslam bufferbloat the required bandwidth sacrifice seems excessive…
> But again you are right, there might be users who do not mind to go to this
> length. For this reason I occasionally recommend to start the bandwidth at
> 50% to certainly rule out overhead/encapsulation accounting issues (mind
> you take 50% as starting point from which to ramp up…)
> >
> > Best Regards
> >         Sebastian.
> >
> >
> > >
> > > David Lang
> >
> >
>
>

[-- Attachment #2: Type: text/html, Size: 5937 bytes --]

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Sebastian Moeller]

Hi Dendari,

thanks,


> On Apr 24, 2017, at 14:08, Dendari Marini <dendari92@gmail.com> wrote:
> 
> Hello,
> 
> Could you share the two output plots somewhere, so I can have a look at those? (Also I might want tto ask for the text file that actually was generated by the ping collector script, just so I can run and confirm/de-bug things my self).
> 
> Sure thing. The plot images: http://imgur.com/a/qDtA0 

	Okay, these _look_ reasonable on first sight...


> And the output text file: https://drive.google.com/open?id=0B7vEuplJWEIkc1ozbUZRSGstajQ

…but this actually shows that 8.8.8.8 truncated the size of the ICMP responses (the runs of “bytes=92” in the log file also it only returned sizes from 44 to 92 so not exactly what we asked for). I would not really trust these results, even though the RTTs should be dominated by your slow egress link. Please redo these against a different ICMP target server (netperf-east.bufferbloat.net might do, depending on your location, that on is on the east coast of the U.S.). I have to admit, that the hrping/windows overhead extraction did not have much exercise so far and might be not be robust enough. I will have a look at your log file, but this really requires a different target server. In the unix script there is the following:


    echo "To run measurements supply the TARGET IP address as first agument to ${0} this script."
    echo "Use traceroute 8.8.8.8 to get a list of increasingly distant hosts, pick the first host out of your network (ideally the DSLAM)."
    echo "Test whether the selected host responds to ping: 'ping -s16 -c 1 target.IP.address.quad' : this needs to actually return non zero RTTs."
    echo "If the hosts does not reply to the pings take the next host from the traceroute (moving closer to 8.8.8.8), repeat until you find a replying host."
    echo "Once the main script is started have a quick look at the logfile, to see whether the RTTs stay close to the initial test RTT."
    echo "If the RTTs have increased a lot, the PINGPERIOD might be too short, and the host might have put us on a slow path; either increase PINGPERIOD or try the next host..."
    echo ""
    echo "Here is the traceroute (might take a while):"
    echo ""
    traceroute 8.8.8.8
    echo ""
    echo "Alternatively you might try to use googles infrastructure by running: ${0} gstatic.com "
    echo "Please note that gstatic.com might not return arbitrarily sized ICMP probes, so check the log file care-fully.”

Which unfortunately does not appear in the windows script yet, but this still seems great advise, please note that at the time this was written, 8.8.8.8 still replied with “correctly”’ sized ICMP echo responses just like gstatic.com di initially, now both seem to truncate their responses. This is somewhat to be expected, but maybe somewhere along the path there is another server that will still respond properly...    


Best Regards
	Sebastian



> 
> On 24 April 2017 at 13:34, Sebastian Moeller <moeller0@gmx.de> wrote:
> Hello,
> 
> 
> > On Apr 24, 2017, at 10:41, Dendari Marini <dendari92@gmail.com> wrote:
> >
> > Hello,
> >
> > Probably correct, but you do not have to resort to believing, you can actually try to measure that ;) In case I have been too subtle before, have a look at https://github.com/moeller0/ATM_overhead_detector and follow the instructions there...
> >
> > I just used your script and it estimated an overhead of 20 bytes, so should I use "overhead 20 atm" or am I missing something? In the last few days I've been using "pppoe-llcsnap" ("overhead 40 atm") without any evident issue, should I change it?
> 
>         Hmm, 20 seems rather interesting and something I never saw before. Could you share the two output plots somewhere, so I can have a look at those? (Also I might want tto ask for the text file that actually was generated by the ping collector script, just so I can run and confirm/de-bug things my self). I am not saying 20 is impossible, just that it is improbable enough to require more scrutiny.
> 
> 
> Best Regards
>         Sebastian
> 
> 
> >
> > FWIW here's a quick example on ingress ppp that I tested using connmark
> > the connmarks (1 or 2 or unmarked) being set by iptables rules on outbound
> > connections/traffic classes.
> >
> > Unfortunately I'm really not sure how to apply those settings to my case, it's something I've never done so some hand-holding is probably needed, sorry. At the moment I've limited the Steam bandwidth using the built-in Basic Queue and DPI features from the ER-X. They're easy to set up but aren't really ideal, would rather prefer Cake would take care about it more dynamically.
> >
> > Anyway about the Steam IP addresses I've noticed, in the almost three weeks of testing, they're almost always the same IP blocks (most of which can be found on the Steam Support website, https://support.steampowered.com/kb_article.php?ref=8571-GLVN-8711). I believe it would be a good starting point for limiting Steam, what do you think?
> >
> > On 24 April 2017 at 09:55, Sebastian Moeller <moeller0@gmx.de> wrote:
> > Hi David,
> >
> > > On Apr 23, 2017, at 14:32, David Lang <david@lang.hm> wrote:
> > >
> > > On Sun, 23 Apr 2017, Sebastian Moeller wrote:
> > >
> > >>> About the per-host fairness download issue: while it's kinda resolved I still feel like it's mainly related to Steam, as normally downloading files from PC1 and PC2 halved the speed as expected even at full bandwidth (so no overhead, no -15%).
> > >>
> > >>      This might be true, but for cake to meaningfully resolve bufferbloat you absolutely _must_ take care to account for encapsulation and overhead one way or another.
> > >
> > > well, one way to account for this overhead is to set the allowed bandwidth low enough. Being precise on this overhead lets you get closer to the actual line rate, but if you have enough bandwidth, it may not really matter (i.e. if you have a 100Mb connection and only get 70Mb out of it, you probably won't notice unless you go looking)
> >
> >         Violent agreement. But note that with AAL5’s rule to always use an integer number of ATM cells per user packet the required bandwidth sacrifice to statically cover the worst case gets ludicrous (theoretical worst case: requiring 2 53 byte ATM cells for on 49 Byte data packet: 100 * 49 / (53 * 2) = 46.2% and this is on top of any potential unaccounted overhead inside the 49 Byte packet). Luckily the ATM padding issue is not as severe for bigger packets… but still to statically fully solve modem/dslam bufferbloat the required bandwidth sacrifice seems excessive… But again you are right, there might be users who do not mind to go to this length. For this reason I occasionally recommend to start the bandwidth at 50% to certainly rule out overhead/encapsulation accounting issues (mind you take 50% as starting point from which to ramp up…)
> >
> > Best Regards
> >         Sebastian.
> >
> >
> > >
> > > David Lang
> >
> >
> 
> 

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Dendari Marini]

[-- Attachment #1: Type: text/plain, Size: 7836 bytes --]

Hi Sebastian,

Did few quick tests using your recommendations and the results is still
"overhead 20". Plot: http://imgur.com/a/5c3iv Text:
https://drive.google.com/open?id=0B7vEuplJWEIkRVBhX2xGUmNUMVE (NOTE: I
used PINGSPERSIZE = 100 instead of 1000, as I didn't have much time)

On 24 April 2017 at 14:35, Sebastian Moeller <moeller0@gmx.de> wrote:

> Hi Dendari,
>
> thanks,
>
>
> > On Apr 24, 2017, at 14:08, Dendari Marini <dendari92@gmail.com> wrote:
> >
> > Hello,
> >
> > Could you share the two output plots somewhere, so I can have a look at
> those? (Also I might want tto ask for the text file that actually was
> generated by the ping collector script, just so I can run and
> confirm/de-bug things my self).
> >
> > Sure thing. The plot images: http://imgur.com/a/qDtA0
>
>         Okay, these _look_ reasonable on first sight...
>
>
> > And the output text file: https://drive.google.com/open?id=
> 0B7vEuplJWEIkc1ozbUZRSGstajQ
>
> …but this actually shows that 8.8.8.8 truncated the size of the ICMP
> responses (the runs of “bytes=92” in the log file also it only returned
> sizes from 44 to 92 so not exactly what we asked for). I would not really
> trust these results, even though the RTTs should be dominated by your slow
> egress link. Please redo these against a different ICMP target server (
> netperf-east.bufferbloat.net might do, depending on your location, that
> on is on the east coast of the U.S.). I have to admit, that the
> hrping/windows overhead extraction did not have much exercise so far and
> might be not be robust enough. I will have a look at your log file, but
> this really requires a different target server. In the unix script there is
> the following:
>
>
>     echo "To run measurements supply the TARGET IP address as first
> agument to ${0} this script."
>     echo "Use traceroute 8.8.8.8 to get a list of increasingly distant
> hosts, pick the first host out of your network (ideally the DSLAM)."
>     echo "Test whether the selected host responds to ping: 'ping -s16 -c 1
> target.IP.address.quad' : this needs to actually return non zero RTTs."
>     echo "If the hosts does not reply to the pings take the next host from
> the traceroute (moving closer to 8.8.8.8), repeat until you find a replying
> host."
>     echo "Once the main script is started have a quick look at the
> logfile, to see whether the RTTs stay close to the initial test RTT."
>     echo "If the RTTs have increased a lot, the PINGPERIOD might be too
> short, and the host might have put us on a slow path; either increase
> PINGPERIOD or try the next host..."
>     echo ""
>     echo "Here is the traceroute (might take a while):"
>     echo ""
>     traceroute 8.8.8.8
>     echo ""
>     echo "Alternatively you might try to use googles infrastructure by
> running: ${0} gstatic.com "
>     echo "Please note that gstatic.com might not return arbitrarily sized
> ICMP probes, so check the log file care-fully.”
>
> Which unfortunately does not appear in the windows script yet, but this
> still seems great advise, please note that at the time this was written,
> 8.8.8.8 still replied with “correctly”’ sized ICMP echo responses just like
> gstatic.com di initially, now both seem to truncate their responses. This
> is somewhat to be expected, but maybe somewhere along the path there is
> another server that will still respond properly...
>
>
> Best Regards
>         Sebastian
>
>
>
> >
> > On 24 April 2017 at 13:34, Sebastian Moeller <moeller0@gmx.de> wrote:
> > Hello,
> >
> >
> > > On Apr 24, 2017, at 10:41, Dendari Marini <dendari92@gmail.com> wrote:
> > >
> > > Hello,
> > >
> > > Probably correct, but you do not have to resort to believing, you can
> actually try to measure that ;) In case I have been too subtle before, have
> a look at https://github.com/moeller0/ATM_overhead_detector and follow
> the instructions there...
> > >
> > > I just used your script and it estimated an overhead of 20 bytes, so
> should I use "overhead 20 atm" or am I missing something? In the last few
> days I've been using "pppoe-llcsnap" ("overhead 40 atm") without any
> evident issue, should I change it?
> >
> >         Hmm, 20 seems rather interesting and something I never saw
> before. Could you share the two output plots somewhere, so I can have a
> look at those? (Also I might want tto ask for the text file that actually
> was generated by the ping collector script, just so I can run and
> confirm/de-bug things my self). I am not saying 20 is impossible, just that
> it is improbable enough to require more scrutiny.
> >
> >
> > Best Regards
> >         Sebastian
> >
> >
> > >
> > > FWIW here's a quick example on ingress ppp that I tested using connmark
> > > the connmarks (1 or 2 or unmarked) being set by iptables rules on
> outbound
> > > connections/traffic classes.
> > >
> > > Unfortunately I'm really not sure how to apply those settings to my
> case, it's something I've never done so some hand-holding is probably
> needed, sorry. At the moment I've limited the Steam bandwidth using the
> built-in Basic Queue and DPI features from the ER-X. They're easy to set up
> but aren't really ideal, would rather prefer Cake would take care about it
> more dynamically.
> > >
> > > Anyway about the Steam IP addresses I've noticed, in the almost three
> weeks of testing, they're almost always the same IP blocks (most of which
> can be found on the Steam Support website, https://support.steampowered.
> com/kb_article.php?ref=8571-GLVN-8711). I believe it would be a good
> starting point for limiting Steam, what do you think?
> > >
> > > On 24 April 2017 at 09:55, Sebastian Moeller <moeller0@gmx.de> wrote:
> > > Hi David,
> > >
> > > > On Apr 23, 2017, at 14:32, David Lang <david@lang.hm> wrote:
> > > >
> > > > On Sun, 23 Apr 2017, Sebastian Moeller wrote:
> > > >
> > > >>> About the per-host fairness download issue: while it's kinda
> resolved I still feel like it's mainly related to Steam, as normally
> downloading files from PC1 and PC2 halved the speed as expected even at
> full bandwidth (so no overhead, no -15%).
> > > >>
> > > >>      This might be true, but for cake to meaningfully resolve
> bufferbloat you absolutely _must_ take care to account for encapsulation
> and overhead one way or another.
> > > >
> > > > well, one way to account for this overhead is to set the allowed
> bandwidth low enough. Being precise on this overhead lets you get closer to
> the actual line rate, but if you have enough bandwidth, it may not really
> matter (i.e. if you have a 100Mb connection and only get 70Mb out of it,
> you probably won't notice unless you go looking)
> > >
> > >         Violent agreement. But note that with AAL5’s rule to always
> use an integer number of ATM cells per user packet the required bandwidth
> sacrifice to statically cover the worst case gets ludicrous (theoretical
> worst case: requiring 2 53 byte ATM cells for on 49 Byte data packet: 100 *
> 49 / (53 * 2) = 46.2% and this is on top of any potential unaccounted
> overhead inside the 49 Byte packet). Luckily the ATM padding issue is not
> as severe for bigger packets… but still to statically fully solve
> modem/dslam bufferbloat the required bandwidth sacrifice seems excessive…
> But again you are right, there might be users who do not mind to go to this
> length. For this reason I occasionally recommend to start the bandwidth at
> 50% to certainly rule out overhead/encapsulation accounting issues (mind
> you take 50% as starting point from which to ramp up…)
> > >
> > > Best Regards
> > >         Sebastian.
> > >
> > >
> > > >
> > > > David Lang
> > >
> > >
> >
> >
>
>

[-- Attachment #2: Type: text/html, Size: 9969 bytes --]

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Sebastian Moeller]

Hi Dendari,

> On Apr 24, 2017, at 15:49, Dendari Marini <dendari92@gmail.com> wrote:
> 
> Hi Sebastian,
> 
> Did few quick tests using your recommendations and the results is still "overhead 20". Plot: http://imgur.com/a/5c3iv Text: https://drive.google.com/open?id=0B7vEuplJWEIkRVBhX2xGUmNUMVE (NOTE: I used PINGSPERSIZE = 100 instead of 1000, as I didn't have much time)

	The stair plot is quite nice, so 100 repetitions are suffficient, the issue is more that the size the hrping reports is not the same that gnu ping reports and hence thigns are off. So I still do not trust the 20 bytes report, but over all the data q\acquisition worked, I just need to figure out what hrping actually reports as “bytes” and then correct in the code accordingly. Weird, I believe I have a hrping dataset that worked correctly… 
Well scratch that hrping reports in its header:
with 16-160 bytes data (44-188 bytes IP):

so I need to handle hrping log files differently from normal ones (where I add the 20 + 8 IPv4 and ICMP header manually)… it seems that never worked correctly.. thanks for testing I will get back to you with an update later (this week)

Best Regards
	Sebastian





> 
> On 24 April 2017 at 14:35, Sebastian Moeller <moeller0@gmx.de> wrote:
> Hi Dendari,
> 
> thanks,
> 
> 
> > On Apr 24, 2017, at 14:08, Dendari Marini <dendari92@gmail.com> wrote:
> >
> > Hello,
> >
> > Could you share the two output plots somewhere, so I can have a look at those? (Also I might want tto ask for the text file that actually was generated by the ping collector script, just so I can run and confirm/de-bug things my self).
> >
> > Sure thing. The plot images: http://imgur.com/a/qDtA0
> 
>         Okay, these _look_ reasonable on first sight...
> 
> 
> > And the output text file: https://drive.google.com/open?id=0B7vEuplJWEIkc1ozbUZRSGstajQ
> 
> …but this actually shows that 8.8.8.8 truncated the size of the ICMP responses (the runs of “bytes=92” in the log file also it only returned sizes from 44 to 92 so not exactly what we asked for). I would not really trust these results, even though the RTTs should be dominated by your slow egress link. Please redo these against a different ICMP target server (netperf-east.bufferbloat.net might do, depending on your location, that on is on the east coast of the U.S.). I have to admit, that the hrping/windows overhead extraction did not have much exercise so far and might be not be robust enough. I will have a look at your log file, but this really requires a different target server. In the unix script there is the following:
> 
> 
>     echo "To run measurements supply the TARGET IP address as first agument to ${0} this script."
>     echo "Use traceroute 8.8.8.8 to get a list of increasingly distant hosts, pick the first host out of your network (ideally the DSLAM)."
>     echo "Test whether the selected host responds to ping: 'ping -s16 -c 1 target.IP.address.quad' : this needs to actually return non zero RTTs."
>     echo "If the hosts does not reply to the pings take the next host from the traceroute (moving closer to 8.8.8.8), repeat until you find a replying host."
>     echo "Once the main script is started have a quick look at the logfile, to see whether the RTTs stay close to the initial test RTT."
>     echo "If the RTTs have increased a lot, the PINGPERIOD might be too short, and the host might have put us on a slow path; either increase PINGPERIOD or try the next host..."
>     echo ""
>     echo "Here is the traceroute (might take a while):"
>     echo ""
>     traceroute 8.8.8.8
>     echo ""
>     echo "Alternatively you might try to use googles infrastructure by running: ${0} gstatic.com "
>     echo "Please note that gstatic.com might not return arbitrarily sized ICMP probes, so check the log file care-fully.”
> 
> Which unfortunately does not appear in the windows script yet, but this still seems great advise, please note that at the time this was written, 8.8.8.8 still replied with “correctly”’ sized ICMP echo responses just like gstatic.com di initially, now both seem to truncate their responses. This is somewhat to be expected, but maybe somewhere along the path there is another server that will still respond properly...
> 
> 
> Best Regards
>         Sebastian
> 
> 
> 
> >
> > On 24 April 2017 at 13:34, Sebastian Moeller <moeller0@gmx.de> wrote:
> > Hello,
> >
> >
> > > On Apr 24, 2017, at 10:41, Dendari Marini <dendari92@gmail.com> wrote:
> > >
> > > Hello,
> > >
> > > Probably correct, but you do not have to resort to believing, you can actually try to measure that ;) In case I have been too subtle before, have a look at https://github.com/moeller0/ATM_overhead_detector and follow the instructions there...
> > >
> > > I just used your script and it estimated an overhead of 20 bytes, so should I use "overhead 20 atm" or am I missing something? In the last few days I've been using "pppoe-llcsnap" ("overhead 40 atm") without any evident issue, should I change it?
> >
> >         Hmm, 20 seems rather interesting and something I never saw before. Could you share the two output plots somewhere, so I can have a look at those? (Also I might want tto ask for the text file that actually was generated by the ping collector script, just so I can run and confirm/de-bug things my self). I am not saying 20 is impossible, just that it is improbable enough to require more scrutiny.
> >
> >
> > Best Regards
> >         Sebastian
> >
> >
> > >
> > > FWIW here's a quick example on ingress ppp that I tested using connmark
> > > the connmarks (1 or 2 or unmarked) being set by iptables rules on outbound
> > > connections/traffic classes.
> > >
> > > Unfortunately I'm really not sure how to apply those settings to my case, it's something I've never done so some hand-holding is probably needed, sorry. At the moment I've limited the Steam bandwidth using the built-in Basic Queue and DPI features from the ER-X. They're easy to set up but aren't really ideal, would rather prefer Cake would take care about it more dynamically.
> > >
> > > Anyway about the Steam IP addresses I've noticed, in the almost three weeks of testing, they're almost always the same IP blocks (most of which can be found on the Steam Support website, https://support.steampowered.com/kb_article.php?ref=8571-GLVN-8711). I believe it would be a good starting point for limiting Steam, what do you think?
> > >
> > > On 24 April 2017 at 09:55, Sebastian Moeller <moeller0@gmx.de> wrote:
> > > Hi David,
> > >
> > > > On Apr 23, 2017, at 14:32, David Lang <david@lang.hm> wrote:
> > > >
> > > > On Sun, 23 Apr 2017, Sebastian Moeller wrote:
> > > >
> > > >>> About the per-host fairness download issue: while it's kinda resolved I still feel like it's mainly related to Steam, as normally downloading files from PC1 and PC2 halved the speed as expected even at full bandwidth (so no overhead, no -15%).
> > > >>
> > > >>      This might be true, but for cake to meaningfully resolve bufferbloat you absolutely _must_ take care to account for encapsulation and overhead one way or another.
> > > >
> > > > well, one way to account for this overhead is to set the allowed bandwidth low enough. Being precise on this overhead lets you get closer to the actual line rate, but if you have enough bandwidth, it may not really matter (i.e. if you have a 100Mb connection and only get 70Mb out of it, you probably won't notice unless you go looking)
> > >
> > >         Violent agreement. But note that with AAL5’s rule to always use an integer number of ATM cells per user packet the required bandwidth sacrifice to statically cover the worst case gets ludicrous (theoretical worst case: requiring 2 53 byte ATM cells for on 49 Byte data packet: 100 * 49 / (53 * 2) = 46.2% and this is on top of any potential unaccounted overhead inside the 49 Byte packet). Luckily the ATM padding issue is not as severe for bigger packets… but still to statically fully solve modem/dslam bufferbloat the required bandwidth sacrifice seems excessive… But again you are right, there might be users who do not mind to go to this length. For this reason I occasionally recommend to start the bandwidth at 50% to certainly rule out overhead/encapsulation accounting issues (mind you take 50% as starting point from which to ramp up…)
> > >
> > > Best Regards
> > >         Sebastian.
> > >
> > >
> > > >
> > > > David Lang
> > >
> > >
> >
> >
> 
> 

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Sebastian Moeller]

Okay,

so the fix was simply to reduce the hrping reported sizes by 20, as hrping reports the full IP size of its probe, while gnu/bsd ping only reports ICMP size so for the hrping case the sizes where 20 bytes to large. So in essence 40 is the correct number for the ADSL-leg of your link. Your ISP might have an upstream shaper that might even have a larger overhead*, but I really doubt that. Interestingly enough even the truncated 8.8.8.8 data gets processed reasonably sanely (in the end it only affects a single datapoint the last one so the detector is amazingly robust enough).
Sorry for having you act as beta tester, and thanks for helping debug this ;) Anyway I committed that change and after a new pull reported values should also be correct for the windows case… (in my defense I only had PTM vdsl2 data available when adding the processing for hrping data and obviously did not look to closely at the output as that showed as expected no ATM carrier, so the precise numerical value was not meaningful anyway). I guess that shows that the hrping script probably has not seen any real world usage.

Best Regards
	Sebastian



*) my code only ever should see the overhead on the ATM link assuming its probe traffic does not fully saturate the potential upstream shaper…




> On Apr 24, 2017, at 15:49, Dendari Marini <dendari92@gmail.com> wrote:
> 
> Hi Sebastian,
> 
> Did few quick tests using your recommendations and the results is still "overhead 20". Plot: http://imgur.com/a/5c3iv Text: https://drive.google.com/open?id=0B7vEuplJWEIkRVBhX2xGUmNUMVE (NOTE: I used PINGSPERSIZE = 100 instead of 1000, as I didn't have much time)
> 
> On 24 April 2017 at 14:35, Sebastian Moeller <moeller0@gmx.de> wrote:
> Hi Dendari,
> 
> thanks,
> 
> 
> > On Apr 24, 2017, at 14:08, Dendari Marini <dendari92@gmail.com> wrote:
> >
> > Hello,
> >
> > Could you share the two output plots somewhere, so I can have a look at those? (Also I might want tto ask for the text file that actually was generated by the ping collector script, just so I can run and confirm/de-bug things my self).
> >
> > Sure thing. The plot images: http://imgur.com/a/qDtA0
> 
>         Okay, these _look_ reasonable on first sight...
> 
> 
> > And the output text file: https://drive.google.com/open?id=0B7vEuplJWEIkc1ozbUZRSGstajQ
> 
> …but this actually shows that 8.8.8.8 truncated the size of the ICMP responses (the runs of “bytes=92” in the log file also it only returned sizes from 44 to 92 so not exactly what we asked for). I would not really trust these results, even though the RTTs should be dominated by your slow egress link. Please redo these against a different ICMP target server (netperf-east.bufferbloat.net might do, depending on your location, that on is on the east coast of the U.S.). I have to admit, that the hrping/windows overhead extraction did not have much exercise so far and might be not be robust enough. I will have a look at your log file, but this really requires a different target server. In the unix script there is the following:
> 
> 
>     echo "To run measurements supply the TARGET IP address as first agument to ${0} this script."
>     echo "Use traceroute 8.8.8.8 to get a list of increasingly distant hosts, pick the first host out of your network (ideally the DSLAM)."
>     echo "Test whether the selected host responds to ping: 'ping -s16 -c 1 target.IP.address.quad' : this needs to actually return non zero RTTs."
>     echo "If the hosts does not reply to the pings take the next host from the traceroute (moving closer to 8.8.8.8), repeat until you find a replying host."
>     echo "Once the main script is started have a quick look at the logfile, to see whether the RTTs stay close to the initial test RTT."
>     echo "If the RTTs have increased a lot, the PINGPERIOD might be too short, and the host might have put us on a slow path; either increase PINGPERIOD or try the next host..."
>     echo ""
>     echo "Here is the traceroute (might take a while):"
>     echo ""
>     traceroute 8.8.8.8
>     echo ""
>     echo "Alternatively you might try to use googles infrastructure by running: ${0} gstatic.com "
>     echo "Please note that gstatic.com might not return arbitrarily sized ICMP probes, so check the log file care-fully.”
> 
> Which unfortunately does not appear in the windows script yet, but this still seems great advise, please note that at the time this was written, 8.8.8.8 still replied with “correctly”’ sized ICMP echo responses just like gstatic.com di initially, now both seem to truncate their responses. This is somewhat to be expected, but maybe somewhere along the path there is another server that will still respond properly...
> 
> 
> Best Regards
>         Sebastian
> 
> 
> 
> >
> > On 24 April 2017 at 13:34, Sebastian Moeller <moeller0@gmx.de> wrote:
> > Hello,
> >
> >
> > > On Apr 24, 2017, at 10:41, Dendari Marini <dendari92@gmail.com> wrote:
> > >
> > > Hello,
> > >
> > > Probably correct, but you do not have to resort to believing, you can actually try to measure that ;) In case I have been too subtle before, have a look at https://github.com/moeller0/ATM_overhead_detector and follow the instructions there...
> > >
> > > I just used your script and it estimated an overhead of 20 bytes, so should I use "overhead 20 atm" or am I missing something? In the last few days I've been using "pppoe-llcsnap" ("overhead 40 atm") without any evident issue, should I change it?
> >
> >         Hmm, 20 seems rather interesting and something I never saw before. Could you share the two output plots somewhere, so I can have a look at those? (Also I might want tto ask for the text file that actually was generated by the ping collector script, just so I can run and confirm/de-bug things my self). I am not saying 20 is impossible, just that it is improbable enough to require more scrutiny.
> >
> >
> > Best Regards
> >         Sebastian
> >
> >
> > >
> > > FWIW here's a quick example on ingress ppp that I tested using connmark
> > > the connmarks (1 or 2 or unmarked) being set by iptables rules on outbound
> > > connections/traffic classes.
> > >
> > > Unfortunately I'm really not sure how to apply those settings to my case, it's something I've never done so some hand-holding is probably needed, sorry. At the moment I've limited the Steam bandwidth using the built-in Basic Queue and DPI features from the ER-X. They're easy to set up but aren't really ideal, would rather prefer Cake would take care about it more dynamically.
> > >
> > > Anyway about the Steam IP addresses I've noticed, in the almost three weeks of testing, they're almost always the same IP blocks (most of which can be found on the Steam Support website, https://support.steampowered.com/kb_article.php?ref=8571-GLVN-8711). I believe it would be a good starting point for limiting Steam, what do you think?
> > >
> > > On 24 April 2017 at 09:55, Sebastian Moeller <moeller0@gmx.de> wrote:
> > > Hi David,
> > >
> > > > On Apr 23, 2017, at 14:32, David Lang <david@lang.hm> wrote:
> > > >
> > > > On Sun, 23 Apr 2017, Sebastian Moeller wrote:
> > > >
> > > >>> About the per-host fairness download issue: while it's kinda resolved I still feel like it's mainly related to Steam, as normally downloading files from PC1 and PC2 halved the speed as expected even at full bandwidth (so no overhead, no -15%).
> > > >>
> > > >>      This might be true, but for cake to meaningfully resolve bufferbloat you absolutely _must_ take care to account for encapsulation and overhead one way or another.
> > > >
> > > > well, one way to account for this overhead is to set the allowed bandwidth low enough. Being precise on this overhead lets you get closer to the actual line rate, but if you have enough bandwidth, it may not really matter (i.e. if you have a 100Mb connection and only get 70Mb out of it, you probably won't notice unless you go looking)
> > >
> > >         Violent agreement. But note that with AAL5’s rule to always use an integer number of ATM cells per user packet the required bandwidth sacrifice to statically cover the worst case gets ludicrous (theoretical worst case: requiring 2 53 byte ATM cells for on 49 Byte data packet: 100 * 49 / (53 * 2) = 46.2% and this is on top of any potential unaccounted overhead inside the 49 Byte packet). Luckily the ATM padding issue is not as severe for bigger packets… but still to statically fully solve modem/dslam bufferbloat the required bandwidth sacrifice seems excessive… But again you are right, there might be users who do not mind to go to this length. For this reason I occasionally recommend to start the bandwidth at 50% to certainly rule out overhead/encapsulation accounting issues (mind you take 50% as starting point from which to ramp up…)
> > >
> > > Best Regards
> > >         Sebastian.
> > >
> > >
> > > >
> > > > David Lang
> > >
> > >
> >
> >
> 
> 

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Andy Furniss]

Dendari Marini wrote:

>> FWIW here's a quick example on ingress ppp that I tested using
>> connmark the connmarks (1 or 2 or unmarked) being set by iptables
>> rules on outbound connections/traffic classes.
> 
> 
> Unfortunately I'm really not sure how to apply those settings to my
> case, it's something I've never done so some hand-holding is probably
> needed, sorry. At the moment I've limited the Steam bandwidth using
> the built-in Basic Queue and DPI features from the ER-X. They're easy
> to set up but aren't really ideal, would rather prefer Cake would
> take care about it more dynamically.
> 
> Anyway about the Steam IP addresses I've noticed, in the almost three
> weeks of testing, they're almost always the same IP blocks (most of
> which can be found on the Steam Support website, 
> https://support.steampowered.com/kb_article.php?ref=8571-GLVN-8711).
> I believe it would be a good starting point for limiting Steam, what
> do you think?

I think the easiest and most robust way to shape ingress traffic for you
would be to do it on the LAN side.
If you have multiple interfaces facing LAN then use ifb.
I've never used it myself, but to mark multiple address ranges the
easiest way would be to use iptables with ipset - there will be many
examples to be found on the internet.
Even if you do mark (well set dscp cs1) for steam servers you will still
need to be backed off from your ingress rate enough or it still won't
work as the queue will build up too much on the ISP side of the
bottleneck. Shaping ingress is quite different to doing egress as you
are at the wrong end of the bottleneck.

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Dendari Marini]

[-- Attachment #1: Type: text/plain, Size: 3821 bytes --]

Hello,

I think the easiest and most robust way to shape ingress traffic for you
> would be to do it on the LAN side.
> If you have multiple interfaces facing LAN then use ifb.
> I've never used it myself, but to mark multiple address ranges the
> easiest way would be to use iptables with ipset - there will be many
> examples to be found on the internet.
> Even if you do mark (well set dscp cs1) for steam servers you will still
> need to be backed off from your ingress rate enough or it still won't
> work as the queue will build up too much on the ISP side of the
> bottleneck. Shaping ingress is quite different to doing egress as you
> are at the wrong end of the bottleneck.


In my case switch0 should work for the LAN, just wondering if setting cake
on both pppoe0 (for outbound traffic) and switch0 (fro inbound traffic) at
the same time should work without issues? Hmm, probably gonna try it and
see if it does. I'm gonna look up iptables and ipset and see what I can
find, thanks for the help.

Also I have done some more testing, I was able to limit Steam connections
just to one thanks to some console commands ("@cMaxContentServersToRequest"
and "@cCSClientMaxNumSocketsPerHost") and while the situation improved (no
more packet loss, latency variation within 10ms, but still seeing ping
spikes of ~50ms) it's still not what I'd consider ideal, which would be
like with any other download. So my guess is there's something else going
on other than just the multiple connections, which are definitely big part
of the problem but not the only thing.

Anyway these are my current settings for Cake and I've been using them for
the last four days without issues:

*qdisc cake 8005: root refcnt 2 bandwidth 950Kbit diffserv3 triple-isolate
nat wash rtt 100.0ms atm overhead 40 via-ethernet*
*qdisc cake 8006: root refcnt 2 bandwidth 17500Kbit diffserv3
triple-isolate nat wash ingress rtt 100.0ms atm overhead 40 via-ethernet*

I will try Andy's suggestions and see what I can do.

@Sebastian No problem, I tested your updated script and it is indeed
reporting overhead 40.

On 25 April 2017 at 12:26, Andy Furniss <adf.lists@gmail.com> wrote:

> Dendari Marini wrote:
>
> FWIW here's a quick example on ingress ppp that I tested using
>>> connmark the connmarks (1 or 2 or unmarked) being set by iptables
>>> rules on outbound connections/traffic classes.
>>>
>>
>>
>> Unfortunately I'm really not sure how to apply those settings to my
>> case, it's something I've never done so some hand-holding is probably
>> needed, sorry. At the moment I've limited the Steam bandwidth using
>> the built-in Basic Queue and DPI features from the ER-X. They're easy
>> to set up but aren't really ideal, would rather prefer Cake would
>> take care about it more dynamically.
>>
>> Anyway about the Steam IP addresses I've noticed, in the almost three
>> weeks of testing, they're almost always the same IP blocks (most of
>> which can be found on the Steam Support website,
>> https://support.steampowered.com/kb_article.php?ref=8571-GLVN-8711).
>> I believe it would be a good starting point for limiting Steam, what
>> do you think?
>>
>
> I think the easiest and most robust way to shape ingress traffic for you
> would be to do it on the LAN side.
> If you have multiple interfaces facing LAN then use ifb.
> I've never used it myself, but to mark multiple address ranges the
> easiest way would be to use iptables with ipset - there will be many
> examples to be found on the internet.
> Even if you do mark (well set dscp cs1) for steam servers you will still
> need to be backed off from your ingress rate enough or it still won't
> work as the queue will build up too much on the ISP side of the
> bottleneck. Shaping ingress is quite different to doing egress as you
> are at the wrong end of the bottleneck.
>

[-- Attachment #2: Type: text/html, Size: 5409 bytes --]

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Andy Furniss]

Dendari Marini wrote:

> Also I have done some more testing, I was able to limit Steam 
> connections just to one thanks to some console commands 
> ("@cMaxContentServersToRequest" and
> "@cCSClientMaxNumSocketsPerHost") and while the situation improved
> (no more packet loss, latency variation within 10ms, but still seeing
> ping spikes of ~50ms) it's still not what I'd consider ideal, which
> would be like with any other download. So my guess is there's
> something else going on other than just the multiple connections,
> which are definitely big part of the problem but not the only thing.
> 
> Anyway these are my current settings for Cake and I've been using 
> them for the last four days without issues:
> 
> *qdisc cake 8005: root refcnt 2 bandwidth 950Kbit diffserv3 
> triple-isolate nat wash rtt 100.0ms atm overhead 40 via-ethernet* 
> *qdisc cake 8006: root refcnt 2 bandwidth 17500Kbit diffserv3 
> triple-isolate nat wash ingress rtt 100.0ms atm overhead 40 
> via-ethernet*

I still think that once via-ethernet appears you really need the raw
parameter. On egress ppp it likely subtracts 22 bytes on ifb that is
attached to ingress ppp 14 bytes.
If you shape on real eth (eg. lan side) or ifb hooked from real eth then
you shouldn't use raw.

Thinking about it, mostly you will luck into it not making any
difference, so the test I suggested earlier won't work. The reason being
that whether the overhead is 40 or 40 - 22 an MTU sized packet or an
empty ack/single sack will end up the same due to atm. sack > 1 will be
wrong as of course will be other traffic of certain sizes, so a
carefully crafted test would be needed to show the issue.

Talking of acks/sacks 17500 vs 950kbit doesn't even have room for one
sack per packet, though mostly there will be < 1 ack per packet.

Depending on your sync rate 17500 may be too close for ingress.

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Dendari Marini]

[-- Attachment #1: Type: text/plain, Size: 2761 bytes --]

Hello,

Some good news and some bad news.

The good news is that using switch0 as inbound and pppoe0 as outbound
works, and I was able to set up Steam as bulk using the interface on the
ER-X (used DSCP 8 and used a custom DPI category). I confirmed this was
working by looking at the bulk traffic increasing (using the "tc -s qdisc"
command) and by starting another download (Steam gets pretty much nothing
in this case).

The bad news is this isn't enough to fix my gaming issue (still having ping
spikes, latency variation and packet loss), and even using it with Steam
configured to use just one connection didn't change much from my previous
testing.

So I'm  really confused :\
What could cause ping spikes in this case (assuming the multiple
connections aren't the issue)?

On 25 April 2017 at 14:58, Andy Furniss <adf.lists@gmail.com> wrote:

> Dendari Marini wrote:
>
> Also I have done some more testing, I was able to limit Steam connections
>> just to one thanks to some console commands ("@cMaxContentServersToRequest"
>> and
>> "@cCSClientMaxNumSocketsPerHost") and while the situation improved
>> (no more packet loss, latency variation within 10ms, but still seeing
>> ping spikes of ~50ms) it's still not what I'd consider ideal, which
>> would be like with any other download. So my guess is there's
>> something else going on other than just the multiple connections,
>> which are definitely big part of the problem but not the only thing.
>>
>> Anyway these are my current settings for Cake and I've been using them
>> for the last four days without issues:
>>
>> *qdisc cake 8005: root refcnt 2 bandwidth 950Kbit diffserv3
>> triple-isolate nat wash rtt 100.0ms atm overhead 40 via-ethernet* *qdisc
>> cake 8006: root refcnt 2 bandwidth 17500Kbit diffserv3 triple-isolate nat
>> wash ingress rtt 100.0ms atm overhead 40 via-ethernet*
>>
>
> I still think that once via-ethernet appears you really need the raw
> parameter. On egress ppp it likely subtracts 22 bytes on ifb that is
> attached to ingress ppp 14 bytes.
> If you shape on real eth (eg. lan side) or ifb hooked from real eth then
> you shouldn't use raw.
>
> Thinking about it, mostly you will luck into it not making any
> difference, so the test I suggested earlier won't work. The reason being
> that whether the overhead is 40 or 40 - 22 an MTU sized packet or an
> empty ack/single sack will end up the same due to atm. sack > 1 will be
> wrong as of course will be other traffic of certain sizes, so a
> carefully crafted test would be needed to show the issue.
>
> Talking of acks/sacks 17500 vs 950kbit doesn't even have room for one
> sack per packet, though mostly there will be < 1 ack per packet.
>
> Depending on your sync rate 17500 may be too close for ingress.
>

[-- Attachment #2: Type: text/html, Size: 3464 bytes --]

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Jonathan Morton]

> On 25 Apr, 2017, at 21:22, Dendari Marini <dendari92@gmail.com> wrote:
> 
> The good news is that using switch0 as inbound and pppoe0 as outbound works, and I was able to set up Steam as bulk using the interface on the ER-X (used DSCP 8 and used a custom DPI category). I confirmed this was working by looking at the bulk traffic increasing (using the "tc -s qdisc" command) and by starting another download (Steam gets pretty much nothing in this case).
> 
> The bad news is this isn't enough to fix my gaming issue (still having ping spikes, latency variation and packet loss), and even using it with Steam configured to use just one connection didn't change much from my previous testing.
> 
> So I'm  really confused :\ 
> What could cause ping spikes in this case (assuming the multiple connections aren't the issue)? 

As noted, it’s far more difficult to control latency from downstream of a bottleneck link.  If a bulk sender decides to send burstily, those bursts will always collect in the dumb queue at the far end and delay other traffic.  The only true solution is to install a smart queue at the upstream end - but that’s not under your control.

You may see some improvement from wholesale reducing the inbound bandwidth, to say 10Mbit.  This is especially true given the high asymmetry of your connection, which might require dropped acks upstream to keep filled downstream - and dropped acks will tend to increase burstiness of sending on unpaced senders.

You should also try to ensure ECN is fully enabled on your LAN hosts, especially the ones running Steam.  This will help to reduce retransmissions and loss-recovery cycles.

 - Jonathan Morton

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Dendari Marini]

[-- Attachment #1: Type: text/plain, Size: 1395 bytes --]

On 25 April 2017 at 21:10, Jonathan Morton <chromatix99@gmail.com> wrote:

>
> You may see some improvement from wholesale reducing the inbound
> bandwidth, to say 10Mbit.  This is especially true given the high asymmetry
> of your connection, which might require dropped acks upstream to keep
> filled downstream - and dropped acks will tend to increase burstiness of
> sending on unpaced senders.
>
> You should also try to ensure ECN is fully enabled on your LAN hosts,
> especially the ones running Steam.  This will help to reduce
> retransmissions and loss-recovery cycles.
>
>  - Jonathan Morton
>
>
Well, the only improvement I've seen when limiting the bandwidth with Steam
has been at lower than 1Mbps, don't think I want to go that far. In all
honesty I wouldn't limit it to 10Mbit either, with the overhead it means
half of my total bandwidth, not a trade-off I'm willing to do.

Still, the issue is real and it seems Steam is the only application I can
reproduce it. I've seen reports about Battle.net and Windows Updates doing
the same thing (because they should open multiple concurrent connections),
but I can't reproduce it, at least not in the way Steam does.

Anyway I'm gonna take a "pause" from all of this, I've wasted the last
three weeks ago just for trying resolving it but unfortunately still
nothing. Thanks all for your help, if there's any news I'll report it here.

[-- Attachment #2: Type: text/html, Size: 1992 bytes --]

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Andy Furniss]

Dendari Marini wrote:
> On 25 April 2017 at 21:10, Jonathan Morton <chromatix99@gmail.com>
> wrote:
> 
>> 
>> You may see some improvement from wholesale reducing the inbound 
>> bandwidth, to say 10Mbit.  This is especially true given the high
>> asymmetry of your connection, which might require dropped acks
>> upstream to keep filled downstream - and dropped acks will tend to
>> increase burstiness of sending on unpaced senders.
>> 
>> You should also try to ensure ECN is fully enabled on your LAN
>> hosts, especially the ones running Steam.  This will help to
>> reduce retransmissions and loss-recovery cycles.
>> 
>> - Jonathan Morton
>> 
>> 
> Well, the only improvement I've seen when limiting the bandwidth with
> Steam has been at lower than 1Mbps, don't think I want to go that
> far. In all honesty I wouldn't limit it to 10Mbit either, with the
> overhead it means half of my total bandwidth, not a trade-off I'm
> willing to do.

That is strange, if you are running the ping tests from the same PC
maybe there is something strange going on with windows.

> 
> Still, the issue is real and it seems Steam is the only application I
> can reproduce it. I've seen reports about Battle.net and Windows
> Updates doing the same thing (because they should open multiple
> concurrent connections), but I can't reproduce it, at least not in
> the way Steam does.
> 
> Anyway I'm gonna take a "pause" from all of this, I've wasted the
> last three weeks ago just for trying resolving it but unfortunately
> still nothing. Thanks all for your help, if there's any news I'll
> report it here.
> 

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Benjamin Cronce]

[-- Attachment #1: Type: text/plain, Size: 1897 bytes --]

What's your RTT(ping) to the different services, like Steam and Windows
Update? Some ISPs have local CDNs that can give incredibly low latency
relative to the provisioned bandwidth, which can cause bad things to happen
with TCP.

On Tue, Apr 25, 2017 at 3:44 PM, Dendari Marini <dendari92@gmail.com> wrote:

>
> On 25 April 2017 at 21:10, Jonathan Morton <chromatix99@gmail.com> wrote:
>
>>
>> You may see some improvement from wholesale reducing the inbound
>> bandwidth, to say 10Mbit.  This is especially true given the high asymmetry
>> of your connection, which might require dropped acks upstream to keep
>> filled downstream - and dropped acks will tend to increase burstiness of
>> sending on unpaced senders.
>>
>> You should also try to ensure ECN is fully enabled on your LAN hosts,
>> especially the ones running Steam.  This will help to reduce
>> retransmissions and loss-recovery cycles.
>>
>>  - Jonathan Morton
>>
>>
> Well, the only improvement I've seen when limiting the bandwidth with
> Steam has been at lower than 1Mbps, don't think I want to go that far. In
> all honesty I wouldn't limit it to 10Mbit either, with the overhead it
> means half of my total bandwidth, not a trade-off I'm willing to do.
>
> Still, the issue is real and it seems Steam is the only application I can
> reproduce it. I've seen reports about Battle.net and Windows Updates doing
> the same thing (because they should open multiple concurrent connections),
> but I can't reproduce it, at least not in the way Steam does.
>
> Anyway I'm gonna take a "pause" from all of this, I've wasted the last
> three weeks ago just for trying resolving it but unfortunately still
> nothing. Thanks all for your help, if there's any news I'll report it here.
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>
>

[-- Attachment #2: Type: text/html, Size: 2946 bytes --]

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Dendari Marini]

[-- Attachment #1: Type: text/plain, Size: 3331 bytes --]

Hello,


> Um, I wasn't sure if I should mention it, because it doesn't seem like it
> should be able to cause these kinds of issues. But, if you're using steam *on
> linux*, there's a known bug where it makes hundreds (thousands?) of DNS
> queries per second, during downloads, which can cause issues if the DNS
> server on your router starts throttling. I don't know how or if that should
> affect the apparent performance of cake in different tests. But the
> workaround is to have a local DNS

cache like dnsmasq on your host (and of course it's not an issue on Windows
> machines).




All of my testing were done on Windows machines. My main test PC1 is using
the latest Windows 10 update (Creators Update), while the other PC2 is
using the latest Windows 8.1.


> That is strange, if you are running the ping tests from the same PC
> maybe there is something strange going on with windows.


>
I tested on both of my PCs. Also I didn't just do ping tests, but used real
application like TeamSpeak or games with in-game network tools to analyze
the issue. You can find my Ubiquiti Forums post with some more info in one
of the earlier mail I sent.


> What's your RTT(ping) to the different services, like Steam and Windows
> Update? Some ISPs have local CDNs that can give incredibly low latency
> relative to the provisioned bandwidth, which can cause bad things to happen
> with TCP.
>

>
I tried Battle.net and Steam (manually starting a Windows Update is rather
difficult) and it seems Battle.net servers are closer compared to the Steam
ones (and as I said I don't have the same issues with Battle.net).


>
> On Tue, Apr 25, 2017 at 3:44 PM, Dendari Marini <dendari92@gmail.com>
> wrote:
>
>>
>> On 25 April 2017 at 21:10, Jonathan Morton <chromatix99@gmail.com> wrote:
>>
>>>
>>> You may see some improvement from wholesale reducing the inbound
>>> bandwidth, to say 10Mbit.  This is especially true given the high asymmetry
>>> of your connection, which might require dropped acks upstream to keep
>>> filled downstream - and dropped acks will tend to increase burstiness of
>>> sending on unpaced senders.
>>>
>>> You should also try to ensure ECN is fully enabled on your LAN hosts,
>>> especially the ones running Steam.  This will help to reduce
>>> retransmissions and loss-recovery cycles.
>>>
>>>  - Jonathan Morton
>>>
>>>
>> Well, the only improvement I've seen when limiting the bandwidth with
>> Steam has been at lower than 1Mbps, don't think I want to go that far. In
>> all honesty I wouldn't limit it to 10Mbit either, with the overhead it
>> means half of my total bandwidth, not a trade-off I'm willing to do.
>>
>> Still, the issue is real and it seems Steam is the only application I can
>> reproduce it. I've seen reports about Battle.net and Windows Updates doing
>> the same thing (because they should open multiple concurrent connections),
>> but I can't reproduce it, at least not in the way Steam does.
>>
>> Anyway I'm gonna take a "pause" from all of this, I've wasted the last
>> three weeks ago just for trying resolving it but unfortunately still
>> nothing. Thanks all for your help, if there's any news I'll report it here.
>>
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 6261 bytes --]

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Andy Furniss]

Dendari Marini wrote:

>> That is strange, if you are running the ping tests from the same
>> PC maybe there is something strange going on with windows.

Context lost .... I was referring to 1mbit test failing for you.

> I tested on both of my PCs. Also I didn't just do ping tests, but
> used real application like TeamSpeak or games with in-game network
> tools to analyze the issue. You can find my Ubiquiti Forums post with
> some more info in one of the earlier mail I sent.

OK I haven't looked yet, but have been trying to sim some things.

With the ingress param shaping at 1mbit 5 tcps (cubic or bbr) really
destroys latency.

With the caveat that my test may be flawed, I am currently suspecting
that cake cobalt head + ingress param and a low rate is buggy.

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Jonathan Morton]

> On 29 Apr, 2017, at 18:11, Andy Furniss <adf.lists@gmail.com> wrote:
> 
> With the ingress param shaping at 1mbit 5 tcps (cubic or bbr) really
> destroys latency.
> 
> With the caveat that my test may be flawed, I am currently suspecting
> that cake cobalt head + ingress param and a low rate is buggy.

That’s odd, since I’m currently dogfooding it at 512Kbit, and it works fine like that.  Not to the point of wanting to play online games while torrenting and downloading Steam updates, but that sort of limitation comes with the territory.

With a game updater that uses *80* web-seeds simultaneously (a libtorrent quirk which should get patched in the next version), I can still reliably use my Web browser and e-mail on a second machine; these are things that start to fail intermittently over about 2 seconds RTT, and I’ve measured this ISP at 45 seconds without modification.

The key thing to remember is that in ingress mode, you *must* reduce the shaped rate to some (large) fraction of the bottleneck link, otherwise it won’t control the queue at all.  For example, I’m reasonably sure my current link is dumb-shaped to 576Kbit at the ISP.  The smaller the fraction, the better the control of latency Cake can achieve.

This is in contrast to egress mode, where you want to match the link capacity as closely as possible to get maximum performance; latency control remains ideal as long as you never actually *exceed* the true link capacity.

 - Jonathan Morton

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Andy Furniss]

Jonathan Morton wrote:
> 
>> On 29 Apr, 2017, at 18:11, Andy Furniss <adf.lists@gmail.com>
>> wrote:
>> 
>> With the ingress param shaping at 1mbit 5 tcps (cubic or bbr)
>> really destroys latency.
>> 
>> With the caveat that my test may be flawed, I am currently
>> suspecting that cake cobalt head + ingress param and a low rate is
>> buggy.
> 
> That’s odd, since I’m currently dogfooding it at 512Kbit, and it
> works fine like that.  Not to the point of wanting to play online
> games while torrenting and downloading Steam updates, but that sort
> of limitation comes with the territory.
> 
> With a game updater that uses *80* web-seeds simultaneously (a
> libtorrent quirk which should get patched in the next version), I can
> still reliably use my Web browser and e-mail on a second machine;
> these are things that start to fail intermittently over about 2
> seconds RTT, and I’ve measured this ISP at 45 seconds without
> modification.
> 
> The key thing to remember is that in ingress mode, you *must* reduce
> the shaped rate to some (large) fraction of the bottleneck link,
> otherwise it won’t control the queue at all.  For example, I’m
> reasonably sure my current link is dumb-shaped to 576Kbit at the ISP.
> The smaller the fraction, the better the control of latency Cake can
> achieve.
> 
> This is in contrast to egress mode, where you want to match the link
> capacity as closely as possible to get maximum performance; latency
> control remains ideal as long as you never actually *exceed* the true
> link capacity.

It was a rather artificial test with cake set at 1mbit behind hfsc at
18mbit - just trying to recreate one of Dendari's tests. With the
ingress parameter latency was hurt quite badly compared to without,
which was unexpected. There were a lot of drops - but it seemed like
they were hurting the ping flow. Putting ping into voice didn't help.

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Andy Furniss]

Andy Furniss wrote:
> Jonathan Morton wrote:
>> 
>>> On 29 Apr, 2017, at 18:11, Andy Furniss <adf.lists@gmail.com> 
>>> wrote:
>>> 
>>> With the ingress param shaping at 1mbit 5 tcps (cubic or bbr) 
>>> really destroys latency.
>>> 
>>> With the caveat that my test may be flawed, I am currently 
>>> suspecting that cake cobalt head + ingress param and a low rate
>>> is buggy.
>> 
>> That’s odd, since I’m currently dogfooding it at 512Kbit, and it 
>> works fine like that.  Not to the point of wanting to play online 
>> games while torrenting and downloading Steam updates, but that
>> sort of limitation comes with the territory.
>> 
>> With a game updater that uses *80* web-seeds simultaneously (a 
>> libtorrent quirk which should get patched in the next version), I
>> can still reliably use my Web browser and e-mail on a second
>> machine; these are things that start to fail intermittently over
>> about 2 seconds RTT, and I’ve measured this ISP at 45 seconds
>> without modification.
>> 
>> The key thing to remember is that in ingress mode, you *must*
>> reduce the shaped rate to some (large) fraction of the bottleneck
>> link, otherwise it won’t control the queue at all.  For example,
>> I’m reasonably sure my current link is dumb-shaped to 576Kbit at
>> the ISP. The smaller the fraction, the better the control of
>> latency Cake can achieve.
>> 
>> This is in contrast to egress mode, where you want to match the
>> link capacity as closely as possible to get maximum performance;
>> latency control remains ideal as long as you never actually
>> *exceed* the true link capacity.
> 
> It was a rather artificial test with cake set at 1mbit behind hfsc
> at 18mbit - just trying to recreate one of Dendari's tests. With the 
> ingress parameter latency was hurt quite badly compared to without, 
> which was unexpected. There were a lot of drops - but it seemed like 
> they were hurting the ping flow. Putting ping into voice didn't
> help.

Also seen on a very simple test case with cake on eth. In this case brr
is worse than cubic, but both are "bad".

I am guessing that the ingress param just shouldn't be used when it's
not (as you said) a large fraction of the bottleneck.

I suspect that with low bandwidth + many drops + pretending the drops
were sent, that the whole queue goes over rate, rather than just the
individual flows.

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Jonathan Morton]

> I am guessing that the ingress param just shouldn't be used when it's
> not (as you said) a large fraction of the bottleneck.

That might be a reasonable conclusion, actually.  Here the governing bottleneck is clearly Cake’s shaper as configured, not the physical link.

> I suspect that with low bandwidth + many drops + pretending the drops
> were sent, that the whole queue goes over rate, rather than just the
> individual flows.

Yes, at present only the overall shaper gets the special accounting.  In principle it might also be valid to modify the DRR++ system similarly, so that the fairness metric is also applied to the physical link being controlled, rather than to the delivery rate.

But as long as the difference between the physical link rate and the shaped rate is reasonably small, it works well enough for now.

 - Jonathan Morton

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Andy Furniss]

Dendari Marini wrote:

>> What's your RTT(ping) to the different services, like Steam and 
>> Windows Update? Some ISPs have local CDNs that can give incredibly 
>> low latency relative to the provisioned bandwidth, which can cause 
>> bad things to happen with TCP.

> I tried Battle.net and Steam (manually starting a Windows Update is 
> rather difficult) and it seems Battle.net servers are closer
> compared to the Steam ones (and as I said I don't have the same
> issues with Battle.net).

Well it seems distance is important for BBR. It seems to have a design
whereby your rtt to the server determines how badly it will bork your
latency. Unlike cubic it doesn't take loss/ecn as a hint to get out of
its exponential phase, which is IMHO not a friendly thing to do, I mean
didn't they think people have to share connections :-(

Playing with a sim, something like dash that grabs a meg waits then gets
another with a new connection needs crazy amount of back off to avoid
borking latency every chunk. The amount of disruption getting worse the
higher the RTT of the TCP.

If you don't have a close CDN plus a low latency DSL gaming is not going
to be nice sharing with repeatedly starting up bbr.

With constant download muti-thread sims - like steam, there is sometimes
a case where every 10 seconds there will be a spike as bbr does probes.
This doesn't happen as much with staggered starts for the TCPs. It does
happen with single connections.

By default cake uses rtt parameter of 100ms - this seems to cause
excessive drops/marks causing upstream bandwidth issues on a connection
like yours. Increasing to say 300 reduces drops/marks but does make some
latency spikes a bit worse.

In summary BBR if you are not close to the server is a nightmare to
shape for latency on ingress. I am testing with 40ms rtt, which I don't
thing is excessive. Where I live, luckily youtube/steam are only 8ms and
things are a lot nicer when emulating that.

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Jonathan Morton]

> On 1 May, 2017, at 14:32, Andy Furniss <adf.lists@gmail.com> wrote:
> 
> Well it seems distance is important for BBR. It seems to have a design
> whereby your rtt to the server determines how badly it will bork your
> latency. Unlike cubic it doesn't take loss/ecn as a hint to get out of
> its exponential phase, which is IMHO not a friendly thing to do, I mean
> didn't they think people have to share connections :-(
> 
> Playing with a sim, something like dash that grabs a meg waits then gets
> another with a new connection needs crazy amount of back off to avoid
> borking latency every chunk. The amount of disruption getting worse the
> higher the RTT of the TCP.

Just to be sure - are you running sch_fq on the egress interface(s) of the BBR sender?  If not, *you are not running BBR*.  BBR needs the pacing functionality of sch_fq to work correctly.

 - Jonathan Morton

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Andy Furniss]

Jonathan Morton wrote:
> 
>> On 1 May, 2017, at 14:32, Andy Furniss <adf.lists@gmail.com> 
>> wrote:
>> 
>> Well it seems distance is important for BBR. It seems to have a 
>> design whereby your rtt to the server determines how badly it will 
>> bork your latency. Unlike cubic it doesn't take loss/ecn as a hint 
>> to get out of its exponential phase, which is IMHO not a friendly 
>> thing to do, I mean didn't they think people have to share 
>> connections :-(
>> 
>> Playing with a sim, something like dash that grabs a meg waits
>> then gets another with a new connection needs crazy amount of back
>> off to avoid borking latency every chunk. The amount of disruption 
>> getting worse the higher the RTT of the TCP.
> 
> Just to be sure - are you running sch_fq on the egress interface(s) 
> of the BBR sender?  If not, *you are not running BBR*.  BBR needs
> the pacing functionality of sch_fq to work correctly.
> 

No, I was trying to pretend that the flow was not local (though looking
I see sch_fq can handle routed).

Reading random high level descriptions of BBR I didn't see it mentioned.

I did read that bbr tries to probe 2x rtt, which is what I see in a pure
buffer.

Does google/valve/every bbr user run it?

I mean Dendari is clearly having issues with valve. I can't really test
latency properly with steam on my real wan as my ISP does QOS.

I'll try to cook up another sim, but will probably run out of interfaces.

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Jonathan Morton]

> On 1 May, 2017, at 16:03, Andy Furniss <adf.lists@gmail.com> wrote:
> 
> Does google/valve/every bbr user run it?

By definition, yes.  Without sch_fq, there is no pacing available for BBR to use, and pacing is the mode it normally tries to operate in.  Without pacing, you are not in fact running BBR, but something quite different and inferior.

Please redo your measurements with the correct configuration at the sender.

 - Jonathan Morton

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Andy Furniss]

Jonathan Morton wrote:
> 
>> On 1 May, 2017, at 16:03, Andy Furniss <adf.lists@gmail.com>
>> wrote:
>> 
>> Does google/valve/every bbr user run it?
> 
> By definition, yes.  Without sch_fq, there is no pacing available for
> BBR to use, and pacing is the mode it normally tries to operate in.
> Without pacing, you are not in fact running BBR, but something quite
> different and inferior.
> 
> Please redo your measurements with the correct configuration at the
> sender.

Well I can't really test cake like this. If I need to put fq on eth of
sender then I can't double queue to sim cake behind ISP.

What I can do though is use ingress on target box just to sim ISP buffer
and compare how it gets filled.

So sender, bbr just fq on eth. netem is on ingress.

Target/downloader hfsc on ingress with bfifo sim 18meg dsl oh 40.

Differences to previous setup - One tcp is now very low "ISP" buffer
fullness, before it was a bit higher.

5 tcps are a bit different but may fill the buffer 280ms for a while
before eventually falling back to about 60ms in the case of a staggered
start. Simultaneous start straight to 60ms sometimes falling to 0 then
rising.

Testing repeated one meg downloads looks just the same as my previous
set up (ie if I just looked at "ISP" buffer by setting cake high).

The latency spikes to roughly 2x the netem delay per download. Given
this is the same as seen with my non pacing sim then I expect cake would
have trouble stopping this (though it did make the spikes shorter
time wise and a bit lower depending on backoff amount).

You would miss a lot of this just pinging normally. I am testing as if
on a 30Hz gameserver and plotting pings live (gnuplot).

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Andy Furniss]

Jonathan Morton wrote:
> 
>> On 25 Apr, 2017, at 21:22, Dendari Marini <dendari92@gmail.com>
>> wrote:
>> 
>> The good news is that using switch0 as inbound and pppoe0 as
>> outbound works, and I was able to set up Steam as bulk using the
>> interface on the ER-X (used DSCP 8 and used a custom DPI category).
>> I confirmed this was working by looking at the bulk traffic
>> increasing (using the "tc -s qdisc" command) and by starting
>> another download (Steam gets pretty much nothing in this case).
>> 
>> The bad news is this isn't enough to fix my gaming issue (still
>> having ping spikes, latency variation and packet loss), and even
>> using it with Steam configured to use just one connection didn't
>> change much from my previous testing.
>> 
>> So I'm  really confused :\ What could cause ping spikes in this
>> case (assuming the multiple connections aren't the issue)?
> 
> As noted, it’s far more difficult to control latency from downstream
> of a bottleneck link.  If a bulk sender decides to send burstily,
> those bursts will always collect in the dumb queue at the far end and
> delay other traffic.  The only true solution is to install a smart
> queue at the upstream end - but that’s not under your control.
> 
> You may see some improvement from wholesale reducing the inbound
> bandwidth, to say 10Mbit.  This is especially true given the high
> asymmetry of your connection, which might require dropped acks
> upstream to keep filled downstream - and dropped acks will tend to
> increase burstiness of sending on unpaced senders.
> 
> You should also try to ensure ECN is fully enabled on your LAN hosts,
> especially the ones running Steam.  This will help to reduce
> retransmissions and loss-recovery cycles.

Yea, good idea - hopefully steam servers will honor that.

Plus remember what I said about raw - you have an upstream fail case
with sacks - also see below.

I just looked at a tcpdump I made last night doing a steam d/l, mainly
to see if I got servers in the list from the steam link posted, I did.

The other stand out thing is there are far more sacks than I would
normally expect to see.

tcpdump -nnr steam2.pcap src host 192.168.0.224 | grep "sack" | wc -l
reading from file steam2.pcap, link-type EN10MB (Ethernet)
28298

tcpdump -nnr steam2.pcap src host 192.168.0.224 | wc -l
reading from file steam2.pcap, link-type EN10MB (Ethernet)
44094

incoming count -

tcpdump -nnr steam2.pcap dst host 192.168.0.224 | wc -l
reading from file steam2.pcap, link-type EN10MB (Ethernet)
59319

That's a crazy amount especially as being in recovery means ack per packet.

My connection is quite different from the OPs of course, ptm 66 meg sync
with cake at 60mbit for this test and the pcap was only over 12 seconds.
The servers are only 8ms away from me, 5 connections.

I wonder if they are using bbr or something.

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Neil Shepperd]

[-- Attachment #1: Type: text/plain, Size: 3842 bytes --]

Um, I wasn't sure if I should mention it, because it doesn't seem like it
should be able to cause these kinds of issues. But, if you're using steam *on
linux*, there's a known bug where it makes hundreds (thousands?) of DNS
queries per second, during downloads, which can cause issues if the DNS
server on your router starts throttling. I don't know how or if that should
affect the apparent performance of cake in different tests. But the
workaround is to have a local DNS cache like dnsmasq on your host (and of
course it's not an issue on Windows machines).

On Tue, 25 Apr 2017 at 17:06 Andy Furniss <adf.lists@gmail.com> wrote:

> Jonathan Morton wrote:
> >
> >> On 25 Apr, 2017, at 21:22, Dendari Marini <dendari92@gmail.com>
> >> wrote:
> >>
> >> The good news is that using switch0 as inbound and pppoe0 as
> >> outbound works, and I was able to set up Steam as bulk using the
> >> interface on the ER-X (used DSCP 8 and used a custom DPI category).
> >> I confirmed this was working by looking at the bulk traffic
> >> increasing (using the "tc -s qdisc" command) and by starting
> >> another download (Steam gets pretty much nothing in this case).
> >>
> >> The bad news is this isn't enough to fix my gaming issue (still
> >> having ping spikes, latency variation and packet loss), and even
> >> using it with Steam configured to use just one connection didn't
> >> change much from my previous testing.
> >>
> >> So I'm  really confused :\ What could cause ping spikes in this
> >> case (assuming the multiple connections aren't the issue)?
> >
> > As noted, it’s far more difficult to control latency from downstream
> > of a bottleneck link.  If a bulk sender decides to send burstily,
> > those bursts will always collect in the dumb queue at the far end and
> > delay other traffic.  The only true solution is to install a smart
> > queue at the upstream end - but that’s not under your control.
> >
> > You may see some improvement from wholesale reducing the inbound
> > bandwidth, to say 10Mbit.  This is especially true given the high
> > asymmetry of your connection, which might require dropped acks
> > upstream to keep filled downstream - and dropped acks will tend to
> > increase burstiness of sending on unpaced senders.
> >
> > You should also try to ensure ECN is fully enabled on your LAN hosts,
> > especially the ones running Steam.  This will help to reduce
> > retransmissions and loss-recovery cycles.
>
> Yea, good idea - hopefully steam servers will honor that.
>
> Plus remember what I said about raw - you have an upstream fail case
> with sacks - also see below.
>
> I just looked at a tcpdump I made last night doing a steam d/l, mainly
> to see if I got servers in the list from the steam link posted, I did.
>
> The other stand out thing is there are far more sacks than I would
> normally expect to see.
>
> tcpdump -nnr steam2.pcap src host 192.168.0.224 | grep "sack" | wc -l
> reading from file steam2.pcap, link-type EN10MB (Ethernet)
> 28298
>
> tcpdump -nnr steam2.pcap src host 192.168.0.224 | wc -l
> reading from file steam2.pcap, link-type EN10MB (Ethernet)
> 44094
>
> incoming count -
>
> tcpdump -nnr steam2.pcap dst host 192.168.0.224 | wc -l
> reading from file steam2.pcap, link-type EN10MB (Ethernet)
> 59319
>
> That's a crazy amount especially as being in recovery means ack per packet.
>
> My connection is quite different from the OPs of course, ptm 66 meg sync
> with cake at 60mbit for this test and the pcap was only over 12 seconds.
> The servers are only 8ms away from me, 5 connections.
>
> I wonder if they are using bbr or something.
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>

[-- Attachment #2: Type: text/html, Size: 4669 bytes --]

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Andy Furniss]

Neil Shepperd wrote:
> Um, I wasn't sure if I should mention it, because it doesn't seem
> like it should be able to cause these kinds of issues. But, if you're
> using steam *on linux*, there's a known bug where it makes hundreds
> (thousands?) of DNS queries per second, during downloads, which can
> cause issues if the DNS server on your router starts throttling. I
> don't know how or if that should affect the apparent performance of
> cake in different tests. But the workaround is to have a local DNS
> cache like dnsmasq on your host (and of course it's not an issue on
> Windows machines).

Interesting, I didn't know that, I did the pcap from my Linux router
box, steam it's self was running on Windows. Judging by the use of
hrping earlier I assume Dendari is also running windows.

I am 99.9% booted into Linux on my desktop - but it's pure 64bit LFS and
I am far too lazy to build a multilib LFS just for steam :-)

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Sebastian Moeller]

Hi Andy,

> On Apr 25, 2017, at 14:58, Andy Furniss <adf.lists@gmail.com> wrote:
> 
> Dendari Marini wrote:
> 
>> Also I have done some more testing, I was able to limit Steam connections just to one thanks to some console commands ("@cMaxContentServersToRequest" and
>> "@cCSClientMaxNumSocketsPerHost") and while the situation improved
>> (no more packet loss, latency variation within 10ms, but still seeing
>> ping spikes of ~50ms) it's still not what I'd consider ideal, which
>> would be like with any other download. So my guess is there's
>> something else going on other than just the multiple connections,
>> which are definitely big part of the problem but not the only thing.
>> Anyway these are my current settings for Cake and I've been using them for the last four days without issues:
>> *qdisc cake 8005: root refcnt 2 bandwidth 950Kbit diffserv3 triple-isolate nat wash rtt 100.0ms atm overhead 40 via-ethernet* *qdisc cake 8006: root refcnt 2 bandwidth 17500Kbit diffserv3 triple-isolate nat wash ingress rtt 100.0ms atm overhead 40 via-ethernet*
> 
> I still think that once via-ethernet appears you really need the raw
> parameter.

	Why? As far as I can tell the default mode is to adjust for the kernel auto-added overhead (which seems to be 14 bytes for pakets headed for an ethernet device and zero for packets headed for a ppp device). If he adds raw, the kernel willl make its additions silently…

Independent of that matter one needs to specify different overheads on ethN and pppeN interfaces simply as the packets on the pppoN interface have not yet been encpsulated and are 8 byte smaller than on the respective ethN interface. I am by now thoroghly confused, so I might have missed the subtleties in which either cake or my understanding is wrong.

> On egress ppp it likely subtracts 22 bytes on ifb that is
> attached to ingress ppp 14 bytes.

	My understanding is again that on pppoe devices the kernel adds zero bytes auto matically and attaching the ifb does not seem to change that?

> If you shape on real eth (eg. lan side) or ifb hooked from real eth then
> you shouldn't use raw.

	Ah, I thought that the non-raw mode inquires at the kernel what overhead it intends to account for silently and simply undoes that probably by changing the overhead passed to the kernel so that the sum of both match what the user requested?

> 
> Thinking about it, mostly you will luck into it not making any
> difference, so the test I suggested earlier won't work. The reason being
> that whether the overhead is 40 or 40 - 22 an MTU sized packet or an
> empty ack/single sack will end up the same due to atm. sack > 1 will be
> wrong as of course will be other traffic of certain sizes, so a
> carefully crafted test would be needed to show the issue.
> 
> Talking of acks/sacks 17500 vs 950kbit doesn't even have room for one
> sack per packet, though mostly there will be < 1 ack per packet.

Intersting, looking into this I learned that SACK can cause up to 40 bytes of TCP options added to the packet...

> 
> Depending on your sync rate 17500 may be too close for ingress.

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Andy Furniss]

Sebastian Moeller wrote:
> Hi Andy,
> 
>> On Apr 25, 2017, at 14:58, Andy Furniss <adf.lists@gmail.com>
>> wrote:
>> 
>> Dendari Marini wrote:
>> 
>>> Also I have done some more testing, I was able to limit Steam
>>> connections just to one thanks to some console commands
>>> ("@cMaxContentServersToRequest" and 
>>> "@cCSClientMaxNumSocketsPerHost") and while the situation
>>> improved (no more packet loss, latency variation within 10ms, but
>>> still seeing ping spikes of ~50ms) it's still not what I'd
>>> consider ideal, which would be like with any other download. So
>>> my guess is there's something else going on other than just the
>>> multiple connections, which are definitely big part of the
>>> problem but not the only thing. Anyway these are my current
>>> settings for Cake and I've been using them for the last four days
>>> without issues: *qdisc cake 8005: root refcnt 2 bandwidth 950Kbit
>>> diffserv3 triple-isolate nat wash rtt 100.0ms atm overhead 40
>>> via-ethernet* *qdisc cake 8006: root refcnt 2 bandwidth 17500Kbit
>>> diffserv3 triple-isolate nat wash ingress rtt 100.0ms atm
>>> overhead 40 via-ethernet*
>> 
>> I still think that once via-ethernet appears you really need the
>> raw parameter.
> 
> Why? As far as I can tell the default mode is to adjust for the
> kernel auto-added overhead (which seems to be 14 bytes for pakets
> headed for an ethernet device and zero for packets headed for a ppp
> device). If he adds raw, the kernel willl make its additions
> silently…

I shape on pppoe - though it's ptm. My overhead is 34, it seems as soon
as you add overhead as a parameter via-ethernet appears and in the case
of pppoe apparently does the wrong thing. Recalling my now closed issue
about overheads on github, cake beleives what the kernel tells it about
overheads and this seems to 22. In fact cake does not see the traffic as
+22 it sees ip length.

If I don't use raw below my shaper fails. Also note that when using raw
the overhead gets increased by 22 from that specified.

tc qdisc add dev ppp0 handle 1:0 root cake bandwidth 19690kbit raw 
overhead 34 diffserv4 dual-srchost nat rtt 200ms

asr[/home/andy]# tc -s qdisc ls dev ppp0
qdisc cake 1: root refcnt 2 bandwidth 19690Kbit diffserv4 dual-srchost 
nat rtt 200.0ms noatm overhead 56 via-ethernet
  Sent 1311201393 bytes 5277696 pkt (dropped 128, overlimits 1606291 
requeues 0)
  backlog 0b 0p requeues 0
  memory used: 115328b of 4Mb
  capacity estimate: 19690Kbit
                  Bulk   Best Effort      Video       Voice
   thresh      1230Kbit   19690Kbit    9845Kbit    4922Kbit
   target        14.8ms      10.0ms      10.0ms      10.0ms
   interval     204.8ms     200.0ms     200.0ms     200.0ms
   pk_delay         6us        25us         2us       209us
   av_delay         3us         2us         2us        31us
   sp_delay         1us         1us         1us         3us
   pkts          927088     4133045      118552       99139
   bytes      974738943   320755191     3326412    12572807
   way_inds        7798       33031           0           2
   way_miss       45787       84501          40        8687
   way_cols           0           0           0           0
   drops              0         128           0           0
   marks              0           0           0           0
   sp_flows           1           0           0           0
   bk_flows           1           0           0           0
   un_flows           0           0           0           0
   max_len         1500        1500          84        1428



> 
> Independent of that matter one needs to specify different overheads
> on ethN and pppeN interfaces simply as the packets on the pppoN
> interface have not yet been encpsulated and are 8 byte smaller than
> on the respective ethN interface. I am by now thoroghly confused, so
> I might have missed the subtleties in which either cake or my
> understanding is wrong.
> 
>> On egress ppp it likely subtracts 22 bytes on ifb that is attached
>> to ingress ppp 14 bytes.
> 
> My understanding is again that on pppoe devices the kernel adds zero
> bytes auto matically and attaching the ifb does not seem to change
> that?

The packet size is ip length as seen by cake on ifb redirected from
pppoe - but this time it seems the difference is 14 not 22 ...

tc qdisc add dev ifb0 handle 1:0 root cake bandwidth 60mbit raw overhead 
34 diffserv4 nat dual-dsthost

asr[/home/andy]# tc -s qdisc ls dev ifb0
qdisc cake 1: root refcnt 2 bandwidth 60Mbit diffserv4 dual-dsthost nat 
rtt 100.0ms noatm overhead 48 via-ethernet
  Sent 14315263754 bytes 12978554 pkt (dropped 51315, overlimits 
18015371 requeues 0)
  backlog 0b 0p requeues 0
  memory used: 1006656b of 4Mb
  capacity estimate: 60Mbit
                  Bulk   Best Effort      Video       Voice
   thresh      3750Kbit      60Mbit      30Mbit      15Mbit
   target         5.0ms       5.0ms       5.0ms       5.0ms
   interval     100.0ms     100.0ms     100.0ms     100.0ms
   pk_delay       243us       1.2ms        42us        82us
   av_delay        18us       821us         7us         8us
   sp_delay         2us         3us         4us         3us
   pkts         1942903    10462709      272777      351480
   bytes      130598812 14085111253     8318580   168132633
   way_inds         220       46306           0           0
   way_miss       68580      114792         188       84890
   way_cols           0           0           0           0
   drops              0       51308           0           7
   marks              0           0           0           0
   sp_flows           0           0           1           0
   bk_flows           0           0           0           0
   un_flows           0           0           0           0
   max_len         1500        1500        1500        1441



> 
>> If you shape on real eth (eg. lan side) or ifb hooked from real eth
>> then you shouldn't use raw.
> 
> Ah, I thought that the non-raw mode inquires at the kernel what
> overhead it intends to account for silently and simply undoes that
> probably by changing the overhead passed to the kernel so that the
> sum of both match what the user requested?

I think that things change when you add overhead XX (though I may need
to retest that on a normal eth), but the reason I thought not to use raw
on lan side was that cake really will see the packets as +14 so you want
via-ethernet to subtract 14.

> 
>> 
>> Thinking about it, mostly you will luck into it not making any 
>> difference, so the test I suggested earlier won't work. The reason
>> being that whether the overhead is 40 or 40 - 22 an MTU sized
>> packet or an empty ack/single sack will end up the same due to atm.
>> sack > 1 will be wrong as of course will be other traffic of
>> certain sizes, so a carefully crafted test would be needed to show
>> the issue.
>> 
>> Talking of acks/sacks 17500 vs 950kbit doesn't even have room for
>> one sack per packet, though mostly there will be < 1 ack per
>> packet.
> 
> Intersting, looking into this I learned that SACK can cause up to 40
> bytes of TCP options added to the packet...

Yea, not nice with overhead of 40 as 3 cells for multi sacks - but even
at 2 cells Dendari doesn't have enough bandwidth for 1 ack per packet
during recovery.

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Andy Furniss]

Andy Furniss wrote:

>> My understanding is again that on pppoe devices the kernel adds zero
>> bytes auto matically and attaching the ifb does not seem to change
>> that?
> 
> The packet size is ip length as seen by cake on ifb redirected from
> pppoe - but this time it seems the difference is 14 not 22 ...

To be clear below is redirecting ingress qdisc on ppp0 to ifb0.

> 
> tc qdisc add dev ifb0 handle 1:0 root cake bandwidth 60mbit raw overhead 
> 34 diffserv4 nat dual-dsthost
> 
> asr[/home/andy]# tc -s qdisc ls dev ifb0
> qdisc cake 1: root refcnt 2 bandwidth 60Mbit diffserv4 dual-dsthost nat 
> rtt 100.0ms noatm overhead 48 via-ethernet
>   Sent 14315263754 bytes 12978554 pkt (dropped 51315, overlimits 
> 18015371 requeues 0)
>   backlog 0b 0p requeues 0
>   memory used: 1006656b of 4Mb
>   capacity estimate: 60Mbit
>                   Bulk   Best Effort      Video       Voice
>    thresh      3750Kbit      60Mbit      30Mbit      15Mbit
>    target         5.0ms       5.0ms       5.0ms       5.0ms
>    interval     100.0ms     100.0ms     100.0ms     100.0ms
>    pk_delay       243us       1.2ms        42us        82us
>    av_delay        18us       821us         7us         8us
>    sp_delay         2us         3us         4us         3us
>    pkts         1942903    10462709      272777      351480
>    bytes      130598812 14085111253     8318580   168132633
>    way_inds         220       46306           0           0
>    way_miss       68580      114792         188       84890
>    way_cols           0           0           0           0
>    drops              0       51308           0           7
>    marks              0           0           0           0
>    sp_flows           0           0           1           0
>    bk_flows           0           0           0           0
>    un_flows           0           0           0           0
>    max_len         1500        1500        1500        1441

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Andy Furniss]

Andy Furniss wrote:
> Andy Furniss wrote:
> 
>>> My understanding is again that on pppoe devices the kernel adds
>>> zero bytes auto matically and attaching the ifb does not seem to
>>> change that?
>> 
>> The packet size is ip length as seen by cake on ifb redirected
>> from pppoe - but this time it seems the difference is 14 not 22
>> ...
> 
> To be clear below is redirecting ingress qdisc on ppp0 to ifb0.

>> max_len         1500        1500        1500        1441

Oops sent to early.

Another clarification the 1500s really are ip length I run mini-jumbos
so the mtu on ppp0 is 1500.

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Andy Furniss]

Dendari Marini wrote:
> Hello, thank you all for your replies!
> 
> For the overhead I'm gonna use "pppoe-llcsnap" (or "overhead 40 atm),
> as I believe it's the one which should work best for my connection.

I seem to have confused my self as to whether you are using raw or not.

If you are not then I think that shaping on pppoe just overhead 40 atm
will be 22 bytes too little ....

Maybe what version of tc and or cake matters here, I don't know.

Maybe others could comment on that.

One simple test is set your up bandwidth just a little below your syn
rate and upload while pinging somewhere - if you are over then the pings
will rise.
In theory you should be able to run up almost at line rate - back off a
tiny bit to allow for LCP.
As you have a highly asymmetric connection you don't really want to
sacrifice more upload than is needed. A lot of your up bandwidth will be
needed just for acks.

> About the per-host fairness download issue: while it's kinda resolved
> I still feel like it's mainly related to Steam, as normally
> downloading files from PC1 and PC2 halved the speed as expected even
> at full bandwidth (so no overhead, no -15%).

Shaping too close to the real bandwidth is somewhat exposed more when
there are many connections involved.

> Anyway back to Steam: assuming the IP addresses aren't a big issue,
> what steps should I follow to start filtering its traffic so it's
> considered background? Would the DPI from the ER-X be any helpful?

It's a shame steam doesn't allow you to tell it to set tos/dscp like
some torrent clients do.

You could then use connmark to classify the incoming. Whatever method
you find to identify steam downloads it may be a bit hit and miss with
the dscp marking as on ingress you need to get tc to call iptables,
which seems to need a bit of luck with versions matching up properly.

Maybe on a pure router (ie. no significant traffic from wan to router)
you could consider shaping inbound traffic on the lan side.

FWIW here's a quick example on ingress ppp that I tested using connmark
the connmarks (1 or 2 or unmarked) being set by iptables rules on outbound
connections/traffic classes.

tc qdisc add dev ppp0 ingress
tc filter add dev ppp0 parent ffff: prio 1 protocol ip u32 match u32 0 0 
action connmark continue
tc filter add dev ppp0 parent ffff: prio 2 protocol ip handle 1 fw 
action xt -j DSCP --set-dscp-class cs1 action mirred egress redirect dev 
ifb0
tc filter add dev ppp0 parent ffff: prio 3 protocol ip handle 2 fw 
action xt -j DSCP --set-dscp-class cs4 action mirred egress redirect dev 
ifb0
tc filter add dev ppp0 parent ffff: prio 4 protocol ip u32 match u32 0 0 
  action mirred egress redirect dev ifb0

If you use ipv6 you would need to handle that as this is ipv4 only.

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Sebastian Moeller]

Hi Dendari,

> On Apr 22, 2017, at 10:25, Dendari Marini <dendari92@gmail.com> wrote:
> 
> Hi Sebastian,
> 
> Only now I noticed your message, it seems we were writing at the same time.
> 
> So please add “atm overhead 32" to cake on eth0 or “atm overhead 40” to cake instances on pppoe (these packets do not have the PPPoE header added yet and hence appear 8 bytes to small).
> 
> Thanks for your help, will definitely use them. Just wondering if I use "pppoe-vcmux/bridged-llcsnap" on eth0 or "pppoe-llcsnap" on pppoe0 would have the same effect? Or are there some other "under-the-hood" changes when using them?

	The compound keywords like pppoe-llcsnap are effectively short-hands for “atm overhead N” with N depending on the name. Some people prefer the symbolic names to the numeric specification (see Jonathan’s mail), I prefer the “full truth” of the numeric specifier. Which keyword corresponds exactly to “atm overhead 32” or “atm overhead 40” I do not know by heart, I would either look into the source code of the cake module for iproute2/tc or cake’s man page. That is if I would actually want to use the symbolic keywords. It really is a question of personal preference, I guess...

> 
> Question: if you set the shaper’s to 50% of line rate (8.75/0.5?) do you still see that unfairness? And if you add “atm overhead 40” to cake on pppoe0 and set the shaper to 90% of line rates (15.75/0.9) how does the Steam affect per-host fairness? Also how transient are these connections team uses?
> 
> Actually did more testing about this and it seems that as far I have set the bandwidth to ~15Mbps (so ~15% less of my max speed) and use the "nat" parameter, the per-host fairness works even without the "dual-host" and "overhead" parameters. I definitely find this very interesting, is this behaviour caused by the way Steam downloads games?

	Cake defaults to triple-isolate, which with nat active will sort of avoid any host IP internal or external hogging the link. Correct overhead specification is more important to correctly model what bits are actually send over the bottleneck, only if this is precisely specified will cake reduce bufferbloat even with tight shaper settings (close to the true bottleneck). I then to think of per-host-isolation and link-layer-accounting as two rather orthogonal issues, that both affect a link’s performance...


> 
> As far as I can tell cake can drill down to the required IP/TCP/UDP fields independent of whether there are VLAN tags or PPPoE headers so cake should not care (except for the different overhead specifications you need to add as stated above). BUT if instantiated on eth0 cake will see pppoe LCP packets and might decide to drop them, which can take down the link, so out of caution I would still instantiate on pppoe in your case.
> 
> Yeah, with further testing it seems the interface wasn't the culprit but I'll still do all my testing on pppoe0 just to be safe.

	Okay, good to know.

> 
> Anyway I was wondering if there's some kind of manual for Cake and the various parameters, I'm looking to set it up best way possible but there are some parameters which I'm not sure what they do (one of them being "ingress"). Also while reading on the bufferbloat.net Cake page I noticed a possible "fix" for BitTorrent (by setting it as "background", https://www.bufferbloat.net/projects/codel/wiki/Cake/#diffserv-support), I'm wondering if this can be done with Steam too?

	Well in theory using the correct DSCP markings together with a matching diffserv mode in cake should allow to relegate any wanted traffic patterns to a nicely bandwidth-yielding background queue. The challenge really is to properly label the packets, which especially on ingress is hard.

Best Regards
	Sebastian


> 
> On 21 April 2017 at 15:27, Dendari Marini <dendari92@gmail.com> wrote:
> Hello, I have an update.
> 
> The download test (PC1 downloading Steam game, PC2 downloading a file) seems to work fine now, even while using higher bandwidth value. So that seems resolved, probably the interface change was the fix.
> 
> Now about the gaming test (PC1 downloading Steam game, PC2 online gaming), this is the one I'm mostly looking forward to fix and the main reason I bought the ER-X. Unfortunately Cake doesn't seem to do much in this case, specifically I'm still getting high latency spikes and packet loss on PC2. One weird thing: starting a download on PC2 will decrease the lantecy/p.loss issues, not sure if it's because of the Steam speed being halved on PC1 or something else.
> 
> On 21 April 2017 at 10:34, Dendari Marini <dendari92@gmail.com> wrote:
> Hello, thanks for all of your replies.
> 
> First of all, my connection encapsulation should be ATM LLC and it can actually reach up to 17.5/1 Mbps, but that's kinda best case scenario which is why I wanted to play it safe with just 16/.9 (which I should reach more consistently).
> 
> Back to the Steam issue. Unfortunately I can't seem to get really consistent results, mainly because sometimes it's downloading the game from just a few connections and other times it's downloading from dozens and dozens connections. The latter is the one giving me more issues both in terms of latency/packet loss and in terms of evenly splitting the bandwidth across the hosts.
> 
> One thing that seems to give better results is changing the interface where Cake is used from eth0 to pppoe0. When I used fq_codel it seemed to give better results when using eth0 and so I went ahead and did the same with Cake.
> 
> Anyway more testing needed, will report if I notice any consistent result.
> 
> By the way this is the thread I opened on the Ubiquiti forums talking about this issue (not sure if it can give you some more info): https://community.ubnt.com/t5/EdgeMAX/Smart-Queue-seemingly-not-working-for-Steam-downloads/td-p/1890405
> Also the thread where I got Cake for the ER-X from: https://community.ubnt.com/t5/EdgeMAX/Cake-and-FQ-PIE-compiled-for-the-EdgeRouter-devices/td-p/1679844
> 
> On 20 April 2017 at 20:36, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> > On Apr 20, 2017, at 18:05, Dendari Marini <dendari92@gmail.com> wrote:
> >
> > Hello, thanks for your reply.
> >
> > Looks like most of your options are okay, including the correct “dual” modes and “ingress” mode in the right place.  However, I think you need to adjust your bandwidth and overhead settings, otherwise Cake isn’t reliably in control of the bottleneck queues.  Try these to begin with:
> >
> > … bandwidth 850Kbit conservative dual-srchost nat
> >
> > … bandwidth 15Mbit conservative dual-dsthost nat ingress
> >
> > That should give you correct operation, and you can fine-tune from there.
> >
> > Just did quick test with your settings. First thing I noticed is my final download bandwidth is about 12Mbps, Steam on PC1 downloads at 1.4-1.5MB/s while downloading a file on PC2 seems to max out at ~250KB/s. From my understanding I should see each PC download at ~700KB/s, or am I mistaken?
> 
> Assuming you measured good put in [M|K]iBytes this adds up to  1.5+0.25 = 1.75 * 1024^2 * 8 = 14680064 Bits or (1.4+0.25) * 8 *1024^2 / 1000^2 = 13.84 Mbps which seems a bit high for a 16Mbps ADSL link. I would ecpext something like 16 * (48/53)  * ((1500 - 8 - 20 -20) / (1500 + 32)) = 13.73 Mbps TCP/IPv4 goodput… so you seem to be running close to theoretical maximum of your link (assuming I am not totally off with the overhead (estimated ADSL overhead on top of MTU: 6 destination MAC + 6 source MAC + 2 ethertype + 3 ATM LLC + 5 ATM SNAP + 2 ATM pad + 8 ATM AAL5 SAR 32 bytes). But with your shaper set at 15Mbps without the atm option you will actually accept up to 15 * (53/48) = 16.5625 Mbps on the wire, which probably is above your link bandwidth. This fits well with the really low number of drops in your cake stats, you simply never have cake feel that shaping is needed?
> 
> Best Regards
> 
> 
> 
> 
> >
> > On 20 April 2017 at 17:32, Jonathan Morton <chromatix99@gmail.com> wrote:
> >
> >> On 20 Apr, 2017, at 18:23, Dendari Marini <dendari92@gmail.com> wrote:
> >>
> >>> Could you post the output of calling “tc -s qdisc” here on the list please? That should allow to figure out what you actually told cake to do ;0
> >
> >> qdisc cake 8001: dev eth0 root refcnt 2 bandwidth 900Kbit diffserv3 dual-srchost nat rtt 100.0ms raw
> >
> >> qdisc cake 8002: dev ifb4eth0 root refcnt 2 bandwidth 16Mbit diffserv3 dual-dsthost nat ingress rtt 100.0ms raw
> >
> > Looks like most of your options are okay, including the correct “dual” modes and “ingress” mode in the right place.  However, I think you need to adjust your bandwidth and overhead settings, otherwise Cake isn’t reliably in control of the bottleneck queues.  Try these to begin with:
> >
> > … bandwidth 850Kbit conservative dual-srchost nat
> >
> > … bandwidth 15Mbit conservative dual-dsthost nat ingress
> >
> > That should give you correct operation, and you can fine-tune from there.
> >
> > - Jonathan Morton
> >
> >
> >
> 
> 
> 
> 

Re: [Cake] Getting Cake to work better with Steam and similar applications

[Sebastian Moeller]

Hi Dendari,


> On Apr 20, 2017, at 17:23, Dendari Marini <dendari92@gmail.com> wrote:
> 
> qdisc cake 8002: dev ifb4eth0 root refcnt 2 bandwidth 16Mbit diffserv3 dual-dsthost nat ingress rtt 100.0ms raw 

It seems you are lacking the proper link layer accounting, with an ADSL line you in all likelihood will need to add the atm keyword for both directions. I also concur with Jonathan that you shou;d set the correct overhead parameter, as a first test adding “atm overhead 48” to your cake invocation should help to test this theory. If this helps you could try to inquire from your ISP which encapsulation is used (or use https://github.com/moeller0/ATM_overhead_detector to try to empirically figure out the actual overhead).

Best Regards