From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-x22a.google.com (mail-wm0-x22a.google.com [IPv6:2a00:1450:400c:c09::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 145103B2A4 for ; Sat, 22 Apr 2017 04:25:21 -0400 (EDT) Received: by mail-wm0-x22a.google.com with SMTP id w64so30098938wma.0 for ; Sat, 22 Apr 2017 01:25:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=pTHB0QryTw2yWZUwcSNOPhbk+SMykjBwp1tTEkd4IvA=; b=Q5AQK5aBrGLa4A9U72tjw+XcYJqV9ELYdgl3EGxyrNwMWZAkmr/FYpwvyc5NMT+NHd d8HsLWVCttNsVEwTTdo+Ye1F5vbXE1kwqmdqPeCv8ZSh7C5YGEgImIdQnkhqbI27cMB6 pGZ0BoPcYxTeG6bwlNWMIlNWpxGPqz6fZYGFrZ/eiywl7BE0qfznQknG+qBPnlPTizZ9 1ZLYeNsPns2Lp4h6fjvvejEqWgjWaWRfVZ/68LuDl84lqvV1NcQV8pVXlmJwGWnnY5ev ufQHHzt/LUhOOWXAWzoOIGrJMnY+13pFeT+ZROtIH6tHHevFiuzm2Uhq01RXF392HoE9 Vhhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=pTHB0QryTw2yWZUwcSNOPhbk+SMykjBwp1tTEkd4IvA=; b=qafTGQNdNY56lrU5GAziQnhO8xzaQ2RyAteYR7XxQZsVEQo7p31fYu+R9u6nD4tfzi btuMp0b/TVu07jT6P6Z3BkDM5cO8CPVFV47fguDOxCZKECOVHDagdofqaWrSHZFk/lNi e08uxVt6R1jjoqyPA8DLgvubA/u/Vlw5tXNjPtyI+1RxsXtrYf1nGLvi0EsAzxeCjkhL HpEMFj5CZfC4AMEjci1TDjlyflqylO5XS/jb8qRwGWN/TydaVGHQU0nl/E4q0ojOWo5c rjEpEiWeXyL3Dy1N8pLB/15O3uv2INARJr/HaqOinx7UKsTU7aNDAoZ/Yk23vmHPz0us gCrw== X-Gm-Message-State: AN3rC/70RooJKJVRER/ha9SBh7laUlGTubU2uCmU9k8v1RsSYg68V6CP SesBMU/NXYIwl16mPcltkhONAZ0V9g== X-Received: by 10.28.195.7 with SMTP id t7mr2181931wmf.62.1492849520740; Sat, 22 Apr 2017 01:25:20 -0700 (PDT) MIME-Version: 1.0 Received: by 10.80.169.59 with HTTP; Sat, 22 Apr 2017 01:25:20 -0700 (PDT) In-Reply-To: References: <05C0B0C7-4337-4115-AC6B-DA81392FCB34@gmail.com> <22E633CF-5EE0-4B0F-89A8-B790E730FB6C@gmx.de> From: Dendari Marini Date: Sat, 22 Apr 2017 10:25:20 +0200 Message-ID: To: Sebastian Moeller Cc: Jonathan Morton , cake@lists.bufferbloat.net Content-Type: multipart/alternative; boundary=001a114a0e6c630dd0054dbd1d6c Subject: Re: [Cake] Getting Cake to work better with Steam and similar applications X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Apr 2017 08:25:22 -0000 --001a114a0e6c630dd0054dbd1d6c Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Sebastian, Only now I noticed your message, it seems we were writing at the same time. So please add =E2=80=9Catm overhead 32" to cake on eth0 or =E2=80=9Catm ove= rhead 40=E2=80=9D to > cake instances on pppoe (these packets do not have the PPPoE header added > yet and hence appear 8 bytes to small). Thanks for your help, will definitely use them. Just wondering if I use "pppoe-vcmux/bridged-llcsnap" on eth0 or "pppoe-llcsnap" on pppoe0 would have the same effect? Or are there some other "under-the-hood" changes when using them? Question: if you set the shaper=E2=80=99s to 50% of line rate (8.75/0.5?) d= o you > still see that unfairness? And if you add =E2=80=9Catm overhead 40=E2=80= =9D to cake on > pppoe0 and set the shaper to 90% of line rates (15.75/0.9) how does the > Steam affect per-host fairness? Also how transient are these connections > team uses? Actually did more testing about this and it seems that as far I have set the bandwidth to ~15Mbps (so ~15% less of my max speed) and use the "nat" parameter, the per-host fairness works even without the "dual-host" and "overhead" parameters. I definitely find this very interesting, is this behaviour caused by the way Steam downloads games? As far as I can tell cake can drill down to the required IP/TCP/UDP fields > independent of whether there are VLAN tags or PPPoE headers so cake shoul= d > not care (except for the different overhead specifications you need to ad= d > as stated above). BUT if instantiated on eth0 cake will see pppoe LCP > packets and might decide to drop them, which can take down the link, so o= ut > of caution I would still instantiate on pppoe in your case. Yeah, with further testing it seems the interface wasn't the culprit but I'll still do all my testing on pppoe0 just to be safe. Anyway I was wondering if there's some kind of manual for Cake and the various parameters, I'm looking to set it up best way possible but there are some parameters which I'm not sure what they do (one of them being "ingress"). Also while reading on the bufferbloat.net Cake page I noticed a possible "fix" for BitTorrent (by setting it as "background", https://www.bufferbloat.net/projects/codel/wiki/Cake/#diffserv-support), I'm wondering if this can be done with Steam too? On 21 April 2017 at 15:27, Dendari Marini wrote: > Hello, I have an update. > > The download test (PC1 downloading Steam game, PC2 downloading a file) > seems to work fine now, even while using higher bandwidth value. So that > seems resolved, probably the interface change was the fix. > > Now about the gaming test (PC1 downloading Steam game, PC2 online gaming)= , > this is the one I'm mostly looking forward to fix and the main reason I > bought the ER-X. Unfortunately Cake doesn't seem to do much in this case, > specifically I'm still getting high latency spikes and packet loss on PC2= . > One weird thing: starting a download on PC2 will decrease the > lantecy/p.loss issues, not sure if it's because of the Steam speed being > halved on PC1 or something else. > > On 21 April 2017 at 10:34, Dendari Marini wrote: > >> Hello, thanks for all of your replies. >> >> First of all, my connection encapsulation should be ATM LLC and it can >> actually reach up to 17.5/1 Mbps, but that's kinda best case scenario wh= ich >> is why I wanted to play it safe with just 16/.9 (which I should reach mo= re >> consistently). >> >> Back to the Steam issue. Unfortunately I can't seem to get really >> consistent results, mainly because sometimes it's downloading the game f= rom >> just a few connections and other times it's downloading from dozens and >> dozens connections. The latter is the one giving me more issues both in >> terms of latency/packet loss and in terms of evenly splitting the bandwi= dth >> across the hosts. >> >> One thing that seems to give better results is changing the interface >> where Cake is used from eth0 to pppoe0. When I used fq_codel it seemed t= o >> give better results when using eth0 and so I went ahead and did the same >> with Cake. >> >> Anyway more testing needed, will report if I notice any consistent resul= t. >> >> By the way this is the thread I opened on the Ubiquiti forums talking >> about this issue (not sure if it can give you some more info): >> https://community.ubnt.com/t5/EdgeMAX/Smart-Queue-seemingly- >> not-working-for-Steam-downloads/td-p/1890405 >> Also the thread where I got Cake for the ER-X from: >> https://community.ubnt.com/t5/EdgeMAX/Cake-and-FQ-PIE- >> compiled-for-the-EdgeRouter-devices/td-p/1679844 >> >> On 20 April 2017 at 20:36, Sebastian Moeller wrote: >> >>> >>> > On Apr 20, 2017, at 18:05, Dendari Marini wrote= : >>> > >>> > Hello, thanks for your reply. >>> > >>> > Looks like most of your options are okay, including the correct =E2= =80=9Cdual=E2=80=9D >>> modes and =E2=80=9Cingress=E2=80=9D mode in the right place. However, = I think you need to >>> adjust your bandwidth and overhead settings, otherwise Cake isn=E2=80= =99t reliably >>> in control of the bottleneck queues. Try these to begin with: >>> > >>> > =E2=80=A6 bandwidth 850Kbit conservative dual-srchost nat >>> > >>> > =E2=80=A6 bandwidth 15Mbit conservative dual-dsthost nat ingress >>> > >>> > That should give you correct operation, and you can fine-tune from >>> there. >>> > >>> > Just did quick test with your settings. First thing I noticed is my >>> final download bandwidth is about 12Mbps, Steam on PC1 downloads at >>> 1.4-1.5MB/s while downloading a file on PC2 seems to max out at ~250KB/= s. >>> From my understanding I should see each PC download at ~700KB/s, or am = I >>> mistaken? >>> >>> Assuming you measured good put in [M|K]iBytes this adds up to 1.5+0.25 >>> =3D 1.75 * 1024^2 * 8 =3D 14680064 Bits or (1.4+0.25) * 8 *1024^2 / 100= 0^2 =3D >>> 13.84 Mbps which seems a bit high for a 16Mbps ADSL link. I would ecpex= t >>> something like 16 * (48/53) * ((1500 - 8 - 20 -20) / (1500 + 32)) =3D = 13.73 >>> Mbps TCP/IPv4 goodput=E2=80=A6 so you seem to be running close to theor= etical >>> maximum of your link (assuming I am not totally off with the overhead >>> (estimated ADSL overhead on top of MTU: 6 destination MAC + 6 source MA= C + >>> 2 ethertype + 3 ATM LLC + 5 ATM SNAP + 2 ATM pad + 8 ATM AAL5 SAR 32 >>> bytes). But with your shaper set at 15Mbps without the atm option you w= ill >>> actually accept up to 15 * (53/48) =3D 16.5625 Mbps on the wire, which >>> probably is above your link bandwidth. This fits well with the really l= ow >>> number of drops in your cake stats, you simply never have cake feel tha= t >>> shaping is needed? >>> >>> Best Regards >>> >>> >>> >>> >>> > >>> > On 20 April 2017 at 17:32, Jonathan Morton >>> wrote: >>> > >>> >> On 20 Apr, 2017, at 18:23, Dendari Marini >>> wrote: >>> >> >>> >>> Could you post the output of calling =E2=80=9Ctc -s qdisc=E2=80=9D = here on the list >>> please? That should allow to figure out what you actually told cake to = do ;0 >>> > >>> >> qdisc cake 8001: dev eth0 root refcnt 2 bandwidth 900Kbit diffserv3 >>> dual-srchost nat rtt 100.0ms raw >>> > >>> >> qdisc cake 8002: dev ifb4eth0 root refcnt 2 bandwidth 16Mbit >>> diffserv3 dual-dsthost nat ingress rtt 100.0ms raw >>> > >>> > Looks like most of your options are okay, including the correct =E2= =80=9Cdual=E2=80=9D >>> modes and =E2=80=9Cingress=E2=80=9D mode in the right place. However, = I think you need to >>> adjust your bandwidth and overhead settings, otherwise Cake isn=E2=80= =99t reliably >>> in control of the bottleneck queues. Try these to begin with: >>> > >>> > =E2=80=A6 bandwidth 850Kbit conservative dual-srchost nat >>> > >>> > =E2=80=A6 bandwidth 15Mbit conservative dual-dsthost nat ingress >>> > >>> > That should give you correct operation, and you can fine-tune from >>> there. >>> > >>> > - Jonathan Morton >>> > >>> > >>> > >>> >>> >> > --001a114a0e6c630dd0054dbd1d6c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Sebastian,

Only now I noticed your m= essage, it seems we were writing at the same time.

So please add =E2=80=9Catm overhead 32" to cake on eth0 or =E2=80= =9Catm overhead 40=E2=80=9D to cake instances on pppoe (these packets do no= t have the PPPoE header added yet and hence appear 8 bytes to small).

Thanks for your help, will definitely use= them. Just wondering if I use "pppoe-vcmux/bridged-llcsnap" on e= th0 or "pppoe-llcsnap" on pppoe0 would have the same effect? Or a= re there some other "under-the-hood" changes when using them?

Question: if you set the shaper=E2=80=99s to 50% o= f line rate (8.75/0.5?) do you still see that unfairness? And if you add = =E2=80=9Catm overhead 40=E2=80=9D to cake on pppoe0 and set the shaper to 9= 0% of line rates (15.75/0.9) how does the Steam affect per-host fairness? A= lso how transient are these connections team uses?
=
Actually did more testing about this and it seems that as fa= r I have set the bandwidth to ~15Mbps (so ~15% less of my max speed) and us= e the "nat" parameter, the per-host fairness works even without t= he "dual-host" and "overhead" parameters. I definitely = find this very interesting, is this behaviour caused by the way Steam downl= oads games?

As far as I can tell cake can drill= down to the required IP/TCP/UDP fields independent of whether there are VL= AN tags or PPPoE headers so cake should not care (except for the different = overhead specifications you need to add as stated above). BUT if instantiat= ed on eth0 cake will see pppoe LCP packets and might decide to drop them, w= hich can take down the link, so out of caution I would still instantiate on= pppoe in your case.

Yeah, with furt= her testing it seems the interface wasn't the culprit but I'll stil= l do all my testing on pppoe0 just to be safe.

Any= way I was wondering if there's some kind of manual for Cake and the var= ious parameters, I'm looking to set it up best way possible but there a= re some parameters which I'm not sure what they do (one of them being &= quot;ingress"). Also while reading on the bufferbloat.net Cake page I noticed a possible "fix" f= or BitTorrent (by setting it as "background",=C2=A0https:= //www.bufferbloat.net/projects/codel/wiki/Cake/#diffserv-support), I= 9;m wondering if this can be done with Steam too?

On 21 April 2017 at 15:27, Dend= ari Marini <dendari92@gmail.com> wrote:
Hello, I have an update.

The download test (PC1 downloading Steam game, PC2 downloading a file) se= ems to work fine now, even while using higher bandwidth value. So that seem= s resolved, probably the interface change was the fix.

=
Now about the gaming test (PC1 downloading Steam game, PC2 online gami= ng), this is the one I'm mostly looking forward to fix and the main rea= son I bought the ER-X. Unfortunately Cake doesn't seem to do much in th= is case, specifically I'm still getting high latency spikes and packet = loss on PC2. One weird thing: starting a download on PC2 will decrease the = lantecy/p.loss issues, not sure if it's because of the Steam speed bein= g halved on PC1 or something else.

On 21 = April 2017 at 10:34, Dendari Marini <dendari92@gmail.com> = wrote:
Hello, thanks for= all of your replies.

First of all, my connection=C2=A0<= span style=3D"font-size:12.8px">encapsulation should be ATM LLC and it=C2=A0can actually reach up to 17.5/1 Mbps, but that's kinda best cas= e scenario which is why I wanted to play it safe with just 16/.9 (which I s= hould reach more consistently).
=
Back to the Steam issue. Unfortunately I can't se= em to get really consistent results, mainly because sometimes it's down= loading the game from just a few connections and other times it's downl= oading from dozens and dozens connections. The latter is the one giving me = more issues both in terms of latency/packet loss and in terms of evenly spl= itting the bandwidth across the hosts.

One thi= ng that seems to give better results is changing the interface where Cake i= s used from eth0 to pppoe0. When I used fq_codel it seemed to give better r= esults when using eth0 and so I went ahead and did the same with Cake.

Anyway more testing needed, will report if I notice an= y consistent result.

By the way this is the thread= I opened on the Ubiquiti forums talking about this issue (not sure if it c= an give you some more info): https://community.ubnt.com/t5/EdgeMAX/Smart-Queue-seemin= gly-not-working-for-Steam-downloads/td-p/1890405

On 20 April 2017 at 20:36, Sebastian Moeller <moeller0@gmx.de> wrote:

> On Apr 20, 2017, at 18:05, Dendari Marini <dendari92@gmail.com> wrote:
>
> Hello, thanks for your r= eply.
>
> Looks like most of your options are okay, including the correct =E2=80= =9Cdual=E2=80=9D modes and =E2=80=9Cingress=E2=80=9D mode in the right plac= e.=C2=A0 However, I think you need to adjust your bandwidth and overhead se= ttings, otherwise Cake isn=E2=80=99t reliably in control of the bottleneck = queues.=C2=A0 Try these to begin with:
>
> =E2=80=A6 bandwidth 850Kbit conservative dual-srchost nat
>
> =E2=80=A6 bandwidth 15Mbit conservative dual-dsthost nat ingress
>
> That should give you correct operation, and you can fine-tune from the= re.
>
> Just did quick test with your settings. First thing I noticed is my fi= nal download bandwidth is about 12Mbps, Steam on PC1 downloads at 1.4-1.5MB= /s while downloading a file on PC2 seems to max out at ~250KB/s. From my un= derstanding I should see each PC download at ~700KB/s, or am I mistaken?
Assuming you measured good pu= t in [M|K]iBytes this adds up to=C2=A0 1.5+0.25 =3D 1.75 * 1024^2 * 8 =3D 1= 4680064 Bits or (1.4+0.25) * 8 *1024^2 / 1000^2 =3D 13.84 Mbps which seems = a bit high for a 16Mbps ADSL link. I would ecpext something like 16 * (48/5= 3)=C2=A0 * ((1500 - 8 - 20 -20) / (1500 + 32)) =3D 13.73 Mbps TCP/IPv4 good= put=E2=80=A6 so you seem to be running close to theoretical maximum of your= link (assuming I am not totally off with the overhead (estimated ADSL over= head on top of MTU: 6 destination MAC + 6 source MAC + 2 ethertype + 3 ATM = LLC + 5 ATM SNAP + 2 ATM pad + 8 ATM AAL5 SAR 32 bytes). But with your shap= er set at 15Mbps without the atm option you will actually accept up to 15 *= (53/48) =3D 16.5625 Mbps on the wire, which probably is above your link ba= ndwidth. This fits well with the really low number of drops in your cake st= ats, you simply never have cake feel that shaping is needed?

Best Regards




>
> On 20 Apri= l 2017 at 17:32, Jonathan Morton <chromatix99@gmail.com> wrote:
>
>> On 20 Apr, 2017, at 18:23, Dendari Marini <dendari92@gmail.com> wrote:
>>
>>> Could you post the output of calling =E2=80=9Ctc -s qdisc=E2= =80=9D here on the list please? That should allow to figure out what you ac= tually told cake to do ;0
>
>> qdisc cake 8001: dev eth0 root refcnt 2 bandwidth 900Kbit diffserv= 3 dual-srchost nat rtt 100.0ms raw
>
>> qdisc cake 8002: dev ifb4eth0 root refcnt 2 bandwidth 16Mbit diffs= erv3 dual-dsthost nat ingress rtt 100.0ms raw
>
> Looks like most of your options are okay, including the correct =E2=80= =9Cdual=E2=80=9D modes and =E2=80=9Cingress=E2=80=9D mode in the right plac= e.=C2=A0 However, I think you need to adjust your bandwidth and overhead se= ttings, otherwise Cake isn=E2=80=99t reliably in control of the bottleneck = queues.=C2=A0 Try these to begin with:
>
> =E2=80=A6 bandwidth 850Kbit conservative dual-srchost nat
>
> =E2=80=A6 bandwidth 15Mbit conservative dual-dsthost nat ingress
>
> That should give you correct operation, and you can fine-tune from the= re.
>
> - Jonathan Morton
>
>
>




--001a114a0e6c630dd0054dbd1d6c--