From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 289D33B2A3 for ; Sat, 22 Apr 2017 10:12:14 -0400 (EDT) Received: from [192.168.42.241] ([80.135.73.48]) by mail.gmx.com (mrgmx101 [212.227.17.168]) with ESMTPSA (Nemesis) id 0McR00-1ckGOS1AwK-00Hfpx; Sat, 22 Apr 2017 16:12:12 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) From: Sebastian Moeller In-Reply-To: Date: Sat, 22 Apr 2017 16:12:11 +0200 Cc: Jonathan Morton , cake@lists.bufferbloat.net Content-Transfer-Encoding: quoted-printable Message-Id: <14DA7DD2-B0E8-49AB-9DAE-BB9D51F39CD0@gmx.de> References: <05C0B0C7-4337-4115-AC6B-DA81392FCB34@gmail.com> <22E633CF-5EE0-4B0F-89A8-B790E730FB6C@gmx.de> To: Dendari Marini X-Mailer: Apple Mail (2.3273) X-Provags-ID: V03:K0:77ty30e5Xb5DvzZkApiZtrvSV6aJKfZSUaMACAp4LdCLBJ5s9jU cF+5ECx21X2oJajv87oMC3WdXPCe5Hm9o1j2s93KQ3LFbEpUEtGORXVzWROHwcOKl05a3uD jTAe/EmmnXZ5um7JRh6Tkr9mJIFlr8ksmGzdsXXZbquUFbVgQkvtt79RNbBhLO//Y2aSPA7 fzRswqV+qogd3cxYg0lVQ== X-UI-Out-Filterresults: notjunk:1;V01:K0:BPJkewD5WY8=:rmn/QAho3ntHC92oTNXpNl qbmOLYNDqIoaWDiiAC92pO0mzbWr76u/4S0m/uUc3+tMn7cpQc+J5LDuaOuEXNHSciRurXjdk pOe6GmG1OPJvRuLorl6b0JTlczPBj1Sx7LjDZP1VXqXbxW7LH0fe+542QSCTVGexfY7+Aw0iI Uink4PcLWnh1xKgGgJ/i1KlK5kcuBZRWmE7qY5n4v3pUNSZgzH5KNg0P8voRHDKgVTFrYN8LT MqncI8CmtpHytpjNF5H1jcyxUzqQoubn5gaxVSd+P8PfmFSUkaD3Uorp/ZP48KedJbhtAj+Vf vwFAdLkcxHvBtXBHLZpb2Nn3q0a7sRAwZQUQjsJWkYVwUy+iM7QE14Uw4YJe1nsz2tAyUetho T7CuLO9cUzAvxGFIfxBtuNpP67VG46KEp3ylY+SWd5IOFJZ6mrhaZpeR85vt63eun2aqySaUj PTl8j1Pw28Valo7xZuMxTvjTz827aP051oKIT2HDFm/1WmGWc0KZKuH4vgSmHQr73zVnENds6 NcAkK9gvrVtL3+afjIqO9pxPZ/ZLG1F/q/JFJ2hckglNSZID96ViAl4mVfiijm2d5EJYbXsrL f5VfrpiyHw4rMrXc6HPHsxyvKFmE7QyH9CDUC5Ioh62s9dg50VzLrwS0uEF1Ai+CPnErJTKxn GP5L4PTC5thekniplxTB/cdr4eKOisAo9DKsMPasTc1zTgZBc4ar/q/ZjjS7doqiSfwFq2OA6 n/MNFuXaDwUkG7FGD+1yCSvbhViAt+5CY22l4gb3xZM/A3zPsKCKQpp3FmXIKCvelyhFQx3j+ uJofj+B Subject: Re: [Cake] Getting Cake to work better with Steam and similar applications X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Apr 2017 14:12:14 -0000 Hi Dendari, > On Apr 22, 2017, at 10:25, Dendari Marini wrote: >=20 > Hi Sebastian, >=20 > Only now I noticed your message, it seems we were writing at the same = time. >=20 > So please add =E2=80=9Catm overhead 32" to cake on eth0 or =E2=80=9Catm = overhead 40=E2=80=9D to cake instances on pppoe (these packets do not = have the PPPoE header added yet and hence appear 8 bytes to small). >=20 > Thanks for your help, will definitely use them. Just wondering if I = use "pppoe-vcmux/bridged-llcsnap" on eth0 or "pppoe-llcsnap" on pppoe0 = would have the same effect? Or are there some other "under-the-hood" = changes when using them? The compound keywords like pppoe-llcsnap are effectively = short-hands for =E2=80=9Catm overhead N=E2=80=9D with N depending on the = name. Some people prefer the symbolic names to the numeric specification = (see Jonathan=E2=80=99s mail), I prefer the =E2=80=9Cfull truth=E2=80=9D = of the numeric specifier. Which keyword corresponds exactly to =E2=80=9Cat= m overhead 32=E2=80=9D or =E2=80=9Catm overhead 40=E2=80=9D I do not = know by heart, I would either look into the source code of the cake = module for iproute2/tc or cake=E2=80=99s man page. That is if I would = actually want to use the symbolic keywords. It really is a question of = personal preference, I guess... >=20 > Question: if you set the shaper=E2=80=99s to 50% of line rate = (8.75/0.5?) do you still see that unfairness? And if you add =E2=80=9Catm = overhead 40=E2=80=9D to cake on pppoe0 and set the shaper to 90% of line = rates (15.75/0.9) how does the Steam affect per-host fairness? Also how = transient are these connections team uses? >=20 > Actually did more testing about this and it seems that as far I have = set the bandwidth to ~15Mbps (so ~15% less of my max speed) and use the = "nat" parameter, the per-host fairness works even without the = "dual-host" and "overhead" parameters. I definitely find this very = interesting, is this behaviour caused by the way Steam downloads games? Cake defaults to triple-isolate, which with nat active will sort = of avoid any host IP internal or external hogging the link. Correct = overhead specification is more important to correctly model what bits = are actually send over the bottleneck, only if this is precisely = specified will cake reduce bufferbloat even with tight shaper settings = (close to the true bottleneck). I then to think of per-host-isolation = and link-layer-accounting as two rather orthogonal issues, that both = affect a link=E2=80=99s performance... >=20 > As far as I can tell cake can drill down to the required IP/TCP/UDP = fields independent of whether there are VLAN tags or PPPoE headers so = cake should not care (except for the different overhead specifications = you need to add as stated above). BUT if instantiated on eth0 cake will = see pppoe LCP packets and might decide to drop them, which can take down = the link, so out of caution I would still instantiate on pppoe in your = case. >=20 > Yeah, with further testing it seems the interface wasn't the culprit = but I'll still do all my testing on pppoe0 just to be safe. Okay, good to know. >=20 > Anyway I was wondering if there's some kind of manual for Cake and the = various parameters, I'm looking to set it up best way possible but there = are some parameters which I'm not sure what they do (one of them being = "ingress"). Also while reading on the bufferbloat.net Cake page I = noticed a possible "fix" for BitTorrent (by setting it as "background", = https://www.bufferbloat.net/projects/codel/wiki/Cake/#diffserv-support), = I'm wondering if this can be done with Steam too? Well in theory using the correct DSCP markings together with a = matching diffserv mode in cake should allow to relegate any wanted = traffic patterns to a nicely bandwidth-yielding background queue. The = challenge really is to properly label the packets, which especially on = ingress is hard. Best Regards Sebastian >=20 > On 21 April 2017 at 15:27, Dendari Marini wrote: > Hello, I have an update. >=20 > The download test (PC1 downloading Steam game, PC2 downloading a file) = seems to work fine now, even while using higher bandwidth value. So that = seems resolved, probably the interface change was the fix. >=20 > Now about the gaming test (PC1 downloading Steam game, PC2 online = gaming), this is the one I'm mostly looking forward to fix and the main = reason I bought the ER-X. Unfortunately Cake doesn't seem to do much in = this case, specifically I'm still getting high latency spikes and packet = loss on PC2. One weird thing: starting a download on PC2 will decrease = the lantecy/p.loss issues, not sure if it's because of the Steam speed = being halved on PC1 or something else. >=20 > On 21 April 2017 at 10:34, Dendari Marini wrote: > Hello, thanks for all of your replies. >=20 > First of all, my connection encapsulation should be ATM LLC and it can = actually reach up to 17.5/1 Mbps, but that's kinda best case scenario = which is why I wanted to play it safe with just 16/.9 (which I should = reach more consistently). >=20 > Back to the Steam issue. Unfortunately I can't seem to get really = consistent results, mainly because sometimes it's downloading the game = from just a few connections and other times it's downloading from dozens = and dozens connections. The latter is the one giving me more issues both = in terms of latency/packet loss and in terms of evenly splitting the = bandwidth across the hosts. >=20 > One thing that seems to give better results is changing the interface = where Cake is used from eth0 to pppoe0. When I used fq_codel it seemed = to give better results when using eth0 and so I went ahead and did the = same with Cake. >=20 > Anyway more testing needed, will report if I notice any consistent = result. >=20 > By the way this is the thread I opened on the Ubiquiti forums talking = about this issue (not sure if it can give you some more info): = https://community.ubnt.com/t5/EdgeMAX/Smart-Queue-seemingly-not-working-fo= r-Steam-downloads/td-p/1890405 > Also the thread where I got Cake for the ER-X from: = https://community.ubnt.com/t5/EdgeMAX/Cake-and-FQ-PIE-compiled-for-the-Edg= eRouter-devices/td-p/1679844 >=20 > On 20 April 2017 at 20:36, Sebastian Moeller wrote: >=20 > > On Apr 20, 2017, at 18:05, Dendari Marini = wrote: > > > > Hello, thanks for your reply. > > > > Looks like most of your options are okay, including the correct = =E2=80=9Cdual=E2=80=9D modes and =E2=80=9Cingress=E2=80=9D mode in the = right place. However, I think you need to adjust your bandwidth and = overhead settings, otherwise Cake isn=E2=80=99t reliably in control of = the bottleneck queues. Try these to begin with: > > > > =E2=80=A6 bandwidth 850Kbit conservative dual-srchost nat > > > > =E2=80=A6 bandwidth 15Mbit conservative dual-dsthost nat ingress > > > > That should give you correct operation, and you can fine-tune from = there. > > > > Just did quick test with your settings. First thing I noticed is my = final download bandwidth is about 12Mbps, Steam on PC1 downloads at = 1.4-1.5MB/s while downloading a file on PC2 seems to max out at = ~250KB/s. =46rom my understanding I should see each PC download at = ~700KB/s, or am I mistaken? >=20 > Assuming you measured good put in [M|K]iBytes this adds up to = 1.5+0.25 =3D 1.75 * 1024^2 * 8 =3D 14680064 Bits or (1.4+0.25) * 8 = *1024^2 / 1000^2 =3D 13.84 Mbps which seems a bit high for a 16Mbps ADSL = link. I would ecpext something like 16 * (48/53) * ((1500 - 8 - 20 -20) = / (1500 + 32)) =3D 13.73 Mbps TCP/IPv4 goodput=E2=80=A6 so you seem to = be running close to theoretical maximum of your link (assuming I am not = totally off with the overhead (estimated ADSL overhead on top of MTU: 6 = destination MAC + 6 source MAC + 2 ethertype + 3 ATM LLC + 5 ATM SNAP + = 2 ATM pad + 8 ATM AAL5 SAR 32 bytes). But with your shaper set at 15Mbps = without the atm option you will actually accept up to 15 * (53/48) =3D = 16.5625 Mbps on the wire, which probably is above your link bandwidth. = This fits well with the really low number of drops in your cake stats, = you simply never have cake feel that shaping is needed? >=20 > Best Regards >=20 >=20 >=20 >=20 > > > > On 20 April 2017 at 17:32, Jonathan Morton = wrote: > > > >> On 20 Apr, 2017, at 18:23, Dendari Marini = wrote: > >> > >>> Could you post the output of calling =E2=80=9Ctc -s qdisc=E2=80=9D = here on the list please? That should allow to figure out what you = actually told cake to do ;0 > > > >> qdisc cake 8001: dev eth0 root refcnt 2 bandwidth 900Kbit diffserv3 = dual-srchost nat rtt 100.0ms raw > > > >> qdisc cake 8002: dev ifb4eth0 root refcnt 2 bandwidth 16Mbit = diffserv3 dual-dsthost nat ingress rtt 100.0ms raw > > > > Looks like most of your options are okay, including the correct = =E2=80=9Cdual=E2=80=9D modes and =E2=80=9Cingress=E2=80=9D mode in the = right place. However, I think you need to adjust your bandwidth and = overhead settings, otherwise Cake isn=E2=80=99t reliably in control of = the bottleneck queues. Try these to begin with: > > > > =E2=80=A6 bandwidth 850Kbit conservative dual-srchost nat > > > > =E2=80=A6 bandwidth 15Mbit conservative dual-dsthost nat ingress > > > > That should give you correct operation, and you can fine-tune from = there. > > > > - Jonathan Morton > > > > > > >=20 >=20 >=20 >=20