From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-x22b.google.com (mail-wr0-x22b.google.com [IPv6:2a00:1450:400c:c0c::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 502873B2A3 for ; Sat, 22 Apr 2017 17:56:47 -0400 (EDT) Received: by mail-wr0-x22b.google.com with SMTP id z109so71850948wrb.1 for ; Sat, 22 Apr 2017 14:56:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=oMa14RgS3Eq5/RLzHXEwRtevLiJsNWNnkOb/UypR1ck=; b=lG7uvFP7lfHHCsnIsQhiWgPWvW/UTrR5o9t3hOIz8AxnRWDeQD0OsUy6rADIpD6zrx 84et3erZFPMb0TkMme5eKWk7QA4Xt3Y0a4R6/yJSDQ0pjT6smLbWiUsZLB1d1Azmahfr revWItam5rfVEcNxOf9EjVTKhxcK4ZcP2uX/urV63F4ys++AV9oaUYNtnTJI6uf02glk TcdmsFoJFqVjACt1ivAkq8l++lGQ7D3znHUN7X0Td4/wRhmQI5c+8GOsVgHr+8IivV/g ULvKGoDkqJWgGSQJF/PR+AdLzgOXZrfiyvZwbleOkZIg5Ve50Jr0e0hOdNVwK2DlEC6s 55qQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=oMa14RgS3Eq5/RLzHXEwRtevLiJsNWNnkOb/UypR1ck=; b=MW0XDSs7WuzAzjFit24rctTkpSTi0Vbp1Sv/iH0NP5TlvJ8/POaz301cosckdrfNI7 XMjZ7AffGveZkhsQiomepaeVAPRokr2gahxM6u0fY1ZPgpyhytxNsT+wAvak2XMvNI7a 2ss0FOpHSgZ29kfIcg+942jyoUz6QWjTuUZtOx7e3h17pSkO7/6kT9kZFfqq4BsiSUZZ MCkC89SJRFpViWs8jMEjYzZF8XgnGR7N/wmfpT/QVmTAWSqMsNj/c0mRsuFeLn730Y9Q kONuBDPXUnFYlS7GbOPV8TnuzBM/1KUkkMeKMcNpfyZsZ+ixOgolThYgeiACLXcFVbNl Nvhw== X-Gm-Message-State: AN3rC/7HVx68q0nuEk3CA5khnGxgja68f7L9J3YdvK5mjwAEYbX6x5/s t+KzEB7AQuBQMH8ArmkUqYjwvQp/Qg== X-Received: by 10.223.173.23 with SMTP id p23mr2948980wrc.117.1492898205995; Sat, 22 Apr 2017 14:56:45 -0700 (PDT) MIME-Version: 1.0 Received: by 10.80.169.59 with HTTP; Sat, 22 Apr 2017 14:56:45 -0700 (PDT) In-Reply-To: <0BA3EE91-C5BC-4155-9D5D-D15D34490A1A@gmx.de> References: <05C0B0C7-4337-4115-AC6B-DA81392FCB34@gmail.com> <22E633CF-5EE0-4B0F-89A8-B790E730FB6C@gmx.de> <0BA3EE91-C5BC-4155-9D5D-D15D34490A1A@gmx.de> From: Dendari Marini Date: Sat, 22 Apr 2017 23:56:45 +0200 Message-ID: To: Sebastian Moeller Cc: Jonathan Morton , cake@lists.bufferbloat.net Content-Type: multipart/alternative; boundary=f403045cef584119df054dc873d3 Subject: Re: [Cake] Getting Cake to work better with Steam and similar applications X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Apr 2017 21:56:47 -0000 --f403045cef584119df054dc873d3 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hello, thank you all for your replies! For the overhead I'm gonna use "pppoe-llcsnap" (or "overhead 40 atm), as I believe it's the one which should work best for my connection. About the per-host fairness download issue: while it's kinda resolved I still feel like it's mainly related to Steam, as normally downloading files from PC1 and PC2 halved the speed as expected even at full bandwidth (so no overhead, no -15%). Anyway back to Steam: assuming the IP addresses aren't a big issue, what steps should I follow to start filtering its traffic so it's considered background? Would the DPI from the ER-X be any helpful? On 22 April 2017 at 18:47, Sebastian Moeller wrote: > > > On Apr 22, 2017, at 11:36, Jonathan Morton > wrote: > > > >>> So please add =E2=80=9Catm overhead 32" to cake on eth0 or =E2=80=9Ca= tm overhead 40=E2=80=9D > to cake instances on pppoe (these packets do not have the PPPoE header > added yet and hence appear 8 bytes to small). > >> > >> Thanks for your help, will definitely use them. Just wondering if I us= e > "pppoe-vcmux/bridged-llcsnap" on eth0 or "pppoe-llcsnap" on pppoe0 would > have the same effect? Or are there some other "under-the-hood" changes wh= en > using them? > > > > On the pppoe interface, use pppoe-vcmux if your modem is set to use > VC-MUX, or pppoe-llcsnap if it=E2=80=99s set to use LLC-SNAP (they might = be > described using slightly different terms, but should still be recognisabl= e > as one or the other). This probably depends on your ISP, and may further > vary regionally within the same ISP. > > In my experience it is rather bothersome trying to get that > information from one=E2=80=99s ISP, this is why I would recommend to foll= ow the > instructions on https://github.com/moeller0/ATM_overhead_detector and > empirically measure the actual overhead. In that case one ends up with th= e > numeric overhead, hence my inclination to use that number directly instea= d > of looking into a table to translate that back into a symbolic keyword=E2= =80=A6 > especially since say for an overhead of 32 (and 36) there are two differe= nt > encapsulation schees that add up to that number: > > case 32 > disp('Connection: Bridged, LLC/SNAP RFC-1483/2684'); > disp('Protocol (bytes): Ethernet Header (14), ATM LLC (3)= , > ATM SNAP (5), ATM pad (2), ATM AAL5 SAR (8) : Total 32'); > > disp('Connection: PPPoE, VC/Mux RFC-2684'); > disp('Protocol (bytes): PPP (2), PPPoE (6), Ethernet > Header (14), ATM pad (2), ATM AAL5 SAR (8) : Total 32=E2=80=99); > > good luck divining which of those is in use if all you know is the numeri= c > overhead... > > > > > > > I really prefer to use the self-explanatory keywords (which is why I > added them in the first place) instead of opaque magic numbers. This is = a > point on which Sebastian has long disagreed with me. > > True, but I am not going to re-hash that here again ;) > > > > >>> Question: if you set the shaper=E2=80=99s to 50% of line rate (8.75/0= .5?) do > you still see that unfairness? And if you add =E2=80=9Catm overhead 40=E2= =80=9D to cake on > pppoe0 and set the shaper to 90% of line rates (15.75/0.9) how does the > Steam affect per-host fairness? Also how transient are these connections > team uses? > >> > >> Actually did more testing about this and it seems that as far I have > set the bandwidth to ~15Mbps (so ~15% less of my max speed) and use the > "nat" parameter, the per-host fairness works even without the "dual-host" > and "overhead" parameters. I definitely find this very interesting, is th= is > behaviour caused by the way Steam downloads games? > > > > By default, Cake uses triple-isolate mode, which uses information about > both source and destination hosts to perform per-host isolation; this > usually works well regardless of which side of the connection has the LAN > hosts. The =E2=80=9Cdual=E2=80=9D modes let you specify that fact explic= itly, making it a > little more robust and predictable. > > > > Without overhead compensation, Cake will actually use more of the > physical link than it thinks it does - by default it only accounts for ra= w > IP or Ethernet packets, depending on the type of interface it=E2=80=99s a= ttached > to. With full-size packets as in a bulk download, the difference is > relatively small, so the 15% margin is just about sufficient to make thin= gs > work. But with small packets mixed in, the difference grows, such that > Cake might no longer control the bottleneck with some traffic mixes. > > All true, to elaborate a bit on the ATM specific issue, due to > AAL5=E2=80=99s insistence that each ethernet frame is packaged into an in= teger > number of ATM cells (where the unused octets are simply padded out) the > worst case is something like 100%, if a hypothetical packet would only > require 49 Bytes, it will still require two ATM cells of 53 bytes... > > > > > The =E2=80=9Cconservative=E2=80=9D keyword I recommended earlier (which= is exactly > equivalent to Sebastian=E2=80=99s recommendation of =E2=80=9Catm overhead= 48=E2=80=9D) reverses > that situation; Cake will then always end up using *less* of the physical > link than it accounts for, which is safe for troubleshooting with. The > keyword is there specifically so that we do=E2=80=99t have to figure out = the > precise overhead profile before tackling more substantive issues. > > Due to the boundary observation above, one other option is to > start with the shaper set to 50% of link rate, that should have sufficien= t > wiggle room for all realistic overheads=E2=80=A6 (but honestly on a known= ATM link > I would always run the ATM_overhead_detector to get the precise number). > > > > > At any rate, it has nothing to do with Steam specifically. > > > >>> As far as I can tell cake can drill down to the required IP/TCP/UDP > fields independent of whether there are VLAN tags or PPPoE headers so cak= e > should not care (except for the different overhead specifications you nee= d > to add as stated above). BUT if instantiated on eth0 cake will see pppoe > LCP packets and might decide to drop them, which can take down the link, = so > out of caution I would still instantiate on pppoe in your case. > >> > >> Yeah, with further testing it seems the interface wasn't the culprit > but I'll still do all my testing on pppoe0 just to be safe. > >> > >> Anyway I was wondering if there's some kind of manual for Cake and the > various parameters, I'm looking to set it up best way possible but there > are some parameters which I'm not sure what they do (one of them being > "ingress=E2=80=9D). > > > > With the correct version of iproute2 installed, just issue =E2=80=9Cman > tc-cake=E2=80=9D. That=E2=80=99s the official documentation. > > > > Currently it doesn=E2=80=99t have the ingress keyword yet. That=E2=80= =99ll be fixed > soon. > > > >> Also while reading on the bufferbloat.net Cake page I noticed a > possible "fix" for BitTorrent (by setting it as "background", > https://www.bufferbloat.net/projects/codel/wiki/Cake/#diffserv-support), > I'm wondering if this can be done with Steam too? > > > > It=E2=80=99s possible, if you can figure out which traffic is Steam in = the first > place, and write filters to match on it. This is complicated by the fact > that Valve runs a sophisticated CDN to handle their rather impressive > bandwidth load. > > > > - Jonathan Morton > > > > --f403045cef584119df054dc873d3 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hello, thank you all for your replies!

= For the overhead I'm gonna use "pppoe-llcsnap" (or "over= head 40 atm), as I believe it's the one which should work best for my c= onnection.

About the per-host fairness download is= sue: while it's kinda resolved I still feel like it's mainly relate= d to Steam, as normally downloading files from PC1 and PC2 halved the speed= as expected even at full bandwidth (so no overhead, no -15%).
Anyway back to Steam: assuming the IP addresses aren't a b= ig issue, what steps should I follow to start filtering its traffic so it&#= 39;s considered background? Would the DPI from the ER-X be any helpful?

On 22 Apr= il 2017 at 18:47, Sebastian Moeller <moeller0@gmx.de> wrote:

> On Apr 22, 2017, at 11:36, Jonathan Morton <chromatix99@gmail.com> wrote:
>
>>> So please add =E2=80=9Catm overhead 32" to cake on eth0 o= r =E2=80=9Catm overhead 40=E2=80=9D to cake instances on pppoe (these packe= ts do not have the PPPoE header added yet and hence appear 8 bytes to small= ).
>>
>> Thanks for your help, will definitely use them. Just wondering if = I use "pppoe-vcmux/bridged-llcsnap" on eth0 or "pppoe-llcsna= p" on pppoe0 would have the same effect? Or are there some other "= ;under-the-hood" changes when using them?
>
> On the pppoe interface, use pppoe-vcmux if your modem is set to use VC= -MUX, or pppoe-llcsnap if it=E2=80=99s set to use LLC-SNAP (they might be d= escribed using slightly different terms, but should still be recognisable a= s one or the other).=C2=A0 This probably depends on your ISP, and may furth= er vary regionally within the same ISP.

=C2=A0 =C2=A0 =C2=A0 =C2=A0 In my experience it is rather bothersome= trying to get that information from one=E2=80=99s ISP, this is why I would= recommend to follow the instructions on https://gi= thub.com/moeller0/ATM_overhead_detector and empirically measure th= e actual overhead. In that case one ends up with the numeric overhead, henc= e my inclination to use that number directly instead of looking into a tabl= e to translate that back into a symbolic keyword=E2=80=A6 especially since = say for an overhead of 32 (and 36) there are two different encapsulation sc= hees that add up to that number:

case 32
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 disp('Connectio= n: Bridged, LLC/SNAP RFC-1483/2684');
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 disp('Protocol = (bytes): Ethernet Header (14), ATM LLC (3), ATM SNAP (5), ATM pad (2), ATM = AAL5 SAR (8) : Total 32');

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 disp('Connectio= n: PPPoE, VC/Mux RFC-2684');
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 disp('Protocol = (bytes): PPP (2), PPPoE (6), Ethernet Header (14), ATM pad (2), ATM AAL5 SA= R (8) : Total 32=E2=80=99);

good luck divining which of those is in use if all you know is the numeric = overhead...



>
> I really prefer to use the self-explanatory keywords (which is why I a= dded them in the first place) instead of opaque magic numbers.=C2=A0 This i= s a point on which Sebastian has long disagreed with me.

=C2=A0 =C2=A0 =C2=A0 =C2=A0 True, but I am not going to re-hash that= here again ;)

>
>>> Question: if you set the shaper=E2=80=99s to 50% of line rate = (8.75/0.5?) do you still see that unfairness? And if you add =E2=80=9Catm o= verhead 40=E2=80=9D to cake on pppoe0 and set the shaper to 90% of line rat= es (15.75/0.9) how does the Steam affect per-host fairness? Also how transi= ent are these connections team uses?
>>
>> Actually did more testing about this and it seems that as far I ha= ve set the bandwidth to ~15Mbps (so ~15% less of my max speed) and use the = "nat" parameter, the per-host fairness works even without the &qu= ot;dual-host" and "overhead" parameters. I definitely find t= his very interesting, is this behaviour caused by the way Steam downloads g= ames?
>
> By default, Cake uses triple-isolate mode, which uses information abou= t both source and destination hosts to perform per-host isolation; this usu= ally works well regardless of which side of the connection has the LAN host= s.=C2=A0 The =E2=80=9Cdual=E2=80=9D modes let you specify that fact explici= tly, making it a little more robust and predictable.
>
> Without overhead compensation, Cake will actually use more of the phys= ical link than it thinks it does - by default it only accounts for raw IP o= r Ethernet packets, depending on the type of interface it=E2=80=99s attache= d to.=C2=A0 With full-size packets as in a bulk download, the difference is= relatively small, so the 15% margin is just about sufficient to make thing= s work.=C2=A0 But with small packets mixed in, the difference grows, such t= hat Cake might no longer control the bottleneck with some traffic mixes.
=C2=A0 =C2=A0 =C2=A0 =C2=A0 All true, to elaborate a bit on the ATM = specific issue, due to AAL5=E2=80=99s insistence that each ethernet frame i= s packaged into an integer number of ATM cells (where the unused octets are= simply padded out) the worst case is something like 100%, if a hypothetica= l packet would only require 49 Bytes, it will still require two ATM cells o= f 53 bytes...

>
> The =E2=80=9Cconservative=E2=80=9D keyword I recommended earlier (whic= h is exactly equivalent to Sebastian=E2=80=99s recommendation of =E2=80=9Ca= tm overhead 48=E2=80=9D) reverses that situation; Cake will then always end= up using *less* of the physical link than it accounts for, which is safe f= or troubleshooting with.=C2=A0 The keyword is there specifically so that we= do=E2=80=99t have to figure out the precise overhead profile before tackli= ng more substantive issues.

=C2=A0 =C2=A0 =C2=A0 =C2=A0 Due to the boundary observation above, o= ne other option is to start with the shaper set to 50% of link rate, that s= hould have sufficient wiggle room for all realistic overheads=E2=80=A6 (but= honestly on a known ATM link I would always run the ATM_overhead_detector = to get the precise number).

>
> At any rate, it has nothing to do with Steam specifically.
>
>>> As far as I can tell cake can drill down to the required IP/TC= P/UDP fields independent of whether there are VLAN tags or PPPoE headers so= cake should not care (except for the different overhead specifications you= need to add as stated above). BUT if instantiated on eth0 cake will see pp= poe LCP packets and might decide to drop them, which can take down the link= , so out of caution I would still instantiate on pppoe in your case.
>>
>> Yeah, with further testing it seems the interface wasn't the c= ulprit but I'll still do all my testing on pppoe0 just to be safe.
>>
>> Anyway I was wondering if there's some kind of manual for Cake= and the various parameters, I'm looking to set it up best way possible= but there are some parameters which I'm not sure what they do (one of = them being "ingress=E2=80=9D).
>
> With the correct version of iproute2 installed, just issue =E2=80=9Cma= n tc-cake=E2=80=9D.=C2=A0 That=E2=80=99s the official documentation.
>
> Currently it doesn=E2=80=99t have the ingress keyword yet.=C2=A0 That= =E2=80=99ll be fixed soon.
>
>> Also while reading on the bufferbloat.net Cake page I noticed a= possible "fix" for BitTorrent (by setting it as "background= ", https://www.bufferbloa= t.net/projects/codel/wiki/Cake/#diffserv-support), I'm wo= ndering if this can be done with Steam too?
>
> It=E2=80=99s possible, if you can figure out which traffic is Steam in= the first place, and write filters to match on it.=C2=A0 This is complicat= ed by the fact that Valve runs a sophisticated CDN to handle their rather i= mpressive bandwidth load.
>
> - Jonathan Morton
>


--f403045cef584119df054dc873d3--