From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-x236.google.com (mail-wm0-x236.google.com [IPv6:2a00:1450:400c:c09::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 2C47A3B2A3 for ; Tue, 25 Apr 2017 18:06:34 -0400 (EDT) Received: by mail-wm0-x236.google.com with SMTP id r190so17160064wme.1 for ; Tue, 25 Apr 2017 15:06:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=OSh4ym1FqPSIFtDE4UCzRXvhEM1Mup4PJ8Ev1JBMk/c=; b=p9qXU/kOTNEjaUaofl+H5gEjnLke5eAc7b/CE6MzQEIuXaeXIb6NhW2HxzIWa6T+s2 u6yQ2hnz3gYhGmwBV5xpv08C5xytNLoMLSQoful+c446ptBsaIKUdCjONyOht5hRth7E XX1mNN7Ibh3zFaGbWz69gjPklUqQXARizrmd3rpDA7k6dA6bef6EAjv3M12dZNLKJ82j eYzMZNyOo0ncBrA/GYdizIdz5EgTqajQhEyDnkUL2z8h+mnVwt2eGzKpBGAxbLHZocH2 2MH3165PWiGZSYg2q0GZDVifpF7xzUXvQLW7D7hq5KKq9alQHorscfgz05O339i1Tv2B A3Mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=OSh4ym1FqPSIFtDE4UCzRXvhEM1Mup4PJ8Ev1JBMk/c=; b=pYOXnlHeyoP3Al/WJ0dLqed1H2ZSHDs0ORqMhvEIjgb+VrlLyMjL6HA5kEhvekmOzy a0wF2wLjXybLF8byowgt5m5uT8yf+M45umwznjWBfMExVUqAg3ZOoGAcLK3eFl+xQNfZ DrkyIK0hKlCJ6XcIwdsxAlztJmF+3BcfDjIvWsStZ+UcsGebF13zWu4ePLmcSG4VomGk QRt8aQicch01rOf2YIw/Y4iVK3184NKD4qeLFPldqV0Vg6yUz1WVY7h9YxGUIZ37qNHw hyfsXiFtLSro816QiAHsUDraSy3P++EhYnyjmANhdO0sQ6aVj4v1Qk4/w/nKQUY9/gYF 3bSg== X-Gm-Message-State: AN3rC/6pb+/l6np/ivlqu/FF9S7ZJZ4t/FTQGhdlj3KFiXF94Ae3nAul 3vuOV0Dx4dDYsw== X-Received: by 10.28.31.196 with SMTP id f187mr14009249wmf.117.1493157993119; Tue, 25 Apr 2017 15:06:33 -0700 (PDT) Received: from [192.168.0.3] (185.182.7.51.dyn.plus.net. [51.7.182.185]) by smtp.gmail.com with ESMTPSA id p197sm4509992wmb.34.2017.04.25.15.06.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 25 Apr 2017 15:06:32 -0700 (PDT) To: Sebastian Moeller Cc: Dendari Marini , cake@lists.bufferbloat.net References: <22E633CF-5EE0-4B0F-89A8-B790E730FB6C@gmx.de> <0BA3EE91-C5BC-4155-9D5D-D15D34490A1A@gmx.de> <00DDAA0B-7D99-489B-BA2D-1F20289409B3@gmx.de> <2FFBF256-2932-4FC7-AD1F-0D7CEE111809@gmx.de> <3fbfd0ee-7b41-0f83-8b44-ce7eed6a0562@gmail.com> <06A39EE4-4EF5-477B-B737-98195BA5CFAB@gmx.de> From: Andy Furniss Message-ID: <9be60376-7d48-c374-751e-c12a9d73cef9@gmail.com> Date: Tue, 25 Apr 2017 23:06:31 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0 SeaMonkey/2.51a2 MIME-Version: 1.0 In-Reply-To: <06A39EE4-4EF5-477B-B737-98195BA5CFAB@gmx.de> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: [Cake] Getting Cake to work better with Steam and similar applications X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Apr 2017 22:06:34 -0000 Sebastian Moeller wrote: > Hi Andy, > >> On Apr 25, 2017, at 14:58, Andy Furniss >> wrote: >> >> Dendari Marini wrote: >> >>> Also I have done some more testing, I was able to limit Steam >>> connections just to one thanks to some console commands >>> ("@cMaxContentServersToRequest" and >>> "@cCSClientMaxNumSocketsPerHost") and while the situation >>> improved (no more packet loss, latency variation within 10ms, but >>> still seeing ping spikes of ~50ms) it's still not what I'd >>> consider ideal, which would be like with any other download. So >>> my guess is there's something else going on other than just the >>> multiple connections, which are definitely big part of the >>> problem but not the only thing. Anyway these are my current >>> settings for Cake and I've been using them for the last four days >>> without issues: *qdisc cake 8005: root refcnt 2 bandwidth 950Kbit >>> diffserv3 triple-isolate nat wash rtt 100.0ms atm overhead 40 >>> via-ethernet* *qdisc cake 8006: root refcnt 2 bandwidth 17500Kbit >>> diffserv3 triple-isolate nat wash ingress rtt 100.0ms atm >>> overhead 40 via-ethernet* >> >> I still think that once via-ethernet appears you really need the >> raw parameter. > > Why? As far as I can tell the default mode is to adjust for the > kernel auto-added overhead (which seems to be 14 bytes for pakets > headed for an ethernet device and zero for packets headed for a ppp > device). If he adds raw, the kernel willl make its additions > silently… I shape on pppoe - though it's ptm. My overhead is 34, it seems as soon as you add overhead as a parameter via-ethernet appears and in the case of pppoe apparently does the wrong thing. Recalling my now closed issue about overheads on github, cake beleives what the kernel tells it about overheads and this seems to 22. In fact cake does not see the traffic as +22 it sees ip length. If I don't use raw below my shaper fails. Also note that when using raw the overhead gets increased by 22 from that specified. tc qdisc add dev ppp0 handle 1:0 root cake bandwidth 19690kbit raw overhead 34 diffserv4 dual-srchost nat rtt 200ms asr[/home/andy]# tc -s qdisc ls dev ppp0 qdisc cake 1: root refcnt 2 bandwidth 19690Kbit diffserv4 dual-srchost nat rtt 200.0ms noatm overhead 56 via-ethernet Sent 1311201393 bytes 5277696 pkt (dropped 128, overlimits 1606291 requeues 0) backlog 0b 0p requeues 0 memory used: 115328b of 4Mb capacity estimate: 19690Kbit Bulk Best Effort Video Voice thresh 1230Kbit 19690Kbit 9845Kbit 4922Kbit target 14.8ms 10.0ms 10.0ms 10.0ms interval 204.8ms 200.0ms 200.0ms 200.0ms pk_delay 6us 25us 2us 209us av_delay 3us 2us 2us 31us sp_delay 1us 1us 1us 3us pkts 927088 4133045 118552 99139 bytes 974738943 320755191 3326412 12572807 way_inds 7798 33031 0 2 way_miss 45787 84501 40 8687 way_cols 0 0 0 0 drops 0 128 0 0 marks 0 0 0 0 sp_flows 1 0 0 0 bk_flows 1 0 0 0 un_flows 0 0 0 0 max_len 1500 1500 84 1428 > > Independent of that matter one needs to specify different overheads > on ethN and pppeN interfaces simply as the packets on the pppoN > interface have not yet been encpsulated and are 8 byte smaller than > on the respective ethN interface. I am by now thoroghly confused, so > I might have missed the subtleties in which either cake or my > understanding is wrong. > >> On egress ppp it likely subtracts 22 bytes on ifb that is attached >> to ingress ppp 14 bytes. > > My understanding is again that on pppoe devices the kernel adds zero > bytes auto matically and attaching the ifb does not seem to change > that? The packet size is ip length as seen by cake on ifb redirected from pppoe - but this time it seems the difference is 14 not 22 ... tc qdisc add dev ifb0 handle 1:0 root cake bandwidth 60mbit raw overhead 34 diffserv4 nat dual-dsthost asr[/home/andy]# tc -s qdisc ls dev ifb0 qdisc cake 1: root refcnt 2 bandwidth 60Mbit diffserv4 dual-dsthost nat rtt 100.0ms noatm overhead 48 via-ethernet Sent 14315263754 bytes 12978554 pkt (dropped 51315, overlimits 18015371 requeues 0) backlog 0b 0p requeues 0 memory used: 1006656b of 4Mb capacity estimate: 60Mbit Bulk Best Effort Video Voice thresh 3750Kbit 60Mbit 30Mbit 15Mbit target 5.0ms 5.0ms 5.0ms 5.0ms interval 100.0ms 100.0ms 100.0ms 100.0ms pk_delay 243us 1.2ms 42us 82us av_delay 18us 821us 7us 8us sp_delay 2us 3us 4us 3us pkts 1942903 10462709 272777 351480 bytes 130598812 14085111253 8318580 168132633 way_inds 220 46306 0 0 way_miss 68580 114792 188 84890 way_cols 0 0 0 0 drops 0 51308 0 7 marks 0 0 0 0 sp_flows 0 0 1 0 bk_flows 0 0 0 0 un_flows 0 0 0 0 max_len 1500 1500 1500 1441 > >> If you shape on real eth (eg. lan side) or ifb hooked from real eth >> then you shouldn't use raw. > > Ah, I thought that the non-raw mode inquires at the kernel what > overhead it intends to account for silently and simply undoes that > probably by changing the overhead passed to the kernel so that the > sum of both match what the user requested? I think that things change when you add overhead XX (though I may need to retest that on a normal eth), but the reason I thought not to use raw on lan side was that cake really will see the packets as +14 so you want via-ethernet to subtract 14. > >> >> Thinking about it, mostly you will luck into it not making any >> difference, so the test I suggested earlier won't work. The reason >> being that whether the overhead is 40 or 40 - 22 an MTU sized >> packet or an empty ack/single sack will end up the same due to atm. >> sack > 1 will be wrong as of course will be other traffic of >> certain sizes, so a carefully crafted test would be needed to show >> the issue. >> >> Talking of acks/sacks 17500 vs 950kbit doesn't even have room for >> one sack per packet, though mostly there will be < 1 ack per >> packet. > > Intersting, looking into this I learned that SACK can cause up to 40 > bytes of TCP options added to the packet... Yea, not nice with overhead of 40 as 3 cells for multi sacks - but even at 2 cells Dendari doesn't have enough bandwidth for 1 ack per packet during recovery.