[Cake] Getting Cake to work better with Steam and similar applications

Andy Furniss adf.lists at gmail.com
Tue Apr 25 18:06:31 EDT 2017

Sebastian Moeller wrote:
> Hi Andy,
>> On Apr 25, 2017, at 14:58, Andy Furniss <adf.lists at gmail.com>
>> wrote:
>> Dendari Marini wrote:
>>> Also I have done some more testing, I was able to limit Steam
>>> connections just to one thanks to some console commands
>>> ("@cMaxContentServersToRequest" and 
>>> "@cCSClientMaxNumSocketsPerHost") and while the situation
>>> improved (no more packet loss, latency variation within 10ms, but
>>> still seeing ping spikes of ~50ms) it's still not what I'd
>>> consider ideal, which would be like with any other download. So
>>> my guess is there's something else going on other than just the
>>> multiple connections, which are definitely big part of the
>>> problem but not the only thing. Anyway these are my current
>>> settings for Cake and I've been using them for the last four days
>>> without issues: *qdisc cake 8005: root refcnt 2 bandwidth 950Kbit
>>> diffserv3 triple-isolate nat wash rtt 100.0ms atm overhead 40
>>> via-ethernet* *qdisc cake 8006: root refcnt 2 bandwidth 17500Kbit
>>> diffserv3 triple-isolate nat wash ingress rtt 100.0ms atm
>>> overhead 40 via-ethernet*
>> I still think that once via-ethernet appears you really need the
>> raw parameter.
> Why? As far as I can tell the default mode is to adjust for the
> kernel auto-added overhead (which seems to be 14 bytes for pakets
> headed for an ethernet device and zero for packets headed for a ppp
> device). If he adds raw, the kernel willl make its additions
> silently…

I shape on pppoe - though it's ptm. My overhead is 34, it seems as soon
as you add overhead as a parameter via-ethernet appears and in the case
of pppoe apparently does the wrong thing. Recalling my now closed issue
about overheads on github, cake beleives what the kernel tells it about
overheads and this seems to 22. In fact cake does not see the traffic as
+22 it sees ip length.

If I don't use raw below my shaper fails. Also note that when using raw
the overhead gets increased by 22 from that specified.

tc qdisc add dev ppp0 handle 1:0 root cake bandwidth 19690kbit raw 
overhead 34 diffserv4 dual-srchost nat rtt 200ms

asr[/home/andy]# tc -s qdisc ls dev ppp0
qdisc cake 1: root refcnt 2 bandwidth 19690Kbit diffserv4 dual-srchost 
nat rtt 200.0ms noatm overhead 56 via-ethernet
  Sent 1311201393 bytes 5277696 pkt (dropped 128, overlimits 1606291 
requeues 0)
  backlog 0b 0p requeues 0
  memory used: 115328b of 4Mb
  capacity estimate: 19690Kbit
                  Bulk   Best Effort      Video       Voice
   thresh      1230Kbit   19690Kbit    9845Kbit    4922Kbit
   target        14.8ms      10.0ms      10.0ms      10.0ms
   interval     204.8ms     200.0ms     200.0ms     200.0ms
   pk_delay         6us        25us         2us       209us
   av_delay         3us         2us         2us        31us
   sp_delay         1us         1us         1us         3us
   pkts          927088     4133045      118552       99139
   bytes      974738943   320755191     3326412    12572807
   way_inds        7798       33031           0           2
   way_miss       45787       84501          40        8687
   way_cols           0           0           0           0
   drops              0         128           0           0
   marks              0           0           0           0
   sp_flows           1           0           0           0
   bk_flows           1           0           0           0
   un_flows           0           0           0           0
   max_len         1500        1500          84        1428

> Independent of that matter one needs to specify different overheads
> on ethN and pppeN interfaces simply as the packets on the pppoN
> interface have not yet been encpsulated and are 8 byte smaller than
> on the respective ethN interface. I am by now thoroghly confused, so
> I might have missed the subtleties in which either cake or my
> understanding is wrong.
>> On egress ppp it likely subtracts 22 bytes on ifb that is attached
>> to ingress ppp 14 bytes.
> My understanding is again that on pppoe devices the kernel adds zero
> bytes auto matically and attaching the ifb does not seem to change
> that?

The packet size is ip length as seen by cake on ifb redirected from
pppoe - but this time it seems the difference is 14 not 22 ...

tc qdisc add dev ifb0 handle 1:0 root cake bandwidth 60mbit raw overhead 
34 diffserv4 nat dual-dsthost

asr[/home/andy]# tc -s qdisc ls dev ifb0
qdisc cake 1: root refcnt 2 bandwidth 60Mbit diffserv4 dual-dsthost nat 
rtt 100.0ms noatm overhead 48 via-ethernet
  Sent 14315263754 bytes 12978554 pkt (dropped 51315, overlimits 
18015371 requeues 0)
  backlog 0b 0p requeues 0
  memory used: 1006656b of 4Mb
  capacity estimate: 60Mbit
                  Bulk   Best Effort      Video       Voice
   thresh      3750Kbit      60Mbit      30Mbit      15Mbit
   target         5.0ms       5.0ms       5.0ms       5.0ms
   interval     100.0ms     100.0ms     100.0ms     100.0ms
   pk_delay       243us       1.2ms        42us        82us
   av_delay        18us       821us         7us         8us
   sp_delay         2us         3us         4us         3us
   pkts         1942903    10462709      272777      351480
   bytes      130598812 14085111253     8318580   168132633
   way_inds         220       46306           0           0
   way_miss       68580      114792         188       84890
   way_cols           0           0           0           0
   drops              0       51308           0           7
   marks              0           0           0           0
   sp_flows           0           0           1           0
   bk_flows           0           0           0           0
   un_flows           0           0           0           0
   max_len         1500        1500        1500        1441

>> If you shape on real eth (eg. lan side) or ifb hooked from real eth
>> then you shouldn't use raw.
> Ah, I thought that the non-raw mode inquires at the kernel what
> overhead it intends to account for silently and simply undoes that
> probably by changing the overhead passed to the kernel so that the
> sum of both match what the user requested?

I think that things change when you add overhead XX (though I may need
to retest that on a normal eth), but the reason I thought not to use raw
on lan side was that cake really will see the packets as +14 so you want
via-ethernet to subtract 14.

>> Thinking about it, mostly you will luck into it not making any 
>> difference, so the test I suggested earlier won't work. The reason
>> being that whether the overhead is 40 or 40 - 22 an MTU sized
>> packet or an empty ack/single sack will end up the same due to atm.
>> sack > 1 will be wrong as of course will be other traffic of
>> certain sizes, so a carefully crafted test would be needed to show
>> the issue.
>> Talking of acks/sacks 17500 vs 950kbit doesn't even have room for
>> one sack per packet, though mostly there will be < 1 ack per
>> packet.
> Intersting, looking into this I learned that SACK can cause up to 40
> bytes of TCP options added to the packet...

Yea, not nice with overhead of 40 as 3 cells for multi sacks - but even
at 2 cells Dendari doesn't have enough bandwidth for 1 ack per packet
during recovery.

More information about the Cake mailing list