[Cake] low bandwidth default params best effort vs voice latency.

[Cake] low bandwidth default params best effort vs voice latency.

[Andy Furniss]

In the UK quite a lot of people have a 40/2 vdsl2 product.

Thankfully not me, ugh, it doesn't even have enough bandwidth for
sack per incoming in recovery - but "pretending" I wanted to see what
cake was like.

tc qdisc add dev enp6s0 handle 1:0 root cake bandwidth 1969230bit 
overhead 34 dual-srchost diffserv3

tc -s qdisc ls dev enp6s0
qdisc cake 1: root refcnt 2 bandwidth 1969Kbit diffserv3 dual-srchost 
rtt 100.0ms noatm overhead 34 via-ethernet
  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
  backlog 0b 0p requeues 0
  memory used: 0b of 4Mb
  capacity estimate: 1969Kbit
                  Bulk   Best Effort      Voice
   thresh     123072bit    1969Kbit   492304bit
   target       147.6ms       9.2ms      36.9ms
   interval     295.2ms     104.2ms      73.8ms

Now we have 3 xbox ones in the house and I know that when we play
soldiers together they each do 60 pps upstream just for the game
(more if you include voice).

So I use fping x3 to sim this from 3 different ips and then also
do a single upload (netperf) with a fourth address.

fping -S 192.168.0.59 -c 1000 -b 150 -p 16 -i 16 -Q 1 asr & fping .......

With all traffic through Best effort, 1000 pings x3 looks like

asr : xmt/rcv/%loss = 1000/1000/0%, min/avg/max = 0.55/4.57/8.77
asr : xmt/rcv/%loss = 1000/1000/0%, min/avg/max = 0.57/4.78/8.67
asr : xmt/rcv/%loss = 1000/1000/0%, min/avg/max = 0.60/4.62/8.73

I notice that unlike diffserv4 marking game traffic as real time
interactive (cs4) doesn't go to voice.
So I mark icmp as ef which does go to voice and repeat the test.

It performs slightly worse for max delay.

asr : xmt/rcv/%loss = 1000/1000/0%, min/avg/max = 0.50/4.33/14.1
asr : xmt/rcv/%loss = 1000/1000/0%, min/avg/max = 0.65/4.44/11.2
asr : xmt/rcv/%loss = 1000/1000/0%, min/avg/max = 0.86/4.71/13.6

Admittedly it's only a small proportion of the packets that are
above average but still it's not the result I would have hoped for.

I know cake seeks to be low configuration, but being able to tweak
things like target/interval and thresh per bin I think would give
greater flexibility. On thresh for example, a single xbox hosting
a game may need a bit more than 492k and (IMHO) it should be up to the
user to decide things like that.

Re: [Cake] low bandwidth default params best effort vs voice latency.

[Jonathan Morton]

> On 4 Mar, 2017, at 20:21, Andy Furniss <adf.lists@gmail.com> wrote:
> 
> So I mark icmp as ef which does go to voice and repeat the test.
> 
> It performs slightly worse for max delay.

Hmm.

If I’m reading those fping commands right, you’re sending a total of 36KB/s in ICMP (4x 60pps 150B), which should fit into the 25% Voice allocation on a 2Mbps (250KB/s) uplink.  Under these circumstances, Voice should still be prioritised relative to Best Effort.  Adding over 10ms of RTT is not what’s expected.

Let me think carefully about how this filters through the priority layer.  I may need to tweak it a bit.

 - Jonathan Morton

Re: [Cake] low bandwidth default params best effort vs voice latency.

[Andy Furniss]

Jonathan Morton wrote:
>
>> On 4 Mar, 2017, at 20:21, Andy Furniss <adf.lists@gmail.com> wrote:
>>
>> So I mark icmp as ef which does go to voice and repeat the test.
>>
>> It performs slightly worse for max delay.
>
> Hmm.
>
> If I’m reading those fping commands right, you’re sending a total of 36KB/s in ICMP (4x 60pps 150B), which should fit into the 25% Voice allocation on a 2Mbps (250KB/s) uplink.  Under these circumstances, Voice should still be prioritised relative to Best Effort.  Adding over 10ms of RTT is not what’s expected.
>
> Let me think carefully about how this filters through the priority layer.  I may need to tweak it a bit.

Thanks, I should be under rate, though for clarity I am using 3 pingers
so 3x 62pps x (150 + 28 + 34) with my overheads.

Re: [Cake] low bandwidth default params best effort vs voice latency.

[Jonathan Morton]

Okay, I think I’ve worked out what is happening.

At 250KB/s, it takes 6ms to get one 1500-byte bulk packet down the pipe.  This is unavoidable, so having a bulk flow competing with your game traffic will always increase your peak latency by that much.

With three independent game streams in play, it’s possible for them all to transmit simultaneously *and* to coincide with a bulk packet having just been sent.  With overheads, it will take a total of 8.5ms to get all four packets through.  This corresponds nicely to your best-effort results; you’re getting very close to the theoretical best performance there.

So Diffserv marking actually can’t improve your performance in this particular case.  But it shouldn’t make it worse either.  You’re actually seeing a nearly 6ms increase in peak latency, which corresponds neatly to an additional bulk packet ending up ahead of the game traffic in the queue.

That’s not supposed to happen, but I think I can see how it *can* happen with the current Diffserv logic.  It’s a weighted DRR, much like what is used between flow queues a little further down - but it *doesn’t* have a special bonus for sparse tins.  That’s something I clearly need to fix.

To help remind me, please do open an issue on the Github project.

 - Jonathan Morton

Re: [Cake] low bandwidth default params best effort vs voice latency.

[Dave Taht]

One thing the original sqm-scripts did (with fq_codel) was explicitly
deprioritize ping with a filter.

I had written  in my original "wondershaper must die" rant how stupid
it was to prioritize ping upwards  to "impress your friends" - as that
was the case in many, many a wshaper implementation I'd seen.

I only just now noticed that rant does not have the images inline.
"Oh, man, ping's prioritized:"

https://www.bufferbloat.net/projects/attachments/131229050304_wshaper-800-220.svg

As for explicit depriorization, well, I took a lot of flack from some
ietfers about lowering it below best effort, and I felt the additional
classification needed was excessive for cake, directly. Some OSes
already do mark it with a background class.  I still think it's a good
idea as it makes ping floods against (for example) your entire 10.
network (which happened to me under worm attack once) - vanish.

Re: [Cake] low bandwidth default params best effort vs voice latency.

[Andy Furniss]

Jonathan Morton wrote:
> Okay, I think I’ve worked out what is happening.
>
> At 250KB/s, it takes 6ms to get one 1500-byte bulk packet down the
> pipe.  This is unavoidable, so having a bulk flow competing with your
> game traffic will always increase your peak latency by that much.
>
> With three independent game streams in play, it’s possible for them
> all to transmit simultaneously *and* to coincide with a bulk packet
> having just been sent.  With overheads, it will take a total of 8.5ms
> to get all four packets through.  This corresponds nicely to your
> best-effort results; you’re getting very close to the theoretical
> best performance there.

Yea, the pleasures of low bandwidth - not that I would have called 2mbit
low a few years ago, and 8ms isn't that bad.

Historically, I once had an asymmetric mss clamping setup so up tcp
was smaller than down.

> So Diffserv marking actually can’t improve your performance in this
> particular case.  But it shouldn’t make it worse either.  You’re
> actually seeing a nearly 6ms increase in peak latency, which
> corresponds neatly to an additional bulk packet ending up ahead of
> the game traffic in the queue.
>
> That’s not supposed to happen, but I think I can see how it *can*
> happen with the current Diffserv logic.  It’s a weighted DRR, much
> like what is used between flow queues a little further down - but it
> *doesn’t* have a special bonus for sparse tins.  That’s something I
> clearly need to fix.
>
> To help remind me, please do open an issue on the Github project.

OK, I opened
https://github.com/dtaht/sch_cake/issues/52

Re: [Cake] low bandwidth default params best effort vs voice latency.

[Andy Furniss]

Dave Taht wrote:
> One thing the original sqm-scripts did (with fq_codel) was explicitly
> deprioritize ping with a filter.
>
> I had written  in my original "wondershaper must die" rant how stupid
> it was to prioritize ping upwards  to "impress your friends" - as that
> was the case in many, many a wshaper implementation I'd seen.
>
> I only just now noticed that rant does not have the images inline.
> "Oh, man, ping's prioritized:"
>
> https://www.bufferbloat.net/projects/attachments/131229050304_wshaper-800-220.svg
>
> As for explicit depriorization, well, I took a lot of flack from some
> ietfers about lowering it below best effort, and I felt the additional
> classification needed was excessive for cake, directly. Some OSes
> already do mark it with a background class.  I still think it's a good
> idea as it makes ping floods against (for example) your entire 10.
> network (which happened to me under worm attack once) - vanish.

Yea, fair point, but in this case it's just a simulation tool as I don't
know how to get delay times from a udp stream.

I still think cs4 should go to voice on diffserv3 as it does on diffserv4.

Another observation with flent udp_flood, I notice that 
flent-london.bufferbloat.net
is tcp only, is this intended? The test still works, despite the port
unreachables coming back, just no bandwidth reading.