[Cake] Cake tin behaviour - discuss....

[Cake] Cake tin behaviour - discuss....

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 633 bytes --]

I’m confused as to what the ‘correct' behaviour should be under the following (real life) scenario:

Downstream VDSL wan link 79.5Mbit/s cake shaped to 77Mbit/s - diffserv4, so Bulk 4.8Mbit, Best effort 77Mbit, Video 38.5, Voice 19.

Download from ‘onedrive’ from 1 box, using 5 flows, classified as Bulk.  Little other traffic going on, sits there at circa 70Mbit, no problem.

If I started another download on another box, say 5 flows, classified as Best Effort, what rates would you expect the Bulk & Best effort tins to flow at?

Cheers,

Kevin D-B

gpg: 012C ACB2 28C6 C53E 9775  9123 B3A2 389B 9DE2 334A


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Cake] Cake tin behaviour - discuss....

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 5143 bytes --]


I'll bite.
 
Assuming a lot of things (you seem to be a microsoft user, One Drive, so your OS's network stack isn't necessarily very good at all).
 
35/35 split.
 
Why?
 
THere are no "bursts" in the fundamental flows (disk transfer rates are far higher than 80 Mb/sec., so the only burstiness would come from OS schedulers on either end).
 
There should be next to zero queueing in the bottleneck, and without queue depth, best efforts and bulk are happy to sync to that share, and stable once sawtoothing around an average of 35.
 
What's more important is what studying this teaches us:
 
1. diffserv only makes a difference when queues are allowed to build in switches/routers. But the whole goal of cake is to make the queues zero length.
 
2. TCP's optimal state is to adjust rate to insure that there is no queueing delay *inside* the network. (Well, Kleinrock says it should be just under 1 packet's worth of delay at the bottleneck router, and a small fraction of 1 packet on each router that is not bottlenecked.
 
3. in terms of "end to end" control, diffserv is about the worst possible mechanism for creating "differentiated service". It is based on a very old (pre-Jacobson AIMD) idea about inter-networking sharing of capacity. Because Van finally demonstrated (unfortunately it didn't penetrate the transport layer thick skulls who invented diffserv) that when *sharing* a path's capacity, the work has to be done at the *endpoints* - simply adjusting the window size on the receive end can do it - to slow the sending rate to the point where the network buffering drops to a mean of < 1 packet at the bottleneck.
 
4. diffserv is an example of attempting to "put a function in the network" that cannot be provided fully (or much at all) by the network equipment. The function (differentiating service quality) requires attention at the TCP level, not the IP layer, with the receive end and the transmit end cooperating. As one of the creators of The End-to-end argument, this is why I continue to be frustrated at the whole "diffserv" effort, which has wasted decades of sporadic research projects, all failing. My co-author, Dave Clark, has an equally strong critique of diffserv - which is that there is no actual quantitative and *universal* definition of all its "code points" across all AS's in the network. And there never will be because of commercial considerations - even if there were *only* two code points for performance (high and low), there is no way to provide pricing incentives for routers to follow those definitions in ANY algorithm they use.
 
5. There is paradoxically intense interest in *router vendors* and network operators to have any "feature" they can seel that claims to improve game performance or create "very low latency" priceable service. You can see this in the current "5G" pitches about being able to robotically do telesurgery with sub millisecond latency (faster than the speed of light, note, but the marketers don't care about truth), merely because they have "5G magic". To have a differentiation for your company's Brand, all you have to say is "I support diffserv", and the rubes will buy it. It doesn't work, but you can blame it on the fact that the problem is the other networks on the path, not your fancy routers.
 
6. If you have a dozen independent flows through a particular router, most likely those flows will be between pairs that do not, and pragmatically cannot, know anything about the other flows sharing the bottleneck. Yet to achieve differentiation among flows, somehow each flow must adjust its *own* rate to share *unequally* with the other flows.
There is 0.000 information bits/second about the differentiated service requirements shared between the distinct flows.
 
7. Every time I or others have pointed out that diffserv cannot work, we get met with nasty, very nasty personal attacks. I even wrote a short paper about it, which was killed by the referees (apparently diffserv fanboys). So we generally have just waited for the idea to die.
But it just won't die. It has never worked. But that just makes people want to imagine it will work if they only hold their breath real deep and wish.
 
On Saturday, April 25, 2020 7:07am, "Kevin Darbyshire-Bryant" <kevin@darbyshire-bryant.me.uk> said:



> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
> I’m confused as to what the ‘correct' behaviour should be under the
> following (real life) scenario:
> 
> Downstream VDSL wan link 79.5Mbit/s cake shaped to 77Mbit/s - diffserv4, so Bulk
> 4.8Mbit, Best effort 77Mbit, Video 38.5, Voice 19.
> 
> Download from ‘onedrive’ from 1 box, using 5 flows, classified as
> Bulk. Little other traffic going on, sits there at circa 70Mbit, no problem.
> 
> If I started another download on another box, say 5 flows, classified as Best
> Effort, what rates would you expect the Bulk & Best effort tins to flow at?
> 
> Cheers,
> 
> Kevin D-B
> 
> gpg: 012C ACB2 28C6 C53E 9775 9123 B3A2 389B 9DE2 334A
> 
> 

[-- Attachment #2: Type: text/html, Size: 8599 bytes --]

Re: [Cake] Cake tin behaviour - discuss....

[Jonathan Morton]

> On 25 Apr, 2020, at 2:07 pm, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
> 
> Download from ‘onedrive’ from 1 box, using 5 flows, classified as Bulk.  Little other traffic going on, sits there at circa 70Mbit, no problem.
> 
> If I started another download on another box, say 5 flows, classified as Best Effort, what rates would you expect the Bulk & Best effort tins to flow at?

Approximately speaking, Cake should give the Best Effort traffic priority over Bulk, until the latter is squashed down to its tin's capacity.  So you may see 5-10Mbps of Bulk and 65-70Mbps of Best Effort, depending on some short-term effects.

This assumes that the Diffserv marking actually reaches Cake, of course.

 - Jonathan Morton

Re: [Cake] Cake tin behaviour - discuss....

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 3350 bytes --]



> On 25 Apr 2020, at 16:25, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> On 25 Apr, 2020, at 2:07 pm, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
>> 
>> Download from ‘onedrive’ from 1 box, using 5 flows, classified as Bulk.  Little other traffic going on, sits there at circa 70Mbit, no problem.
>> 
>> If I started another download on another box, say 5 flows, classified as Best Effort, what rates would you expect the Bulk & Best effort tins to flow at?
> 
> Approximately speaking, Cake should give the Best Effort traffic priority over Bulk, until the latter is squashed down to its tin's capacity.  So you may see 5-10Mbps of Bulk and 65-70Mbps of Best Effort, depending on some short-term effects.
> 
> This assumes that the Diffserv marking actually reaches Cake, of course.

Thanks Jonathan.  I can assure you diffserv markings are reaching cake both egress & ingress due to my pet ‘act_ctinfo/connmark -savedscp’ project.  Amongst other monitoring methods a simple 'watch -t tc -s qdisc show dev $1’ albeit with a slightly modified cake module & tc to report per tin traffic as a percentage of total & per tin % of threshold is used.

eg:
                   Bulk  Best Effort        Video        Voice
  thresh       4812Kbit       77Mbit    38500Kbit    19250Kbit
  target          5.0ms        5.0ms        5.0ms        5.0ms
  interval      100.0ms      100.0ms      100.0ms      100.0ms
  pk_delay        961us        167us        311us        164us
  av_delay        453us         78us        141us         75us
  sp_delay         51us         12us         17us          9us
  backlog         9084b           0b           0b           0b
  pkts         60618617      2006708       460725        11129
  bytes     91414263264   2453185010    636385583      5205008
  traffic%           89            0            0            0
  traftin%         1435            0            0            0
  way_inds      2703134         8957          169          111
  way_miss          922         6192          104          525
  way_cols            0            0            0            0
  drops            8442          230           37            0
  marks               5            0            0            0
  ack_drop            0            0            0            0
  sp_flows            2            3            1            3
  bk_flows            1            0            0            0
  un_flows            0            0            0            0
  max_len         66616        12112         9084         3360
  quantum           300         1514         1174          587

Your expectation is that Best Effort would exert downward pressure on Bulk traffic reducing bulk traffic to about bulk threshold level which is my expectation also.  Tin priority then host (fairness), then flow.

As you may have guessed, that’s not quite what I’m seeing but as I’ve managed to see the issue when using ‘flowblind’ am now much less inclined to point the finger at host fairness & friends.  I remain confused why ‘bulk’ is exceeding its allocation though in what should be pressure from best effort but it ends up going all over the place and being a bit unstable.  Odd.

BTW: The ‘onedrive’ client box is actually running linux.


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Cake] Cake tin behaviour - discuss....

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 3193 bytes --]


Question: what's the "lag under load" experienced when these two loads are filling the capacity of the bottleneck router (the DSL link)?
I'm wondering whether your cake setup is deliberately building up a big queue within the router for any of the 10 bulk/best efforts flows.
 
On Saturday, April 25, 2020 4:34pm, "Kevin Darbyshire-Bryant" <kevin@darbyshire-bryant.me.uk> said:



> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
> 
> 
> > On 25 Apr 2020, at 16:25, Jonathan Morton <chromatix99@gmail.com>
> wrote:
> >
> >> On 25 Apr, 2020, at 2:07 pm, Kevin Darbyshire-Bryant
> <kevin@darbyshire-bryant.me.uk> wrote:
> >>
> >> Download from ‘onedrive’ from 1 box, using 5 flows,
> classified as Bulk. Little other traffic going on, sits there at circa 70Mbit, no
> problem.
> >>
> >> If I started another download on another box, say 5 flows, classified as
> Best Effort, what rates would you expect the Bulk & Best effort tins to flow at?
> >
> > Approximately speaking, Cake should give the Best Effort traffic priority
> over Bulk, until the latter is squashed down to its tin's capacity. So you may
> see 5-10Mbps of Bulk and 65-70Mbps of Best Effort, depending on some short-term
> effects.
> >
> > This assumes that the Diffserv marking actually reaches Cake, of course.
> 
> Thanks Jonathan. I can assure you diffserv markings are reaching cake both egress
> & ingress due to my pet ‘act_ctinfo/connmark -savedscp’ project. 
> Amongst other monitoring methods a simple 'watch -t tc -s qdisc show dev $1’
> albeit with a slightly modified cake module & tc to report per tin traffic as a
> percentage of total & per tin % of threshold is used.
> 
> eg:
> Bulk Best Effort Video Voice
> thresh 4812Kbit 77Mbit 38500Kbit 19250Kbit
> target 5.0ms 5.0ms 5.0ms 5.0ms
> interval 100.0ms 100.0ms 100.0ms 100.0ms
> pk_delay 961us 167us 311us 164us
> av_delay 453us 78us 141us 75us
> sp_delay 51us 12us 17us 9us
> backlog 9084b 0b 0b 0b
> pkts 60618617 2006708 460725 11129
> bytes 91414263264 2453185010 636385583 5205008
> traffic% 89 0 0 0
> traftin% 1435 0 0 0
> way_inds 2703134 8957 169 111
> way_miss 922 6192 104 525
> way_cols 0 0 0 0
> drops 8442 230 37 0
> marks 5 0 0 0
> ack_drop 0 0 0 0
> sp_flows 2 3 1 3
> bk_flows 1 0 0 0
> un_flows 0 0 0 0
> max_len 66616 12112 9084 3360
> quantum 300 1514 1174 587
> 
> Your expectation is that Best Effort would exert downward pressure on Bulk traffic
> reducing bulk traffic to about bulk threshold level which is my expectation also. 
> Tin priority then host (fairness), then flow.
> 
> As you may have guessed, that’s not quite what I’m seeing but as
> I’ve managed to see the issue when using ‘flowblind’ am now much
> less inclined to point the finger at host fairness & friends. I remain confused
> why ‘bulk’ is exceeding its allocation though in what should be
> pressure from best effort but it ends up going all over the place and being a bit
> unstable. Odd.
> 
> BTW: The ‘onedrive’ client box is actually running linux.
> 
> 

[-- Attachment #2: Type: text/html, Size: 4347 bytes --]

Re: [Cake] Cake tin behaviour - discuss....

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 607 bytes --]



> On 25 Apr 2020, at 21:56, David P. Reed <dpreed@deepplum.com> wrote:
> 
> Question: what's the "lag under load" experienced when these two loads are filling the capacity of the bottleneck router (the DSL link)?
> I'm wondering whether your cake setup is deliberately building up a big queue within the router for any of the 10 bulk/best efforts flows.

https://www.thinkbroadband.com/broadband/monitoring/quality/share/3dec809ecef5cd52f574b6be5da9af28326845d6-25-04-2020

I don’t reckon it’s bad for the past 24 hours, one peak at 50ms.  Avg latency increase by about 6ms during load.



[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Cake] Cake tin behaviour - discuss....

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 1838 bytes --]


Very interesting. However, I'm curious about what is being "ping'ed" from outside.
 
I would bet that the ping comes in on your router interface and is reflected immediately back. Which would mean that it might not at all be going through the Cake layer. That depends on the details of your setup, which you didn't share.
 
As you probably know, Cake works by packet shaping in the box where it runs, in the Linux stack. If the ping responder is on the ISP side of Cake, it will not be measuring lag-under-load *inside* cake.
 
End-to-end lag-under-load on multiple paths sharing a bottleneck is the problem Cake was invented to solve. (Jonathan - you agree?) Yes, it will move that congestion "inside" itself, pulling it out of the bottleneck itself. There it drops and ECN's "as if" the bottleneck were working correctly, rather than being "bufferbloated".
 
So it would be interesting to learn more about the topology of your test to interpret this ping.  A more interesting ping would be along the fujl path that the other flows are taking. Your ISP can't provide that.
 
 
On Saturday, April 25, 2020 5:31pm, "Kevin Darbyshire-Bryant" <kevin@darbyshire-bryant.me.uk> said:



> 
> 
> > On 25 Apr 2020, at 21:56, David P. Reed <dpreed@deepplum.com> wrote:
> >
> > Question: what's the "lag under load" experienced when these two loads are
> filling the capacity of the bottleneck router (the DSL link)?
> > I'm wondering whether your cake setup is deliberately building up a big queue
> within the router for any of the 10 bulk/best efforts flows.
> 
> https://www.thinkbroadband.com/broadband/monitoring/quality/share/3dec809ecef5cd52f574b6be5da9af28326845d6-25-04-2020
> 
> I don’t reckon it’s bad for the past 24 hours, one peak at 50ms. Avg
> latency increase by about 6ms during load.
> 
> 
> 

[-- Attachment #2: Type: text/html, Size: 3359 bytes --]

Re: [Cake] Cake tin behaviour - discuss....

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 2764 bytes --]



> On 26 Apr 2020, at 14:53, David P. Reed <dpreed@deepplum.com> wrote:
> 
> Very interesting. However, I'm curious about what is being "ping'ed" from outside.
> 
> I would bet that the ping comes in on your router interface and is reflected immediately back. Which would mean that it might not at all be going through the Cake layer. That depends on the details of your setup, which you didn't share.

The address being pinged from the external ‘ping box’ is that of the globally routable IPv6 WAN interface on my APU2 router.  The ping packet is going through 2 instances of cake, one on ingress (ifb4eth0), one on egress (eth0).

DSCP is applied to the packets by tc filter action act_ctinfo JUST before cake gets to see the packets.  I know DSCP is affecting cake tin selection because I see cake’s tin byte/packet counters adjust accordingly.  icmp/icmpv6 traffic is marked as BE by default AND also explicitly by some ip’n’tables rules that set it so.

> As you probably know, Cake works by packet shaping in the box where it runs, in the Linux stack. If the ping responder is on the ISP side of Cake, it will not be measuring lag-under-load *inside* cake.

I think I answered that above, however just for good measure, I’ve set up another ‘ping latency’ test to a box that is definitely on my LAN side, so it’ll go: ingress (cake) eth0 (wan) -> egress eth1 (lan) -> switches -> device under test -> ingress eth1 (lan) -> egress (cake) eth0 (wan)

Note that the DSCP applied by cake on egress is ignored by the ISP.  Similarly, it’s a very rare thing to see a non 0 DSCP come in from them.  I’m using DSCP ‘internally’ purely to provide CAKE with some traffic identification and hence clue as to how to shape it.

> 
> End-to-end lag-under-load on multiple paths sharing a bottleneck is the problem Cake was invented to solve. (Jonathan - you agree?) Yes, it will move that congestion "inside" itself, pulling it out of the bottleneck itself. There it drops and ECN's "as if" the bottleneck were working correctly, rather than being "bufferbloated".
> 
> So it would be interesting to learn more about the topology of your test to interpret this ping.  A more interesting ping would be along the fujl path that the other flows are taking. Your ISP can't provide that.

My question was trying to determine what cake was doing:

bandwidth / per host fairness / tin weighting or
bandwidth / tin weighting / per host fairness

I was expecting the latter and Jonathan has confirmed my expectation to be the correct one.  The results I saw under some circumstance appeared more toward the former, which boggled the mind.


Cheers,

Kevin D-B

gpg: 012C ACB2 28C6 C53E 9775  9123 B3A2 389B 9DE2 334A


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]