Re: [Cake] cake default target is too low for bbr?

Re: [Cake] cake default target is too low for bbr?

[xnor]

Hello,

As I understand it, increase in RTT due to queueing of packets is the 
main feedback mechanism for BBR.
So dropping packets, which I already consider harmful, is really harmful 
with BBR because you're not telling the sender to slow down.

Instead, with a controlled delay qdisc like cake or codel, you're 
telling the sender to keep sending the data faster because the qdisc 
keeps the increase in RTT minimal. To make things worse, you're throwing 
away perfectly good packets with no effect other than wasting bandwidth.


I'm not sure how BBR handles jitter or how small of an increase in RTT 
it detects as "congestion" but from my limited testing, I've noticed no 
problems with steam downloads in the bulk tin with 15 ms target in cake.

Re: [Cake] cake default target is too low for bbr?

[David Lang]

On Fri, 28 Apr 2017, xnor wrote:

> As I understand it, increase in RTT due to queueing of packets is the 
> main feedback mechanism for BBR.
> So dropping packets, which I already consider harmful, is really harmful 
> with BBR because you're not telling the sender to slow down.

If BBR does not slow down when packets are dropped, it's too hostile to use on a 
public network. The only way for a public network to respond to a flood of 
traffic higher than what it can handle is to drop packets (with a possible 
warning via ECN shortly before packets get dropped). If BBR doesn't slow down, 
it's just going to be wasting bandwidth.

> Instead, with a controlled delay qdisc like cake or codel, you're 
> telling the sender to keep sending the data faster because the qdisc 
> keeps the increase in RTT minimal. To make things worse, you're throwing 
> away perfectly good packets with no effect other than wasting bandwidth.

you are mistaking how cake and codel work. They are working at the link endpoint 
adjacent to where the bandwidth is most limited. They drop packets before they 
get send over the most contrained link. The fact that the packets eat up some 
bandwidth on the non-constrained link prior to the bottleneck doesn't matter, 
the bandwidth is available by definition (otherwise it would be a constrained 
link to the endpoint before it)

you have the systems A-H sending connected in series. If the bandwidth between D 
and E is the bottleneck, it doesn't matter that you wast the bandwidth between 
A-B-C-D because the bandwidth would otherwise be idle. D drops packets only when 
there is not bandwidth between D and E to handle them.

David Lang

>
> I'm not sure how BBR handles jitter or how small of an increase in RTT 
> it detects as "congestion" but from my limited testing, I've noticed no 
> problems with steam downloads in the bulk tin with 15 ms target in cake.
>
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>

Re: [Cake] cake default target is too low for bbr?

[Andy Furniss]

David Lang wrote:
> On Fri, 28 Apr 2017, xnor wrote:
> 
>> As I understand it, increase in RTT due to queueing of packets is 
>> the main feedback mechanism for BBR. So dropping packets, which I 
>> already consider harmful, is really harmful with BBR because
>> you're not telling the sender to slow down.
> 
> If BBR does not slow down when packets are dropped, it's too hostile 
> to use on a public network. The only way for a public network to 
> respond to a flood of traffic higher than what it can handle is to 
> drop packets (with a possible warning via ECN shortly before packets 
> get dropped). If BBR doesn't slow down, it's just going to be
> wasting bandwidth.
> 
>> Instead, with a controlled delay qdisc like cake or codel, you're 
>> telling the sender to keep sending the data faster because the 
>> qdisc keeps the increase in RTT minimal. To make things worse, 
>> you're throwing away perfectly good packets with no effect other 
>> than wasting bandwidth.
> 
> you are mistaking how cake and codel work. They are working at the 
> link endpoint adjacent to where the bandwidth is most limited. They 
> drop packets before they get send over the most contrained link. The 
> fact that the packets eat up some bandwidth on the non-constrained 
> link prior to the bottleneck doesn't matter, the bandwidth is 
> available by definition (otherwise it would be a constrained link to 
> the endpoint before it)

Well cake may be used on ingress, OK so that is a "hack" but it's better
than nothing. The ingress parameter does help a bit - but not much.

Reducing bufferbloat is a good ideal, but it seems sometimes being too
aggressive can hurt (I've also needed to increase rtt somewhat higher
than the actual latency to get upstream tests to fill bandwidth).

If target is supposed to work 5-10% of rtt value, then maybe cake could
let users choose 10% rather than hard code 5% - to avoid the need to set
excessive rtt values just to raise target.

Re: [Cake] cake default target is too low for bbr?

[xnor]

>On Fri, 28 Apr 2017, xnor wrote:
>
>>As I understand it, increase in RTT due to queueing of packets is the 
>>main feedback mechanism for BBR.
>>So dropping packets, which I already consider harmful, is really 
>>harmful with BBR because you're not telling the sender to slow down.
>
>If BBR does not slow down when packets are dropped, it's too hostile to 
>use on a public network. The only way for a public network to respond 
>to a flood of traffic higher than what it can handle is to drop packets 
>(with a possible warning via ECN shortly before packets get dropped). 
>If BBR doesn't slow down, it's just going to be wasting bandwidth.
No it isn't. Packet loss does not equal conguestion - it never did. 
Dropping packets to signal congestion is an ugly hack for 
implementations that are too dumb to understand any proper congestion 
control mechanism.
Dropping due to queues being full normally doesn't happen before the RTT 
has significantly increased.

BBR fixes both of these problems:
a) it ignores packet loss on unreliable links and therefore achieves 
much higher throughput than conventional congestion control algorithms 
(that wrongly assume congestion on packet loss)
An experiment with 10 Gbps bottleneck, 100 ms RTT and 1% packet loss (as 
described in the net-next commit) showed ~3000x higher throughput with 
BBR compared to cubic.

b) it reacts to increase in RTT.
An experiment with 10 Mbps bottleneck, 40 ms RTT and a typical 1000 
packet buffer, increase in RTT with BBR is ~3 ms while with cubic it is 
over 1000 ms.


>
>>Instead, with a controlled delay qdisc like cake or codel, you're 
>>telling the sender to keep sending the data faster because the qdisc 
>>keeps the increase in RTT minimal. To make things worse, you're 
>>throwing away perfectly good packets with no effect other than wasting 
>>bandwidth.
>
>you are mistaking how cake and codel work. They are working at the link 
>endpoint adjacent to where the bandwidth is most limited. They drop 
>packets before they get send over the most contrained link. The fact 
>that the packets eat up some bandwidth on the non-constrained link 
>prior to the bottleneck doesn't matter, the bandwidth is available by 
>definition (otherwise it would be a constrained link to the endpoint 
>before it)
No, my ISP isn't using either cake or codel and neither is Andy's.
We're talking about ingress traffic "shaping" of downloads here, so 
we're not working at the point in front of the most constrained link. 
Instead we work behind the most contrained link.

That's why dropping packets is counterproductive, but as I've mentioned 
before it should be avoided anyway (except for the broken parts of the 
Internet where it still is a necessary evil).

Re: [Cake] cake default target is too low for bbr?

[Andy Furniss]

xnor wrote:
> 
>> On Fri, 28 Apr 2017, xnor wrote:
>> 
>>> As I understand it, increase in RTT due to queueing of packets is
>>> the main feedback mechanism for BBR. So dropping packets, which I
>>> already consider harmful, is really harmful with BBR because
>>> you're not telling the sender to slow down.
>> 
>> If BBR does not slow down when packets are dropped, it's too
>> hostile to use on a public network. The only way for a public
>> network to respond to a flood of traffic higher than what it can
>> handle is to drop packets (with a possible warning via ECN shortly
>> before packets get dropped). If BBR doesn't slow down, it's just
>> going to be wasting bandwidth.
> No it isn't. Packet loss does not equal conguestion - it never did. 
> Dropping packets to signal congestion is an ugly hack for 
> implementations that are too dumb to understand any proper congestion
>  control mechanism.

Hmm, I bet a lot of carrier links are policed rather than smart queue.

It also seems (OK one quick possibly flawed test), that bbr ignores ECN
as well as drops in the sense that marked is just as high as dropped.

> Dropping due to queues being full normally doesn't happen before the
> RTT has significantly increased.

Not so good on ingress though - where normaly (ie. non bbr) a drop is
handy to signal early to back off - letting a buffer fill up too much is
also somewhat going to fill the remote buffer that you would like to
keep empty.

> 
> BBR fixes both of these problems: a) it ignores packet loss on
> unreliable links and therefore achieves much higher throughput than
> conventional congestion control algorithms (that wrongly assume
> congestion on packet loss) An experiment with 10 Gbps bottleneck, 100
> ms RTT and 1% packet loss (as described in the net-next commit)
> showed ~3000x higher throughput with BBR compared to cubic.

So on a congested policed link it will make matters worse for "normal"
tcp users - maybe everyone will need to use it soon.

> b) it reacts to increase in RTT. An experiment with 10 Mbps
> bottleneck, 40 ms RTT and a typical 1000 packet buffer, increase in
> RTT with BBR is ~3 ms while with cubic it is over 1000 ms.

That is a nice aspect (though at 60mbit hfsc + 80ms bfifo I tested with
5 tcps it was IIRC 20ms vs 80 for cubic). I deliberately test using ifb
on my PC because I want to pretend to be a router - IME (OK it was a
while ago) testing on eth directly gives different results - like the
locally generated tcp is backing off and giving different results.

Re: [Cake] cake default target is too low for bbr?

[Andy Furniss]

Andy Furniss wrote:

>> b) it reacts to increase in RTT. An experiment with 10 Mbps
>> bottleneck, 40 ms RTT and a typical 1000 packet buffer, increase in
>> RTT with BBR is ~3 ms while with cubic it is over 1000 ms.
> 
> That is a nice aspect (though at 60mbit hfsc + 80ms bfifo I tested with
> 5 tcps it was IIRC 20ms vs 80 for cubic). I deliberately test using ifb
> on my PC because I want to pretend to be a router - IME (OK it was a
> while ago) testing on eth directly gives different results - like the
> locally generated tcp is backing off and giving different results.

I retested this with 40ms latency (netem) with hfsc + 1000 pfifo on ifb.

5 tcps netperf bbr =

64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=40.611 ms
64 bytes from 192.168.0.1: icmp_seq=3 ttl=64 time=162.566 ms
64 bytes from 192.168.0.1: icmp_seq=4 ttl=64 time=163.854 ms
64 bytes from 192.168.0.1: icmp_seq=5 ttl=64 time=143.220 ms
64 bytes from 192.168.0.1: icmp_seq=6 ttl=64 time=139.458 ms
64 bytes from 192.168.0.1: icmp_seq=7 ttl=64 time=139.466 ms
64 bytes from 192.168.0.1: icmp_seq=8 ttl=64 time=139.570 ms
64 bytes from 192.168.0.1: icmp_seq=9 ttl=64 time=139.876 ms
64 bytes from 192.168.0.1: icmp_seq=10 ttl=64 time=139.592 ms
64 bytes from 192.168.0.1: icmp_seq=11 ttl=64 time=139.580 ms
64 bytes from 192.168.0.1: icmp_seq=12 ttl=64 time=139.458 ms
64 bytes from 192.168.0.1: icmp_seq=13 ttl=64 time=40.625 ms

Of course cubic was totally horrible -

64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=40.605 ms
64 bytes from 192.168.0.1: icmp_seq=3 ttl=64 time=209.066 ms
64 bytes from 192.168.0.1: icmp_seq=4 ttl=64 time=261.172 ms
64 bytes from 192.168.0.1: icmp_seq=5 ttl=64 time=316.063 ms
64 bytes from 192.168.0.1: icmp_seq=6 ttl=64 time=363.004 ms
64 bytes from 192.168.0.1: icmp_seq=7 ttl=64 time=417.586 ms
64 bytes from 192.168.0.1: icmp_seq=8 ttl=64 time=569.068 ms
64 bytes from 192.168.0.1: icmp_seq=9 ttl=64 time=784.740 ms
64 bytes from 192.168.0.1: icmp_seq=10 ttl=64 time=1064.652 ms
64 bytes from 192.168.0.1: icmp_seq=12 ttl=64 time=1204.801 ms
64 bytes from 192.168.0.1: icmp_seq=13 ttl=64 time=973.802 ms
64 bytes from 192.168.0.1: icmp_seq=14 ttl=64 time=155.116 ms
64 bytes from 192.168.0.1: icmp_seq=15 ttl=64 time=40.414 ms

With hfsc on root of eth both cubic and bbr were the same -

64 bytes from 192.168.0.1: icmp_seq=3 ttl=64 time=40.619 ms
64 bytes from 192.168.0.1: icmp_seq=4 ttl=64 time=52.775 ms
64 bytes from 192.168.0.1: icmp_seq=5 ttl=64 time=53.217 ms
64 bytes from 192.168.0.1: icmp_seq=6 ttl=64 time=52.742 ms
64 bytes from 192.168.0.1: icmp_seq=7 ttl=64 time=53.176 ms
64 bytes from 192.168.0.1: icmp_seq=8 ttl=64 time=53.179 ms
64 bytes from 192.168.0.1: icmp_seq=9 ttl=64 time=53.027 ms
64 bytes from 192.168.0.1: icmp_seq=10 ttl=64 time=53.179 ms
64 bytes from 192.168.0.1: icmp_seq=11 ttl=64 time=52.623 ms
64 bytes from 192.168.0.1: icmp_seq=12 ttl=64 time=53.383 ms
64 bytes from 192.168.0.1: icmp_seq=13 ttl=64 time=52.906 ms
64 bytes from 192.168.0.1: icmp_seq=14 ttl=64 time=40.600 ms

So tcp was getting push back = unrepresentative result if you
want to test as if a queue on a remote router.

Re: [Cake] cake default target is too low for bbr?

[Andy Furniss]

Andy Furniss wrote:
> Andy Furniss wrote:
> 
>>> b) it reacts to increase in RTT. An experiment with 10 Mbps
>>> bottleneck, 40 ms RTT and a typical 1000 packet buffer, increase in
>>> RTT with BBR is ~3 ms while with cubic it is over 1000 ms.
>>
>> That is a nice aspect (though at 60mbit hfsc + 80ms bfifo I tested with
>> 5 tcps it was IIRC 20ms vs 80 for cubic). I deliberately test using ifb
>> on my PC because I want to pretend to be a router - IME (OK it was a
>> while ago) testing on eth directly gives different results - like the
>> locally generated tcp is backing off and giving different results.
> 
> I retested this with 40ms latency (netem) with hfsc + 1000 pfifo on ifb.

Forgot to put this was was 10mbit not 60.

Re: [Cake] cake default target is too low for bbr?

[Jonathan Morton]

> On 29 Apr, 2017, at 01:26, Andy Furniss <adf.lists@gmail.com> wrote:
> 
>>>> As I understand it, increase in RTT due to queueing of packets is
>>>> the main feedback mechanism for BBR. So dropping packets, which I
>>>> already consider harmful, is really harmful with BBR because
>>>> you're not telling the sender to slow down.

Actually, BBR considers mainly a measurement of “delivery rate”, and paces its sending to match that.  It does *not* primarily rely on a congestion window as most TCPs do; one is provided only as a safety net of last resort.

Measurements of RTT are mostly used for two purposes: to size the congestion window so that it doesn’t interfere with normal operation; and to estimate when “queue draining” is complete after a bandwidth probe cycle.

>>> If BBR does not slow down when packets are dropped, it's too
>>> hostile to use on a public network. The only way for a public
>>> network to respond to a flood of traffic higher than what it can
>>> handle is to drop packets (with a possible warning via ECN shortly
>>> before packets get dropped). If BBR doesn't slow down, it's just
>>> going to be wasting bandwidth.

>> No it isn't. Packet loss does not equal conguestion - it never did. Dropping packets to signal congestion is an ugly hack for implementations that are too dumb to understand any proper congestion
>> control mechanism.
> 
> Hmm, I bet a lot of carrier links are policed rather than smart queue.

Policing should theoretically produce a consistent delivery rate, which is what BBR needs to work effectively.  A wrinkle here is that most policers and shapers to date rely on a token-bucket algorithm which permits bursts at rates well above the policy, and BBR has to attempt to infer the policy rate from the medium-term behaviour.

> It also seems (OK one quick possibly flawed test), that bbr ignores ECN
> as well as drops in the sense that marked is just as high as dropped.

Yes, BBR ignores ECN, which I consider to be an unfortunate feature; it could quite reasonably be used to terminate bandwidth probes early, before they build up a significant queue (which then needs to be drained).

Cake works very well with BBR, provided it is deployed at the upstream end of the bottleneck link.  In this position, Cake happily absorbs the temporary standing queue caused by bandwidth probes, and the deficit-mode shaper means that BBR tends to see a consistent delivery rate, which it considers ideal.  In practice it matters little whether the BBR sender negotiates ECN or not, in this case.

When deployed at the downstream end of the bottleneck link, Cake works less well with BBR - but that’s true to some extent of all TCPs.  In ingress mode, at least, dropping packets effectively causes a reduction in the delivery rate, which should influence BBR more strongly to correct itself.  But if ECN is negotiated, these packet drops do not occur.  In both cases, the temporary standing queue collects in the upstream dumb queue, unless Cake is configured to a conservative enough margin below the bottleneck rate.  Cake does everything it reasonably can here, but the topology is fundamentally unfavourable.

 - Jonathan Morton

Re: [Cake] cake default target is too low for bbr?

[Andy Furniss]

Jonathan Morton wrote:
> 
>> On 29 Apr, 2017, at 01:26, Andy Furniss <adf.lists@gmail.com>
>> wrote:
>> 
>>>>> As I understand it, increase in RTT due to queueing of
>>>>> packets is the main feedback mechanism for BBR. So dropping
>>>>> packets, which I already consider harmful, is really harmful
>>>>> with BBR because you're not telling the sender to slow down.
> 
> Actually, BBR considers mainly a measurement of “delivery rate”, and
> paces its sending to match that.  It does *not* primarily rely on a
> congestion window as most TCPs do; one is provided only as a safety
> net of last resort.
> 
> Measurements of RTT are mostly used for two purposes: to size the
> congestion window so that it doesn’t interfere with normal operation;
> and to estimate when “queue draining” is complete after a bandwidth
> probe cycle.

Interesting.

> 
>>>> If BBR does not slow down when packets are dropped, it's too 
>>>> hostile to use on a public network. The only way for a public 
>>>> network to respond to a flood of traffic higher than what it
>>>> can handle is to drop packets (with a possible warning via ECN
>>>> shortly before packets get dropped). If BBR doesn't slow down,
>>>> it's just going to be wasting bandwidth.
> 
>>> No it isn't. Packet loss does not equal conguestion - it never
>>> did. Dropping packets to signal congestion is an ugly hack for
>>> implementations that are too dumb to understand any proper
>>> congestion control mechanism.
>> 
>> Hmm, I bet a lot of carrier links are policed rather than smart
>> queue.
> 
> Policing should theoretically produce a consistent delivery rate,
> which is what BBR needs to work effectively.  A wrinkle here is that
> most policers and shapers to date rely on a token-bucket algorithm
> which permits bursts at rates well above the policy, and BBR has to
> attempt to infer the policy rate from the medium-term behaviour.

Ok, but it's not really going to work (be fair) when a big link with
000s of users is policed overall. Of course this shouldn't really happen
- but contention exists at ISP level and local level for DOCSIS cable users.

>> It also seems (OK one quick possibly flawed test), that bbr ignores
>> ECN as well as drops in the sense that marked is just as high as
>> dropped.
> 
> Yes, BBR ignores ECN, which I consider to be an unfortunate feature;
> it could quite reasonably be used to terminate bandwidth probes
> early, before they build up a significant queue (which then needs to
> be drained).

Now that is unfortunate - so ECN is effectively deprecated by BBR :-(

> 
> Cake works very well with BBR, provided it is deployed at the
> upstream end of the bottleneck link.  In this position, Cake happily
> absorbs the temporary standing queue caused by bandwidth probes, and
> the deficit-mode shaper means that BBR tends to see a consistent
> delivery rate, which it considers ideal.  In practice it matters
> little whether the BBR sender negotiates ECN or not, in this case.
> 
> When deployed at the downstream end of the bottleneck link, Cake
> works less well with BBR - but that’s true to some extent of all
> TCPs.  In ingress mode, at least, dropping packets effectively causes
> a reduction in the delivery rate, which should influence BBR more
> strongly to correct itself.  But if ECN is negotiated, these packet
> drops do not occur.  In both cases, the temporary standing queue
> collects in the upstream dumb queue, unless Cake is configured to a
> conservative enough margin below the bottleneck rate.  Cake does
> everything it reasonably can here, but the topology is fundamentally
> unfavourable.

Further testing with the aim of reducing drops seems to indicate that
it's not so much target that matters but RTT.

Each output two netperf (x5) runs one marked cs1, one unmarked.

Sending through bulk with higher target and best effort with lower
target isn't much different. Using ingress param for this test, tcp
throughput is low 1.6 mbit (x5)

qdisc cake 1: dev ifb0 root refcnt 2 bandwidth 16Mbit diffserv3 
dual-srchost ingress rtt 100.0ms atm overhead 40 via-ethernet
  Sent 20770678 bytes 13819 pkt (dropped 9485, overlimits 36449 requeues 0)
  backlog 0b 0p requeues 0
  memory used: 153Kb of 4Mb
  capacity estimate: 16Mbit
                  Bulk   Best Effort      Voice
   thresh         1Mbit      16Mbit       4Mbit
   target        18.2ms       5.0ms       5.0ms
   interval     113.2ms     100.0ms      10.0ms
   pk_delay      13.0ms       9.6ms         0us
   av_delay      10.0ms       3.9ms         0us
   sp_delay         6us        13us         0us
   pkts           11654       11650           0
   bytes       17565164    17563044           0
   way_inds           0           0           0
   way_miss          10          10           0
   way_cols           0           0           0
   drops           4511        4974           0
   marks              0           0           0
   sp_flows           4           5           0
   bk_flows           2           0           0
   un_flows           0           0           0
   max_len         1514        1514           0

With RTT at 300ms throughput is 2.33 mbit and less drops for similar target.

qdisc cake 1: dev ifb0 root refcnt 2 bandwidth 16Mbit diffserv3 
dual-srchost ingress rtt 300.0ms atm overhead 40 via-ethernet
  Sent 31265716 bytes 20758 pkt (dropped 2563, overlimits 43619 requeues 0)
  backlog 0b 0p requeues 0
  memory used: 153Kb of 4Mb
  capacity estimate: 16Mbit
                  Bulk   Best Effort      Voice
   thresh         1Mbit      16Mbit       4Mbit
   target        18.2ms      15.0ms      15.0ms
   interval     303.2ms     300.0ms      30.0ms
   pk_delay      21.2ms      20.1ms         0us
   av_delay      18.9ms      17.0ms         0us
   sp_delay         4us         7us         0us
   pkts           11656       11665           0
   bytes       17564952    17579378           0
   way_inds           0           0           0
   way_miss          10          10           0
   way_cols           0           0           0
   drops           1206        1357           0
   marks              0           0           0
   sp_flows           5           5           0
   bk_flows           0           1           0
   un_flows           0           0           0
   max_len         1514        1514           0

It's even better for loss/throughput (2.44 x5) with higher rtt.

qdisc cake 1: dev ifb0 root refcnt 2 bandwidth 16Mbit diffserv3 
dual-srchost ingress rtt 400.0ms atm overhead 40 via-ethernet
  Sent 32594660 bytes 21626 pkt (dropped 1677, overlimits 44556 requeues 0)
  backlog 0b 0p requeues 0
  memory used: 153Kb of 4Mb
  capacity estimate: 16Mbit
                  Bulk   Best Effort      Voice
   thresh         1Mbit      16Mbit       4Mbit
   target        20.0ms      20.0ms      20.0ms
   interval     400.0ms     400.0ms      40.0ms
   pk_delay      21.9ms      20.3ms         0us
   av_delay      19.7ms      18.8ms         0us
   sp_delay         4us         4us         0us
   pkts           11640       11663           0
   bytes       17550552    17583086           0
   way_inds           0           0           0
   way_miss          10          10           0
   way_cols           0           0           0
   drops            836         841           0
   marks              0           0           0
   sp_flows           5           5           0
   bk_flows           0           1           0
   un_flows           0           0           0
   max_len         1514        1514           0


I haven't yet tried cooking up a test that includes a double
queue one 10% faster fifo to see how much the latency is affected.

Re: [Cake] cake default target is too low for bbr?

[Andy Furniss]

Andy Furniss wrote:
> Andy Furniss wrote:
> 
>>> b) it reacts to increase in RTT. An experiment with 10 Mbps 
>>> bottleneck, 40 ms RTT and a typical 1000 packet buffer, increase
>>> in RTT with BBR is ~3 ms while with cubic it is over 1000 ms.
>> 
>> That is a nice aspect (though at 60mbit hfsc + 80ms bfifo I tested
>> with 5 tcps it was IIRC 20ms vs 80 for cubic). I deliberately test
>> using ifb on my PC because I want to pretend to be a router - IME
>> (OK it was a while ago) testing on eth directly gives different
>> results - like the locally generated tcp is backing off and giving
>> different results.
> 
> I retested this with 40ms latency (netem) with hfsc + 1000 pfifo on
> ifb.

So, as Jonathan pointed out to me in another thread bbr needs fq and it
seems fq only wotks on root of a real eth, which means thay are invalid
tests.

I will soon (need to find a crossover cable!) be able to see using a
third sender how cake varies shaping bbr in simulated ingress.

I can test now how bbr fills buffers - some slightly strange results,
one netperf ends up being "good" = buffer only a few ms.

5 netperfs started together are not so good but nothing like cubic.

5 netperfs started with a gap of a second or two are initially terrible,
filling the buffer for about 30 seconds, then eventually falling back to
lower occupancy.

TODO - maybe this is a netperf artifact like bbr/fq thinks it is app
limited.

The worse thing about bbr + longer RTT I see so far is that its design
seems to be to deliberately bork latency by 2x rtt during initial
bandwidth probe. It does drain afterwards, but for something like dash
generating a regular spike is not very game friendly and the spec
"boasts" that unlike cubic a loss in the exponential phase is ignored,
making ingress shaping somewhat less effective.

Re: [Cake] cake default target is too low for bbr?

[Jim Gettys]

[-- Attachment #1: Type: text/plain, Size: 2541 bytes --]

On Wed, May 3, 2017 at 5:50 AM, Andy Furniss <adf.lists@gmail.com> wrote:

> Andy Furniss wrote:
>
>> Andy Furniss wrote:
>>
>> b) it reacts to increase in RTT. An experiment with 10 Mbps bottleneck,
>>>> 40 ms RTT and a typical 1000 packet buffer, increase
>>>> in RTT with BBR is ~3 ms while with cubic it is over 1000 ms.
>>>>
>>>
>>> That is a nice aspect (though at 60mbit hfsc + 80ms bfifo I tested
>>> with 5 tcps it was IIRC 20ms vs 80 for cubic). I deliberately test
>>> using ifb on my PC because I want to pretend to be a router - IME
>>> (OK it was a while ago) testing on eth directly gives different
>>> results - like the locally generated tcp is backing off and giving
>>> different results.
>>>
>>
>> I retested this with 40ms latency (netem) with hfsc + 1000 pfifo on
>> ifb.
>>
>
> So, as Jonathan pointed out to me in another thread bbr needs fq and it
> seems fq only wotks on root of a real eth, which means thay are invalid
> tests.
>

​Specifically, BBR needs packet pacing to work properly: the algorithm
depends on the packets being properly paced.

Today, fq is the only qdisc supporting pacing.

The right answer would be to add packet pacing to cake/fq_codel directly.
Until that is done, we don't know how BBR will work in our world.
                                              - Jim​

>
> I will soon (need to find a crossover cable!) be able to see using a
> third sender how cake varies shaping bbr in simulated ingress.
>
> I can test now how bbr fills buffers - some slightly strange results,
> one netperf ends up being "good" = buffer only a few ms.
>
> 5 netperfs started together are not so good but nothing like cubic.
>
> 5 netperfs started with a gap of a second or two are initially terrible,
> filling the buffer for about 30 seconds, then eventually falling back to
> lower occupancy.
>
> TODO - maybe this is a netperf artifact like bbr/fq thinks it is app
> limited.
>
> The worse thing about bbr + longer RTT I see so far is that its design
> seems to be to deliberately bork latency by 2x rtt during initial
> bandwidth probe. It does drain afterwards, but for something like dash
> generating a regular spike is not very game friendly and the spec
> "boasts" that unlike cubic a loss in the exponential phase is ignored,
> making ingress shaping somewhat less effective.
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>

[-- Attachment #2: Type: text/html, Size: 4122 bytes --]

Re: [Cake] cake default target is too low for bbr?

[Andy Furniss]

Jim Gettys wrote:
> On Wed, May 3, 2017 at 5:50 AM, Andy Furniss <adf.lists@gmail.com>
> wrote:
> 
>> Andy Furniss wrote:
>> 
>>> Andy Furniss wrote:
>>> 
>>> b) it reacts to increase in RTT. An experiment with 10 Mbps
>>> bottleneck,
>>>>> 40 ms RTT and a typical 1000 packet buffer, increase in RTT
>>>>> with BBR is ~3 ms while with cubic it is over 1000 ms.
>>>>> 
>>>> 
>>>> That is a nice aspect (though at 60mbit hfsc + 80ms bfifo I
>>>> tested with 5 tcps it was IIRC 20ms vs 80 for cubic). I
>>>> deliberately test using ifb on my PC because I want to pretend
>>>> to be a router - IME (OK it was a while ago) testing on eth
>>>> directly gives different results - like the locally generated
>>>> tcp is backing off and giving different results.
>>>> 
>>> 
>>> I retested this with 40ms latency (netem) with hfsc + 1000 pfifo
>>> on ifb.
>>> 
>> 
>> So, as Jonathan pointed out to me in another thread bbr needs fq
>> and it seems fq only wotks on root of a real eth, which means thay
>> are invalid tests.
>> 
> 
> ​Specifically, BBR needs packet pacing to work properly: the
> algorithm depends on the packets being properly paced.
> 
> Today, fq is the only qdisc supporting pacing.
> 
> The right answer would be to add packet pacing to cake/fq_codel
> directly. Until that is done, we don't know how BBR will work in our
> world. - Jim​

I guess you mean so cake could be used on egress of sender (in place of fq)?

That's not really the test that I intend to do, which is more like -

[boxA bbr+fq] -> [boxB simulate ISP buffer] -> [boxC cake ingress shape]
a bit lower than "line" rate and see how much "ISP" buffer gets filled.

Also compare bbr, cubic and netem different rtts etc.

>> 
>> I will soon (need to find a crossover cable!) be able to see using
>> a third sender how cake varies shaping bbr in simulated ingress.
>> 
>> I can test now how bbr fills buffers - some slightly strange
>> results, one netperf ends up being "good" = buffer only a few ms.
>> 
>> 5 netperfs started together are not so good but nothing like
>> cubic.
>> 
>> 5 netperfs started with a gap of a second or two are initially
>> terrible, filling the buffer for about 30 seconds, then eventually
>> falling back to lower occupancy.
>> 
>> TODO - maybe this is a netperf artifact like bbr/fq thinks it is
>> app limited.
>> 
>> The worse thing about bbr + longer RTT I see so far is that its
>> design seems to be to deliberately bork latency by 2x rtt during
>> initial bandwidth probe. It does drain afterwards, but for
>> something like dash generating a regular spike is not very game
>> friendly and the spec "boasts" that unlike cubic a loss in the
>> exponential phase is ignored, making ingress shaping somewhat less
>> effective.

Re: [Cake] cake default target is too low for bbr?

[Jim Gettys]

[-- Attachment #1: Type: text/plain, Size: 3280 bytes --]

On Thu, May 4, 2017 at 6:22 AM, Andy Furniss <adf.lists@gmail.com> wrote:

> Jim Gettys wrote:
>
>> On Wed, May 3, 2017 at 5:50 AM, Andy Furniss <adf.lists@gmail.com>
>> wrote:
>>
>> Andy Furniss wrote:
>>>
>>> Andy Furniss wrote:
>>>>
>>>> b) it reacts to increase in RTT. An experiment with 10 Mbps
>>>> bottleneck,
>>>>
>>>>> 40 ms RTT and a typical 1000 packet buffer, increase in RTT
>>>>>> with BBR is ~3 ms while with cubic it is over 1000 ms.
>>>>>>
>>>>>>
>>>>> That is a nice aspect (though at 60mbit hfsc + 80ms bfifo I
>>>>> tested with 5 tcps it was IIRC 20ms vs 80 for cubic). I
>>>>> deliberately test using ifb on my PC because I want to pretend
>>>>> to be a router - IME (OK it was a while ago) testing on eth
>>>>> directly gives different results - like the locally generated
>>>>> tcp is backing off and giving different results.
>>>>>
>>>>>
>>>> I retested this with 40ms latency (netem) with hfsc + 1000 pfifo
>>>> on ifb.
>>>>
>>>>
>>> So, as Jonathan pointed out to me in another thread bbr needs fq
>>> and it seems fq only wotks on root of a real eth, which means thay
>>> are invalid tests.
>>>
>>>
>> ​Specifically, BBR needs packet pacing to work properly: the
>> algorithm depends on the packets being properly paced.
>>
>> Today, fq is the only qdisc supporting pacing.
>>
>> The right answer would be to add packet pacing to cake/fq_codel
>> directly. Until that is done, we don't know how BBR will work in our
>> world. - Jim​
>>
>
> I guess you mean so cake could be used on egress of sender (in place of
> fq)?
>

​Yes.
​

>
> That's not really the test that I intend to do, which is more like -
>
> [boxA bbr+fq] -> [boxB simulate ISP buffer] -> [boxC cake ingress shape]
> a bit lower than "line" rate and see how much "ISP" buffer gets filled.
>
> Also compare bbr, cubic and netem different rtts etc.


​Ok.  The usual warnings about netem being dangerous apply, though netem
can be useful if run on a separate machine.  Netem is an attractive
nuisance, but has caused lots of results to be ultimately useless....  Be
careful.
                              - Jim
​

>
>
>
>>> I will soon (need to find a crossover cable!) be able to see using
>>> a third sender how cake varies shaping bbr in simulated ingress.
>>>
>>> I can test now how bbr fills buffers - some slightly strange
>>> results, one netperf ends up being "good" = buffer only a few ms.
>>>
>>> 5 netperfs started together are not so good but nothing like
>>> cubic.
>>>
>>> 5 netperfs started with a gap of a second or two are initially
>>> terrible, filling the buffer for about 30 seconds, then eventually
>>> falling back to lower occupancy.
>>>
>>> TODO - maybe this is a netperf artifact like bbr/fq thinks it is
>>> app limited.
>>>
>>> The worse thing about bbr + longer RTT I see so far is that its
>>> design seems to be to deliberately bork latency by 2x rtt during
>>> initial bandwidth probe. It does drain afterwards, but for
>>> something like dash generating a regular spike is not very game
>>> friendly and the spec "boasts" that unlike cubic a loss in the
>>> exponential phase is ignored, making ingress shaping somewhat less
>>> effective.
>>>
>>

[-- Attachment #2: Type: text/html, Size: 5210 bytes --]

Re: [Cake] cake default target is too low for bbr?

[Andy Furniss]

Jim Gettys wrote:
> On Thu, May 4, 2017 at 6:22 AM, Andy Furniss <adf.lists@gmail.com> wrote:
> 
>> Jim Gettys wrote:
>>
>>> On Wed, May 3, 2017 at 5:50 AM, Andy Furniss <adf.lists@gmail.com>
>>> wrote:
>>>
>>> Andy Furniss wrote:
>>>>
>>>> Andy Furniss wrote:
>>>>>
>>>>> b) it reacts to increase in RTT. An experiment with 10 Mbps
>>>>> bottleneck,
>>>>>
>>>>>> 40 ms RTT and a typical 1000 packet buffer, increase in RTT
>>>>>>> with BBR is ~3 ms while with cubic it is over 1000 ms.
>>>>>>>
>>>>>>>
>>>>>> That is a nice aspect (though at 60mbit hfsc + 80ms bfifo I
>>>>>> tested with 5 tcps it was IIRC 20ms vs 80 for cubic). I
>>>>>> deliberately test using ifb on my PC because I want to pretend
>>>>>> to be a router - IME (OK it was a while ago) testing on eth
>>>>>> directly gives different results - like the locally generated
>>>>>> tcp is backing off and giving different results.
>>>>>>
>>>>>>
>>>>> I retested this with 40ms latency (netem) with hfsc + 1000 pfifo
>>>>> on ifb.
>>>>>
>>>>>
>>>> So, as Jonathan pointed out to me in another thread bbr needs fq
>>>> and it seems fq only wotks on root of a real eth, which means thay
>>>> are invalid tests.
>>>>
>>>>
>>> ​Specifically, BBR needs packet pacing to work properly: the
>>> algorithm depends on the packets being properly paced.
>>>
>>> Today, fq is the only qdisc supporting pacing.
>>>
>>> The right answer would be to add packet pacing to cake/fq_codel
>>> directly. Until that is done, we don't know how BBR will work in our
>>> world. - Jim​
>>>
>>
>> I guess you mean so cake could be used on egress of sender (in place of
>> fq)?
>>
> 
> ​Yes.
> ​
> 
>>
>> That's not really the test that I intend to do, which is more like -
>>
>> [boxA bbr+fq] -> [boxB simulate ISP buffer] -> [boxC cake ingress shape]
>> a bit lower than "line" rate and see how much "ISP" buffer gets filled.
>>
>> Also compare bbr, cubic and netem different rtts etc.
> 
> 
> ​Ok.  The usual warnings about netem being dangerous apply, though netem
> can be useful if run on a separate machine.  Netem is an attractive
> nuisance, but has caused lots of results to be ultimately useless....  Be
> careful.
>                                - Jim

Yea, I saw the warning about netem on the website - tricky as I would
need 4 boxes to really isolate it, and I've only got three, so it's not
ideal.

I tested with it delaying acks on ingress of sender which had fq on 
egress. Also did some tcpdumps with different setups to see if it was
clumping acks - it seemed smooth.

With this setup my results are much the same as before = bbr is harder
to shape on ingress vs cubic, the longer the rtt to sender the worse it
is.

I was testing 18mbit dsl sim with mainly 16mbit ingress. Repeatedly
grabbing 1 or 2 meg files (like dash) spikes latency every time.
With rtt of 40ms unshaped it will burst to 80+ms, 16mbit behind 18mbit
spikes 50ms. IIRC to get spikes below 20ms I needed 12mbit = too low.

For continuous downloads, after initial spike a single tcp wasn't too 
bad, even 5 can be controlled latency wise at 16mbit.

The problem with 5 vs 1 is the number of drops, 1 not too bad, but 5
will drop 1/10 so tcp ends up sacking almost per packet, which is not
good for those with limited upstream.

ECN marks more packets than would be dropped, almost all in the 5 case
so also causes many upstream acks.

This is with cakes ingress parameter and default rtt of 100ms.

Changing to 300ms does reduce the drops/marks as in my previous post,
but it makes controlling the startup spikes slightly less effective -
though without going really low they weren't controlled very well anyway.

Re: [Cake] cake default target is too low for bbr?

[Andy Furniss]

Andy Furniss wrote:

> OK so ecn may cure - but people may no know how or want that on.

So ecn doesn't really help with upstream bandwidth issues as it still
does 1 ack per packet when marked - though they are at least not any
longer like sacks.