Re: [Cake] total download rate with many flows

Re: [Cake] total download rate with many flows

[Pete Heist]

> From: Jonathan Morton <chromatix99@gmail.com>
> To: Georgios Amanakis <gamanakis@gmail.com>,
> 	cake@lists.bufferbloat.net
> Subject: Re: [Cake] total download rate with many flows
> Message-ID: <464c2c38-8ecc-b232-19b3-bf3eb014665e@gmail.com>
> Content-Type: text/plain; charset=utf-8; format=flowed
> 
> On 16/11/17 19:06, Georgios Amanakis wrote:
>> I am skimming through the code but cannot see where the failsafe 
>> shaper's rate of advance is set at one-quarter rate. Could you point 
>> this out?
> 
> Oddly enough, I can't find it either.  It could be that the code wasn't 
> pushed yet - or even that it only exists on my MBP hard drive, which I 
> still don't have a working MBP to put in again.  Due to HFS+ 
> compression, I can't actually read the file by attaching it to a Linux 
> machine.

I have a vintage 2007 MBP running El Capitan that isn’t doing much. If it’s a matter of getting to data or some other critical stuff, would this help?

Its battery is shot because I stupidly neglected to charge it for too long (for the second time, which multiplies the dumbness), but the machine otherwise works.

If it’s a matter of getting to data, would this help? If so, we can see if there’s a way to arrange getting it to you from Czech and back. I’d want to be careful not to pay VAT to re-import my own possession. The postal authorities here can be sticklers. Just let me know if this helps somehow...

Pete

Re: [Cake] total download rate with many flows

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 363 bytes --]

It's a kind offer, and since I'm in Finland (also EU), there shouldn't be
any problem with VAT.

However, I just managed to find a copy of the failsafe code on one of my
other machines, and merged it to cobalt branch.  I honestly thought it had
been pushed already, and that the only reason for it being otherwise was
that it must be stranded.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 435 bytes --]

Re: [Cake] total download rate with many flows

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 523 bytes --]


> On Nov 20, 2017, at 9:06 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
> It's a kind offer, and since I'm in Finland (also EU), there shouldn't be any problem with VAT.
> 
> However, I just managed to find a copy of the failsafe code on one of my other machines, and merged it to cobalt branch.  I honestly thought it had been pushed already, and that the only reason for it being otherwise was that it must be stranded.
> 

Great news, well, the offer stands if there’s something to deal with later…


[-- Attachment #2: Type: text/html, Size: 983 bytes --]

Re: [Cake] total download rate with many flows

[Jonathan Morton]

On 16/11/17 19:06, Georgios Amanakis wrote:
> I am skimming through the code but cannot see where the failsafe 
> shaper's rate of advance is set at one-quarter rate. Could you point 
> this out?

Oddly enough, I can't find it either.  It could be that the code wasn't 
pushed yet - or even that it only exists on my MBP hard drive, which I 
still don't have a working MBP to put in again.  Due to HFS+ 
compression, I can't actually read the file by attaching it to a Linux 
machine.

-- 
  - Jonathan Morton

Re: [Cake] total download rate with many flows

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 1234 bytes --]

I'm curious as to why you think Cobalt is more aggressive than Codel.  It
does use more accurate approximations to the mathematical ideal than the
"reference" codel does.

It is however very odd that the Diffserv mode has any effect on this at
all.  It could be explained if a lot of the traffic is marked CS1, since
the Bulk tin has looser AQM parameters.  That suggests that selecting
'satellite' might help similarly.

Something worth trying would be to alter the failsafe shaper's rate of
advance.  Currently it has a one-quarter rate, which might be too
restrictive.  Tests at one-half and three-quarters might therefore be
interesting.  Otherwise, I don't think trying to modify the way ingress
mode works will do the right things.

Ultimately, this arises because Cake is having to drop packets in order to
signal congestion, and when there's a lot of flows, a lot of signals must
be sent to get through to them all.  With ECN working, it doesn't need to
waste bandwidth merely to signal.

The only other reasonable approach is to somehow reduce the signalling rate
under heavy flow load.  That requires informing Cobalt of the number of
bulk flows, and using that to scale the signalling frequency somehow.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 1390 bytes --]

Re: [Cake] total download rate with many flows

[Dave Taht]

On Tue, Nov 14, 2017 at 8:04 PM, George Amanakis <g_amanakis@yahoo.com> wrote:
> The problem I am describing in this thread appears only when using
> besteffort. If diffserv3 is used (without any further prioritization by
> manipulating DSCP bits) downstream is more stable and higher in throughput
> than with besteffort.
>
> Also, in my case disabling BLUE (by commenting out cake_queue_{empty,full}
> and all p_drop parts in sch_cake.c and cobalt.c) does not make any
> difference.

The only other thing I can think of is replacing the codel algorithm
in cake with the one in fq_codel.


>
>
> On 11/14/2017 5:23 PM, Dave Taht wrote:
>>
>> because cobalt is more aggressive than codel. I've never believed in it.
>>
>> I'm sitting here trying to figure out how to get good ole regular cake
>> to compile again.
>>
>> I just merged the new ack stuff into the codel branch.
>>
>>
>> On Tue, Nov 14, 2017 at 2:13 PM, G. Amanakis via Cake
>> <cake@lists.bufferbloat.net> wrote:
>>>
>>> _______________________________________________
>>> Cake mailing list
>>> Cake@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cake
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: "G. Amanakis" <g_amanakis@yahoo.com>
>>> To: Dave Taht <dave@taht.net>, George Amanakis via Cake
>>> <cake@lists.bufferbloat.net>
>>> Cc:
>>> Bcc:
>>> Date: Tue, 14 Nov 2017 17:13:16 -0500
>>> Subject: Re: [Cake] total download rate with many flows
>>> Yes, it is. I am building from the cobalt branch. What makes you believe
>>> otherwise? Would you expect a different behaviour?
>>>
>>> On November 14, 2017 3:11:04 PM EST, Dave Taht <dave@taht.net> wrote:
>>>>
>>>> George Amanakis via Cake <cake@lists.bufferbloat.net> writes:
>>>>
>>>>>   From: George Amanakis <g_amanakis@yahoo.com>
>>>>>   Subject: Re: [Cake] total download rate with many flows
>>>>>   To: David Lang <david@lang.hm>
>>>>>   Cc: cake@lists.bufferbloat.net
>>>>>   Date: Mon, 13 Nov 2017 21:49:53 -0500 (17 hours, 20 minutes, 42
>>>>> seconds ago)
>>>>>
>>>>>   Dear David,
>>>>>
>>>>>   I agree. My point is that currently ingress mode seems to be dropping
>>>>> more
>>>>>   packets than necessary to keep senders from bottlenecking the
>>>>> connection (when
>>>>>   there is a large number of concurrent flows, >8). And right now,
>>>>> ingress mode is
>>>>>   the only mode that achieves this in situations such as Windows
>>>>> updates.
>>>>
>>>>
>>>> Is cobalt enabled in your build?
>>>
>>>
>>> --
>>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>>
>>
>>
>



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] total download rate with many flows

[George Amanakis]

The problem I am describing in this thread appears only when using 
besteffort. If diffserv3 is used (without any further prioritization by 
manipulating DSCP bits) downstream is more stable and higher in 
throughput than with besteffort.

Also, in my case disabling BLUE (by commenting out 
cake_queue_{empty,full} and all p_drop parts in sch_cake.c and cobalt.c) 
does not make any difference.


On 11/14/2017 5:23 PM, Dave Taht wrote:
> because cobalt is more aggressive than codel. I've never believed in it.
>
> I'm sitting here trying to figure out how to get good ole regular cake
> to compile again.
>
> I just merged the new ack stuff into the codel branch.
>
>
> On Tue, Nov 14, 2017 at 2:13 PM, G. Amanakis via Cake
> <cake@lists.bufferbloat.net> wrote:
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
>>
>>
>> ---------- Forwarded message ----------
>> From: "G. Amanakis" <g_amanakis@yahoo.com>
>> To: Dave Taht <dave@taht.net>, George Amanakis via Cake <cake@lists.bufferbloat.net>
>> Cc:
>> Bcc:
>> Date: Tue, 14 Nov 2017 17:13:16 -0500
>> Subject: Re: [Cake] total download rate with many flows
>> Yes, it is. I am building from the cobalt branch. What makes you believe otherwise? Would you expect a different behaviour?
>>
>> On November 14, 2017 3:11:04 PM EST, Dave Taht <dave@taht.net> wrote:
>>> George Amanakis via Cake <cake@lists.bufferbloat.net> writes:
>>>
>>>>   From: George Amanakis <g_amanakis@yahoo.com>
>>>>   Subject: Re: [Cake] total download rate with many flows
>>>>   To: David Lang <david@lang.hm>
>>>>   Cc: cake@lists.bufferbloat.net
>>>>   Date: Mon, 13 Nov 2017 21:49:53 -0500 (17 hours, 20 minutes, 42 seconds ago)
>>>>
>>>>   Dear David,
>>>>
>>>>   I agree. My point is that currently ingress mode seems to be dropping more
>>>>   packets than necessary to keep senders from bottlenecking the connection (when
>>>>   there is a large number of concurrent flows, >8). And right now, ingress mode is
>>>>   the only mode that achieves this in situations such as Windows updates.
>>>
>>> Is cobalt enabled in your build?
>>
>> --
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>
>
>

Re: [Cake] total download rate with many flows

[Dave Taht]

because cobalt is more aggressive than codel. I've never believed in it.

I'm sitting here trying to figure out how to get good ole regular cake
to compile again.

I just merged the new ack stuff into the codel branch.


On Tue, Nov 14, 2017 at 2:13 PM, G. Amanakis via Cake
<cake@lists.bufferbloat.net> wrote:
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>
>
> ---------- Forwarded message ----------
> From: "G. Amanakis" <g_amanakis@yahoo.com>
> To: Dave Taht <dave@taht.net>, George Amanakis via Cake <cake@lists.bufferbloat.net>
> Cc:
> Bcc:
> Date: Tue, 14 Nov 2017 17:13:16 -0500
> Subject: Re: [Cake] total download rate with many flows
> Yes, it is. I am building from the cobalt branch. What makes you believe otherwise? Would you expect a different behaviour?
>
> On November 14, 2017 3:11:04 PM EST, Dave Taht <dave@taht.net> wrote:
>>
>> George Amanakis via Cake <cake@lists.bufferbloat.net> writes:
>>
>>>  From: George Amanakis <g_amanakis@yahoo.com>
>>>  Subject: Re: [Cake] total download rate with many flows
>>>  To: David Lang <david@lang.hm>
>>>  Cc: cake@lists.bufferbloat.net
>>>  Date: Mon, 13 Nov 2017 21:49:53 -0500 (17 hours, 20 minutes, 42 seconds ago)
>>>
>>>  Dear David,
>>>
>>>  I agree. My point is that currently ingress mode seems to be dropping more
>>>  packets than necessary to keep senders from bottlenecking the connection (when
>>>  there is a large number of concurrent flows, >8). And right now, ingress mode is
>>>  the only mode that achieves this in situations such as Windows updates.
>>
>>
>> Is cobalt enabled in your build?
>
>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] total download rate with many flows

[G. Amanakis]

[-- Attachment #1: Type: text/plain, Size: 995 bytes --]

Yes, it is. I am building from the cobalt branch. What makes you believe otherwise? Would you expect a different behaviour?

On November 14, 2017 3:11:04 PM EST, Dave Taht <dave@taht.net> wrote:
>George Amanakis via Cake <cake@lists.bufferbloat.net> writes:
>
>> From: George Amanakis <g_amanakis@yahoo.com>
>> Subject: Re: [Cake] total download rate with many flows
>> To: David Lang <david@lang.hm>
>> Cc: cake@lists.bufferbloat.net
>> Date: Mon, 13 Nov 2017 21:49:53 -0500 (17 hours, 20 minutes, 42
>seconds ago)
>>
>> Dear David,
>>
>> I agree. My point is that currently ingress mode seems to be dropping
>more
>> packets than necessary to keep senders from bottlenecking the
>connection (when
>> there is a large number of concurrent flows, >8). And right now,
>ingress mode is
>> the only mode that achieves this in situations such as Windows
>updates.
>
>Is cobalt enabled in your build?

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 1429 bytes --]

Re: [Cake] total download rate with many flows

[Dave Taht]

George Amanakis via Cake <cake@lists.bufferbloat.net> writes:

> From: George Amanakis <g_amanakis@yahoo.com>
> Subject: Re: [Cake] total download rate with many flows
> To: David Lang <david@lang.hm>
> Cc: cake@lists.bufferbloat.net
> Date: Mon, 13 Nov 2017 21:49:53 -0500 (17 hours, 20 minutes, 42 seconds ago)
>
> Dear David,
>
> I agree. My point is that currently ingress mode seems to be dropping more
> packets than necessary to keep senders from bottlenecking the connection (when
> there is a large number of concurrent flows, >8). And right now, ingress mode is
> the only mode that achieves this in situations such as Windows updates.

Is cobalt enabled in your build?

Re: [Cake] total download rate with many flows

[George Amanakis]

Dear David,

I agree. My point is that currently ingress mode seems to be dropping 
more packets than necessary to keep senders from bottlenecking the 
connection (when there is a large number of concurrent flows, >8). And 
right now, ingress mode is the only mode that achieves this in 
situations such as Windows updates.

George


On 11/13/2017 9:08 PM, David Lang wrote:
> Ingres and Egress are fundamentally different due to the fact that in 
> ingress mode you are having to throw away data that has successfully 
> traversed the bottleneck (deliberatly wasting your limited resource 
> now to avoid having senders bottleneck the queue on the far side of 
> the link)
>
> in Egress mode, you aren't doing that, and so you can get much closer 
> to the actual bandwidth.
>
> In addition, in Ingress mode, you are always working via second-order 
> effects, you can't slow a transmission directly like you can in egress 
> mode, all you can do is drop packets and wait until the sender 
> notices, retransmits and slows down. Egress has full control of the 
> queues and can send things in any order, and may be able to 
> continually fill the pipe and avoid any timeouts and retransmissions 
> (if the flow is short enough)
>
> David Lang
>

Re: [Cake] total download rate with many flows

[David Lang]

Ingres and Egress are fundamentally different due to the fact that in ingress 
mode you are having to throw away data that has successfully traversed the 
bottleneck (deliberatly wasting your limited resource now to avoid having 
senders bottleneck the queue on the far side of the link)

in Egress mode, you aren't doing that, and so you can get much closer to the 
actual bandwidth.

In addition, in Ingress mode, you are always working via second-order effects, 
you can't slow a transmission directly like you can in egress mode, all you can 
do is drop packets and wait until the sender notices, retransmits and slows 
down. Egress has full control of the queues and can send things in any order, 
and may be able to continually fill the pipe and avoid any timeouts and 
retransmissions (if the flow is short enough)

David Lang

Re: [Cake] total download rate with many flows

[George Amanakis]

I meant proportionally to (1-1/sqrt(x)).


On 11/13/2017 8:51 PM, George Amanakis wrote:
> I am exploring this idea further. If q->time_next_packet is 
> incremented for dropped packets proportionally to (1-1/x), where x is 
> the count of all flows in the tin that is being served, ingress mode 
> works much more smoothly: latency is still <50ms and throughput is 
> very near to the set limit.
>
> I *tried* to make a patch from latest cobalt.
>
> =============8<=============
> diff --git a/sch_cake.c b/sch_cake.c
> index 82f264f..752783a 100644
> --- a/sch_cake.c
> +++ b/sch_cake.c
> @@ -145,6 +145,7 @@ struct cake_flow {
>         struct list_head  flowchain;
>         s32               deficit;
>         struct cobalt_vars cvars;
> +       struct cobalt_vars cvars2;
>         u16               srchost; /* index into cake_host table */
>         u16               dsthost;
>         u8                set;
> @@ -254,6 +255,7 @@ struct cake_sched_data {
>         u32             avg_window_bytes;
>         u32             avg_peak_bandwidth;
>         u64             last_reconfig_time;
> +       u32             drop_len;
>  };
>
>  enum {
> @@ -820,7 +822,7 @@ static unsigned int cake_drop(struct Qdisc *sch, 
> struct sk_buff **to_free)
>         sch->qstats.drops++;
>
>         if(q->rate_flags & CAKE_FLAG_INGRESS)
> -               cake_advance_shaper(q, b, cake_overhead(q, len), now);
> +               q->drop_len += len;
>
>  #if LINUX_VERSION_CODE < KERNEL_VERSION(4, 8, 0)
>         kfree_skb(skb);
> @@ -1274,7 +1276,9 @@ retry:
>                 /* drop this packet, get another one */
>                 if(q->rate_flags & CAKE_FLAG_INGRESS) {
>                         len = cake_overhead(q, qdisc_pkt_len(skb));
> -                       cake_advance_shaper(q, b, len, now);
> +                       flow->cvars2.count = 
> b->bulk_flow_count+b->sparse_flow_count+b->decaying_flow_count+b->unresponsive_flow_count;
> +                       cobalt_invsqrt(&(flow->cvars2));
> +                       q->drop_len += (len - reciprocal_scale(len, 
> flow->cvars2.rec_inv_sqrt));
>                         flow->deficit -= len;
>                         b->tin_deficit -= len;
>                 }
> @@ -1286,8 +1290,6 @@ retry:
>                 qdisc_qstats_drop(sch);
>                 kfree_skb(skb);
>  #endif
> -               if(q->rate_flags & CAKE_FLAG_INGRESS)
> -                       goto retry;
>         }
>
>         b->tin_ecn_mark += !!flow->cvars.ecn_marked;
> @@ -1340,7 +1342,7 @@ static void cake_advance_shaper(struct 
> cake_sched_data *q, struct cake_tin_data
>         if(q->rate_ns) {
>                 s64 tdiff1 = b->tin_time_next_packet - now;
>                 s64 tdiff2 = (len * (u64)b->tin_rate_ns) >> 
> b->tin_rate_shft;
> -               s64 tdiff3 = (len * (u64)q->rate_ns) >> q->rate_shft;
> +               s64 tdiff3 = ((q->drop_len + len) * (u64)q->rate_ns) 
> >> q->rate_shft;
>
>                 if(tdiff1 < 0)
>                         b->tin_time_next_packet += tdiff2;
> @@ -1348,6 +1350,7 @@ static void cake_advance_shaper(struct 
> cake_sched_data *q, struct cake_tin_data
>                         b->tin_time_next_packet = now + tdiff2;
>
>                 q->time_next_packet += tdiff3;
> +               q->drop_len = 0;
>         }
>  }
>
> @@ -1711,6 +1714,7 @@ static void cake_reconfigure(struct Qdisc *sch)
>  {
>         struct cake_sched_data *q = qdisc_priv(sch);
>         int c, ft;
> +       q->drop_len=0;
>
>         switch (q->tin_mode) {
>         case CAKE_MODE_BESTEFFORT:
> @@ -1941,6 +1945,7 @@ static int cake_init(struct Qdisc *sch, struct 
> nlattr *opt)
>
>                         INIT_LIST_HEAD(&flow->flowchain);
>                         cobalt_vars_init(&flow->cvars);
> +                       cobalt_vars_init(&flow->cvars2);
>
>                         q->overflow_heap[k].t = i;
>                         q->overflow_heap[k].b = j;
>
> =============8<=============
>
>
>
> On 11/11/2017 10:48 PM, George Amanakis wrote:
>> I totally understand what you are saying. However, I believe cake's 
>> egress and ingress modes currently behave as two extremes. One could 
>> argue that neither of them is the golden mean. With a patch in 
>> ingress mode (see below) and a single host using 32 flows to download 
>> I managed to increase throughput from ~7Mbps to ~10Mbps (configured 
>> limit 12200kbps) while latency increased from ~10ms to ~50ms, which 
>> would still be acceptable. As a comparison egress mode in the same 
>> setup gives me throughput ~11.5Mbps and latency ~500ms.
>>
>> I would like to hear your thoughts about this idea: the patch is 
>> incrementing q->time_next_packet for dropped packets differently than 
>> for passed-through ones. Please focus on the idea, not the actual 
>> implementation :) (also pasted in https://pastebin.com/SZ14WiYw)
>>
>> =============8<=============
>>
>> diff --git a/sch_cake.c b/sch_cake.c
>> index 82f264f..a3a4a88 100644
>> --- a/sch_cake.c
>> +++ b/sch_cake.c
>> @@ -769,6 +769,7 @@ static void cake_heapify_up(struct 
>> cake_sched_data *q, u16 i)
>>  }
>>
>>  static void cake_advance_shaper(struct cake_sched_data *q, struct 
>> cake_tin_data *b, u32 len, u64 now);
>> +static void cake_advance_shaper2(struct cake_sched_data *q, struct 
>> cake_tin_data *b, u32 len, u64 now);
>>
>>  #if LINUX_VERSION_CODE < KERNEL_VERSION(4, 8, 0)
>>  static unsigned int cake_drop(struct Qdisc *sch)
>> @@ -1274,7 +1275,7 @@ retry:
>>                 /* drop this packet, get another one */
>>                 if(q->rate_flags & CAKE_FLAG_INGRESS) {
>>                         len = cake_overhead(q, qdisc_pkt_len(skb));
>> -                       cake_advance_shaper(q, b, len, now);
>> +                       cake_advance_shaper2(q, b, len, now);
>>                         flow->deficit -= len;
>>                         b->tin_deficit -= len;
>>                 }
>> @@ -1286,8 +1287,6 @@ retry:
>>                 qdisc_qstats_drop(sch);
>>                 kfree_skb(skb);
>>  #endif
>> -               if(q->rate_flags & CAKE_FLAG_INGRESS)
>> -                       goto retry;
>>         }
>>
>>         b->tin_ecn_mark += !!flow->cvars.ecn_marked;
>> @@ -1351,6 +1350,24 @@ static void cake_advance_shaper(struct 
>> cake_sched_data *q, struct cake_tin_data
>>         }
>>  }
>>
>> +static void cake_advance_shaper2(struct cake_sched_data *q, struct 
>> cake_tin_data *b, u32 len, u64 now)
>> +{
>> +       /* charge packet bandwidth to this tin, lower tins,
>> +        * and to the global shaper.
>> +        */
>> +       if(q->rate_ns) {
>> +               s64 tdiff1 = b->tin_time_next_packet - now;
>> +               s64 tdiff2 = (len * (u64)b->tin_rate_ns) >> 
>> b->tin_rate_shft;
>> +               s64 tdiff3 = (len * (u64)q->rate_ns) >> q->rate_shft;
>> +
>> +               if(tdiff1 < 0)
>> +                       b->tin_time_next_packet += tdiff2;
>> +               else if(tdiff1 < tdiff2)
>> +                       b->tin_time_next_packet = now + tdiff2;
>> +
>> +               q->time_next_packet += (tdiff3*27)>>5;
>> +       }
>> +}
>>  static void cake_reset(struct Qdisc *sch)
>>  {
>>         u32 c;
>>
>> =============8<=============
>>
>> On 11/10/2017 4:50 PM, Jonathan Morton wrote:
>>>
>>> In fact, that's why I put a failsafe into ingress mode, so that it 
>>> would never stall completely.  It can happen, however, that 
>>> throughput is significantly reduced when the drop rate is high.
>>>
>>> If throughput is more important to you than induced latency, switch 
>>> to egress mode.
>>>
>>> Unfortunately it's not possible to guarantee both low latency and 
>>> high throughput when operating downstream of the bottleneck link.  
>>> ECN gives you better results, though.
>>>
>>> - Jonathan Morton
>>>
>>
>

Re: [Cake] total download rate with many flows

[George Amanakis]

I am exploring this idea further. If q->time_next_packet is incremented 
for dropped packets proportionally to (1-1/x), where x is the count of 
all flows in the tin that is being served, ingress mode works much more 
smoothly: latency is still <50ms and throughput is very near to the set 
limit.

I *tried* to make a patch from latest cobalt.

=============8<=============
diff --git a/sch_cake.c b/sch_cake.c
index 82f264f..752783a 100644
--- a/sch_cake.c
+++ b/sch_cake.c
@@ -145,6 +145,7 @@ struct cake_flow {
         struct list_head  flowchain;
         s32               deficit;
         struct cobalt_vars cvars;
+       struct cobalt_vars cvars2;
         u16               srchost; /* index into cake_host table */
         u16               dsthost;
         u8                set;
@@ -254,6 +255,7 @@ struct cake_sched_data {
         u32             avg_window_bytes;
         u32             avg_peak_bandwidth;
         u64             last_reconfig_time;
+       u32             drop_len;
  };

  enum {
@@ -820,7 +822,7 @@ static unsigned int cake_drop(struct Qdisc *sch, 
struct sk_buff **to_free)
         sch->qstats.drops++;

         if(q->rate_flags & CAKE_FLAG_INGRESS)
-               cake_advance_shaper(q, b, cake_overhead(q, len), now);
+               q->drop_len += len;

  #if LINUX_VERSION_CODE < KERNEL_VERSION(4, 8, 0)
         kfree_skb(skb);
@@ -1274,7 +1276,9 @@ retry:
                 /* drop this packet, get another one */
                 if(q->rate_flags & CAKE_FLAG_INGRESS) {
                         len = cake_overhead(q, qdisc_pkt_len(skb));
-                       cake_advance_shaper(q, b, len, now);
+                       flow->cvars2.count = 
b->bulk_flow_count+b->sparse_flow_count+b->decaying_flow_count+b->unresponsive_flow_count;
+                       cobalt_invsqrt(&(flow->cvars2));
+                       q->drop_len += (len - reciprocal_scale(len, 
flow->cvars2.rec_inv_sqrt));
                         flow->deficit -= len;
                         b->tin_deficit -= len;
                 }
@@ -1286,8 +1290,6 @@ retry:
                 qdisc_qstats_drop(sch);
                 kfree_skb(skb);
  #endif
-               if(q->rate_flags & CAKE_FLAG_INGRESS)
-                       goto retry;
         }

         b->tin_ecn_mark += !!flow->cvars.ecn_marked;
@@ -1340,7 +1342,7 @@ static void cake_advance_shaper(struct 
cake_sched_data *q, struct cake_tin_data
         if(q->rate_ns) {
                 s64 tdiff1 = b->tin_time_next_packet - now;
                 s64 tdiff2 = (len * (u64)b->tin_rate_ns) >> 
b->tin_rate_shft;
-               s64 tdiff3 = (len * (u64)q->rate_ns) >> q->rate_shft;
+               s64 tdiff3 = ((q->drop_len + len) * (u64)q->rate_ns) >> 
q->rate_shft;

                 if(tdiff1 < 0)
                         b->tin_time_next_packet += tdiff2;
@@ -1348,6 +1350,7 @@ static void cake_advance_shaper(struct 
cake_sched_data *q, struct cake_tin_data
                         b->tin_time_next_packet = now + tdiff2;

                 q->time_next_packet += tdiff3;
+               q->drop_len = 0;
         }
  }

@@ -1711,6 +1714,7 @@ static void cake_reconfigure(struct Qdisc *sch)
  {
         struct cake_sched_data *q = qdisc_priv(sch);
         int c, ft;
+       q->drop_len=0;

         switch (q->tin_mode) {
         case CAKE_MODE_BESTEFFORT:
@@ -1941,6 +1945,7 @@ static int cake_init(struct Qdisc *sch, struct 
nlattr *opt)

                         INIT_LIST_HEAD(&flow->flowchain);
                         cobalt_vars_init(&flow->cvars);
+                       cobalt_vars_init(&flow->cvars2);

                         q->overflow_heap[k].t = i;
                         q->overflow_heap[k].b = j;

=============8<=============



On 11/11/2017 10:48 PM, George Amanakis wrote:
> I totally understand what you are saying. However, I believe cake's 
> egress and ingress modes currently behave as two extremes. One could 
> argue that neither of them is the golden mean. With a patch in ingress 
> mode (see below) and a single host using 32 flows to download I 
> managed to increase throughput from ~7Mbps to ~10Mbps (configured 
> limit 12200kbps) while latency increased from ~10ms to ~50ms, which 
> would still be acceptable. As a comparison egress mode in the same 
> setup gives me throughput ~11.5Mbps and latency ~500ms.
>
> I would like to hear your thoughts about this idea: the patch is 
> incrementing q->time_next_packet for dropped packets differently than 
> for passed-through ones. Please focus on the idea, not the actual 
> implementation :) (also pasted in https://pastebin.com/SZ14WiYw)
>
> =============8<=============
>
> diff --git a/sch_cake.c b/sch_cake.c
> index 82f264f..a3a4a88 100644
> --- a/sch_cake.c
> +++ b/sch_cake.c
> @@ -769,6 +769,7 @@ static void cake_heapify_up(struct cake_sched_data 
> *q, u16 i)
>  }
>
>  static void cake_advance_shaper(struct cake_sched_data *q, struct 
> cake_tin_data *b, u32 len, u64 now);
> +static void cake_advance_shaper2(struct cake_sched_data *q, struct 
> cake_tin_data *b, u32 len, u64 now);
>
>  #if LINUX_VERSION_CODE < KERNEL_VERSION(4, 8, 0)
>  static unsigned int cake_drop(struct Qdisc *sch)
> @@ -1274,7 +1275,7 @@ retry:
>                 /* drop this packet, get another one */
>                 if(q->rate_flags & CAKE_FLAG_INGRESS) {
>                         len = cake_overhead(q, qdisc_pkt_len(skb));
> -                       cake_advance_shaper(q, b, len, now);
> +                       cake_advance_shaper2(q, b, len, now);
>                         flow->deficit -= len;
>                         b->tin_deficit -= len;
>                 }
> @@ -1286,8 +1287,6 @@ retry:
>                 qdisc_qstats_drop(sch);
>                 kfree_skb(skb);
>  #endif
> -               if(q->rate_flags & CAKE_FLAG_INGRESS)
> -                       goto retry;
>         }
>
>         b->tin_ecn_mark += !!flow->cvars.ecn_marked;
> @@ -1351,6 +1350,24 @@ static void cake_advance_shaper(struct 
> cake_sched_data *q, struct cake_tin_data
>         }
>  }
>
> +static void cake_advance_shaper2(struct cake_sched_data *q, struct 
> cake_tin_data *b, u32 len, u64 now)
> +{
> +       /* charge packet bandwidth to this tin, lower tins,
> +        * and to the global shaper.
> +        */
> +       if(q->rate_ns) {
> +               s64 tdiff1 = b->tin_time_next_packet - now;
> +               s64 tdiff2 = (len * (u64)b->tin_rate_ns) >> 
> b->tin_rate_shft;
> +               s64 tdiff3 = (len * (u64)q->rate_ns) >> q->rate_shft;
> +
> +               if(tdiff1 < 0)
> +                       b->tin_time_next_packet += tdiff2;
> +               else if(tdiff1 < tdiff2)
> +                       b->tin_time_next_packet = now + tdiff2;
> +
> +               q->time_next_packet += (tdiff3*27)>>5;
> +       }
> +}
>  static void cake_reset(struct Qdisc *sch)
>  {
>         u32 c;
>
> =============8<=============
>
> On 11/10/2017 4:50 PM, Jonathan Morton wrote:
>>
>> In fact, that's why I put a failsafe into ingress mode, so that it 
>> would never stall completely.  It can happen, however, that 
>> throughput is significantly reduced when the drop rate is high.
>>
>> If throughput is more important to you than induced latency, switch 
>> to egress mode.
>>
>> Unfortunately it's not possible to guarantee both low latency and 
>> high throughput when operating downstream of the bottleneck link.  
>> ECN gives you better results, though.
>>
>> - Jonathan Morton
>>
>

Re: [Cake] total download rate with many flows

[George Amanakis]

I totally understand what you are saying. However, I believe cake's 
egress and ingress modes currently behave as two extremes. One could 
argue that neither of them is the golden mean. With a patch in ingress 
mode (see below) and a single host using 32 flows to download I managed 
to increase throughput from ~7Mbps to ~10Mbps (configured limit 
12200kbps) while latency increased from ~10ms to ~50ms, which would 
still be acceptable. As a comparison egress mode in the same setup gives 
me throughput ~11.5Mbps and latency ~500ms.

I would like to hear your thoughts about this idea: the patch is 
incrementing q->time_next_packet for dropped packets differently than 
for passed-through ones. Please focus on the idea, not the actual 
implementation :) (also pasted in https://pastebin.com/SZ14WiYw)

=============8<=============

diff --git a/sch_cake.c b/sch_cake.c
index 82f264f..a3a4a88 100644
--- a/sch_cake.c
+++ b/sch_cake.c
@@ -769,6 +769,7 @@ static void cake_heapify_up(struct cake_sched_data 
*q, u16 i)
  }

  static void cake_advance_shaper(struct cake_sched_data *q, struct 
cake_tin_data *b, u32 len, u64 now);
+static void cake_advance_shaper2(struct cake_sched_data *q, struct 
cake_tin_data *b, u32 len, u64 now);

  #if LINUX_VERSION_CODE < KERNEL_VERSION(4, 8, 0)
  static unsigned int cake_drop(struct Qdisc *sch)
@@ -1274,7 +1275,7 @@ retry:
                 /* drop this packet, get another one */
                 if(q->rate_flags & CAKE_FLAG_INGRESS) {
                         len = cake_overhead(q, qdisc_pkt_len(skb));
-                       cake_advance_shaper(q, b, len, now);
+                       cake_advance_shaper2(q, b, len, now);
                         flow->deficit -= len;
                         b->tin_deficit -= len;
                 }
@@ -1286,8 +1287,6 @@ retry:
                 qdisc_qstats_drop(sch);
                 kfree_skb(skb);
  #endif
-               if(q->rate_flags & CAKE_FLAG_INGRESS)
-                       goto retry;
         }

         b->tin_ecn_mark += !!flow->cvars.ecn_marked;
@@ -1351,6 +1350,24 @@ static void cake_advance_shaper(struct 
cake_sched_data *q, struct cake_tin_data
         }
  }

+static void cake_advance_shaper2(struct cake_sched_data *q, struct 
cake_tin_data *b, u32 len, u64 now)
+{
+       /* charge packet bandwidth to this tin, lower tins,
+        * and to the global shaper.
+        */
+       if(q->rate_ns) {
+               s64 tdiff1 = b->tin_time_next_packet - now;
+               s64 tdiff2 = (len * (u64)b->tin_rate_ns) >> 
b->tin_rate_shft;
+               s64 tdiff3 = (len * (u64)q->rate_ns) >> q->rate_shft;
+
+               if(tdiff1 < 0)
+                       b->tin_time_next_packet += tdiff2;
+               else if(tdiff1 < tdiff2)
+                       b->tin_time_next_packet = now + tdiff2;
+
+               q->time_next_packet += (tdiff3*27)>>5;
+       }
+}
  static void cake_reset(struct Qdisc *sch)
  {
         u32 c;

=============8<=============

On 11/10/2017 4:50 PM, Jonathan Morton wrote:
>
> In fact, that's why I put a failsafe into ingress mode, so that it 
> would never stall completely.  It can happen, however, that throughput 
> is significantly reduced when the drop rate is high.
>
> If throughput is more important to you than induced latency, switch to 
> egress mode.
>
> Unfortunately it's not possible to guarantee both low latency and high 
> throughput when operating downstream of the bottleneck link.  ECN 
> gives you better results, though.
>
> - Jonathan Morton
>

Re: [Cake] total download rate with many flows

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 467 bytes --]

In fact, that's why I put a failsafe into ingress mode, so that it would
never stall completely.  It can happen, however, that throughput is
significantly reduced when the drop rate is high.

If throughput is more important to you than induced latency, switch to
egress mode.

Unfortunately it's not possible to guarantee both low latency and high
throughput when operating downstream of the bottleneck link.  ECN gives you
better results, though.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 553 bytes --]

Re: [Cake] total download rate with many flows

[George Amanakis]

Indeed. Also interplanetary essentially omits cake_advance_shaper for 
dropped packets (since cobalt never drops that way), almost disabling 
ingress mode. Which leads to other hosts having pings to WAN > 500ms 
when one of them is using many flows.

What I think is happening in ingress mode is that codel starts dropping 
packets in many flows simultaneously. Each time this happens, 
q->time_next_packet is advanced, eventually dis-proportionally to the 
actual rate, and this leads to the actual download rate being lower than 
set.

I am a big fan of ingress mode, just if we could correct this behaviour...


On 11/10/2017 3:42 PM, Dave Taht wrote:
> Interplanetary basically disables codel for most applications.
>
>> On 11/1/2017 6:01 PM, Dave Taht wrote:
>>> Awesome. thx. I will try to make time to look at these over the weekend.
> I still haven't had that weekend.
>

Re: [Cake] total download rate with many flows

[Dave Taht]

Interplanetary basically disables codel for most applications.

> On 11/1/2017 6:01 PM, Dave Taht wrote:
>> Awesome. thx. I will try to make time to look at these over the weekend.

I still haven't had that weekend.

Re: [Cake] total download rate with many flows

[George Amanakis]

I did more testing regarding this issue.

I found out that if I use "interplanetary", I do not observe the 
behaviour I mentioned before. I still need to test the responsiveness of 
other hosts while one of them is using too many flows with this setting.

Setup is the same, cake configuration was:

qdisc cake 8007: dev ens4 root refcnt 2 bandwidth 12200Kbit besteffort 
dual-dsthost wash ingress rtt 3600.0s noatm overhead 18 via-ethernet mpu 64
qdisc cake 8008: dev ens3 root refcnt 2 bandwidth 2500Kbit besteffort 
dual-srchost nat wash rtt 3600.0s noatm overhead 18 via-ethernet mpu 64


On 11/1/2017 6:01 PM, Dave Taht wrote:
> Awesome. thx. I will try to make time to look at these over the weekend.

Re: [Cake] total download rate with many flows

[Dave Taht]

Awesome. thx. I will try to make time to look at these over the weekend.

Re: [Cake] total download rate with many flows

[George Amanakis]

Dear Dave,

I could get the following dumps:

*lan.cake.pcap
captured on router, lan iface, with cake

#lan.cake.ping.pcap
captured on router, lan iface, with cake, ping from host other than w10 
update

$lan.nocake.pcap
captured on router, lan iface, without cake

*wan.cake.pcap
captured on router, wan iface, with cake

#wan.cake.ping.pcap
captured on router, wan iface, with cake, ping from host other than w10 
update

$wan.nocake.pcap
captured on router, wan iface, without cake

*win10host.cake.pcapng
captured on w10 updating, with cake

#win10host.cake.ping.pcapng
captured on w10 updating, with cake, ping from another host

$win10host.nocake.pcapng
captured on w10 updating, without cake

===============

ping to google.com, which resolved to an IPv6 address.
The signs(*,#,$) denote packet captures done (nearly) in parallel.

LAN:
qdisc cake 8001: dev ens4 root refcnt 2 bandwidth 12200Kbit besteffort 
dual-dsthost wash ingress rtt 100.0ms noatm overhead 18 via-ethernet mpu 64

WAN:
qdisc cake 8002: dev ens3 root refcnt 2 bandwidth 2500Kbit besteffort 
dual-srchost nat wash rtt 100.0ms noatm overhead 18 via-ethernet mpu 64

===============

Links:
lan.cake.pcap
https://drive.google.com/open?id=0B2bwzgme1zsdbS1uZVdaWXRpVTg

lan.cake.ping.pcap
https://drive.google.com/open?id=0B2bwzgme1zsdd3lwamJFZ0hIZ1E

lan.nocake.pcap
https://drive.google.com/open?id=0B2bwzgme1zsdUzNFZXdOaHVmX3c

wan.cake.pcap
https://drive.google.com/open?id=0B2bwzgme1zsdaURjQ2RuZC1xcU0

wan.cake.ping.pcap
https://drive.google.com/open?id=0B2bwzgme1zsda3FpQmpfQ0M0Nm8

wan.nocake.pcap
https://drive.google.com/open?id=0B2bwzgme1zsdeDhlVWx5OTVhQ28

win10host.cake.pcapng
https://drive.google.com/open?id=0B2bwzgme1zsdMi1nTnlHcnBZZUU

win10host.cake.ping.pcapng
https://drive.google.com/open?id=0B2bwzgme1zsdQmJ1eTEwZ1RKcWc

win10host.nocake.pcapng
https://drive.google.com/open?id=0B2bwzgme1zsdUTJ5OWZsYnlLU1k


Best,
George

Re: [Cake] total download rate with many flows

[George Amanakis]

[-- Attachment #1: Type: text/plain, Size: 1039 bytes --]

I think I have another install not updated to "Fall Creators Update". I 
will try and get a capture using Wireshark. Or should I rather capture 
on the router?

George

On 10/31/2017 9:06 PM, Antoine Deschênes wrote:
> Fall Creators update, killed my whole internet connection for 1 or 2 
> hours two times (two systems to update).
>
> Even the Google homepage would timeout using cake (or fq_codel or no 
> sqm at all), until I figured out the dual-src/dsthost flags in 
> sqm-scripts.
>
> It looks like it had a LOT of simultaneous flows... Maybe they got the 
> P2P update distribution working
>
> Le 31 oct. 2017 8:45 PM, "Dave Taht" <dave@taht.net 
> <mailto:dave@taht.net>> a écrit :
>
>
>     I would *dearly* love a packet capture of a windows 10 update in this
>     case.
>
>     _______________________________________________
>     Cake mailing list
>     Cake@lists.bufferbloat.net <mailto:Cake@lists.bufferbloat.net>
>     https://lists.bufferbloat.net/listinfo/cake
>     <https://lists.bufferbloat.net/listinfo/cake>
>


[-- Attachment #2: Type: text/html, Size: 2527 bytes --]

Re: [Cake] total download rate with many flows

[Antoine Deschênes]

[-- Attachment #1: Type: text/plain, Size: 677 bytes --]

Fall Creators update, killed my whole internet connection for 1 or 2 hours
two times (two systems to update).

Even the Google homepage would timeout using cake (or fq_codel or no sqm at
all), until I figured out the dual-src/dsthost flags in sqm-scripts.

It looks like it had a LOT of simultaneous flows... Maybe they got the P2P
update distribution working

Le 31 oct. 2017 8:45 PM, "Dave Taht" <dave@taht.net> a écrit :

>
> I would *dearly* love a packet capture of a windows 10 update in this
> case.
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>

[-- Attachment #2: Type: text/html, Size: 1472 bytes --]

Re: [Cake] total download rate with many flows

[Dave Taht]

I would *dearly* love a packet capture of a windows 10 update in this
case.