[Cake] More on 'target' corner cases - rate/target/interval confusion?

[Cake] More on 'target' corner cases - rate/target/interval confusion?

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 4223 bytes --]

Hi List,

I'm still not supposed to be here (I'm working) and I'm full of a cold
still however the same/similar questions keep coming up and it prompts a
follow up 'base' question that I don't know the answer to.  So the
background:  Targets.

In the absence of any other instruction and assuming either a)
sufficiently fast rate/unlimited, the first cake classification tin will
be set to 'interval/16', for 100mS this is 6.25mS.  Where a rate *is*
specified then this calculation is performed (I can't get my cold addled
head round this at the moment, but suffice it's an integer division only
dodge and 'magics' up the time for a MTU * 1.5 packet to flow
(byte_target_ns)  Wonderful, no problem here.

cake_set_rate(.....)

         /* convert byte-rate into time-per-byte
         * so it will always unwedge in reasonable time.
         */
        static const u64 MIN_RATE = 64;
        u64 rate_ns = 0;
        u8  rate_shft = 0;
        codel_time_t byte_target_ns;
        u32 byte_target = mtu + (mtu >> 1);

        if (rate) {
                rate_shft = 32;
                rate_ns = ((u64) NSEC_PER_SEC) << rate_shft;
                do_div(rate_ns, max(MIN_RATE, rate));
                while (!!(rate_ns >> 32)) {
                        rate_ns >>= 1;
                        rate_shft--;
                }
        } /* else unlimited, ie. zero delay */

        b->tin_rate_bps  = rate;
        b->tin_rate_ns   = rate_ns;
        b->tin_rate_shft = rate_shft;

        byte_target_ns = (byte_target * rate_ns) >> rate_shft;


Later on we see:


        b->cparams.target = max(byte_target_ns, ns_target);
        b->cparams.interval = max(rtt_est_ns +
                                     b->cparams.target - ns_target,
                                     b->cparams.target * 8);
        b->cparams.threshold = (b->cparams.target >> 15) *
                (b->cparams.interval >> 15) * 2;


This tin's target is set to our calculated 'fat packet' transit time
*OR* our requested time, whichever is the larger.  Similarly interval is
set to our requested target (yes I'm missing some stuff out!) or
target*8, again whichever is larger.  I've no problem with any of this,
though it does silently override requested parameters.  The
problem/question comes when combined with our diffserv/tin
classification, as ably demonstrated by this:

user@computer:~/CODE/tc-adv/tc> sudo tc-adv qdisc replace dev eth0 root cake bandwidth 1Mbit ; sudo tc-adv -s qdisc
qdisc cake 8005: dev eth0 root refcnt 6 bandwidth 1Mbit diffserv4 flows rtt 100.0ms raw 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
capacity estimate: 1Mbit
             Tin 0       Tin 1       Tin 2       Tin 3  
  thresh       1Mbit   937504bit     750Kbit     250Kbit
  target      18.2ms      19.4ms      24.2ms      72.7ms
interval     145.3ms     155.0ms     193.8ms     581.4ms


Note that as rate decreases, target & interval increase as per the above
algorithm.  We can see why:

cake_config_diffserv4(struct Qdisc *sch)
       /* class characteristics */
        cake_set_rate(&q->tins[0], rate, mtu,
                      US2TIME(q->target), US2TIME(q->interval));  //Base
rate
        cake_set_rate(&q->tins[1], rate - (rate >> 4), mtu,
                      US2TIME(q->target), US2TIME(q->interval));  //15/16ths
        cake_set_rate(&q->tins[2], rate - (rate >> 2), mtu,
                      US2TIME(q->target), US2TIME(q->interval));  //3/4 rate
        cake_set_rate(&q->tins[3], rate >> 2, mtu,
                      US2TIME(q->target), US2TIME(q->interval));  //1/4 rate

http://www.bufferbloat.net/projects/codel/wiki/CakeTechnical makes
reference to the bandwidth allocation but makes no reference to its
affect on targets/intervals for that tin.  Can someone please explain if
the above is actually correct and why.  i.e. I'm confused why voice
traffic has a target latency of 72mS over a measurement interval of 1/2
second when I'd have thought keeping voice latency as low as possible
would be the ideal.  I'm sure I'm being exceptionally stupid so I await
education please :-)

Kevin


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4816 bytes --]

Re: [Cake] More on 'target' corner cases - rate/target/interval confusion?

[Dave Taht]

as for the voice class, we always have the problem of
mis-classification into that tin.

Aside from that, I believe you are right, that target/interval should
be tighter on that class in general. At least some versions of
debloat.sh  and the debloat lua script for wifi actually did that.
Some versions also disabled ecn on the voice class also. There is
certainly some thinking out there that argues in favor of dropping
everything but the last few packets per flow in the voice class...
rather than applying something as complex as fq_codel to it.

I appreciate you taking the time for obsessive, deep thoughts like
this. For the past 6 months I've had the workload of hercules,  and
the attention span of a hummingbird on crack.

Nowadays, despite the conventional wisdom, I tend to favor the idea of
allowing ecn, even on voice, now that we have other means of kicking
in drops when ecn is present.

Also, conventional thinking on voice is generally old style single
rate GSM encoded packets,
(something like 120 bytes/20ms).

 where the future I would like to live in is opus 48khz 6 way surround
sound with 2.7 ms latency on the encoder, and I don't know how that
should be treated as to queue depth and jitter, in a DRR (byte based)
system, without testing.

It would be great to upgrade to that sort of opus based flow when
analyzing voice behavior under cake, perhaps driven by an asterisk or
freeswitch or other musical over the internet application (LOLA
anyone?)

https://www.opus-codec.org/
Dave Täht
I just invested five years of my life to making wifi better. And,
now... the FCC wants to make my work, illegal for people to install.
https://www.gofundme.com/savewifi


On Tue, Nov 3, 2015 at 7:13 AM, Kevin Darbyshire-Bryant
<kevin@darbyshire-bryant.me.uk> wrote:
> Hi List,
>
> I'm still not supposed to be here (I'm working) and I'm full of a cold
> still however the same/similar questions keep coming up and it prompts a
> follow up 'base' question that I don't know the answer to.  So the
> background:  Targets.
>
> In the absence of any other instruction and assuming either a)
> sufficiently fast rate/unlimited, the first cake classification tin will
> be set to 'interval/16', for 100mS this is 6.25mS.  Where a rate *is*
> specified then this calculation is performed (I can't get my cold addled
> head round this at the moment, but suffice it's an integer division only
> dodge and 'magics' up the time for a MTU * 1.5 packet to flow
> (byte_target_ns)  Wonderful, no problem here.
>
> cake_set_rate(.....)
>
>          /* convert byte-rate into time-per-byte
>          * so it will always unwedge in reasonable time.
>          */
>         static const u64 MIN_RATE = 64;
>         u64 rate_ns = 0;
>         u8  rate_shft = 0;
>         codel_time_t byte_target_ns;
>         u32 byte_target = mtu + (mtu >> 1);
>
>         if (rate) {
>                 rate_shft = 32;
>                 rate_ns = ((u64) NSEC_PER_SEC) << rate_shft;
>                 do_div(rate_ns, max(MIN_RATE, rate));
>                 while (!!(rate_ns >> 32)) {
>                         rate_ns >>= 1;
>                         rate_shft--;
>                 }
>         } /* else unlimited, ie. zero delay */
>
>         b->tin_rate_bps  = rate;
>         b->tin_rate_ns   = rate_ns;
>         b->tin_rate_shft = rate_shft;
>
>         byte_target_ns = (byte_target * rate_ns) >> rate_shft;
>
>
> Later on we see:
>
>
>         b->cparams.target = max(byte_target_ns, ns_target);
>         b->cparams.interval = max(rtt_est_ns +
>                                      b->cparams.target - ns_target,
>                                      b->cparams.target * 8);
>         b->cparams.threshold = (b->cparams.target >> 15) *
>                 (b->cparams.interval >> 15) * 2;
>
>
> This tin's target is set to our calculated 'fat packet' transit time
> *OR* our requested time, whichever is the larger.  Similarly interval is
> set to our requested target (yes I'm missing some stuff out!) or
> target*8, again whichever is larger.  I've no problem with any of this,
> though it does silently override requested parameters.  The
> problem/question comes when combined with our diffserv/tin
> classification, as ably demonstrated by this:
>
> user@computer:~/CODE/tc-adv/tc> sudo tc-adv qdisc replace dev eth0 root cake bandwidth 1Mbit ; sudo tc-adv -s qdisc
> qdisc cake 8005: dev eth0 root refcnt 6 bandwidth 1Mbit diffserv4 flows rtt 100.0ms raw
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
> capacity estimate: 1Mbit
>              Tin 0       Tin 1       Tin 2       Tin 3
>   thresh       1Mbit   937504bit     750Kbit     250Kbit
>   target      18.2ms      19.4ms      24.2ms      72.7ms
> interval     145.3ms     155.0ms     193.8ms     581.4ms
>
>
> Note that as rate decreases, target & interval increase as per the above
> algorithm.  We can see why:
>
> cake_config_diffserv4(struct Qdisc *sch)
>        /* class characteristics */
>         cake_set_rate(&q->tins[0], rate, mtu,
>                       US2TIME(q->target), US2TIME(q->interval));  //Base
> rate
>         cake_set_rate(&q->tins[1], rate - (rate >> 4), mtu,
>                       US2TIME(q->target), US2TIME(q->interval));  //15/16ths
>         cake_set_rate(&q->tins[2], rate - (rate >> 2), mtu,
>                       US2TIME(q->target), US2TIME(q->interval));  //3/4 rate
>         cake_set_rate(&q->tins[3], rate >> 2, mtu,
>                       US2TIME(q->target), US2TIME(q->interval));  //1/4 rate
>
> http://www.bufferbloat.net/projects/codel/wiki/CakeTechnical makes
> reference to the bandwidth allocation but makes no reference to its
> affect on targets/intervals for that tin.  Can someone please explain if
> the above is actually correct and why.  i.e. I'm confused why voice
> traffic has a target latency of 72mS over a measurement interval of 1/2
> second when I'd have thought keeping voice latency as low as possible
> would be the ideal.  I'm sure I'm being exceptionally stupid so I await
> education please :-)
>
> Kevin
>
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>

Re: [Cake] More on 'target' corner cases - rate/target/interval confusion?

[Toke Høiland-Jørgensen]

[-- Attachment #1: Type: text/plain, Size: 1671 bytes --]

Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> writes:

> Can someone please explain if the above is actually correct and why.
> i.e. I'm confused why voice traffic has a target latency of 72mS over
> a measurement interval of 1/2 second when I'd have thought keeping
> voice latency as low as possible would be the ideal. I'm sure I'm
> being exceptionally stupid so I await education please :-)

Well, I *think* this comes from the 1.5*MTU calculation. I.e. 1.5 * 1500
bytes at 250Kbps is 72 ms. And interval is adjusted to always be
8*target.

Now whether or not this is *correct* I'm not so sure about. The minimum
value set from the MTU is the Linux counterpart to the CoDel NS2 model's
"always keep one packet in the queue". The reasoning is, basically, that
there will be at least one packet in the layers below the qdisc which
we will have to wait for before we can dequeue; so CoDel shouldn't react
until queueing time goes *above* this minimum of waiting for one packet.

Now, in this case we have the rate limiter built right in to the qdisc,
so the queued packet will in fact *not* be in a layer below (because
that is probably going to be a link with a physical rate that is quite a
bit higher than what we're setting here; or if not, we're not doing the
MTU adjustment anyway). So it's not quite clear to me what the right
thing to do here is; as you rightly point out this leads to a very high
target for the VoIP queue which is not necessarily a good idea. Capping
the target before doing the per-tin calculation might be a better idea
(i.e. using the total bandwidth limit as the input to the calculation).
Will think about it some more.

-Toke

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]

Re: [Cake] More on 'target' corner cases - rate/target/interval confusion?

[Sebastian Moeller]

On Nov 3, 2015, at 13:46 , Toke Høiland-Jørgensen <toke@toke.dk> wrote:

> [...]
> Now whether or not this is *correct* I'm not so sure about. The minimum
> value set from the MTU is the Linux counterpart to the CoDel NS2 model's
> "always keep one packet in the queue". The reasoning is, basically, that
> there will be at least one packet in the layers below the qdisc which
> we will have to wait for before we can dequeue; so CoDel shouldn't react
> until queueing time goes *above* this minimum of waiting for one packet.
> [...]

I thought the rationale was that a target be tow the transfer time for a single MTU packet is not too reasonable as the queue will enter dropping state for each (full MTU) packet transferred. But that could be me misunderstanding codel… I guess most VoIP packets will not be full MTU, so on a dedicated VoIP queue the 1.5 * full MTU idea might be sub-optimal...


Best Regards
	Sebastian

Re: [Cake] More on 'target' corner cases - rate/target/interval confusion?

[Toke Høiland-Jørgensen]

Sebastian Moeller <moeller0@gmx.de> writes:

> I thought the rationale was that a target be tow the transfer time for
> a single MTU packet is not too reasonable as the queue will enter
> dropping state for each (full MTU) packet transferred.

Yes, that is exactly it. But that happens when the packet at the head of
the queue is waiting on the packet *before* it, which will be in a lower
layer in the process of being transferred (at low bandwidths that "lower
layer" has in practice been HTB).

However, now that cake itself is doing the delaying (via the packet
transmission scheduling), this is not as clear-cut. The actual packet at
the head of the queue is now the one being delayed. I'm afraid I don't
quite grok how the shaper and CoDel interacts in this case; guess I'll
go read the code.

> I guess most VoIP packets will not be full MTU, so on a dedicated VoIP
> queue the 1.5 * full MTU idea might be sub-optimal...

That is also a (separate IMO) issue; however, as Dave said, we need to
be vary of just assuming that the VoIP queue will be only small VoIP
packets: there's no admission control!

-Toke

Re: [Cake] More on 'target' corner cases - rate/target/interval confusion?

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 1609 bytes --]

At the moment cake is effectively saying that it expects stuff in tin 4 to take 4 times as long to send/its packets are 4 times as big depending which warped view you care to take :-)

That really doesn't make sense to me. 

--
Cheers,

Kevin@Darbyshire-Bryant.me.uk
Sent from my phone, apologies for brevity, spelling & top posting

> On 3 Nov 2015, at 13:56, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Sebastian Moeller <moeller0@gmx.de> writes:
> 
>> I thought the rationale was that a target be tow the transfer time for
>> a single MTU packet is not too reasonable as the queue will enter
>> dropping state for each (full MTU) packet transferred.
> 
> Yes, that is exactly it. But that happens when the packet at the head of
> the queue is waiting on the packet *before* it, which will be in a lower
> layer in the process of being transferred (at low bandwidths that "lower
> layer" has in practice been HTB).
> 
> However, now that cake itself is doing the delaying (via the packet
> transmission scheduling), this is not as clear-cut. The actual packet at
> the head of the queue is now the one being delayed. I'm afraid I don't
> quite grok how the shaper and CoDel interacts in this case; guess I'll
> go read the code.
> 
>> I guess most VoIP packets will not be full MTU, so on a dedicated VoIP
>> queue the 1.5 * full MTU idea might be sub-optimal...
> 
> That is also a (separate IMO) issue; however, as Dave said, we need to
> be vary of just assuming that the VoIP queue will be only small VoIP
> packets: there's no admission control!
> 
> -Toke

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 3062 bytes --]

Re: [Cake] More on 'target' corner cases - rate/target/interval confusion?

[Toke Høiland-Jørgensen]

[-- Attachment #1: Type: text/plain, Size: 489 bytes --]

Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> writes:

> At the moment cake is effectively saying that it expects stuff in tin
> 4 to take 4 times as long to send/its packets are 4 times as big
> depending which warped view you care to take :-)

Yup, exactly. However that can be true because tin 4 is shaped to 1/4
the rate; so for 1 packet, that is actually true.

However, it's not really a shaping, it's a threshold. Or is it? As I
said, I need to go read the code :P

-Toke

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]

Re: [Cake] More on 'target' corner cases - rate/target/interval confusion?

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 1296 bytes --]



On 03/11/15 14:12, Toke Høiland-Jørgensen wrote:
> Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> writes:
>
>> At the moment cake is effectively saying that it expects stuff in tin
>> 4 to take 4 times as long to send/its packets are 4 times as big
>> depending which warped view you care to take :-)
> Yup, exactly. However that can be true because tin 4 is shaped to 1/4
> the rate; so for 1 packet, that is actually true.
>
> However, it's not really a shaping, it's a threshold. Or is it? As I
> said, I need to go read the code :P
>
> -Toke

Following this up:  The existing code is correct, or is certainly much
more correct than some changes I've been experimenting with :-) The
bottom line is that passing the true 'line' rate and hence related
target/intervals for each tin rather breaks the class/tin shaper,
something I proved with the aid of flent.  'b->tin_rate_bps  = rate;' is
really a reporting construct and isn't actually used as an active part
of code.

So I'm gradually understanding things....by breaking them and putting
them back together :-)   The exact mechanism for shaping eludes me at
the moment (I suspect 'threshold') and quite what effect 'target' and
'interval' really have again is something to experiment with.

Fun!



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4816 bytes --]

Re: [Cake] More on 'target' corner cases - rate/target/interval confusion?

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 2710 bytes --]



On 05/11/15 16:41, Kevin Darbyshire-Bryant wrote:
>
> On 03/11/15 14:12, Toke Høiland-Jørgensen wrote:
>> Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> writes:
>>
>>> At the moment cake is effectively saying that it expects stuff in tin
>>> 4 to take 4 times as long to send/its packets are 4 times as big
>>> depending which warped view you care to take :-)
>> Yup, exactly. However that can be true because tin 4 is shaped to 1/4
>> the rate; so for 1 packet, that is actually true.
>>
>> However, it's not really a shaping, it's a threshold. Or is it? As I
>> said, I need to go read the code :P
>>
>> -Toke
> Following this up:  The existing code is correct, or is certainly much
> more correct than some changes I've been experimenting with :-) The
> bottom line is that passing the true 'line' rate and hence related
> target/intervals for each tin rather breaks the class/tin shaper,
> something I proved with the aid of flent.  'b->tin_rate_bps  = rate;' is
> really a reporting construct and isn't actually used as an active part
> of code.
>
> So I'm gradually understanding things....by breaking them and putting
> them back together :-)   The exact mechanism for shaping eludes me at
> the moment (I suspect 'threshold') and quite what effect 'target' and
> 'interval' really have again is something to experiment with.
>
> Fun!
Ok, so more following up. 'threshold' does indeed act as the shaper and
bases its value on target/interval for that tin which is why tweaking
'tin_rate_bps' doesn't help.   I still feel confused as to why 'lower
bandwidth allocation/higher priority/lower jitter' tins end up with
higher targets & intervals.  So, in the interests of experimentation I
came up with another way of setting all tins to the same (tin 0)
target/interval.

I pushed a branch
https://github.com/dtaht/sch_cake/tree/targetintervalcopied   I'm sorry
Jonathan, every time I touch your code I feel like I'm being an absolute
vandal, please forgive the horrors!

If someone is bored, has a suitable test bed etc I'd be really
interested to know what differences are seen with this code.

Whilst I was there I also subtly tweaked the diffserv4 video tin
bandwidth allocation to 11/16ths rather than 3/4.  In combination with
voice (1/4) it meant that (theoretically) best effort could be
completely starved, let alone bulk - it doesn't seem to actually happen
in real life, but setting video to 11/16th & voice to 4/16th means that
there's 1/16th to be fought over between best effort and bulk - and best
effort as a result does seem to get a little bit more of a look-in in
the face of 'more important' competition.


Kevin



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4816 bytes --]

Re: [Cake] More on 'target' corner cases - rate/target/interval confusion?

[Toke Høiland-Jørgensen]

Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> writes:

> If someone is bored, has a suitable test bed etc I'd be really
> interested to know what differences are seen with this code.

I'm leaving today and will be without access to a proper computer until
Thursday. Can incorporate it into my testing when I get back; still need
to design one or more tests to exercise the diffserv capability in
general.

-Toke

Re: [Cake] More on 'target' corner cases - rate/target/interval confusion?

[Jonathan Morton]

> On 8 Nov, 2015, at 12:19, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
> 
> Whilst I was there I also subtly tweaked the diffserv4 video tin
> bandwidth allocation to 11/16ths rather than 3/4.  In combination with
> voice (1/4) it meant that (theoretically) best effort could be
> completely starved, let alone bulk - it doesn't seem to actually happen
> in real life, but setting video to 11/16th & voice to 4/16th means that
> there's 1/16th to be fought over between best effort and bulk - and best
> effort as a result does seem to get a little bit more of a look-in in
> the face of 'more important' competition.

Actually, the “threshold” mechanism doesn’t implement the priority queue at all, but only switches the underlying DRR weights between “priority balance” and “bandwidth balance”.  Since it’s fundamentally DRR, and the weights of the queues never go to zero, there is no possibility of starvation.  Additionally, the threshold mechanism itself “borrows” bandwidth from lower tins to serve higher ones - this is a holdover from an earlier version when there was indeed a separate hard shaper per tin.

It *is* possible that the per-tin choices of target and interval might want tweaking, but it’s probably best to have some hard data for the effects of doing that.  One possibility is that target should be tuned for the worst-case (minimum) bandwidth under maximally saturated conditions, rather than the raw threshold value.

 - Jonathan Morton

Re: [Cake] More on 'target' corner cases - rate/target/interval confusion?

[Sebastian Moeller]

Hi Jonathan,


On Nov 8, 2015, at 17:36 , Jonathan Morton <chromatix99@gmail.com> wrote:

> 
>> On 8 Nov, 2015, at 12:19, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
>> 
>> Whilst I was there I also subtly tweaked the diffserv4 video tin
>> bandwidth allocation to 11/16ths rather than 3/4.  In combination with
>> voice (1/4) it meant that (theoretically) best effort could be
>> completely starved, let alone bulk - it doesn't seem to actually happen
>> in real life, but setting video to 11/16th & voice to 4/16th means that
>> there's 1/16th to be fought over between best effort and bulk - and best
>> effort as a result does seem to get a little bit more of a look-in in
>> the face of 'more important' competition.
> 
> Actually, the “threshold” mechanism doesn’t implement the priority queue at all, but only switches the underlying DRR weights between “priority balance” and “bandwidth balance”.  Since it’s fundamentally DRR, and the weights of the queues never go to zero, there is no possibility of starvation.  Additionally, the threshold mechanism itself “borrows” bandwidth from lower tins to serve higher ones - this is a holdover from an earlier version when there was indeed a separate hard shaper per tin.
> 
> It *is* possible that the per-tin choices of target and interval might want tweaking, but it’s probably best to have some hard data for the effects of doing that.  One possibility is that target should be tuned for the worst-case (minimum) bandwidth under maximally saturated conditions, rather than the raw threshold value.

	Interesting, in sqm-scripts I opted for only adjusting for the total link bandwidth to keep things simple; I always felt this needs reconsideration, so setting the target for each Tin seems reasonable. In sqm-scripts I just added the target increase (target - 5ms) to interval to avoid target >= interval, also I believe that if a full MTU packet takes Xms on the first hop the expected RTTs will be that amount larger compared to a normal/faster link. (So far we have looked at each direction independently, which probably is not fully correct, but at least for asymmetric links I hope we can just ignore one direction).
	I always argued that interval should be increased to keep the target to interval ratio close to the ideal. And I have pushed this idea on the cake list quite offensively ;) Looking at sch_cake and discussions with you, Kevin and Toke make me wonder whether, especially for low bandwidth Tins this is the right thing to do.
	Increasing interval will reduce the responsiveness of the shaper as it will wait longer for flows to react (even though most flows probably will react quickly, it will just take longer to reign in unresponsive flows), so for really slow links adjusting target and then multiply by 8 or 20 will lead to a rather sluggish controller? Increasing target/interval above 10% should simply sacrifice a bit more responsiveness-under-load for more bandwidth utilization? So both adaptation methods have the potential to counter-act cakes design goal. 
	What are the options besides:

1) target set to worst case for each Tin, keep interval at set interval or at target, what ever is larger
2) target set to account for the link’s MTU transfer time, keep interval at set interval or at target, what ever is larger

3) target set to worst case for each Tin, interval at set interval or 8*target, what ever is larger
4) target set to account for the link’s MTU transfer time, interval at set interval or 8*target, what ever is larger

5) target set to worst case for each Tin, interval at set interval plus target
6) target set to account for the link’s MTU transfer time, interval at set interval plus target

What are the relevant bandwidths to test this at?

Best Regards
	Sebastian

> 
> - Jonathan Morton
> 
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] More on 'target' corner cases - rate/target/interval confusion?

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 3489 bytes --]



On 08/11/15 16:36, Jonathan Morton wrote:
>> On 8 Nov, 2015, at 12:19, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
>>
>> Whilst I was there I also subtly tweaked the diffserv4 video tin
>> bandwidth allocation to 11/16ths rather than 3/4.  In combination with
>> voice (1/4) it meant that (theoretically) best effort could be
>> completely starved, let alone bulk - it doesn't seem to actually happen
>> in real life, but setting video to 11/16th & voice to 4/16th means that
>> there's 1/16th to be fought over between best effort and bulk - and best
>> effort as a result does seem to get a little bit more of a look-in in
>> the face of 'more important' competition.
> Actually, the “threshold” mechanism doesn’t implement the priority queue at all, but only switches the underlying DRR weights between “priority balance” and “bandwidth balance”.  Since it’s fundamentally DRR, and the weights of the queues never go to zero, there is no possibility of starvation.  Additionally, the threshold mechanism itself “borrows” bandwidth from lower tins to serve higher ones - this is a holdover from an earlier version when there was indeed a separate hard shaper per tin.
I did say that I felt my coding was more an act of vandalism than
anything as well thought out as your code :-)

> It *is* possible that the per-tin choices of target and interval might want tweaking, but it’s probably best to have some hard data for the effects of doing that.  One possibility is that target should be tuned for the worst-case (minimum) bandwidth under maximally saturated conditions, rather than the raw threshold value.

Oh absolutely.  Real numbers from real testing, couldn't agree more. 
It's very much needed (by people who actually understand what they're
looking at - rules me out)  I think there are two separate questions
that need looking at here:

1) In the presence of a full link, that link having competing 'full'
flows in all 'tins', then how should cake split the link in terms of
bandwidth?  Dave has argued there is not enough focus on 'Best Effort'
(tin 1) and too much focus on 'a bit better than best effort' (tin 2). 
Bulk (tin 0) I can't say I'm worried about.  'Voice' (low latency, low
jitter) (tin 3) we're hoping is also low bandwidth.  Without a full link
it arguably doesn't matter 'cos as you say we 'borrow' bandwidth from
the other tins to help out.  I pushed a tweak to the bandwidth weights
based on the insight you gave me ;-)  It might keep Dave happier (Bulk =
1/16th, Best Effort 8/16th, 'Better than best effort' 7/16th, 'Voice,
low latency & jitter' 2/16th)  Needs testing, needs numbers, based on a
*little* testing with flent it 'behaves' as per those allocations in the
presence of saturated link, saturating all tins.

2) With each increasing priority of tini[0-3], we decrease the
'expected' bandwidth (good) but also as a result increase the target and
interval.  Unless I've missed it, I've not yet seen an explanation at my
level (ie. idiot!) as to why that is the case.  The underlying link rate
doesn't change (it always runs at tin[0]) so it appears that we become
more tolerant of things sitting in and take longer to do anything about
it in tin[3]'s queue than tin[0].  I genuinely don't understand
why....and I'll assume my default position that it's 'cos I'm dumb
rather than an error in what I can see is carefully crafted code.  But I
do remain confused.


Kevin





[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4816 bytes --]

Re: [Cake] More on 'target' corner cases - rate/target/interval confusion?

[Jonathan Morton]

> On 9 Nov, 2015, at 14:12, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
> 
> In the presence of a full link, that link having competing ‘full' flows in all 'tins', then how should cake split the link in terms of bandwidth?

That’s a good question, and one that I think becomes more critical at low bandwidths.  I’ve tended towards generous allocations for that reason, so as to avoid causing trouble to low-latency applications.

The main requirement I keep in mind is that an application should not be able to guarantee itself an excessive bandwidth share simply by selecting a particular DSCP.  At the same time, there are applications for which a relatively large, consistent bandwidth is a requirement for satisfactory performance (consider streaming video), such that best-effort traffic should defer to them.  These are conflicting requirements, so a compromise has to be established somehow.

The current thresholds are at 100%, 15/16, ¾ and ¼.  Under saturated conditions, this gives throughputs of 1/16, 3/16, ½ and ¼.  The “video” class (Tin 2) can usurp ¾ of the bandwidth when competing against any mixture of best-effort and bulk traffic.  This, admittedly, might turn out to be too much, so I could consider setting Tin 2’s threshold at ½ instead of ¾.

And yes, I have long noticed that Flent’s standard RRUL test doesn’t use Tin 2 at all.

> With each increasing priority of tini[0-3], we decrease the 'expected' bandwidth (good) but also as a result increase the target and interval.

Yes, there are a number of counter-intuitive things happening here.

Most of Cake’s latency-reducing power comes from the liberal application of flow isolation, *not* from AQM itself.  Diffserv prioritisation plays a lesser role, and mainly has to do with restoring the desired allocations of bandwidth, replacing the reliance on measure queue fill level that some protocols presently use to stay out from underfoot.  Both these mechanisms primarily control the effects that one flow can have on another, and say little about the latency that a flow causes to itself.

This latter is the domain of AQM, specifically Codel, which is what we’re talking about when we mention “interval" and "target”.  As I mentioned elsewhere recently, Codel is designed specifically to give congestion signals to TCP-like flows, and deals rather less efficiently with unresponsive and anti-responsive flows, which as a result tend to spend some time bouncing off the queue’s hard limit until Codel finds the correct operating point to control them.  There are other AQMs which are designed with unresponsive flows more in mind, but which somehow perform less well with TCP-like flows.

A key design principle of Codel is that no packet whose sojourn time is below target will be signalled.  However, if the sojourn time is consistently above target, signalling begins and increases steadily in frequency.  It is also a fundamental truth that if it takes longer than target to transmit the previous packet, the following packet can have a “congested” sojourn time even if there is consistently precisely one packet in the queue (which is the ideal state).  This is why I constrain “target” to be at least 1.5 packet times at MTU; the difference can be substantial at low bandwidths.

But there are subtleties here too.

If there are multiple flows, isolated into multiple queues, then the effective packet-to-packet time for each queue will be increased proportionately.  Early versions of Codel refused to signal on the last packet in the queue, to account for this.  However, if there were a large number of occupied queues, this meant that the minimum queue fill could be rather high, and this seemed to lead to high induced latencies in fq_codel under heavy load.  Due to the statistical multiplexing effect, it turned out to be sufficient to tune “target” as above for the final output bandwidth (even though this is unknown to fq_codel) and to remove the special status of the last remaining packet.

The same logic could naively be applied to traffic in separate tins.  However, unlike queues for flow isolation, bandwidth is not shared evenly between tins.  More subtly, traffic characteristics also differ - low-latency traffic tends to be unresponsive to TCP-style congestion signalling, and dropping any of it tends to reduce service quality in some way.  Note that network-control traffic (most relevantly NTP) falls into the “voice” category.  Since unresponsive flows aren’t what Codel is meant to deal with, the mere fact that “target” is higher is not meaningful - and in any case this has no effect on the primary flow-isolation mechanism.

The tin-specific tuning of target and interval was introduced when Cake had a separate hard shaper per tin.  It was the obvious design at the time.  Now that Cake uses soft shaping between tins (allowing any tin to use the full link bandwidth if uncontended), it’s possible that choosing identical target and interval for all tins might suffice.  Alternatively, we might choose an even more conservative strategy.

But which - and how do we decide?

As a final point, I haven’t even mentioned “rtt” as the user-specified input to this mayhem.  That parameter must be understood to be *separate* from both “target” and “interval”, even though Codel specifies the latter to be related to expected RTT.  Simply put, the user tells us what the expected RTT is (on the understanding that an order of magnitude variation either way is typical), and we calculate “target” and “interval” to be as consistent with that estimate as is practical, given the link bandwidth and other constraints that he has also specified.  So there is a firm conceptual distinction between the user’s intent and the implementation details.

 - Jonathan Morton

Re: [Cake] More on 'target' corner cases - rate/target/interval confusion?

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 418 bytes --]

On 09/11/15 15:07, Jonathan Morton wrote:
> </snip>  Wrote a whole load of useful info
Jonathan,

Thank you so much for taking the time to explain all that, it's very
much appreciated.  There's a lot to digest and think about.  I shall now
go away and run it through the brain cell.......slowly.  If I come back
before 2 days have elapsed I haven't thought about it for long enough!

Many thanks,

Kevin


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4816 bytes --]

Re: [Cake] More on 'target' corner cases - rate/target/interval confusion?

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 7384 bytes --]

I've had a long think :-)

On 09/11/15 15:07, Jonathan Morton wrote:
>> On 9 Nov, 2015, at 14:12, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
>>
>> In the presence of a full link, that link having competing ‘full' flows in all 'tins', then how should cake split the link in terms of bandwidth?
> That’s a good question, and one that I think becomes more critical at low bandwidths.  I’ve tended towards generous allocations for that reason, so as to avoid causing trouble to low-latency applications.
>
> The main requirement I keep in mind is that an application should not be able to guarantee itself an excessive bandwidth share simply by selecting a particular DSCP.  At the same time, there are applications for which a relatively large, consistent bandwidth is a requirement for satisfactory performance (consider streaming video), such that best-effort traffic should defer to them.  These are conflicting requirements, so a compromise has to be established somehow.
>
> The current thresholds are at 100%, 15/16, ¾ and ¼.  Under saturated conditions, this gives throughputs of 1/16, 3/16, ½ and ¼.  The “video” class (Tin 2) can usurp ¾ of the bandwidth when competing against any mixture of best-effort and bulk traffic.  This, admittedly, might turn out to be too much, so I could consider setting Tin 2’s threshold at ½ instead of ¾.
Indeed it is a compromise :-)  Personally I think 3/4 for video is too
much.  Experimentally I found I could significantly reduce bulk & best
effort (down to 1/16th ish) by having video & voice tins fully active. 
So I tweaked my cake thresholds to set video to 1/2 rather than 3/4 as
you suggested.  Video & Voice can gang together to claim 3/4 bandwidth
still leaving 1/4ish to be battled between bulk & best effort, this is
in my opinion is much better than before.  Without a full voice tin,
video could only claim 1/2 in the presence of best effort traffic. 
There comes a point where the link bandwidth is so low you have to ask
if you're being realistic in trying to run video over it :-)
> And yes, I have long noticed that Flent’s standard RRUL test doesn’t use Tin 2 at all.
>
>> With each increasing priority of tini[0-3], we decrease the 'expected' bandwidth (good) but also as a result increase the target and interval.
> Yes, there are a number of counter-intuitive things happening here.
>
> Most of Cake’s latency-reducing power comes from the liberal application of flow isolation, *not* from AQM itself.  Diffserv prioritisation plays a lesser role, and mainly has to do with restoring the desired allocations of bandwidth, replacing the reliance on measure queue fill level that some protocols presently use to stay out from underfoot.  Both these mechanisms primarily control the effects that one flow can have on another, and say little about the latency that a flow causes to itself.
>
> This latter is the domain of AQM, specifically Codel, which is what we’re talking about when we mention “interval" and "target”.  As I mentioned elsewhere recently, Codel is designed specifically to give congestion signals to TCP-like flows, and deals rather less efficiently with unresponsive and anti-responsive flows, which as a result tend to spend some time bouncing off the queue’s hard limit until Codel finds the correct operating point to control them.  There are other AQMs which are designed with unresponsive flows more in mind, but which somehow perform less well with TCP-like flows.
>
> A key design principle of Codel is that no packet whose sojourn time is below target will be signalled.  However, if the sojourn time is consistently above target, signalling begins and increases steadily in frequency.  It is also a fundamental truth that if it takes longer than target to transmit the previous packet, the following packet can have a “congested” sojourn time even if there is consistently precisely one packet in the queue (which is the ideal state).  This is why I constrain “target” to be at least 1.5 packet times at MTU; the difference can be substantial at low bandwidths.
>
> But there are subtleties here too.
>
> If there are multiple flows, isolated into multiple queues, then the effective packet-to-packet time for each queue will be increased proportionately.  Early versions of Codel refused to signal on the last packet in the queue, to account for this.  However, if there were a large number of occupied queues, this meant that the minimum queue fill could be rather high, and this seemed to lead to high induced latencies in fq_codel under heavy load.  Due to the statistical multiplexing effect, it turned out to be sufficient to tune “target” as above for the final output bandwidth (even though this is unknown to fq_codel) and to remove the special status of the last remaining packet.
>
> The same logic could naively be applied to traffic in separate tins.  However, unlike queues for flow isolation, bandwidth is not shared evenly between tins.  More subtly, traffic characteristics also differ - low-latency traffic tends to be unresponsive to TCP-style congestion signalling, and dropping any of it tends to reduce service quality in some way.  Note that network-control traffic (most relevantly NTP) falls into the “voice” category.  Since unresponsive flows aren’t what Codel is meant to deal with, the mere fact that “target” is higher is not meaningful - and in any case this has no effect on the primary flow-isolation mechanism.
>
> The tin-specific tuning of target and interval was introduced when Cake had a separate hard shaper per tin.  It was the obvious design at the time.  Now that Cake uses soft shaping between tins (allowing any tin to use the full link bandwidth if uncontended), it’s possible that choosing identical target and interval for all tins might suffice.  Alternatively, we might choose an even more conservative strategy.
>
> But which - and how do we decide?
I've no idea!  I tried the following experiment:  set ingress & egress
b/widths to 2mbit & 448kbit respectively (like my parents adsl...on a
good day)  Ran a 'rrul8_splittin' and noted ping box_plot for each tin -
EF was about 30ms higher (120ms) than CS3 (90ms)  Thinking I'd found a
smoking gun and that 120ms was quite close to the EF target of 140ms I
tweaked code to copy tin 0 target across all tins, uploaded,flashed etc
etc, re-ran test......and........ ..... .....  it barely made any
difference at all!  Conclusion 1 - more testing required.  conclusion 2
- my testing strategy is flawed.  conclusion 3 - as you said, flow
isolation is more important than excessively messing with targets and
intervals ;-)


>
> As a final point, I haven’t even mentioned “rtt” as the user-specified input to this mayhem.  That parameter must be understood to be *separate* from both “target” and “interval”, even though Codel specifies the latter to be related to expected RTT.  Simply put, the user tells us what the expected RTT is (on the understanding that an order of magnitude variation either way is typical), and we calculate “target” and “interval” to be as consistent with that estimate as is practical, given the link bandwidth and other constraints that he has also specified.  So there is a firm conceptual distinction between the user’s intent and the implementation details.
>
>  - Jonathan Morton
>



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4816 bytes --]

Re: [Cake] More on 'target' corner cases - rate/target/interval confusion?

[Jonathan Morton]

> Video & Voice can gang together to claim 3/4 bandwidth
> still leaving 1/4ish to be battled between bulk & best effort…

That’s the intended behaviour of the current settings, since Voice borrows all the bandwidth it uses from Video’s allocation.

With Video reduced to a ½ allocation, Video + Voice should also be limited to ½ in total if there is competing saturating traffic.  It’s possible that you haven’t altered the “bandwidth balance” DRR weights to match the new allocations, which might cause such an artefact.

>> The tin-specific tuning of target and interval was introduced when Cake had a separate hard shaper per tin.  It was the obvious design at the time. Now that Cake uses soft shaping between tins (allowing any tin to use the full link bandwidth if uncontended), it’s possible that choosing identical target and interval for all tins might suffice.  Alternatively, we might choose an even more conservative strategy.
>> 
>> But which - and how do we decide?

> I've no idea!  I tried the following experiment:  set ingress & egress
> b/widths to 2mbit & 448kbit respectively (like my parents adsl...on a
> good day)  Ran a 'rrul8_splittin' and noted ping box_plot for each tin -
> EF was about 30ms higher (120ms) than CS3 (90ms)  Thinking I'd found a
> smoking gun and that 120ms was quite close to the EF target of 140ms I
> tweaked code to copy tin 0 target across all tins .....  it barely made any
> difference at all!

Since you’re using saturating traffic in all tins, Voice will be operating in “deprioritised” mode to penalise this abuse, so a higher latency is entirely expected.  If you delete the EF bulk flows but leave the latency measurements in place, you should see a big latency advantage.

As I tried to explain before, the Codel settings affect latency and throughput *within* flows rather than *between* them.  Since you are presently measuring using a between-flows methodology, it’s not surprising that you saw no difference.

 - Jonathan Morton

Re: [Cake] More on 'target' corner cases - rate/target/interval confusion?

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 3837 bytes --]



On 16/11/15 12:41, Jonathan Morton wrote:
>> Video & Voice can gang together to claim 3/4 bandwidth
>> still leaving 1/4ish to be battled between bulk & best effort…
> That’s the intended behaviour of the current settings, since Voice borrows all the bandwidth it uses from Video’s allocation.
>
> With Video reduced to a ½ allocation, Video + Voice should also be limited to ½ in total if there is competing saturating traffic.  It’s possible that you haven’t altered the “bandwidth balance” DRR weights to match the new allocations, which might cause such an artefact.
In which case, and this comes as no surprise at all, I really, really,
really don't have a clue how this works :-)  I apologise, can you explain?

I've noticed a difference between how the drr weights are calculated
between precedence/diffserv8 and diffserv4:

diffserv8:  does 7/8ths rate for each tin, 3/2 priority and 7/8
bandwidth sharing.  Rate & bandwidth sharing decrease, priority increases.

diffserv4:  (and this is from 'master' - none of my fiddling :-)

       /* class characteristics */
        cake_set_rate(&q->tins[0], rate, mtu,
                      US2TIME(q->target), US2TIME(q->interval));
        cake_set_rate(&q->tins[1], rate - (rate >> 4), mtu,
                      US2TIME(q->target), US2TIME(q->interval));
        cake_set_rate(&q->tins[2], rate - (rate >> 2), mtu,
                      US2TIME(q->target), US2TIME(q->interval));
        cake_set_rate(&q->tins[3], rate >> 2, mtu,
                      US2TIME(q->target), US2TIME(q->interval));

        /* priority weights */
        q->tins[0].tin_quantum_prio = quantum >> 4;
        q->tins[1].tin_quantum_prio = quantum;
        q->tins[2].tin_quantum_prio = quantum << 2;
        q->tins[3].tin_quantum_prio = quantum << 4;

        /* bandwidth-sharing weights */
        q->tins[0].tin_quantum_band = (quantum >> 4);
        q->tins[1].tin_quantum_band = (quantum >> 3) + (quantum >> 4);
        q->tins[2].tin_quantum_band = (quantum >> 1);
        q->tins[3].tin_quantum_band = (quantum >> 2);

Rate decreases. Priority increases. bandwidth sharing increases until
voice where it decreases again.  Bandwidth sharing mostly goes the
opposite way round to precedence/diffserv8 - that suggests one of them
is wrong.


>
>>> The tin-specific tuning of target and interval was introduced when Cake had a separate hard shaper per tin.  It was the obvious design at the time. Now that Cake uses soft shaping between tins (allowing any tin to use the full link bandwidth if uncontended), it’s possible that choosing identical target and interval for all tins might suffice.  Alternatively, we might choose an even more conservative strategy.
>>>
>>> But which - and how do we decide?
>> I've no idea!  I tried the following experiment:  set ingress & egress
>> b/widths to 2mbit & 448kbit respectively (like my parents adsl...on a
>> good day)  Ran a 'rrul8_splittin' and noted ping box_plot for each tin -
>> EF was about 30ms higher (120ms) than CS3 (90ms)  Thinking I'd found a
>> smoking gun and that 120ms was quite close to the EF target of 140ms I
>> tweaked code to copy tin 0 target across all tins .....  it barely made any
>> difference at all!
> Since you’re using saturating traffic in all tins, Voice will be operating in “deprioritised” mode to penalise this abuse, so a higher latency is entirely expected.  If you delete the EF bulk flows but leave the latency measurements in place, you should see a big latency advantage.
>
> As I tried to explain before, the Codel settings affect latency and throughput *within* flows rather than *between* them.  Since you are presently measuring using a between-flows methodology, it’s not surprising that you saw no difference.
>
>  - Jonathan Morton
>



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4816 bytes --]

Re: [Cake] More on 'target' corner cases - rate/target/interval confusion?

[Sebastian Moeller]

Hi Kevin, hi Jonathan,

On Nov 16, 2015, at 13:22 , Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:

> I've had a long think :-)
> 
> On 09/11/15 15:07, Jonathan Morton wrote:
>>> On 9 Nov, 2015, at 14:12, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
>>> 
>>> In the presence of a full link, that link having competing ‘full' flows in all 'tins', then how should cake split the link in terms of bandwidth?
>> That’s a good question, and one that I think becomes more critical at low bandwidths.  I’ve tended towards generous allocations for that reason, so as to avoid causing trouble to low-latency applications.
>> 
>> The main requirement I keep in mind is that an application should not be able to guarantee itself an excessive bandwidth share simply by selecting a particular DSCP.  At the same time, there are applications for which a relatively large, consistent bandwidth is a requirement for satisfactory performance (consider streaming video), such that best-effort traffic should defer to them.  These are conflicting requirements, so a compromise has to be established somehow.
>> 
>> The current thresholds are at 100%, 15/16, ¾ and ¼.  Under saturated conditions, this gives throughputs of 1/16, 3/16, ½ and ¼.  The “video” class (Tin 2) can usurp ¾ of the bandwidth when competing against any mixture of best-effort and bulk traffic.  This, admittedly, might turn out to be too much, so I could consider setting Tin 2’s threshold at ½ instead of ¾.
> Indeed it is a compromise :-)  Personally I think 3/4 for video is too
> much.  Experimentally I found I could significantly reduce bulk & best
> effort (down to 1/16th ish) by having video & voice tins fully active. 
> So I tweaked my cake thresholds to set video to 1/2 rather than 3/4 as
> you suggested.  Video & Voice can gang together to claim 3/4 bandwidth
> still leaving 1/4ish to be battled between bulk & best effort, this is
> in my opinion is much better than before.  Without a full voice tin,
> video could only claim 1/2 in the presence of best effort traffic. 
> There comes a point where the link bandwidth is so low you have to ask
> if you're being realistic in trying to run video over it :-)

	I just want to note that the rule of thumb for VoIP is 100Kbps per conversation and direction; if we assume that VoIP will be inside the VO tier than we should make sure that we have at least 100Kbps available in both directions, no? I am not opposed to ample bandwidth in the VI tier, that will allow hoisting all relevant stuff there while leaving the rest including bit torrent in CS0 or CS1 country. (As far as I can tell, which is not that far admittedly, no bit torrent client allows setting DSCP marlins or defaults to CS1, so waiting for them to move themselves to the BK tier is right up there with waiting for Godot… So arguably it seems more worthwhile to move everything else up into a better tier, offering nicer bandwidth might be an incentive to make apps look at the problem space…)


Best Regards
	Sebastian

>> And yes, I have long noticed that Flent’s standard RRUL test doesn’t use Tin 2 at all.
>> 
>>> With each increasing priority of tini[0-3], we decrease the 'expected' bandwidth (good) but also as a result increase the target and interval.
>> Yes, there are a number of counter-intuitive things happening here.
>> 
>> Most of Cake’s latency-reducing power comes from the liberal application of flow isolation, *not* from AQM itself.  Diffserv prioritisation plays a lesser role, and mainly has to do with restoring the desired allocations of bandwidth, replacing the reliance on measure queue fill level that some protocols presently use to stay out from underfoot.  Both these mechanisms primarily control the effects that one flow can have on another, and say little about the latency that a flow causes to itself.
>> 
>> This latter is the domain of AQM, specifically Codel, which is what we’re talking about when we mention “interval" and "target”.  As I mentioned elsewhere recently, Codel is designed specifically to give congestion signals to TCP-like flows, and deals rather less efficiently with unresponsive and anti-responsive flows, which as a result tend to spend some time bouncing off the queue’s hard limit until Codel finds the correct operating point to control them.  There are other AQMs which are designed with unresponsive flows more in mind, but which somehow perform less well with TCP-like flows.
>> 
>> A key design principle of Codel is that no packet whose sojourn time is below target will be signalled.  However, if the sojourn time is consistently above target, signalling begins and increases steadily in frequency.  It is also a fundamental truth that if it takes longer than target to transmit the previous packet, the following packet can have a “congested” sojourn time even if there is consistently precisely one packet in the queue (which is the ideal state).  This is why I constrain “target” to be at least 1.5 packet times at MTU; the difference can be substantial at low bandwidths.
>> 
>> But there are subtleties here too.
>> 
>> If there are multiple flows, isolated into multiple queues, then the effective packet-to-packet time for each queue will be increased proportionately.  Early versions of Codel refused to signal on the last packet in the queue, to account for this.  However, if there were a large number of occupied queues, this meant that the minimum queue fill could be rather high, and this seemed to lead to high induced latencies in fq_codel under heavy load.  Due to the statistical multiplexing effect, it turned out to be sufficient to tune “target” as above for the final output bandwidth (even though this is unknown to fq_codel) and to remove the special status of the last remaining packet.
>> 
>> The same logic could naively be applied to traffic in separate tins.  However, unlike queues for flow isolation, bandwidth is not shared evenly between tins.  More subtly, traffic characteristics also differ - low-latency traffic tends to be unresponsive to TCP-style congestion signalling, and dropping any of it tends to reduce service quality in some way.  Note that network-control traffic (most relevantly NTP) falls into the “voice” category.  Since unresponsive flows aren’t what Codel is meant to deal with, the mere fact that “target” is higher is not meaningful - and in any case this has no effect on the primary flow-isolation mechanism.
>> 
>> The tin-specific tuning of target and interval was introduced when Cake had a separate hard shaper per tin.  It was the obvious design at the time.  Now that Cake uses soft shaping between tins (allowing any tin to use the full link bandwidth if uncontended), it’s possible that choosing identical target and interval for all tins might suffice.  Alternatively, we might choose an even more conservative strategy.
>> 
>> But which - and how do we decide?
> I've no idea!  I tried the following experiment:  set ingress & egress
> b/widths to 2mbit & 448kbit respectively (like my parents adsl...on a
> good day)  Ran a 'rrul8_splittin' and noted ping box_plot for each tin -
> EF was about 30ms higher (120ms) than CS3 (90ms)  Thinking I'd found a
> smoking gun and that 120ms was quite close to the EF target of 140ms I
> tweaked code to copy tin 0 target across all tins, uploaded,flashed etc
> etc, re-ran test......and........ ..... .....  it barely made any
> difference at all!  Conclusion 1 - more testing required.  conclusion 2
> - my testing strategy is flawed.  conclusion 3 - as you said, flow
> isolation is more important than excessively messing with targets and
> intervals ;-)
> 
> 
>> 
>> As a final point, I haven’t even mentioned “rtt” as the user-specified input to this mayhem.  That parameter must be understood to be *separate* from both “target” and “interval”, even though Codel specifies the latter to be related to expected RTT.  Simply put, the user tells us what the expected RTT is (on the understanding that an order of magnitude variation either way is typical), and we calculate “target” and “interval” to be as consistent with that estimate as is practical, given the link bandwidth and other constraints that he has also specified.  So there is a firm conceptual distinction between the user’s intent and the implementation details.
>> 
>> - Jonathan Morton
>> 
> 
> 
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] More on 'target' corner cases - rate/target/interval confusion?

[Sebastian Moeller]

On Nov 9, 2015, at 16:07 , Jonathan Morton <chromatix99@gmail.com> wrote:

> 
>> On 9 Nov, 2015, at 14:12, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
>> 
>> In the presence of a full link, that link having competing ‘full' flows in all 'tins', then how should cake split the link in terms of bandwidth?
> 
> That’s a good question, and one that I think becomes more critical at low bandwidths.  I’ve tended towards generous allocations for that reason, so as to avoid causing trouble to low-latency applications.
> 
> The main requirement I keep in mind is that an application should not be able to guarantee itself an excessive bandwidth share simply by selecting a particular DSCP.  At the same time, there are applications for which a relatively large, consistent bandwidth is a requirement for satisfactory performance (consider streaming video), such that best-effort traffic should defer to them.  These are conflicting requirements, so a compromise has to be established somehow.

	Well, or this can be turned into a policy decision that the user has to make; as always cake should offer a reasonable default ;)


> 
> The current thresholds are at 100%, 15/16, ¾ and ¼.  Under saturated conditions, this gives throughputs of 1/16, 3/16, ½ and ¼.  The “video” class (Tin 2) can usurp ¾ of the bandwidth when competing against any mixture of best-effort and bulk traffic.  This, admittedly, might turn out to be too much, so I could consider setting Tin 2’s threshold at ½ instead of ¾.
> 
> And yes, I have long noticed that Flent’s standard RRUL test doesn’t use Tin 2 at all.
> 
>> With each increasing priority of tini[0-3], we decrease the 'expected' bandwidth (good) but also as a result increase the target and interval.
> 
> Yes, there are a number of counter-intuitive things happening here.
> 
> Most of Cake’s latency-reducing power comes from the liberal application of flow isolation, *not* from AQM itself.  Diffserv prioritisation plays a lesser role, and mainly has to do with restoring the desired allocations of bandwidth, replacing the reliance on measure queue fill level that some protocols presently use to stay out from underfoot.  Both these mechanisms primarily control the effects that one flow can have on another, and say little about the latency that a flow causes to itself.
> 
> This latter is the domain of AQM, specifically Codel, which is what we’re talking about when we mention “interval" and "target”.  As I mentioned elsewhere recently, Codel is designed specifically to give congestion signals to TCP-like flows, and deals rather less efficiently with unresponsive and anti-responsive flows, which as a result tend to spend some time bouncing off the queue’s hard limit until Codel finds the correct operating point to control them.  There are other AQMs which are designed with unresponsive flows more in mind, but which somehow perform less well with TCP-like flows.
> 
> A key design principle of Codel is that no packet whose sojourn time is below target will be signalled.  However, if the sojourn time is consistently above target, signalling begins and increases steadily in frequency.  It is also a fundamental truth that if it takes longer than target to transmit the previous packet, the following packet can have a “congested” sojourn time even if there is consistently precisely one packet in the queue (which is the ideal state).  This is why I constrain “target” to be at least 1.5 packet times at MTU; the difference can be substantial at low bandwidths.
> 
> But there are subtleties here too.
> 
> If there are multiple flows, isolated into multiple queues, then the effective packet-to-packet time for each queue will be increased proportionately.  Early versions of Codel refused to signal on the last packet in the queue, to account for this.  However, if there were a large number of occupied queues, this meant that the minimum queue fill could be rather high, and this seemed to lead to high induced latencies in fq_codel under heavy load.  Due to the statistical multiplexing effect, it turned out to be sufficient to tune “target” as above for the final output bandwidth (even though this is unknown to fq_codel) and to remove the special status of the last remaining packet.
> 
> The same logic could naively be applied to traffic in separate tins.  However, unlike queues for flow isolation, bandwidth is not shared evenly between tins.  More subtly, traffic characteristics also differ - low-latency traffic tends to be unresponsive to TCP-style congestion signalling, and dropping any of it tends to reduce service quality in some way.  Note that network-control traffic (most relevantly NTP) falls into the “voice” category.  Since unresponsive flows aren’t what Codel is meant to deal with, the mere fact that “target” is higher is not meaningful - and in any case this has no effect on the primary flow-isolation mechanism.

	Well, but not everything in the higher priority tiers is not-TCP, I believe even DNS can be operated over TCP, no? And not all non-TCP flows are unresponsive, so there s still some relevance to target of the sub tiers. I have bee arguing for the extend interval to a fixed multiple of target and adapt target to the minimal transfer time in the past, but I am not certain whether that yields better results than just letting target grow higher as long as target < interval. (In the end interval is en estimate of the reaction time window we allow the flows to receive the congestion signals, and the window does just increase as MTU_transfer_time + interval)

> 
> The tin-specific tuning of target and interval was introduced when Cake had a separate hard shaper per tin.  It was the obvious design at the time.  Now that Cake uses soft shaping between tins (allowing any tin to use the full link bandwidth if uncontended), it’s possible that choosing identical target and interval for all tins might suffice.  Alternatively, we might choose an even more conservative strategy.
> 
> But which - and how do we decide?
> 
> As a final point, I haven’t even mentioned “rtt” as the user-specified input to this mayhem.  That parameter must be understood to be *separate* from both “target” and “interval”, even though Codel specifies the latter to be related to expected RTT.  Simply put, the user tells us what the expected RTT is (on the understanding that an order of magnitude variation either way is typical), and we calculate “target” and “interval” to be as consistent with that estimate as is practical, given the link bandwidth and other constraints that he has also specified.  So there is a firm conceptual distinction between the user’s intent and the implementation details.

	Now, I believe the symbolic arguments like interplanetary are a great and obvious way to signal intent, numerically specified parameters however should be honored as well as possible, and changes made to these by cake should be signaled back to the user, preferably with stating the reason….

Best Regards
	Sebastian

> 
> - Jonathan Morton
> 
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake