Re: [Cake] [RFC PATCH 4/5] q_netem: support delivering packets in delayed time slots

Re: [Cake] [RFC PATCH 4/5] q_netem: support delivering packets in delayed time slots

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 1252 bytes --]


> On Nov 17, 2017, at 11:55 PM,dave.taht@gmail.com wrote:
> 
> Slotting is a crude approximation of the behaviors of shared media such
> as cable, wifi, and LTE, which gather up a bunch of packets within a
> varying delay window and deliver them, relative to that, nearly all at
> once.

Nice…

One of the things I also notice in my LAN tests is latencies for different flows staying at more or less fixed (and different) positions relative to the mean in flent results. Those positions, and the mean, can change with each test run. Do you think this could result from the hashing to different hardware queues (four in my case) changing between test runs? And is it worth trying to simulate this effect, or not really?

Just for info, in my case (Intel i210) the hashing is documented starting on page 254 of the specs: https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/i210-ethernet-controller-datasheet.pdf <https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/i210-ethernet-controller-datasheet.pdf> (7.1.2.10.1 RSS Hash Function). For TCP/UDP it uses source and destination addresses and ports. I suppose this could be smoothed over in testing by using a spread of ports for the latency test.

[-- Attachment #2: Type: text/html, Size: 1853 bytes --]

Re: [Cake] [RFC PATCH 4/5] q_netem: support delivering packets in delayed time slots

[Dave Taht]

On Sat, Nov 18, 2017 at 5:18 AM, Pete Heist <peteheist@gmail.com> wrote:
>
> On Nov 17, 2017, at 11:55 PM,dave.taht@gmail.com wrote:
>
>
> Slotting is a crude approximation of the behaviors of shared media such
> as cable, wifi, and LTE, which gather up a bunch of packets within a
> varying delay window and deliver them, relative to that, nearly all at
> once.
>
>
> Nice…

Meh. It really is "crude", and I keeping kicking about ways to somehow
emulate half (or less) duplex, variable rates around a mean, mcast,
etc.

it IS very nice to have a rate limiter that actually behaves a bit
more like wifi, and I hope to also add the new ack fitering stuff to
it.

>
> One of the things I also notice in my LAN tests is latencies for different
> flows staying at more or less fixed (and different) positions relative to
> the mean in flent results. Those positions, and the mean, can change with
> each test run. Do you think this could result from the hashing to different
> hardware queues (four in my case) changing between test runs?

yes if you are using bql probably. Is it sch_mq on top?

> And is it
> worth trying to simulate this effect, or not really?

Dunno. There are a couple ways to turn it off.

> Just for info, in my case (Intel i210) the hashing is documented starting on
> page 254 of the specs:
> https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/i210-ethernet-controller-datasheet.pdf
> (7.1.2.10.1 RSS Hash Function). For TCP/UDP it uses source and destination
> addresses and ports. I suppose this could be smoothed over in testing by
> using a spread of ports for the latency test.



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] [RFC PATCH 4/5] q_netem: support delivering packets in delayed time slots

[Pete Heist]

> On Nov 18, 2017, at 8:02 PM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> On Sat, Nov 18, 2017 at 5:18 AM, Pete Heist <peteheist@gmail.com> wrote:
>> 
>> On Nov 17, 2017, at 11:55 PM,dave.taht@gmail.com wrote:
>> 
>> Slotting is a crude approximation of the behaviors of shared media such
>> as cable, wifi, and LTE, which gather up a bunch of packets within a
>> varying delay window and deliver them, relative to that, nearly all at
>> once.
>> 
>> Nice…
> 
> Meh. It really is "crude", and I keeping kicking about ways to somehow
> emulate half (or less) duplex, variable rates around a mean, mcast,
> etc.
> 
> it IS very nice to have a rate limiter that actually behaves a bit
> more like wifi, and I hope to also add the new ack fitering stuff to
> it.

I guess there may never be a way to make this perfect, only to try to reproduce the behavior that matters enough to make it usable for testing.

>> One of the things I also notice in my LAN tests is latencies for different
>> flows staying at more or less fixed (and different) positions relative to
>> the mean in flent results. Those positions, and the mean, can change with
>> each test run. Do you think this could result from the hashing to different
>> hardware queues (four in my case) changing between test runs?
> 
> yes if you are using bql probably. Is it sch_mq on top?

Yep with bql. I hadn’t thought about mq before. What my qos setup scripts are doing though is replacing the root qdisc (which now I see defaults to mq) with a single cake instance. With bql, should rather be leaving mq and putting four cake instances underneath it?

>> And is it
>> worth trying to simulate this effect, or not really?
> 
> Dunno. There are a couple ways to turn it off.

Fair enough...

Re: [Cake] [RFC PATCH 4/5] q_netem: support delivering packets in delayed time slots

[Dave Taht]

Pete Heist <peteheist@gmail.com> writes:

>> On Nov 18, 2017, at 8:02 PM, Dave Taht <dave.taht@gmail.com> wrote:
>> 
>> On Sat, Nov 18, 2017 at 5:18 AM, Pete Heist <peteheist@gmail.com> wrote:
>>> 
>>> On Nov 17, 2017, at 11:55 PM,dave.taht@gmail.com wrote:
>>> 
>>> Slotting is a crude approximation of the behaviors of shared media such
>>> as cable, wifi, and LTE, which gather up a bunch of packets within a
>>> varying delay window and deliver them, relative to that, nearly all at
>>> once.
>>> 
>>> Nice…
>> 
>> Meh. It really is "crude", and I keeping kicking about ways to somehow
>> emulate half (or less) duplex, variable rates around a mean, mcast,
>> etc.
>> 
>> it IS very nice to have a rate limiter that actually behaves a bit
>> more like wifi, and I hope to also add the new ack fitering stuff to
>> it.
>
> I guess there may never be a way to make this perfect, only to try to reproduce
> the behavior that matters enough to make it usable for testing.
>
>>> One of the things I also notice in my LAN tests is latencies for different
>>> flows staying at more or less fixed (and different) positions relative to
>>> the mean in flent results. Those positions, and the mean, can change with
>>> each test run. Do you think this could result from the hashing to different
>>> hardware queues (four in my case) changing between test runs?
>> 
>> yes if you are using bql probably. Is it sch_mq on top?

Only if you want unlimited mode, and birthday problems, and don't have
cpu to burn.

>
> Yep with bql. I hadn’t thought about mq before. What my qos setup scripts are
> doing though is replacing the root qdisc (which now I see defaults to mq) with a
> single cake instance. With bql, should rather be leaving mq and putting four
> cake instances underneath it?
>
>>> And is it
>>> worth trying to simulate this effect, or not really?
>> 
>> Dunno. There are a couple ways to turn it off.
>
> Fair enough...
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] [RFC PATCH 4/5] q_netem: support delivering packets in delayed time slots

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 647 bytes --]


> On Nov 19, 2017, at 9:41 PM, Dave Taht <dave@taht.net> wrote:
> 
>> Yep with bql. I hadn’t thought about mq before. What my qos setup scripts are
>> doing though is replacing the root qdisc (which now I see defaults to mq) with a
>> single cake instance. With bql, should rather be leaving mq and putting four
>> cake instances underneath it?
> Only if you want unlimited mode, and birthday problems, and don't have
> cpu to burn.

Ok, if I compare the two, latency under load for rrul_be looks “better” (~1-1.5ms instead of ~3ms) with a single “cake unlimited besteffort lan” instance instead of four of them underneath mq.

[-- Attachment #2: Type: text/html, Size: 2291 bytes --]

Re: [Cake] [RFC PATCH 4/5] q_netem: support delivering packets in delayed time slots

[Dave Taht]

Pete Heist <peteheist@gmail.com> writes:

>     On Nov 19, 2017, at 9:41 PM, Dave Taht <dave@taht.net> wrote:
>
>     
>     
>     Yep with bql. I hadn’t thought about mq before. What my qos setup scripts
>         are
>         doing though is replacing the root qdisc (which now I see defaults to
>         mq) with a
>         single cake instance. With bql, should rather be leaving mq and putting
>         four
>         cake instances underneath it?
>
>     Only if you want unlimited mode, and birthday problems, and don't have
>     cpu to burn.
>     
>
> Ok, if I compare the two, latency under load for rrul_be looks “better”
> (~1-1.5ms instead of ~3ms) with a single “cake unlimited besteffort lan”
> instance instead of four of them underneath mq.

Yep. Adding more unmanaged queues doesn't help. However it should spread
the load across more cpus.

[Cake] Cleaning up cake

[Dave Taht]

I just finished a brutal exercise in making the cobalt branch of cake
vastly more checkpatch compliant. (I haven't checked it in yet)

What I am inclined to do next is move it to sch_cobalt.c, and reimport
(somehow) sch_cake.c from the stable commit that is currently shipping
in lede (?), and move that forward to also gain ack filtering.

Then add a q_cobalt equivalent to my iproute2 so we can build, test,
and compare the two versions, sanely.

I'd call this new branch "kobold". Or I'd find some way to fix head.