[Cake] making split_gso_threshold configurable

[Cake] making split_gso_threshold configurable

[Dave Taht]

I really wanted cake to always optimize for low latency. I wanted it to
"just work" at line rate on dsl, on 100mbit, even 10mbit ethernet, to work
against pause frames, etc, without configuration. I wanted to defeat
drivers like the mvneta that can do 64k of software GRO automagically.

Along the way to mainlining it, cake's behavior got changed, to scale
well to 50Gbit by allowing GSO above a 1gbit shaped rate, and *always
allowing GSO at line rate*.

        if (q->rate_bps && q->rate_bps <= CAKE_SPLIT_GSO_THRESHOLD)
                q->rate_flags |= CAKE_FLAG_SPLIT_GSO;
        else
                q->rate_flags &= ~CAKE_FLAG_SPLIT_GSO;

I don't care about 50gbit. I'm delighted, even astonished, it does
50Gbit. I want it to do 1mbit with 13ms interpacket latency when hit
by an IW10 burst from somewhere else, not 130, with hardware pause
frames in use on the dsl modem or elsewhere.

so, I'd like to A) default to always split shaped or unshaped. And B)
expose the split threshold to be configured via userspace. This latter
option would give a tunable for those struggling a bit for cpu to play
with, also.

anyone disagree?




-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] making split_gso_threshold configurable

[Toke Høiland-Jørgensen]

Dave Taht <dave.taht@gmail.com> writes:

> I really wanted cake to always optimize for low latency. I wanted it
> to "just work" at line rate on dsl, on 100mbit, even 10mbit ethernet,
> to work against pause frames, etc, without configuration. I wanted to
> defeat drivers like the mvneta that can do 64k of software GRO
> automagically.

So I think that Eric's point was that the GSO logic itself should
throttle back and not build so big packets when the link rate is lower.
Might be worth it to check if this is actually the case, and if not to
get GSO fixed rather than disabling it.

Also, are there actually any modems that will negotiate ethernet line
rates less than a gigabit?

> Along the way to mainlining it, cake's behavior got changed, to scale
> well to 50Gbit by allowing GSO above a 1gbit shaped rate, and *always
> allowing GSO at line rate*.
>
>         if (q->rate_bps && q->rate_bps <= CAKE_SPLIT_GSO_THRESHOLD)
>                 q->rate_flags |= CAKE_FLAG_SPLIT_GSO;
>         else
>                 q->rate_flags &= ~CAKE_FLAG_SPLIT_GSO;
>
> I don't care about 50gbit. I'm delighted, even astonished, it does
> 50Gbit. I want it to do 1mbit with 13ms interpacket latency when hit
> by an IW10 burst from somewhere else, not 130, with hardware pause
> frames in use on the dsl modem or elsewhere.

Getting pause frames to work properly in this setup kinda requires the
buffering in the modem to be reasonable in the first place...

> so, I'd like to A) default to always split shaped or unshaped. And B)
> expose the split threshold to be configured via userspace. This latter
> option would give a tunable for those struggling a bit for cpu to play
> with, also.

I very much doubt that we can get upstream to accept split as the
default. It may be possible to get it to be user tunable. I don't think
we should deviate from upstream unless we absolutely have to (that was
kinda the whole point of the upstreaming exercise), and I'm not
convinced that this is a good enough reason...

-Toke

Re: [Cake] making split_gso_threshold configurable

[Sebastian Moeller]

Hi Toke,


> On Jul 25, 2018, at 12:02, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Dave Taht <dave.taht@gmail.com> writes:
> 
>> I really wanted cake to always optimize for low latency. I wanted it
>> to "just work" at line rate on dsl, on 100mbit, even 10mbit ethernet,
>> to work against pause frames, etc, without configuration. I wanted to
>> defeat drivers like the mvneta that can do 64k of software GRO
>> automagically.
> 
> So I think that Eric's point was that the GSO logic itself should
> throttle back and not build so big packets when the link rate is lower.
> Might be worth it to check if this is actually the case, and if not to
> get GSO fixed rather than disabling it.
> 
> Also, are there actually any modems that will negotiate ethernet line
> rates less than a gigabit?

	Yes, there are XDSL Modem(routers) that only have fast-ethernet ports. Not sure how rare these are though.

Best Regards
	Sebastian


> 
>> Along the way to mainlining it, cake's behavior got changed, to scale
>> well to 50Gbit by allowing GSO above a 1gbit shaped rate, and *always
>> allowing GSO at line rate*.
>> 
>>        if (q->rate_bps && q->rate_bps <= CAKE_SPLIT_GSO_THRESHOLD)
>>                q->rate_flags |= CAKE_FLAG_SPLIT_GSO;
>>        else
>>                q->rate_flags &= ~CAKE_FLAG_SPLIT_GSO;
>> 
>> I don't care about 50gbit. I'm delighted, even astonished, it does
>> 50Gbit. I want it to do 1mbit with 13ms interpacket latency when hit
>> by an IW10 burst from somewhere else, not 130, with hardware pause
>> frames in use on the dsl modem or elsewhere.
> 
> Getting pause frames to work properly in this setup kinda requires the
> buffering in the modem to be reasonable in the first place...
> 
>> so, I'd like to A) default to always split shaped or unshaped. And B)
>> expose the split threshold to be configured via userspace. This latter
>> option would give a tunable for those struggling a bit for cpu to play
>> with, also.
> 
> I very much doubt that we can get upstream to accept split as the
> default. It may be possible to get it to be user tunable. I don't think
> we should deviate from upstream unless we absolutely have to (that was
> kinda the whole point of the upstreaming exercise), and I'm not
> convinced that this is a good enough reason...
> 
> -Toke
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] making split_gso_threshold configurable

[Dave Taht]

On Wed, Jul 25, 2018 at 3:07 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>
> Hi Toke,
>
>
> > On Jul 25, 2018, at 12:02, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> >
> > Dave Taht <dave.taht@gmail.com> writes:
> >
> >> I really wanted cake to always optimize for low latency. I wanted it
> >> to "just work" at line rate on dsl, on 100mbit, even 10mbit ethernet,
> >> to work against pause frames, etc, without configuration. I wanted to
> >> defeat drivers like the mvneta that can do 64k of software GRO
> >> automagically.
> >
> > So I think that Eric's point was that the GSO logic itself should
> > throttle back and not build so big packets when the link rate is lower.
> > Might be worth it to check if this is actually the case, and if not to
> > get GSO fixed rather than disabling it.

It doesn't and I think I should just put forth the patch and face the
howling from the 40gbit guys.

> >
> > Also, are there actually any modems that will negotiate ethernet line
> > rates less than a gigabit?
>
>         Yes, there are XDSL Modem(routers) that only have fast-ethernet ports. Not sure how rare these are though.

And there are also many devices that ecert hardware flow control.

> Best Regards
>         Sebastian
>
>
> >
> >> Along the way to mainlining it, cake's behavior got changed, to scale
> >> well to 50Gbit by allowing GSO above a 1gbit shaped rate, and *always
> >> allowing GSO at line rate*.
> >>
> >>        if (q->rate_bps && q->rate_bps <= CAKE_SPLIT_GSO_THRESHOLD)
> >>                q->rate_flags |= CAKE_FLAG_SPLIT_GSO;
> >>        else
> >>                q->rate_flags &= ~CAKE_FLAG_SPLIT_GSO;
> >>
> >> I don't care about 50gbit. I'm delighted, even astonished, it does
> >> 50Gbit. I want it to do 1mbit with 13ms interpacket latency when hit
> >> by an IW10 burst from somewhere else, not 130, with hardware pause
> >> frames in use on the dsl modem or elsewhere.
> >
> > Getting pause frames to work properly in this setup kinda requires the
> > buffering in the modem to be reasonable in the first place...
> >
> >> so, I'd like to A) default to always split shaped or unshaped. And B)
> >> expose the split threshold to be configured via userspace. This latter
> >> option would give a tunable for those struggling a bit for cpu to play
> >> with, also.
> >
> > I very much doubt that we can get upstream to accept split as the
> > default. It may be possible to get it to be user tunable. I don't think
> > we should deviate from upstream unless we absolutely have to (that was
> > kinda the whole point of the upstreaming exercise), and I'm not
> > convinced that this is a good enough reason...
> >
> > -Toke
> > _______________________________________________
> > Cake mailing list
> > Cake@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cake
>


-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] making split_gso_threshold configurable

[Toke Høiland-Jørgensen]

Dave Taht <dave.taht@gmail.com> writes:

> On Wed, Jul 25, 2018 at 3:07 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>>
>> Hi Toke,
>>
>>
>> > On Jul 25, 2018, at 12:02, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> >
>> > Dave Taht <dave.taht@gmail.com> writes:
>> >
>> >> I really wanted cake to always optimize for low latency. I wanted it
>> >> to "just work" at line rate on dsl, on 100mbit, even 10mbit ethernet,
>> >> to work against pause frames, etc, without configuration. I wanted to
>> >> defeat drivers like the mvneta that can do 64k of software GRO
>> >> automagically.
>> >
>> > So I think that Eric's point was that the GSO logic itself should
>> > throttle back and not build so big packets when the link rate is lower.
>> > Might be worth it to check if this is actually the case, and if not to
>> > get GSO fixed rather than disabling it.
>
> It doesn't and I think I should just put forth the patch and face the
> howling from the 40gbit guys.

Well, feel free I guess? :)

>> > Also, are there actually any modems that will negotiate ethernet line
>> > rates less than a gigabit?
>>
>>         Yes, there are XDSL Modem(routers) that only have
>>         fast-ethernet ports. Not sure how rare these are though.
>
> And there are also many devices that ecert hardware flow control.

But why would you ever run with such a device in a setup where flow
control gets a chance to kick in? Wouldn't you turn on the shaper anyway
(in which case you already get GSO splitting)?

-Toke