[Cake] tossing acks into the background queue

[Cake] tossing acks into the background queue

[Dave Taht]

ages ago I'd (we'd? I really don't remember - forgive me if I've
forgotten who actually leaned in on it) written a basic ack-filter in
ebpf. this was before cake gained tc actions and my primary use for
the tech was for asymmetric connections, and before the good
ack-filter arrived, and I was (and remain) unfriendly to this level of
dpi.

That said, on a symmetric connection, deprioritizing pure acks to the
5% background queue nd then turning the cake ack-filter loose on it
might actually work.

Am I on drugs/is there any point?



-- 
I tried to build a better future, a few times:
https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org

Dave Täht CEO, TekLibre, LLC

Re: [Cake] tossing acks into the background queue

[Sebastian Moeller]

Hi Dave,


On 23 November 2021 06:03:03 CET, Dave Taht <dave.taht@gmail.com> wrote:
>ages ago I'd (we'd? I really don't remember - forgive me if I've
>forgotten who actually leaned in on it) written a basic ack-filter in
>ebpf. this was before cake gained tc actions and my primary use for
>the tech was for asymmetric connections, and before the good
>ack-filter arrived, and I was (and remain) unfriendly to this level of
>dpi.
>
>That said, on a symmetric connection, deprioritizing pure acks to the
>5% background queue nd then turning the cake ack-filter loose on it
>might actually work.
>
>Am I on drugs/is there any point?

I think at leat when using multiple priority tins forward and reverse traffic should by default use the same tin (I can see non-standard situations that want differential treatment). The argument is that unlike earlier attempts at ingress shaping that tried to throttle reverse ACKs? cake/codel do proper 'hit the brakes' signalling via marking/dropping and we want that signal to reach the other end as quickly as possible, no?

Regards
        Sebastian


>
>
>
>-- 
>I tried to build a better future, a few times:
>https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
>
>Dave Täht CEO, TekLibre, LLC
>_______________________________________________
>Cake mailing list
>Cake@lists.bufferbloat.net
>https://lists.bufferbloat.net/listinfo/cake

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [Cake] tossing acks into the background queue

[Dave Taht]

On Mon, Nov 22, 2021 at 11:07 PM Sebastian Moeller <moeller0@gmx.de> wrote:
>
> Hi Dave,
>
>
> On 23 November 2021 06:03:03 CET, Dave Taht <dave.taht@gmail.com> wrote:
> >ages ago I'd (we'd? I really don't remember - forgive me if I've
> >forgotten who actually leaned in on it) written a basic ack-filter in
> >ebpf. this was before cake gained tc actions and my primary use for
> >the tech was for asymmetric connections, and before the good
> >ack-filter arrived, and I was (and remain) unfriendly to this level of
> >dpi.
> >
> >That said, on a symmetric connection, deprioritizing pure acks to the
> >5% background queue nd then turning the cake ack-filter loose on it
> >might actually work.
> >
> >Am I on drugs/is there any point?
>
> I think at leat when using multiple priority tins forward and reverse traffic should by default use the same tin (I can see non-standard situations that want differential treatment). The argument is that unlike earlier attempts at ingress shaping that tried to throttle reverse ACKs? cake/codel do proper 'hit the brakes' signalling via marking/dropping and we want that signal to reach the other end as quickly as possible, no?

My thought was basically an optional filter that steered all pure acks
(no matter the classification) into the background queue.
Non-pure-acks (sacks) essentially jump the background queue and signal
that loss earlier. The backlog of other acks in background get
delivered out of order, but purely out of order and discarded by the
reciever.

> Regards
>         Sebastian
>
>
> >
> >
> >
> >--
> >I tried to build a better future, a few times:
> >https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
> >
> >Dave Täht CEO, TekLibre, LLC
> >_______________________________________________
> >Cake mailing list
> >Cake@lists.bufferbloat.net
> >https://lists.bufferbloat.net/listinfo/cake
>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.



-- 
I tried to build a better future, a few times:
https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org

Dave Täht CEO, TekLibre, LLC

Re: [Cake] tossing acks into the background queue

[Dave Taht]

The context of my question is basically this:

Is cake baked? Is it done?

Is there anything from libreQos that would be better in C?

On Mon, Nov 22, 2021 at 11:17 PM Dave Taht <dave.taht@gmail.com> wrote:
>
> On Mon, Nov 22, 2021 at 11:07 PM Sebastian Moeller <moeller0@gmx.de> wrote:
> >
> > Hi Dave,
> >
> >
> > On 23 November 2021 06:03:03 CET, Dave Taht <dave.taht@gmail.com> wrote:
> > >ages ago I'd (we'd? I really don't remember - forgive me if I've
> > >forgotten who actually leaned in on it) written a basic ack-filter in
> > >ebpf. this was before cake gained tc actions and my primary use for
> > >the tech was for asymmetric connections, and before the good
> > >ack-filter arrived, and I was (and remain) unfriendly to this level of
> > >dpi.
> > >
> > >That said, on a symmetric connection, deprioritizing pure acks to the
> > >5% background queue nd then turning the cake ack-filter loose on it
> > >might actually work.
> > >
> > >Am I on drugs/is there any point?
> >
> > I think at leat when using multiple priority tins forward and reverse traffic should by default use the same tin (I can see non-standard situations that want differential treatment). The argument is that unlike earlier attempts at ingress shaping that tried to throttle reverse ACKs? cake/codel do proper 'hit the brakes' signalling via marking/dropping and we want that signal to reach the other end as quickly as possible, no?
>
> My thought was basically an optional filter that steered all pure acks
> (no matter the classification) into the background queue.
> Non-pure-acks (sacks) essentially jump the background queue and signal
> that loss earlier. The backlog of other acks in background get
> delivered out of order, but purely out of order and discarded by the
> reciever.
>
> > Regards
> >         Sebastian
> >
> >
> > >
> > >
> > >
> > >--
> > >I tried to build a better future, a few times:
> > >https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
> > >
> > >Dave Täht CEO, TekLibre, LLC
> > >_______________________________________________
> > >Cake mailing list
> > >Cake@lists.bufferbloat.net
> > >https://lists.bufferbloat.net/listinfo/cake
> >
> > --
> > Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
>
>
> --
> I tried to build a better future, a few times:
> https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
>
> Dave Täht CEO, TekLibre, LLC



-- 
I tried to build a better future, a few times:
https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org

Dave Täht CEO, TekLibre, LLC

Re: [Cake] tossing acks into the background queue

[Dave Taht]

https://forum.openwrt.org/t/making-cake-multicore-or-other-new-features/112623

On Mon, Nov 22, 2021 at 11:32 PM Dave Taht <dave.taht@gmail.com> wrote:
>
> The context of my question is basically this:
>
> Is cake baked? Is it done?
>
> Is there anything from libreQos that would be better in C?
>
> On Mon, Nov 22, 2021 at 11:17 PM Dave Taht <dave.taht@gmail.com> wrote:
> >
> > On Mon, Nov 22, 2021 at 11:07 PM Sebastian Moeller <moeller0@gmx.de> wrote:
> > >
> > > Hi Dave,
> > >
> > >
> > > On 23 November 2021 06:03:03 CET, Dave Taht <dave.taht@gmail.com> wrote:
> > > >ages ago I'd (we'd? I really don't remember - forgive me if I've
> > > >forgotten who actually leaned in on it) written a basic ack-filter in
> > > >ebpf. this was before cake gained tc actions and my primary use for
> > > >the tech was for asymmetric connections, and before the good
> > > >ack-filter arrived, and I was (and remain) unfriendly to this level of
> > > >dpi.
> > > >
> > > >That said, on a symmetric connection, deprioritizing pure acks to the
> > > >5% background queue nd then turning the cake ack-filter loose on it
> > > >might actually work.
> > > >
> > > >Am I on drugs/is there any point?
> > >
> > > I think at leat when using multiple priority tins forward and reverse traffic should by default use the same tin (I can see non-standard situations that want differential treatment). The argument is that unlike earlier attempts at ingress shaping that tried to throttle reverse ACKs? cake/codel do proper 'hit the brakes' signalling via marking/dropping and we want that signal to reach the other end as quickly as possible, no?
> >
> > My thought was basically an optional filter that steered all pure acks
> > (no matter the classification) into the background queue.
> > Non-pure-acks (sacks) essentially jump the background queue and signal
> > that loss earlier. The backlog of other acks in background get
> > delivered out of order, but purely out of order and discarded by the
> > reciever.
> >
> > > Regards
> > >         Sebastian
> > >
> > >
> > > >
> > > >
> > > >
> > > >--
> > > >I tried to build a better future, a few times:
> > > >https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
> > > >
> > > >Dave Täht CEO, TekLibre, LLC
> > > >_______________________________________________
> > > >Cake mailing list
> > > >Cake@lists.bufferbloat.net
> > > >https://lists.bufferbloat.net/listinfo/cake
> > >
> > > --
> > > Sent from my Android device with K-9 Mail. Please excuse my brevity.
> >
> >
> >
> > --
> > I tried to build a better future, a few times:
> > https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
> >
> > Dave Täht CEO, TekLibre, LLC
>
>
>
> --
> I tried to build a better future, a few times:
> https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
>
> Dave Täht CEO, TekLibre, LLC



-- 
I tried to build a better future, a few times:
https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org

Dave Täht CEO, TekLibre, LLC

Re: [Cake] tossing acks into the background queue

[Sebastian Moeller]

Hi Dave,

On 23 November 2021 08:32:06 CET, Dave Taht <dave.taht@gmail.com> wrote:
>The context of my question is basically this:
>
>Is cake baked? Is it done?

How about per MAC address fairness (useful for ISPs and to treat IPv4/6 equally)?

How about configurable number of queues (again helpful for ISPs)?

IMHO cake works pretty well, with the biggest issue being its CPU demands. As far as I understand however, that is caused by the shaper component and there low latency and throughput are in direct competition, if we want to lower the CPU latency demands we need to allow for bigger buffers that keep the link busy even if cake itself is not scheduled as precisely as we would desire or as e.g. BQL requires.

Regards
         Sebastian

>
>Is there anything from libreQos that would be better in C?
>
>On Mon, Nov 22, 2021 at 11:17 PM Dave Taht <dave.taht@gmail.com> wrote:
>>
>> On Mon, Nov 22, 2021 at 11:07 PM Sebastian Moeller <moeller0@gmx.de> wrote:
>> >
>> > Hi Dave,
>> >
>> >
>> > On 23 November 2021 06:03:03 CET, Dave Taht <dave.taht@gmail.com> wrote:
>> > >ages ago I'd (we'd? I really don't remember - forgive me if I've
>> > >forgotten who actually leaned in on it) written a basic ack-filter in
>> > >ebpf. this was before cake gained tc actions and my primary use for
>> > >the tech was for asymmetric connections, and before the good
>> > >ack-filter arrived, and I was (and remain) unfriendly to this level of
>> > >dpi.
>> > >
>> > >That said, on a symmetric connection, deprioritizing pure acks to the
>> > >5% background queue nd then turning the cake ack-filter loose on it
>> > >might actually work.
>> > >
>> > >Am I on drugs/is there any point?
>> >
>> > I think at leat when using multiple priority tins forward and reverse traffic should by default use the same tin (I can see non-standard situations that want differential treatment). The argument is that unlike earlier attempts at ingress shaping that tried to throttle reverse ACKs? cake/codel do proper 'hit the brakes' signalling via marking/dropping and we want that signal to reach the other end as quickly as possible, no?
>>
>> My thought was basically an optional filter that steered all pure acks
>> (no matter the classification) into the background queue.
>> Non-pure-acks (sacks) essentially jump the background queue and signal
>> that loss earlier. The backlog of other acks in background get
>> delivered out of order, but purely out of order and discarded by the
>> reciever.
>>
>> > Regards
>> >         Sebastian
>> >
>> >
>> > >
>> > >
>> > >
>> > >--
>> > >I tried to build a better future, a few times:
>> > >https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
>> > >
>> > >Dave Täht CEO, TekLibre, LLC
>> > >_______________________________________________
>> > >Cake mailing list
>> > >Cake@lists.bufferbloat.net
>> > >https://lists.bufferbloat.net/listinfo/cake
>> >
>> > --
>> > Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>
>>
>>
>> --
>> I tried to build a better future, a few times:
>> https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
>>
>> Dave Täht CEO, TekLibre, LLC
>
>
>

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [Cake] tossing acks into the background queue

[Dave Taht]

On Tue, Nov 23, 2021 at 12:06 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>
> Hi Dave,
>
> On 23 November 2021 08:32:06 CET, Dave Taht <dave.taht@gmail.com> wrote:
> >The context of my question is basically this:
> >
> >Is cake baked? Is it done?
>
> How about per MAC address fairness (useful for ISPs and to treat IPv4/6 equally)?
>
> How about configurable number of queues (again helpful for ISPs)?

How about MPLS?

https://www.techtarget.com/searchnetworking/definition/Multiprotocol-Label-Switching-MPLS

>
> IMHO cake works pretty well, with the biggest issue being its CPU demands. As far as I understand however, that is caused by the shaper component and there low latency and throughput are in direct competition, if we want to lower the CPU latency demands we need to allow for bigger buffers that keep the link busy even if cake itself is not scheduled as precisely as we would desire or as e.g. BQL requires.
>
> Regards
>          Sebastian
>
> >
> >Is there anything from libreQos that would be better in C?
> >
> >On Mon, Nov 22, 2021 at 11:17 PM Dave Taht <dave.taht@gmail.com> wrote:
> >>
> >> On Mon, Nov 22, 2021 at 11:07 PM Sebastian Moeller <moeller0@gmx.de> wrote:
> >> >
> >> > Hi Dave,
> >> >
> >> >
> >> > On 23 November 2021 06:03:03 CET, Dave Taht <dave.taht@gmail.com> wrote:
> >> > >ages ago I'd (we'd? I really don't remember - forgive me if I've
> >> > >forgotten who actually leaned in on it) written a basic ack-filter in
> >> > >ebpf. this was before cake gained tc actions and my primary use for
> >> > >the tech was for asymmetric connections, and before the good
> >> > >ack-filter arrived, and I was (and remain) unfriendly to this level of
> >> > >dpi.
> >> > >
> >> > >That said, on a symmetric connection, deprioritizing pure acks to the
> >> > >5% background queue nd then turning the cake ack-filter loose on it
> >> > >might actually work.
> >> > >
> >> > >Am I on drugs/is there any point?
> >> >
> >> > I think at leat when using multiple priority tins forward and reverse traffic should by default use the same tin (I can see non-standard situations that want differential treatment). The argument is that unlike earlier attempts at ingress shaping that tried to throttle reverse ACKs? cake/codel do proper 'hit the brakes' signalling via marking/dropping and we want that signal to reach the other end as quickly as possible, no?
> >>
> >> My thought was basically an optional filter that steered all pure acks
> >> (no matter the classification) into the background queue.
> >> Non-pure-acks (sacks) essentially jump the background queue and signal
> >> that loss earlier. The backlog of other acks in background get
> >> delivered out of order, but purely out of order and discarded by the
> >> reciever.
> >>
> >> > Regards
> >> >         Sebastian
> >> >
> >> >
> >> > >
> >> > >
> >> > >
> >> > >--
> >> > >I tried to build a better future, a few times:
> >> > >https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
> >> > >
> >> > >Dave Täht CEO, TekLibre, LLC
> >> > >_______________________________________________
> >> > >Cake mailing list
> >> > >Cake@lists.bufferbloat.net
> >> > >https://lists.bufferbloat.net/listinfo/cake
> >> >
> >> > --
> >> > Sent from my Android device with K-9 Mail. Please excuse my brevity.
> >>
> >>
> >>
> >> --
> >> I tried to build a better future, a few times:
> >> https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
> >>
> >> Dave Täht CEO, TekLibre, LLC
> >
> >
> >
>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.



-- 
I tried to build a better future, a few times:
https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org

Dave Täht CEO, TekLibre, LLC

Re: [Cake] tossing acks into the background queue

[Sebastian Moeller]

Hi Dave,

On 23 November 2021 09:27:42 CET, Dave Taht <dave.taht@gmail.com> wrote:
>On Tue, Nov 23, 2021 at 12:06 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>>
>> Hi Dave,
>>
>> On 23 November 2021 08:32:06 CET, Dave Taht <dave.taht@gmail.com> wrote:
>> >The context of my question is basically this:
>> >
>> >Is cake baked? Is it done?
>>
>> How about per MAC address fairness (useful for ISPs and to treat IPv4/6 equally)?
>>
>> How about configurable number of queues (again helpful for ISPs)?
>
>How about MPLS?

Not sure that cake will be relevant on MPLS links, but if it should be, I guess peeking into the payload seems required (MPLS headers do not seem to represent flow id), other than that I hear the latest is IPv6 segment routing ;)

Now as far as I can tell, both MPLS or SR will not live at those places where putting in competent AQM promises highest bang for the buck anyways. IMHO as long as the backbone is overprovisioned putting AQM on the entry and exit routers seems the first logical step, so at access and peering links. Sure upping the ante for backbone AQM seems also a good idea, but will not help much if entry/exit are not yet AQMd, no?

Regards
           Sebastian



>
>https://www.techtarget.com/searchnetworking/definition/Multiprotocol-Label-Switching-MPLS
>
>>
>> IMHO cake works pretty well, with the biggest issue being its CPU demands. As far as I understand however, that is caused by the shaper component and there low latency and throughput are in direct competition, if we want to lower the CPU latency demands we need to allow for bigger buffers that keep the link busy even if cake itself is not scheduled as precisely as we would desire or as e.g. BQL requires.
>>
>> Regards
>>          Sebastian
>>
>> >
>> >Is there anything from libreQos that would be better in C?
>> >
>> >On Mon, Nov 22, 2021 at 11:17 PM Dave Taht <dave.taht@gmail.com> wrote:
>> >>
>> >> On Mon, Nov 22, 2021 at 11:07 PM Sebastian Moeller <moeller0@gmx.de> wrote:
>> >> >
>> >> > Hi Dave,
>> >> >
>> >> >
>> >> > On 23 November 2021 06:03:03 CET, Dave Taht <dave.taht@gmail.com> wrote:
>> >> > >ages ago I'd (we'd? I really don't remember - forgive me if I've
>> >> > >forgotten who actually leaned in on it) written a basic ack-filter in
>> >> > >ebpf. this was before cake gained tc actions and my primary use for
>> >> > >the tech was for asymmetric connections, and before the good
>> >> > >ack-filter arrived, and I was (and remain) unfriendly to this level of
>> >> > >dpi.
>> >> > >
>> >> > >That said, on a symmetric connection, deprioritizing pure acks to the
>> >> > >5% background queue nd then turning the cake ack-filter loose on it
>> >> > >might actually work.
>> >> > >
>> >> > >Am I on drugs/is there any point?
>> >> >
>> >> > I think at leat when using multiple priority tins forward and reverse traffic should by default use the same tin (I can see non-standard situations that want differential treatment). The argument is that unlike earlier attempts at ingress shaping that tried to throttle reverse ACKs? cake/codel do proper 'hit the brakes' signalling via marking/dropping and we want that signal to reach the other end as quickly as possible, no?
>> >>
>> >> My thought was basically an optional filter that steered all pure acks
>> >> (no matter the classification) into the background queue.
>> >> Non-pure-acks (sacks) essentially jump the background queue and signal
>> >> that loss earlier. The backlog of other acks in background get
>> >> delivered out of order, but purely out of order and discarded by the
>> >> reciever.
>> >>
>> >> > Regards
>> >> >         Sebastian
>> >> >
>> >> >
>> >> > >
>> >> > >
>> >> > >
>> >> > >--
>> >> > >I tried to build a better future, a few times:
>> >> > >https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
>> >> > >
>> >> > >Dave Täht CEO, TekLibre, LLC
>> >> > >_______________________________________________
>> >> > >Cake mailing list
>> >> > >Cake@lists.bufferbloat.net
>> >> > >https://lists.bufferbloat.net/listinfo/cake
>> >> >
>> >> > --
>> >> > Sent from my Android device with K-9 Mail. Please excuse my brevity.
>> >>
>> >>
>> >>
>> >> --
>> >> I tried to build a better future, a few times:
>> >> https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
>> >>
>> >> Dave Täht CEO, TekLibre, LLC
>> >
>> >
>> >
>>
>> --
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
>
>

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [Cake] tossing acks into the background queue

[Toke Høiland-Jørgensen]

Sebastian Moeller <moeller0@gmx.de> writes:

> Hi Dave,
>
> On 23 November 2021 08:32:06 CET, Dave Taht <dave.taht@gmail.com> wrote:
>>The context of my question is basically this:
>>
>>Is cake baked? Is it done?
>
> How about per MAC address fairness (useful for ISPs and to treat
> IPv4/6 equally)?
>
> How about configurable number of queues (again helpful for ISPs)?

FWIW I don't think CAKE is the right thing for ISPs, except in a
deployment where there's a single CAKE instance per customer. For
anything else (i.e., a single shaper that handles multiple customers),
you really need hierarchical policy enforcement like in a traditional
HTB configuration. And retrofitting this on top of CAKE is going to
conflict with the existing functionality, so it probably has to be a
separate qdisc anyway.

> IMHO cake works pretty well, with the biggest issue being its CPU
> demands. As far as I understand however, that is caused by the shaper
> component and there low latency and throughput are in direct
> competition, if we want to lower the CPU latency demands we need to
> allow for bigger buffers that keep the link busy even if cake itself
> is not scheduled as precisely as we would desire or as e.g. BQL
> requires.

Yes, as link speed increases, batching needs to increase to keep up.
This does not *have* to impact latency, as the faster link should keep
the granularity constant in the time domain. So experimenting with doing
this dynamically in CAKE might be worthwhile, but probably not trivial.

And either way, CAKE is still going to be limited by being single core
only, and fixing that requires some serious surgery that I seem to
recall looking into and giving up at some point :(

-Toke

Re: [Cake] tossing acks into the background queue

[Sebastian Moeller]

Hi Toke,


> On Nov 23, 2021, at 11:39, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Sebastian Moeller <moeller0@gmx.de> writes:
> 
>> Hi Dave,
>> 
>> On 23 November 2021 08:32:06 CET, Dave Taht <dave.taht@gmail.com> wrote:
>>> The context of my question is basically this:
>>> 
>>> Is cake baked? Is it done?
>> 
>> How about per MAC address fairness (useful for ISPs and to treat
>> IPv4/6 equally)?
>> 
>> How about configurable number of queues (again helpful for ISPs)?
> 
> FWIW I don't think CAKE is the right thing for ISPs, except in a
> deployment where there's a single CAKE instance per customer.

Fair point. My other reason for wanting to expose this is to allow easier experimentation, but I can be expected to build from modified sources so that is rather weak.

> For
> anything else (i.e., a single shaper that handles multiple customers),
> you really need hierarchical policy enforcement like in a traditional
> HTB configuration. And retrofitting this on top of CAKE is going to
> conflict with the existing functionality, so it probably has to be a
> separate qdisc anyway.

	I had sort of ignored that ISPs generally do not offer, fair sharing of a link's capacity between all connected users ;)


> 
>> IMHO cake works pretty well, with the biggest issue being its CPU
>> demands. As far as I understand however, that is caused by the shaper
>> component and there low latency and throughput are in direct
>> competition, if we want to lower the CPU latency demands we need to
>> allow for bigger buffers that keep the link busy even if cake itself
>> is not scheduled as precisely as we would desire or as e.g. BQL
>> requires.
> 
> Yes, as link speed increases, batching needs to increase to keep up.

	Yes, all the way through the stack.


> This does not *have* to impact latency, as the faster link should keep
> the granularity constant in the time domain.

	Nit-pick: any batching impacts latency compared to perfect just in time processing, just some impact can easily be accepted/tolerated ;)

> So experimenting with doing
> this dynamically in CAKE might be worthwhile, but probably not trivial.

	We tried to do the same for HTB/fq_codel and testing was a bit inconclusive (then again, affected users where not to dedicated in testing)

> And either way, CAKE is still going to be limited by being single core
> only, and fixing that requires some serious surgery that I seem to
> recall looking into and giving up at some point :(

	That is sad, and pretty much rules out that I could make some progress in that direction. The next level is shaping at ~1Gbps, even though faster access links become available, like 8.5/10 Gbps (XGS-PON is nominally 10G, but after FEC only ~8.5 Gbps actually are usable) or for a lucky few even 25 Gbps ...

Regards
	Sebastian

> 
> -Toke

Re: [Cake] tossing acks into the background queue

[Toke Høiland-Jørgensen]

>> And either way, CAKE is still going to be limited by being single core
>> only, and fixing that requires some serious surgery that I seem to
>> recall looking into and giving up at some point :(
>
> 	That is sad, and pretty much rules out that I could make some
> 	progress in that direction. The next level is shaping at ~1Gbps,
> 	even though faster access links become available, like 8.5/10
> 	Gbps (XGS-PON is nominally 10G, but after FEC only ~8.5 Gbps
> 	actually are usable) or for a lucky few even 25 Gbps ...

Yup. FWIW my hope is that we'll be able to do something about this using
XDP, eventually... :)

-Toke

Re: [Cake] tossing acks into the background queue

[Dave Taht]

On Tue, Nov 23, 2021 at 2:39 AM Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> Sebastian Moeller <moeller0@gmx.de> writes:
>
> > Hi Dave,
> >
> > On 23 November 2021 08:32:06 CET, Dave Taht <dave.taht@gmail.com> wrote:
> >>The context of my question is basically this:
> >>
> >>Is cake baked? Is it done?
> >
> > How about per MAC address fairness (useful for ISPs and to treat
> > IPv4/6 equally)?
> >
> > How about configurable number of queues (again helpful for ISPs)?
>
> FWIW I don't think CAKE is the right thing for ISPs, except in a
> deployment where there's a single CAKE instance per customer. For
> anything else (i.e., a single shaper that handles multiple customers),
> you really need hierarchical policy enforcement like in a traditional
> HTB configuration. And retrofitting this on top of CAKE is going to
> conflict with the existing functionality, so it probably has to be a
> separate qdisc anyway.

What progress has been made on breaking the HTB locks in the last few years?

Given the enormous number of hw tx/rx queues we see today (64+ on
10gbit), trying to charge off
bandwidth per queue in a cake-derived shaper and protecting the merge
with rcu seemed plausible...

>
> > IMHO cake works pretty well, with the biggest issue being its CPU
> > demands. As far as I understand however, that is caused by the shaper
> > component and there low latency and throughput are in direct
> > competition, if we want to lower the CPU latency demands we need to
> > allow for bigger buffers that keep the link busy even if cake itself
> > is not scheduled as precisely as we would desire or as e.g. BQL
> > requires.
>
> Yes, as link speed increases, batching needs to increase to keep up.
> This does not *have* to impact latency, as the faster link should keep
> the granularity constant in the time domain. So experimenting with doing
> this dynamically in CAKE might be worthwhile, but probably not trivial.
>
> And either way, CAKE is still going to be limited by being single core
> only, and fixing that requires some serious surgery that I seem to
> recall looking into and giving up at some point :(

It was so long ago I don't remember what other issues came up at the time.

?

I am seeing nvidia offloading red and htb.

> -Toke



-- 
I tried to build a better future, a few times:
https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org

Dave Täht CEO, TekLibre, LLC

Re: [Cake] tossing acks into the background queue

[Toke Høiland-Jørgensen]

Dave Taht <dave.taht@gmail.com> writes:

> On Tue, Nov 23, 2021 at 2:39 AM Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>
>> Sebastian Moeller <moeller0@gmx.de> writes:
>>
>> > Hi Dave,
>> >
>> > On 23 November 2021 08:32:06 CET, Dave Taht <dave.taht@gmail.com> wrote:
>> >>The context of my question is basically this:
>> >>
>> >>Is cake baked? Is it done?
>> >
>> > How about per MAC address fairness (useful for ISPs and to treat
>> > IPv4/6 equally)?
>> >
>> > How about configurable number of queues (again helpful for ISPs)?
>>
>> FWIW I don't think CAKE is the right thing for ISPs, except in a
>> deployment where there's a single CAKE instance per customer. For
>> anything else (i.e., a single shaper that handles multiple customers),
>> you really need hierarchical policy enforcement like in a traditional
>> HTB configuration. And retrofitting this on top of CAKE is going to
>> conflict with the existing functionality, so it probably has to be a
>> separate qdisc anyway.
>
> What progress has been made on breaking the HTB locks in the last few
> years?

None. Don't see that happening any time soon; just the simple pfifo_fast
qdisc is uncovering all kinds of bugs when running in lockless mode.

Jesper basically solved the contention issue by partitioning the traffic
and running multiple instances:
https://github.com/xdp-project/xdp-cpumap-tc

Doesn't work for bandwidth sharing across instances, though, so it
solves the ISP "separate rates per customer" case, but not the CAKE
"shape a single link" case.

> Given the enormous number of hw tx/rx queues we see today (64+ on
> 10gbit), trying to charge off
> bandwidth per queue in a cake-derived shaper and protecting the merge
> with rcu seemed plausible...

Yeah, that was what I was going to try, but it turned out to be
decidedly non-trivial to make sch_cake itself mq-aware, so I gave up. My
hope is that this will be possible once we get sch_bpf, so we can just
have separate instances but they can share a single atomic var for the
bandwidth sync...

-Toke

Re: [Cake] tossing acks into the background queue

[Sebastian Moeller]

Hi Dave,

On 23 November 2021 08:17:38 CET, Dave Taht <dave.taht@gmail.com> wrote:
>On Mon, Nov 22, 2021 at 11:07 PM Sebastian Moeller <moeller0@gmx.de> wrote:
>>
>> Hi Dave,
>>
>>
>> On 23 November 2021 06:03:03 CET, Dave Taht <dave.taht@gmail.com> wrote:
>> >ages ago I'd (we'd? I really don't remember - forgive me if I've
>> >forgotten who actually leaned in on it) written a basic ack-filter in
>> >ebpf. this was before cake gained tc actions and my primary use for
>> >the tech was for asymmetric connections, and before the good
>> >ack-filter arrived, and I was (and remain) unfriendly to this level of
>> >dpi.
>> >
>> >That said, on a symmetric connection, deprioritizing pure acks to the
>> >5% background queue nd then turning the cake ack-filter loose on it
>> >might actually work.
>> >
>> >Am I on drugs/is there any point?
>>
>> I think at leat when using multiple priority tins forward and reverse traffic should by default use the same tin (I can see non-standard situations that want differential treatment). The argument is that unlike earlier attempts at ingress shaping that tried to throttle reverse ACKs? cake/codel do proper 'hit the brakes' signalling via marking/dropping and we want that signal to reach the other end as quickly as possible, no?
>
>My thought was basically an optional filter that steered all pure acks
>(no matter the classification) into the background queue.
>Non-pure-acks (sacks) essentially jump the background queue and signal
>that loss earlier. The backlog of other acks in background get
>delivered out of order, but purely out of order and discarded by the
>reciever.

Mmmh, not sure whether all connections actually use SACK in the first place?
I am always a bit uneasy when networks try to be clever, if we think that ACK rates are too high, IMHO we should teach the ACK emitters to slow down. Sure, ACK filtering as in cake is a valuable tool if used judiciously in a home network (after all fixing the emitters is not going to happen overnight) but I would appreciate it much less if my ISP would by the same logic fuzz with my packets. (I grudgingly accept that GEO satellite ISPs might have to do some fuzzing with RTTs so much outside of what normal TCPs are prepared for, but I degress)

Regards
         Sebastian



>
>> Regards
>>         Sebastian
>>
>>
>> >
>> >
>> >
>> >--
>> >I tried to build a better future, a few times:
>> >https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
>> >
>> >Dave Täht CEO, TekLibre, LLC
>> >_______________________________________________
>> >Cake mailing list
>> >Cake@lists.bufferbloat.net
>> >https://lists.bufferbloat.net/listinfo/cake
>>
>> --
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
>
>

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.