[Cake] Cake not more CPU efficient than HTB+FQ-CoDel (anymore)?

[Cake] Cake not more CPU efficient than HTB+FQ-CoDel (anymore)?

[Toke Høiland-Jørgensen]

[-- Attachment #1: Type: text/plain, Size: 925 bytes --]

So we've been saying that one of the benefits of Cake is less CPU usage;
but while trying to benchmark this I got results that would seem to
indicate the opposite.

See attached graph + data files. Basically, I setup a shaper on an
Archer C7 with sqm-scripts simplest.qos. Both HTB+FQ-CoDel and Cake
manages to shape at 250 Mbps, where Cake even shows a bit lower latency.
That is good.

However, when I change the configuration to 400 Mbps (more than the
Archer CPU can handle), Cake tops out at ~260 Mbps, while HTB+FQ-CoDel
manages ~305 Mbps and a slightly lower latency. In both cases I see the
characteristic 95% sirq CPU usage in 'top' on the Archer while the test
is running.

So, um, did we cram so many features into Cake that it no longer uses
less CPU? Can anyone confirm these results?

The tests were run on an openwrt nightly image from today, which has the
latest Cake version from the Cobalt branch.


-Toke


[-- Attachment #2: cake-vs-fqcodel-cpulimit.pdf --]
[-- Type: application/pdf, Size: 111833 bytes --]

[-- Attachment #3: tcp_1up-2018-04-11T165952.024206.FQ-CoDel_250_Mbps.flent.gz --]
[-- Type: application/gzip, Size: 16501 bytes --]

[-- Attachment #4: tcp_1up-2018-04-11T170134.227613.Cake_250_Mbps.flent.gz --]
[-- Type: application/gzip, Size: 16249 bytes --]

[-- Attachment #5: tcp_1up-2018-04-11T170457.254899.Cake_400_Mbps.flent.gz --]
[-- Type: application/gzip, Size: 16467 bytes --]

[-- Attachment #6: tcp_1up-2018-04-11T170647.320916.FQ-CoDel_400_Mbps.flent.gz --]
[-- Type: application/gzip, Size: 16560 bytes --]

Re: [Cake] Cake not more CPU efficient than HTB+FQ-CoDel (anymore)?

[Jonathan Morton]

> On 11 Apr, 2018, at 6:24 pm, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> So, um, did we cram so many features into Cake that it no longer uses
> less CPU? Can anyone confirm these results?

To be sure about this, it seems wise to configure Cake to turn off as many of the new features as possible.  That means selecting "besteffort flows nonat" mode at least.

I forget whether simplest.qos correctly uses the built-in shaper with Cake, rather than just layering it with HTB as usual.  If not, then of course Cake will use more CPU, and we should be grateful that it's by a relatively small margin (maybe 15%).

There's also a minor complication in that Cake and fq_codel behave differently when handed superpackets.  A fair comparison requires switching aggregation modes off for both of them.

All of my working hardware tends to saturate either the link or a PCI bus before hitting a CPU limit, so I can't reproduce it locally.

 - Jonathan Morton

Re: [Cake] Cake not more CPU efficient than HTB+FQ-CoDel (anymore)?

[Pete Heist]

For what it’s worth, that’s what I also saw testing Cake on the APU2 late last year, and the ER-X platform earlier. I actually never knew that Cake used less CPU at some point. Sorry for no supporting detail... :)

Pete

> On Apr 11, 2018, at 5:24 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> So we've been saying that one of the benefits of Cake is less CPU usage;
> but while trying to benchmark this I got results that would seem to
> indicate the opposite.
> 
> See attached graph + data files. Basically, I setup a shaper on an
> Archer C7 with sqm-scripts simplest.qos. Both HTB+FQ-CoDel and Cake
> manages to shape at 250 Mbps, where Cake even shows a bit lower latency.
> That is good.
> 
> However, when I change the configuration to 400 Mbps (more than the
> Archer CPU can handle), Cake tops out at ~260 Mbps, while HTB+FQ-CoDel
> manages ~305 Mbps and a slightly lower latency. In both cases I see the
> characteristic 95% sirq CPU usage in 'top' on the Archer while the test
> is running.
> 
> So, um, did we cram so many features into Cake that it no longer uses
> less CPU? Can anyone confirm these results?
> 
> The tests were run on an openwrt nightly image from today, which has the
> latest Cake version from the Cobalt branch.
> 
> 
> -Toke
> 
> <cake-vs-fqcodel-cpulimit.pdf><tcp_1up-2018-04-11T165952.024206.FQ-CoDel_250_Mbps.flent.gz><tcp_1up-2018-04-11T170134.227613.Cake_250_Mbps.flent.gz><tcp_1up-2018-04-11T170457.254899.Cake_400_Mbps.flent.gz><tcp_1up-2018-04-11T170647.320916.FQ-CoDel_400_Mbps.flent.gz>_______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Cake not more CPU efficient than HTB+FQ-CoDel (anymore)?

[Toke Høiland-Jørgensen]

Jonathan Morton <chromatix99@gmail.com> writes:

>> On 11 Apr, 2018, at 6:24 pm, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> 
>> So, um, did we cram so many features into Cake that it no longer uses
>> less CPU? Can anyone confirm these results?
>
> To be sure about this, it seems wise to configure Cake to turn off as
> many of the new features as possible. That means selecting "besteffort
> flows nonat" mode at least.
>
> I forget whether simplest.qos correctly uses the built-in shaper with
> Cake, rather than just layering it with HTB as usual. If not, then of
> course Cake will use more CPU, and we should be grateful that it's by
> a relatively small margin (maybe 15%).

It is definitely using Cake as the shaper; in besteffort mode, but with
nat and triple-isolation enabled I think. I'll run another test tomorrow
with those disabled.

> There's also a minor complication in that Cake and fq_codel behave
> differently when handed superpackets. A fair comparison requires
> switching aggregation modes off for both of them.

I *think* offloads were turned off for those tests; but I'll double
check... Also would be nice to get a measure of the smoothness of the
shaper; will see if I can't extract that from a pcap file or
something...

-Toke

Re: [Cake] Cake not more CPU efficient than HTB+FQ-CoDel (anymore)?

[Toke Høiland-Jørgensen]

Pete Heist <pete@eventide.io> writes:

> For what it’s worth, that’s what I also saw testing Cake on the APU2
> late last year, and the ER-X platform earlier. I actually never knew
> that Cake used less CPU at some point. Sorry for no supporting
> detail... :)

Anecdotal supporting evidence is fine. Just needed a sanity check :)

-Toke

Re: [Cake] Cake not more CPU efficient than HTB+FQ-CoDel (anymore)?

[Jonathan Morton]

> On 11 Apr, 2018, at 8:15 pm, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Also would be nice to get a measure of the smoothness of the
> shaper; will see if I can't extract that from a pcap file or
> something

I remember using tcptrace for that purpose, once upon a time.

 - Jonathan Morton

Re: [Cake] Cake not more CPU efficient than HTB+FQ-CoDel (anymore)?

[Toke Høiland-Jørgensen]

Jonathan Morton <chromatix99@gmail.com> writes:

>> On 11 Apr, 2018, at 8:15 pm, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> 
>> Also would be nice to get a measure of the smoothness of the
>> shaper; will see if I can't extract that from a pcap file or
>> something
>
> I remember using tcptrace for that purpose, once upon a time.

Yup, that was my plan as well. Just not sure whether it has a metric
that is better than "look, the graph is smoother"... ;)

-Toke

Re: [Cake] Cake not more CPU efficient than HTB+FQ-CoDel (anymore)?

[Jonathan Morton]

> On 11 Apr, 2018, at 8:47 pm, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
>>> Also would be nice to get a measure of the smoothness of the
>>> shaper; will see if I can't extract that from a pcap file or
>>> something
>> 
>> I remember using tcptrace for that purpose, once upon a time.
> 
> Yup, that was my plan as well. Just not sure whether it has a metric
> that is better than "look, the graph is smoother"... ;)

It does have a throughput graph available.  However, this is based on raw inter-packet spacing, so it'll have line-rate bursts interspersed by very short gaps if Cake experiences any scheduling latency greater than the serialisation delay of one MTU.

The best counter to that might be to select a low rate, so that the serialisation delay is 1-2ms or so.  It should then be easy to contrast with HTB's behaviour at that same rate.

 - Jonathan Morton

Re: [Cake] Cake not more CPU efficient than HTB+FQ-CoDel (anymore)?

[Jonas Mårtensson]

[-- Attachment #1: Type: text/plain, Size: 1113 bytes --]

On Wed, Apr 11, 2018 at 7:15 PM, Toke Høiland-Jørgensen <toke@toke.dk>
wrote:

> Jonathan Morton <chromatix99@gmail.com> writes:
>
> >> On 11 Apr, 2018, at 6:24 pm, Toke Høiland-Jørgensen <toke@toke.dk>
> wrote:
> >>
> >> So, um, did we cram so many features into Cake that it no longer uses
> >> less CPU? Can anyone confirm these results?
> >
> > To be sure about this, it seems wise to configure Cake to turn off as
> > many of the new features as possible. That means selecting "besteffort
> > flows nonat" mode at least.
> >
> > I forget whether simplest.qos correctly uses the built-in shaper with
> > Cake, rather than just layering it with HTB as usual. If not, then of
> > course Cake will use more CPU, and we should be grateful that it's by
> > a relatively small margin (maybe 15%).
>
> It is definitely using Cake as the shaper; in besteffort mode, but with
> nat and triple-isolation enabled I think. I'll run another test tomorrow
> with those disabled.


Is there any difference between using simplest.qos and piece_of_cake.qos
when Cake is used as qdisc?

/Jonas

[-- Attachment #2: Type: text/html, Size: 1680 bytes --]

Re: [Cake] Cake not more CPU efficient than HTB+FQ-CoDel (anymore)?

[Toke Høiland-Jørgensen]

Jonas Mårtensson <martensson.jonas@gmail.com> writes:

> On Wed, Apr 11, 2018 at 7:15 PM, Toke Høiland-Jørgensen <toke@toke.dk>
> wrote:
>
>> Jonathan Morton <chromatix99@gmail.com> writes:
>>
>> >> On 11 Apr, 2018, at 6:24 pm, Toke Høiland-Jørgensen <toke@toke.dk>
>> wrote:
>> >>
>> >> So, um, did we cram so many features into Cake that it no longer uses
>> >> less CPU? Can anyone confirm these results?
>> >
>> > To be sure about this, it seems wise to configure Cake to turn off as
>> > many of the new features as possible. That means selecting "besteffort
>> > flows nonat" mode at least.
>> >
>> > I forget whether simplest.qos correctly uses the built-in shaper with
>> > Cake, rather than just layering it with HTB as usual. If not, then of
>> > course Cake will use more CPU, and we should be grateful that it's by
>> > a relatively small margin (maybe 15%).
>>
>> It is definitely using Cake as the shaper; in besteffort mode, but with
>> nat and triple-isolation enabled I think. I'll run another test tomorrow
>> with those disabled.
>
>
> Is there any difference between using simplest.qos and piece_of_cake.qos
> when Cake is used as qdisc?

Not really, no. piece_of_cake understands the zero_dscp_ingress option,
simplest.qos does not. I think that is the only difference.

I just used simplest.qos because that made it easier to switch between
fq_codel and cake ;)

-Toke

Re: [Cake] Cake not more CPU efficient than HTB+FQ-CoDel (anymore)?

[Sebastian Moeller]

On April 11, 2018 8:55:12 PM GMT+02:00, "Jonas Mårtensson" <martensson.jonas@gmail.com> wrote:
>On Wed, Apr 11, 2018 at 7:15 PM, Toke Høiland-Jørgensen <toke@toke.dk>
>wrote:
>
>> Jonathan Morton <chromatix99@gmail.com> writes:
>>
>> >> On 11 Apr, 2018, at 6:24 pm, Toke Høiland-Jørgensen <toke@toke.dk>
>> wrote:
>> >>
>> >> So, um, did we cram so many features into Cake that it no longer
>uses
>> >> less CPU? Can anyone confirm these results?
>> >
>> > To be sure about this, it seems wise to configure Cake to turn off
>as
>> > many of the new features as possible. That means selecting
>"besteffort
>> > flows nonat" mode at least.
>> >
>> > I forget whether simplest.qos correctly uses the built-in shaper
>with
>> > Cake, rather than just layering it with HTB as usual. If not, then
>of
>> > course Cake will use more CPU, and we should be grateful that it's
>by
>> > a relatively small margin (maybe 15%).
>>
>> It is definitely using Cake as the shaper; in besteffort mode, but
>with
>> nat and triple-isolation enabled I think. I'll run another test
>tomorrow
>> with those disabled.
>
>
>Is there any difference between using simplest.qos and
>piece_of_cake.qos
>when Cake is used as qdisc?

Yes, IIRC simplest.qos with cake as qdisc will use HTB as shaper and cake as leaf qdisc, while piece_of_cake.qos will use cake both as shaper and leaf qdisc. The former is only useful for comparative testing, for actual usage I recommend the later.

Best Regards
        Sebastian


>
>/Jonas

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [Cake] Cake not more CPU efficient than HTB+FQ-CoDel (anymore)?

[Sebastian Moeller]

Sorry,

just looked at the code and my recollection is wrong. I could have sworn that I purged cake as a shaper from simple.qos when I created piece_of_cake, but apparently that was just a fever dream...

Sorry for the noise.


> On Apr 11, 2018, at 21:26, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> 
> 
> On April 11, 2018 8:55:12 PM GMT+02:00, "Jonas Mårtensson" <martensson.jonas@gmail.com> wrote:
>> On Wed, Apr 11, 2018 at 7:15 PM, Toke Høiland-Jørgensen <toke@toke.dk>
>> wrote:
>> 
>>> Jonathan Morton <chromatix99@gmail.com> writes:
>>> 
>>>>> On 11 Apr, 2018, at 6:24 pm, Toke Høiland-Jørgensen <toke@toke.dk>
>>> wrote:
>>>>> 
>>>>> So, um, did we cram so many features into Cake that it no longer
>> uses
>>>>> less CPU? Can anyone confirm these results?
>>>> 
>>>> To be sure about this, it seems wise to configure Cake to turn off
>> as
>>>> many of the new features as possible. That means selecting
>> "besteffort
>>>> flows nonat" mode at least.
>>>> 
>>>> I forget whether simplest.qos correctly uses the built-in shaper
>> with
>>>> Cake, rather than just layering it with HTB as usual. If not, then
>> of
>>>> course Cake will use more CPU, and we should be grateful that it's
>> by
>>>> a relatively small margin (maybe 15%).
>>> 
>>> It is definitely using Cake as the shaper; in besteffort mode, but
>> with
>>> nat and triple-isolation enabled I think. I'll run another test
>> tomorrow
>>> with those disabled.
>> 
>> 
>> Is there any difference between using simplest.qos and
>> piece_of_cake.qos
>> when Cake is used as qdisc?
> 
> Yes, IIRC simplest.qos with cake as qdisc will use HTB as shaper and cake as leaf qdisc, while piece_of_cake.qos will use cake both as shaper and leaf qdisc. The former is only useful for comparative testing, for actual usage I recommend the later.
> 
> Best Regards
>        Sebastian
> 
> 
>> 
>> /Jonas
> 
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Cake not more CPU efficient than HTB+FQ-CoDel (anymore)?

[Jonas Mårtensson]

[-- Attachment #1: Type: text/plain, Size: 2536 bytes --]

Well, simplest.qos.help says "Simplest possible configuration: HTB rate
limiter with your qdisc attached" so that is probably also a bit misleading.

/Jonas

On Wed, Apr 11, 2018 at 9:30 PM, Sebastian Moeller <moeller0@gmx.de> wrote:

> Sorry,
>
> just looked at the code and my recollection is wrong. I could have sworn
> that I purged cake as a shaper from simple.qos when I created
> piece_of_cake, but apparently that was just a fever dream...
>
> Sorry for the noise.
>
>
> > On Apr 11, 2018, at 21:26, Sebastian Moeller <moeller0@gmx.de> wrote:
> >
> >
> >
> > On April 11, 2018 8:55:12 PM GMT+02:00, "Jonas Mårtensson" <
> martensson.jonas@gmail.com> wrote:
> >> On Wed, Apr 11, 2018 at 7:15 PM, Toke Høiland-Jørgensen <toke@toke.dk>
> >> wrote:
> >>
> >>> Jonathan Morton <chromatix99@gmail.com> writes:
> >>>
> >>>>> On 11 Apr, 2018, at 6:24 pm, Toke Høiland-Jørgensen <toke@toke.dk>
> >>> wrote:
> >>>>>
> >>>>> So, um, did we cram so many features into Cake that it no longer
> >> uses
> >>>>> less CPU? Can anyone confirm these results?
> >>>>
> >>>> To be sure about this, it seems wise to configure Cake to turn off
> >> as
> >>>> many of the new features as possible. That means selecting
> >> "besteffort
> >>>> flows nonat" mode at least.
> >>>>
> >>>> I forget whether simplest.qos correctly uses the built-in shaper
> >> with
> >>>> Cake, rather than just layering it with HTB as usual. If not, then
> >> of
> >>>> course Cake will use more CPU, and we should be grateful that it's
> >> by
> >>>> a relatively small margin (maybe 15%).
> >>>
> >>> It is definitely using Cake as the shaper; in besteffort mode, but
> >> with
> >>> nat and triple-isolation enabled I think. I'll run another test
> >> tomorrow
> >>> with those disabled.
> >>
> >>
> >> Is there any difference between using simplest.qos and
> >> piece_of_cake.qos
> >> when Cake is used as qdisc?
> >
> > Yes, IIRC simplest.qos with cake as qdisc will use HTB as shaper and
> cake as leaf qdisc, while piece_of_cake.qos will use cake both as shaper
> and leaf qdisc. The former is only useful for comparative testing, for
> actual usage I recommend the later.
> >
> > Best Regards
> >        Sebastian
> >
> >
> >>
> >> /Jonas
> >
> > --
> > Sent from my Android device with K-9 Mail. Please excuse my brevity.
> > _______________________________________________
> > Cake mailing list
> > Cake@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cake
>
>

[-- Attachment #2: Type: text/html, Size: 3865 bytes --]

Re: [Cake] Cake not more CPU efficient than HTB+FQ-CoDel (anymore)?

[Toke Høiland-Jørgensen]

[-- Attachment #1: Type: text/plain, Size: 1547 bytes --]

Toke Høiland-Jørgensen <toke@toke.dk> writes:

> Jonathan Morton <chromatix99@gmail.com> writes:
>
>>> On 11 Apr, 2018, at 6:24 pm, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>> 
>>> So, um, did we cram so many features into Cake that it no longer uses
>>> less CPU? Can anyone confirm these results?
>>
>> To be sure about this, it seems wise to configure Cake to turn off as
>> many of the new features as possible. That means selecting "besteffort
>> flows nonat" mode at least.
>>
>> I forget whether simplest.qos correctly uses the built-in shaper with
>> Cake, rather than just layering it with HTB as usual. If not, then of
>> course Cake will use more CPU, and we should be grateful that it's by
>> a relatively small margin (maybe 15%).
>
> It is definitely using Cake as the shaper; in besteffort mode, but with
> nat and triple-isolation enabled I think. I'll run another test tomorrow
> with those disabled.
>
>> There's also a minor complication in that Cake and fq_codel behave
>> differently when handed superpackets. A fair comparison requires
>> switching aggregation modes off for both of them.
>
> I *think* offloads were turned off for those tests; but I'll double
> check... Also would be nice to get a measure of the smoothness of the
> shaper; will see if I can't extract that from a pcap file or
> something...

Right, double checked and ran a few more tests. Setting cake to
flowblind mode help somewhat, but HTB/FQ-CoDel still gets higher
throughput. Graph attached.

-Toke



[-- Attachment #2: cake-vs-fqcodel-cpulimit-moretests.pdf --]
[-- Type: application/pdf, Size: 115775 bytes --]

There is cake qdisc manual

[Y]

Hi, stuff.
I am yutaka.

I found cake qdisc manual.
https://dl.lochnair.net/Bufferbloat/Cake/tc-cake.8.html
Thank you for all Auther :)

yutaka.