* [Cake] Cake memory usage after 1TB
@ 2021-06-24 13:06 Pete Heist
2021-06-24 13:26 ` Pete Heist
2021-06-24 13:59 ` Toke Høiland-Jørgensen
0 siblings, 2 replies; 8+ messages in thread
From: Pete Heist @ 2021-06-24 13:06 UTC (permalink / raw)
To: Cake List
I'm using Cake on an EdgeRouter-X at a site that does around 50-
100GB/day, and we're quite happy with it so far. Egress and ingress
stats are below after ~1TB download and ~100GB upload.
I use "memlimit 8M", and noticed that it's reporting 8389696 bytes used
on egress, which is at the maximum (slightly over actually). Around 6M
is used on ingress. Should I then raise these limits, or is it normal
for it to settle in around the maximum over time?
Pete
Egress:
-------
qdisc cake 8008: dev eth4 root refcnt 2 bandwidth 8Mbit diffserv4 dual-
srchost nat nowash no-ack-filter split-gso rtt 100.0ms raw overhead 0
fwmark 0xf no-sce
Sent 102655686381 bytes 386831397 pkt (dropped 377557, overlimits
230324326 requeues 0)
backlog 0b 0p requeues 0
memory used: 8389696b of 8Mb
capacity estimate: 8Mbit
min/max network layer size: 42 / 1514
min/max overhead-adjusted size: 42 / 1514
average network hdr offset: 14
Bulk Best Effort Video Voice
thresh 500Kbit 8Mbit 4Mbit 2Mbit
target 36.3ms 5.0ms 5.0ms 9.1ms
interval 131.3ms 100.0ms 100.0ms 104.1ms
pk_delay 17.5ms 4.1ms 1.7ms 2.0ms
av_delay 4.5ms 232us 82us 382us
sp_delay 14us 23us 6us 7us
backlog 0b 0b 0b 0b
pkts 165781 324229040 61032588 1781545
bytes 45344666 98316181815 4518083349 291035866
way_inds 42 18492140 7782656 34167
way_miss 4289 6575635 77917 62650
way_cols 0 340 0 0
sce 0 0 0 0
marks 7 19542 1 0
drops 8 377528 15 6
ack_drop 0 0 0 0
sp_flows 0 2 0 1
bk_flows 0 1 0 0
un_flows 0 0 0 0
max_len 1514 1514 1514 1514
quantum 300 300 300 300
Ingress:
--------
qdisc cake 1: dev ifb4eth4 root refcnt 2 bandwidth 40Mbit diffserv4
dual-dsthost nat nowash no-ack-filter split-gso rtt 100.0ms raw
overhead 0 fwmark 0xf no-sce
Sent 1081750548776 bytes 839670608 pkt (dropped 1616216, overlimits
657093747 requeues 0)
backlog 0b 0p requeues 0
memory used: 6027328b of 8Mb
capacity estimate: 40Mbit
min/max network layer size: 60 / 1514
min/max overhead-adjusted size: 60 / 1514
average network hdr offset: 14
Bulk Best Effort Video Voice
thresh 2500Kbit 40Mbit 20Mbit 10Mbit
target 7.3ms 5.0ms 5.0ms 5.0ms
interval 102.3ms 100.0ms 100.0ms 100.0ms
pk_delay 2.0ms 6.5ms 692us 1.1ms
av_delay 248us 2.6ms 88us 139us
sp_delay 12us 16us 7us 6us
backlog 0b 0b 0b 0b
pkts 29132 682090182 155330794 3836716
bytes 25452377 850788764602 230875228776 2357107850
way_inds 3 37939194 21198287 71120
way_miss 9448 6525310 97141 51428
way_cols 0 107 0 0
sce 0 0 0 0
marks 0 206496 203 30
drops 30 1534636 80809 741
ack_drop 0 0 0 0
sp_flows 0 2 0 0
bk_flows 0 1 0 0
un_flows 0 0 0 0
max_len 1484 1514 1514 1514
quantum 300 1220 610 305
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [Cake] Cake memory usage after 1TB
2021-06-24 13:06 [Cake] Cake memory usage after 1TB Pete Heist
@ 2021-06-24 13:26 ` Pete Heist
2021-06-24 13:33 ` Dave Taht
2021-06-24 13:59 ` Toke Høiland-Jørgensen
1 sibling, 1 reply; 8+ messages in thread
From: Pete Heist @ 2021-06-24 13:26 UTC (permalink / raw)
To: Cake List
Hmm, since I see in the code that this could lead to drops, I've
restarted it and upped the limit to 16M. I haven't had reports of
widespread problems, but I'll keep an eye on this more closely. :)
Pete
On Thu, 2021-06-24 at 15:06 +0200, Pete Heist wrote:
> I'm using Cake on an EdgeRouter-X at a site that does around 50-
> 100GB/day, and we're quite happy with it so far. Egress and ingress
> stats are below after ~1TB download and ~100GB upload.
>
> I use "memlimit 8M", and noticed that it's reporting 8389696 bytes
> used
> on egress, which is at the maximum (slightly over actually). Around
> 6M
> is used on ingress. Should I then raise these limits, or is it normal
> for it to settle in around the maximum over time?
>
> Pete
>
>
> Egress:
> -------
>
> qdisc cake 8008: dev eth4 root refcnt 2 bandwidth 8Mbit diffserv4
> dual-
> srchost nat nowash no-ack-filter split-gso rtt 100.0ms raw overhead 0
> fwmark 0xf no-sce
> Sent 102655686381 bytes 386831397 pkt (dropped 377557, overlimits
> 230324326 requeues 0)
> backlog 0b 0p requeues 0
> memory used: 8389696b of 8Mb
> capacity estimate: 8Mbit
> min/max network layer size: 42 / 1514
> min/max overhead-adjusted size: 42 / 1514
> average network hdr offset: 14
>
> Bulk Best Effort Video Voice
> thresh 500Kbit 8Mbit 4Mbit 2Mbit
> target 36.3ms 5.0ms 5.0ms 9.1ms
> interval 131.3ms 100.0ms 100.0ms 104.1ms
> pk_delay 17.5ms 4.1ms 1.7ms 2.0ms
> av_delay 4.5ms 232us 82us 382us
> sp_delay 14us 23us 6us 7us
> backlog 0b 0b 0b 0b
> pkts 165781 324229040 61032588 1781545
> bytes 45344666 98316181815 4518083349 291035866
> way_inds 42 18492140 7782656 34167
> way_miss 4289 6575635 77917 62650
> way_cols 0 340 0 0
> sce 0 0 0 0
> marks 7 19542 1 0
> drops 8 377528 15 6
> ack_drop 0 0 0 0
> sp_flows 0 2 0 1
> bk_flows 0 1 0 0
> un_flows 0 0 0 0
> max_len 1514 1514 1514 1514
> quantum 300 300 300 300
>
> Ingress:
> --------
>
> qdisc cake 1: dev ifb4eth4 root refcnt 2 bandwidth 40Mbit diffserv4
> dual-dsthost nat nowash no-ack-filter split-gso rtt 100.0ms raw
> overhead 0 fwmark 0xf no-sce
> Sent 1081750548776 bytes 839670608 pkt (dropped 1616216, overlimits
> 657093747 requeues 0)
> backlog 0b 0p requeues 0
> memory used: 6027328b of 8Mb
> capacity estimate: 40Mbit
> min/max network layer size: 60 / 1514
> min/max overhead-adjusted size: 60 / 1514
> average network hdr offset: 14
>
> Bulk Best Effort Video Voice
> thresh 2500Kbit 40Mbit 20Mbit 10Mbit
> target 7.3ms 5.0ms 5.0ms 5.0ms
> interval 102.3ms 100.0ms 100.0ms 100.0ms
> pk_delay 2.0ms 6.5ms 692us 1.1ms
> av_delay 248us 2.6ms 88us 139us
> sp_delay 12us 16us 7us 6us
> backlog 0b 0b 0b 0b
> pkts 29132 682090182 155330794 3836716
> bytes 25452377 850788764602 230875228776 2357107850
> way_inds 3 37939194 21198287 71120
> way_miss 9448 6525310 97141 51428
> way_cols 0 107 0 0
> sce 0 0 0 0
> marks 0 206496 203 30
> drops 30 1534636 80809 741
> ack_drop 0 0 0 0
> sp_flows 0 2 0 0
> bk_flows 0 1 0 0
> un_flows 0 0 0 0
> max_len 1484 1514 1514 1514
> quantum 300 1220 610 305
>
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Cake] Cake memory usage after 1TB
2021-06-24 13:26 ` Pete Heist
@ 2021-06-24 13:33 ` Dave Taht
0 siblings, 0 replies; 8+ messages in thread
From: Dave Taht @ 2021-06-24 13:33 UTC (permalink / raw)
To: Pete Heist; +Cc: Cake List
it would be interesting to turn the ack-filter on in this environment.
On Thu, Jun 24, 2021 at 6:27 AM Pete Heist <pete@heistp.net> wrote:
>
> Hmm, since I see in the code that this could lead to drops, I've
> restarted it and upped the limit to 16M. I haven't had reports of
> widespread problems, but I'll keep an eye on this more closely. :)
>
> Pete
>
> On Thu, 2021-06-24 at 15:06 +0200, Pete Heist wrote:
> > I'm using Cake on an EdgeRouter-X at a site that does around 50-
> > 100GB/day, and we're quite happy with it so far. Egress and ingress
> > stats are below after ~1TB download and ~100GB upload.
> >
> > I use "memlimit 8M", and noticed that it's reporting 8389696 bytes
> > used
> > on egress, which is at the maximum (slightly over actually). Around
> > 6M
> > is used on ingress. Should I then raise these limits, or is it normal
> > for it to settle in around the maximum over time?
> >
> > Pete
> >
> >
> > Egress:
> > -------
> >
> > qdisc cake 8008: dev eth4 root refcnt 2 bandwidth 8Mbit diffserv4
> > dual-
> > srchost nat nowash no-ack-filter split-gso rtt 100.0ms raw overhead 0
> > fwmark 0xf no-sce
> > Sent 102655686381 bytes 386831397 pkt (dropped 377557, overlimits
> > 230324326 requeues 0)
> > backlog 0b 0p requeues 0
> > memory used: 8389696b of 8Mb
> > capacity estimate: 8Mbit
> > min/max network layer size: 42 / 1514
> > min/max overhead-adjusted size: 42 / 1514
> > average network hdr offset: 14
> >
> > Bulk Best Effort Video Voice
> > thresh 500Kbit 8Mbit 4Mbit 2Mbit
> > target 36.3ms 5.0ms 5.0ms 9.1ms
> > interval 131.3ms 100.0ms 100.0ms 104.1ms
> > pk_delay 17.5ms 4.1ms 1.7ms 2.0ms
> > av_delay 4.5ms 232us 82us 382us
> > sp_delay 14us 23us 6us 7us
> > backlog 0b 0b 0b 0b
> > pkts 165781 324229040 61032588 1781545
> > bytes 45344666 98316181815 4518083349 291035866
> > way_inds 42 18492140 7782656 34167
> > way_miss 4289 6575635 77917 62650
> > way_cols 0 340 0 0
> > sce 0 0 0 0
> > marks 7 19542 1 0
> > drops 8 377528 15 6
> > ack_drop 0 0 0 0
> > sp_flows 0 2 0 1
> > bk_flows 0 1 0 0
> > un_flows 0 0 0 0
> > max_len 1514 1514 1514 1514
> > quantum 300 300 300 300
> >
> > Ingress:
> > --------
> >
> > qdisc cake 1: dev ifb4eth4 root refcnt 2 bandwidth 40Mbit diffserv4
> > dual-dsthost nat nowash no-ack-filter split-gso rtt 100.0ms raw
> > overhead 0 fwmark 0xf no-sce
> > Sent 1081750548776 bytes 839670608 pkt (dropped 1616216, overlimits
> > 657093747 requeues 0)
> > backlog 0b 0p requeues 0
> > memory used: 6027328b of 8Mb
> > capacity estimate: 40Mbit
> > min/max network layer size: 60 / 1514
> > min/max overhead-adjusted size: 60 / 1514
> > average network hdr offset: 14
> >
> > Bulk Best Effort Video Voice
> > thresh 2500Kbit 40Mbit 20Mbit 10Mbit
> > target 7.3ms 5.0ms 5.0ms 5.0ms
> > interval 102.3ms 100.0ms 100.0ms 100.0ms
> > pk_delay 2.0ms 6.5ms 692us 1.1ms
> > av_delay 248us 2.6ms 88us 139us
> > sp_delay 12us 16us 7us 6us
> > backlog 0b 0b 0b 0b
> > pkts 29132 682090182 155330794 3836716
> > bytes 25452377 850788764602 230875228776 2357107850
> > way_inds 3 37939194 21198287 71120
> > way_miss 9448 6525310 97141 51428
> > way_cols 0 107 0 0
> > sce 0 0 0 0
> > marks 0 206496 203 30
> > drops 30 1534636 80809 741
> > ack_drop 0 0 0 0
> > sp_flows 0 2 0 0
> > bk_flows 0 1 0 0
> > un_flows 0 0 0 0
> > max_len 1484 1514 1514 1514
> > quantum 300 1220 610 305
> >
> >
>
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
--
Latest Podcast:
https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/
Dave Täht CTO, TekLibre, LLC
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Cake] Cake memory usage after 1TB
2021-06-24 13:06 [Cake] Cake memory usage after 1TB Pete Heist
2021-06-24 13:26 ` Pete Heist
@ 2021-06-24 13:59 ` Toke Høiland-Jørgensen
2021-06-24 15:00 ` Pete Heist
2021-06-24 15:07 ` Sebastian Moeller
1 sibling, 2 replies; 8+ messages in thread
From: Toke Høiland-Jørgensen @ 2021-06-24 13:59 UTC (permalink / raw)
To: Pete Heist, Cake List
Pete Heist <pete@heistp.net> writes:
> I'm using Cake on an EdgeRouter-X at a site that does around 50-
> 100GB/day, and we're quite happy with it so far. Egress and ingress
> stats are below after ~1TB download and ~100GB upload.
>
> I use "memlimit 8M", and noticed that it's reporting 8389696 bytes used
> on egress, which is at the maximum (slightly over actually). Around 6M
> is used on ingress. Should I then raise these limits, or is it normal
> for it to settle in around the maximum over time?
That stat is the maximum memory ever used, not the current usage (that
is reported by "backlog 0b 0p"). So if your queue ever overflows it'll
be at its maximum.
-Toke
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Cake] Cake memory usage after 1TB
2021-06-24 13:59 ` Toke Høiland-Jørgensen
@ 2021-06-24 15:00 ` Pete Heist
2021-06-24 15:07 ` Sebastian Moeller
1 sibling, 0 replies; 8+ messages in thread
From: Pete Heist @ 2021-06-24 15:00 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Cake List
On Thu, 2021-06-24 at 15:59 +0200, Toke Høiland-Jørgensen wrote:
> Pete Heist <pete@heistp.net> writes:
>
> > I'm using Cake on an EdgeRouter-X at a site that does around 50-
> > 100GB/day, and we're quite happy with it so far. Egress and ingress
> > stats are below after ~1TB download and ~100GB upload.
> >
> > I use "memlimit 8M", and noticed that it's reporting 8389696 bytes
> > used
> > on egress, which is at the maximum (slightly over actually). Around
> > 6M
> > is used on ingress. Should I then raise these limits, or is it
> > normal
> > for it to settle in around the maximum over time?
>
> That stat is the maximum memory ever used, not the current usage
> (that
> is reported by "backlog 0b 0p"). So if your queue ever overflows
> it'll
> be at its maximum.
Thanks, it's not inconceivable that it reached that at some point, if
there were an unresponsive flow for example. After my bump to 16M, I'll
get notified now if it reaches that, for curiosity's sake.
Pete
>
> -Toke
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Cake] Cake memory usage after 1TB
2021-06-24 13:59 ` Toke Høiland-Jørgensen
2021-06-24 15:00 ` Pete Heist
@ 2021-06-24 15:07 ` Sebastian Moeller
2021-06-24 15:20 ` Toke Høiland-Jørgensen
1 sibling, 1 reply; 8+ messages in thread
From: Sebastian Moeller @ 2021-06-24 15:07 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Pete Heist, Cake List
> On Jun 24, 2021, at 15:59, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> Pete Heist <pete@heistp.net> writes:
>
>> I'm using Cake on an EdgeRouter-X at a site that does around 50-
>> 100GB/day, and we're quite happy with it so far. Egress and ingress
>> stats are below after ~1TB download and ~100GB upload.
>>
>> I use "memlimit 8M", and noticed that it's reporting 8389696 bytes used
>> on egress, which is at the maximum (slightly over actually). Around 6M
>> is used on ingress. Should I then raise these limits, or is it normal
>> for it to settle in around the maximum over time?
>
> That stat is the maximum memory ever used, not the current usage (that
> is reported by "backlog 0b 0p"). So if your queue ever overflows it'll
> be at its maximum.
Is there an easy way to reset that, short of tearing down the qdisc set-up and building it up again?
Regards
Sebastian
>
> -Toke
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Cake] Cake memory usage after 1TB
2021-06-24 15:07 ` Sebastian Moeller
@ 2021-06-24 15:20 ` Toke Høiland-Jørgensen
2021-06-24 15:25 ` Dave Taht
0 siblings, 1 reply; 8+ messages in thread
From: Toke Høiland-Jørgensen @ 2021-06-24 15:20 UTC (permalink / raw)
To: Sebastian Moeller; +Cc: Pete Heist, Cake List
Sebastian Moeller <moeller0@gmx.de> writes:
>> On Jun 24, 2021, at 15:59, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>
>> Pete Heist <pete@heistp.net> writes:
>>
>>> I'm using Cake on an EdgeRouter-X at a site that does around 50-
>>> 100GB/day, and we're quite happy with it so far. Egress and ingress
>>> stats are below after ~1TB download and ~100GB upload.
>>>
>>> I use "memlimit 8M", and noticed that it's reporting 8389696 bytes used
>>> on egress, which is at the maximum (slightly over actually). Around 6M
>>> is used on ingress. Should I then raise these limits, or is it normal
>>> for it to settle in around the maximum over time?
>>
>> That stat is the maximum memory ever used, not the current usage (that
>> is reported by "backlog 0b 0p"). So if your queue ever overflows it'll
>> be at its maximum.
>
> Is there an easy way to reset that, short of tearing down the
> qdisc set-up and building it up again?
Nope, 'fraid not :)
-Toke
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Cake] Cake memory usage after 1TB
2021-06-24 15:20 ` Toke Høiland-Jørgensen
@ 2021-06-24 15:25 ` Dave Taht
0 siblings, 0 replies; 8+ messages in thread
From: Dave Taht @ 2021-06-24 15:25 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Sebastian Moeller, Cake List
extremely interested in ack-filtering results both ways.
On Thu, Jun 24, 2021 at 8:21 AM Toke Høiland-Jørgensen via Cake
<cake@lists.bufferbloat.net> wrote:
>
> Sebastian Moeller <moeller0@gmx.de> writes:
>
> >> On Jun 24, 2021, at 15:59, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> >>
> >> Pete Heist <pete@heistp.net> writes:
> >>
> >>> I'm using Cake on an EdgeRouter-X at a site that does around 50-
> >>> 100GB/day, and we're quite happy with it so far. Egress and ingress
> >>> stats are below after ~1TB download and ~100GB upload.
> >>>
> >>> I use "memlimit 8M", and noticed that it's reporting 8389696 bytes used
> >>> on egress, which is at the maximum (slightly over actually). Around 6M
> >>> is used on ingress. Should I then raise these limits, or is it normal
> >>> for it to settle in around the maximum over time?
> >>
> >> That stat is the maximum memory ever used, not the current usage (that
> >> is reported by "backlog 0b 0p"). So if your queue ever overflows it'll
> >> be at its maximum.
> >
> > Is there an easy way to reset that, short of tearing down the
> > qdisc set-up and building it up again?
>
> Nope, 'fraid not :)
>
> -Toke
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
--
Latest Podcast:
https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/
Dave Täht CTO, TekLibre, LLC
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2021-06-24 15:26 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-24 13:06 [Cake] Cake memory usage after 1TB Pete Heist
2021-06-24 13:26 ` Pete Heist
2021-06-24 13:33 ` Dave Taht
2021-06-24 13:59 ` Toke Høiland-Jørgensen
2021-06-24 15:00 ` Pete Heist
2021-06-24 15:07 ` Sebastian Moeller
2021-06-24 15:20 ` Toke Høiland-Jørgensen
2021-06-24 15:25 ` Dave Taht
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox