From pete at heistp.net Sun Jun 6 06:11:54 2021 From: pete at heistp.net (Pete Heist) Date: Sun, 06 Jun 2021 12:11:54 +0200 Subject: [Cake] customizing Cake's isolation with ipsets, tc-flow and eBPF Message-ID: <22f3032d0dfd47f53d4d6595ee6bd192377fbc6e.camel@heistp.net> I've always wanted a way to customize Cake's host and flow isolation in a way that would be usable e.g. for small ISPs, and this is what I came up with: https://github.com/heistp/cake-custom-isolation ipsets are used to set the skb priority or mark, then tc-flow or a simple eBPF classifier is used in a child filter of cake to get the major and minor class IDs set, which override the host and flow hashes. To show it in action, the cakeiso.sh script sets up a netns environment and runs competition between two "subscribers" and three flows, two TCP flows and one unresponsive UDP flow. Several configurations are run to show what is and isn't possible. If anyone knows of a simpler way than eBPF to get both the major and minor class ID set from ipsets, I'd like to hear it, but the included classifiers are at least very simple one-liners... Pete From toke at toke.dk Sun Jun 6 15:59:14 2021 From: toke at toke.dk (Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?=) Date: Sun, 06 Jun 2021 21:59:14 +0200 Subject: [Cake] customizing Cake's isolation with ipsets, tc-flow and eBPF In-Reply-To: <22f3032d0dfd47f53d4d6595ee6bd192377fbc6e.camel@heistp.net> References: <22f3032d0dfd47f53d4d6595ee6bd192377fbc6e.camel@heistp.net> Message-ID: <87h7iawyr1.fsf@toke.dk> Pete Heist writes: > I've always wanted a way to customize Cake's host and flow isolation in > a way that would be usable e.g. for small ISPs, and this is what I came > up with: > > https://github.com/heistp/cake-custom-isolation > > ipsets are used to set the skb priority or mark, then tc-flow or a > simple eBPF classifier is used in a child filter of cake to get the > major and minor class IDs set, which override the host and flow hashes. Very cool! Awesome to see the customisation options being used for something neat like this! :) > To show it in action, the cakeiso.sh script sets up a netns environment > and runs competition between two "subscribers" and three flows, two TCP > flows and one unresponsive UDP flow. Several configurations are run to > show what is and isn't possible. > > If anyone knows of a simpler way than eBPF to get both the major and > minor class ID set from ipsets, I'd like to hear it, but the included > classifiers are at least very simple one-liners... Well, you could go the other way? Instead of ipset, just do the classification in eBPF and use a BPF map to store the IP addresses. There's even an LPM map type, so you can use arbitrary prefix lengths for each class (or not, and just use a hashmap)... -Toke From pete at heistp.net Sun Jun 6 16:26:42 2021 From: pete at heistp.net (Pete Heist) Date: Sun, 06 Jun 2021 22:26:42 +0200 Subject: [Cake] customizing Cake's isolation with ipsets, tc-flow and eBPF In-Reply-To: <87h7iawyr1.fsf@toke.dk> References: <22f3032d0dfd47f53d4d6595ee6bd192377fbc6e.camel@heistp.net> <87h7iawyr1.fsf@toke.dk> Message-ID: On Sun, 2021-06-06 at 21:59 +0200, Toke Høiland-Jørgensen wrote: > Pete Heist writes: > > > I've always wanted a way to customize Cake's host and flow isolation > > in > > a way that would be usable e.g. for small ISPs, and this is what I > > came > > up with: > > > > https://github.com/heistp/cake-custom-isolation > > > > ipsets are used to set the skb priority or mark, then tc-flow or a > > simple eBPF classifier is used in a child filter of cake to get the > > major and minor class IDs set, which override the host and flow > > hashes. > > Very cool! Awesome to see the customisation options being used for > something neat like this! :) > > > To show it in action, the cakeiso.sh script sets up a netns > > environment > > and runs competition between two "subscribers" and three flows, two > > TCP > > flows and one unresponsive UDP flow. Several configurations are run > > to > > show what is and isn't possible. > > > > If anyone knows of a simpler way than eBPF to get both the major and > > minor class ID set from ipsets, I'd like to hear it, but the included > > classifiers are at least very simple one-liners... > > Well, you could go the other way? Instead of ipset, just do the > classification in eBPF and use a BPF map to store the IP addresses. > There's even an LPM map type, so you can use arbitrary prefix lengths > for each class (or not, and just use a hashmap)... True that, I started something like that at some point: https://github.com/heistp/tc-users/ but I think I got a little overzealous with it. I'm not sure if/when I'll get back to that, but the ipset solution seems to be "good enough" for what I (and my ISP) needs. I'm glad you slipped the tc filter overrides in before Cake went out the door. :) This doesn't do away with the possible need for a full-blown ISP qdisc one day, with configurable subscriber tiers, handling of higher loads, etc, but at least it's something for the little guys. Pete > -Toke From toke at toke.dk Sun Jun 6 17:01:26 2021 From: toke at toke.dk (Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?=) Date: Sun, 06 Jun 2021 23:01:26 +0200 Subject: [Cake] customizing Cake's isolation with ipsets, tc-flow and eBPF In-Reply-To: References: <22f3032d0dfd47f53d4d6595ee6bd192377fbc6e.camel@heistp.net> <87h7iawyr1.fsf@toke.dk> Message-ID: <87eedewvvd.fsf@toke.dk> Pete Heist writes: > On Sun, 2021-06-06 at 21:59 +0200, Toke Høiland-Jørgensen wrote: >> Pete Heist writes: >> >> > I've always wanted a way to customize Cake's host and flow isolation >> > in >> > a way that would be usable e.g. for small ISPs, and this is what I >> > came >> > up with: >> > >> > https://github.com/heistp/cake-custom-isolation >> > >> > ipsets are used to set the skb priority or mark, then tc-flow or a >> > simple eBPF classifier is used in a child filter of cake to get the >> > major and minor class IDs set, which override the host and flow >> > hashes. >> >> Very cool! Awesome to see the customisation options being used for >> something neat like this! :) >> >> > To show it in action, the cakeiso.sh script sets up a netns >> > environment >> > and runs competition between two "subscribers" and three flows, two >> > TCP >> > flows and one unresponsive UDP flow. Several configurations are run >> > to >> > show what is and isn't possible. >> > >> > If anyone knows of a simpler way than eBPF to get both the major and >> > minor class ID set from ipsets, I'd like to hear it, but the included >> > classifiers are at least very simple one-liners... >> >> Well, you could go the other way? Instead of ipset, just do the >> classification in eBPF and use a BPF map to store the IP addresses. >> There's even an LPM map type, so you can use arbitrary prefix lengths >> for each class (or not, and just use a hashmap)... > > True that, I started something like that at some point: > > https://github.com/heistp/tc-users/ > > but I think I got a little overzealous with it. I'm not sure if/when > I'll get back to that, but the ipset solution seems to be "good enough" > for what I (and my ISP) needs. I'm glad you slipped the tc filter > overrides in before Cake went out the door. :) Yes! I would love to take credit for the idea, but this came from the netdev review. I agree though, great that it ended up in there! > This doesn't do away with the possible need for a full-blown ISP qdisc > one day, with configurable subscriber tiers, handling of higher loads, > etc, but at least it's something for the little guys. Yeah; I have something in the pipeline that will hopefully end up being useable for that, but may take a little while to get there :) -Toke From dave.taht at gmail.com Sat Jun 19 17:59:02 2021 From: dave.taht at gmail.com (Dave Taht) Date: Sat, 19 Jun 2021 14:59:02 -0700 Subject: [Cake] access to cmsg from go? Message-ID: anyone have any good ideas here? https://github.com/golang/go/issues/46831 -- Latest Podcast: https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/ Dave Täht CTO, TekLibre, LLC From mattmathis at google.com Sat Jun 19 21:59:06 2021 From: mattmathis at google.com (Matt Mathis) Date: Sat, 19 Jun 2021 18:59:06 -0700 Subject: [Cake] access to cmsg from go? In-Reply-To: References: Message-ID: Is there running code in C? Thanks, --MM-- The best way to predict the future is to create it. - Alan Kay We must not tolerate intolerance; however our response must be carefully measured: too strong would be hypocritical and risks spiraling out of control; too weak risks being mistaken for tacit approval. On Sat, Jun 19, 2021 at 2:59 PM Dave Taht wrote: > anyone have any good ideas here? https://github.com/golang/go/issues/46831 > > -- > Latest Podcast: > https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/ > > Dave Täht CTO, TekLibre, LLC > _______________________________________________ > Cake mailing list > Cake at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake > -------------- next part -------------- An HTML attachment was scrubbed... URL: From halmurray+bufferbloat at sonic.net Sun Jun 20 16:08:33 2021 From: halmurray+bufferbloat at sonic.net (Hal Murray) Date: Sun, 20 Jun 2021 13:08:33 -0700 Subject: [Cake] [Bloat] access to cmsg from go? In-Reply-To: Message from Matt Mathis of "Sat, 19 Jun 2021 18:59:06 -0700." Message-ID: <20210620200833.1AC6A28C157@107-137-68-211.lightspeed.sntcca.sbcglobal.net> > Is there running code in C? Yes. The NTPsec code is full of #ifdef kludgery. The C API started with SO_TIMESTAMP for microsecond precision then added SO_TIMESTAMPNS for nanosecond precision: timeval vs timespec There is also SO_TIMESTAMPNS vs SCM_TIMESTAMPNS The basic idea is to use recvmsg rather than recv. In addition to the data buffer, you feed it another buffer where it can put meta data like timestamps. Then you have to scan that buffer to find the part you want. Both buffers are passed to the kernel via a single msghdr parameter (which also has a pointer for the addr you get via recvfrom). Details are in man recvmsg and man cmsg I thought they were reasonably clear. Without the ifdefs: You have to start by using setsockopt to turn on SO_TIMESTAMPNS. setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPNS, (const void *)&on, sizeof(on))); The setup code for recvmsg: struct msghdr msghdr; struct iovec iovec; char control[100]; /* FIXME: Need space for time stamp plus overhead */ iovec.iov_base = &rb->recv_buffer; iovec.iov_len = sizeof(rb->recv_buffer); memset(&msghdr, '\0', sizeof(msghdr)); msghdr.msg_name = &rb->recv_srcadr; msghdr.msg_namelen = fromlen; msghdr.msg_iov = &iovec; msghdr.msg_iovlen = 1; msghdr.msg_flags = 0; msghdr.msg_control = (void *)&control; msghdr.msg_controllen = sizeof(control); The actual call: buflen = recvmsg(fd, &msghdr, 0); The extract code: struct timespec * tsp; cmsghdr = CMSG_FIRSTHDR(msghdr); if (NULL == cmsghdr) { extra checking because this code is in a subroutine error } if (SCM_TIMESTAMPNS != cmsghdr->cmsg_type) { There is only one -- no need to loop error } tsp = (struct timespec *)CMSG_DATA(cmsghdr); Actual code is in: read_network_packet in ntpd/ntp_io.c https://gitlab.com/NTPsec/ntpsec/-/blob/master/ntpd/ntp_io.c and fetch_packetstamp() in ntpd/ntp_packetstamp.c https://gitlab.com/NTPsec/ntpsec/-/blob/master/ntpd/ntp_packetstamp.c -- These are my opinions. I hate spam. From dpreed at deepplum.com Wed Jun 23 12:09:15 2021 From: dpreed at deepplum.com (David P. Reed) Date: Wed, 23 Jun 2021 12:09:15 -0400 (EDT) Subject: [Cake] =?utf-8?q?=5BCerowrt-devel=5D__access_to_cmsg_from_go=3F?= In-Reply-To: References:

Message-ID: <1624464555.230819179@apps.rackspace.com> (They closed the issue on the golang link.) I'm not a golang user. One language too many for me. It sounds like a library issue. My suggestion would be to use the openness of open source. Generate a patchset that extends the interface properly. Don't try to "improve" what you don't like - communities like stability and backward compatibility. Explain the added semantics in documentation. Then, maintain your fork. I don't know how the golang community works with versioning of libraries, but Python, Rust, Haskell, and NodeJS all have ways to let projects use variants of libraries. Then, submit that patchset upstream to the golang community. Advocate for upstreaming it, and develop a community that uses the patched library. Eventually, you may be able to stop maintaining your variant toolset. Or you will develop an alternat library user base that disagrees with upstream's decisions. (Analogy, Android's Linux Kernel vs. Linus Torvalds's. Google Android rejects to some extent Linus's crew's unwillingness to accept what Android needs as improvements.) This is "modern open source community" practice for getting things done. Pragmatic innovations in shared codebases sometimes have to wait for the original egos to die off. From champetier.etienne at gmail.com Wed Jun 23 15:29:47 2021 From: champetier.etienne at gmail.com (Etienne Champetier) Date: Wed, 23 Jun 2021 15:29:47 -0400 Subject: [Cake] [Cerowrt-devel] access to cmsg from go? In-Reply-To: References: Message-ID: Hello Dave, Le sam. 19 juin 2021 à 17:59, Dave Taht a écrit : > > anyone have any good ideas here? https://github.com/golang/go/issues/46831 (with the list in cc) Some googling that might (or not) help: https://github.com/google/gopacket/blob/master/pcapgo/capture.go https://github.com/gracig/goping/blob/master/pingers/icmpv4/icmpv4_linux.go#L217 Best Etienne > > -- > Latest Podcast: > https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/ > > Dave Täht CTO, TekLibre, LLC > _______________________________________________ > Cerowrt-devel mailing list > Cerowrt-devel at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cerowrt-devel From pete at heistp.net Thu Jun 24 09:06:00 2021 From: pete at heistp.net (Pete Heist) Date: Thu, 24 Jun 2021 15:06:00 +0200 Subject: [Cake] Cake memory usage after 1TB Message-ID: <3d529e0b78ef8ea54021d8f8e93285e9a474cba1.camel@heistp.net> I'm using Cake on an EdgeRouter-X at a site that does around 50- 100GB/day, and we're quite happy with it so far. Egress and ingress stats are below after ~1TB download and ~100GB upload. I use "memlimit 8M", and noticed that it's reporting 8389696 bytes used on egress, which is at the maximum (slightly over actually). Around 6M is used on ingress. Should I then raise these limits, or is it normal for it to settle in around the maximum over time? Pete Egress: ------- qdisc cake 8008: dev eth4 root refcnt 2 bandwidth 8Mbit diffserv4 dual- srchost nat nowash no-ack-filter split-gso rtt 100.0ms raw overhead 0 fwmark 0xf no-sce Sent 102655686381 bytes 386831397 pkt (dropped 377557, overlimits 230324326 requeues 0) backlog 0b 0p requeues 0 memory used: 8389696b of 8Mb capacity estimate: 8Mbit min/max network layer size: 42 / 1514 min/max overhead-adjusted size: 42 / 1514 average network hdr offset: 14 Bulk Best Effort Video Voice thresh 500Kbit 8Mbit 4Mbit 2Mbit target 36.3ms 5.0ms 5.0ms 9.1ms interval 131.3ms 100.0ms 100.0ms 104.1ms pk_delay 17.5ms 4.1ms 1.7ms 2.0ms av_delay 4.5ms 232us 82us 382us sp_delay 14us 23us 6us 7us backlog 0b 0b 0b 0b pkts 165781 324229040 61032588 1781545 bytes 45344666 98316181815 4518083349 291035866 way_inds 42 18492140 7782656 34167 way_miss 4289 6575635 77917 62650 way_cols 0 340 0 0 sce 0 0 0 0 marks 7 19542 1 0 drops 8 377528 15 6 ack_drop 0 0 0 0 sp_flows 0 2 0 1 bk_flows 0 1 0 0 un_flows 0 0 0 0 max_len 1514 1514 1514 1514 quantum 300 300 300 300 Ingress: -------- qdisc cake 1: dev ifb4eth4 root refcnt 2 bandwidth 40Mbit diffserv4 dual-dsthost nat nowash no-ack-filter split-gso rtt 100.0ms raw overhead 0 fwmark 0xf no-sce Sent 1081750548776 bytes 839670608 pkt (dropped 1616216, overlimits 657093747 requeues 0) backlog 0b 0p requeues 0 memory used: 6027328b of 8Mb capacity estimate: 40Mbit min/max network layer size: 60 / 1514 min/max overhead-adjusted size: 60 / 1514 average network hdr offset: 14 Bulk Best Effort Video Voice thresh 2500Kbit 40Mbit 20Mbit 10Mbit target 7.3ms 5.0ms 5.0ms 5.0ms interval 102.3ms 100.0ms 100.0ms 100.0ms pk_delay 2.0ms 6.5ms 692us 1.1ms av_delay 248us 2.6ms 88us 139us sp_delay 12us 16us 7us 6us backlog 0b 0b 0b 0b pkts 29132 682090182 155330794 3836716 bytes 25452377 850788764602 230875228776 2357107850 way_inds 3 37939194 21198287 71120 way_miss 9448 6525310 97141 51428 way_cols 0 107 0 0 sce 0 0 0 0 marks 0 206496 203 30 drops 30 1534636 80809 741 ack_drop 0 0 0 0 sp_flows 0 2 0 0 bk_flows 0 1 0 0 un_flows 0 0 0 0 max_len 1484 1514 1514 1514 quantum 300 1220 610 305 From pete at heistp.net Thu Jun 24 09:26:53 2021 From: pete at heistp.net (Pete Heist) Date: Thu, 24 Jun 2021 15:26:53 +0200 Subject: [Cake] Cake memory usage after 1TB In-Reply-To: <3d529e0b78ef8ea54021d8f8e93285e9a474cba1.camel@heistp.net> References: <3d529e0b78ef8ea54021d8f8e93285e9a474cba1.camel@heistp.net> Message-ID: Hmm, since I see in the code that this could lead to drops, I've restarted it and upped the limit to 16M. I haven't had reports of widespread problems, but I'll keep an eye on this more closely. :) Pete On Thu, 2021-06-24 at 15:06 +0200, Pete Heist wrote: > I'm using Cake on an EdgeRouter-X at a site that does around 50- > 100GB/day, and we're quite happy with it so far. Egress and ingress > stats are below after ~1TB download and ~100GB upload. > > I use "memlimit 8M", and noticed that it's reporting 8389696 bytes > used > on egress, which is at the maximum (slightly over actually). Around > 6M > is used on ingress. Should I then raise these limits, or is it normal > for it to settle in around the maximum over time? > > Pete > > > Egress: > ------- > > qdisc cake 8008: dev eth4 root refcnt 2 bandwidth 8Mbit diffserv4 > dual- > srchost nat nowash no-ack-filter split-gso rtt 100.0ms raw overhead 0 > fwmark 0xf no-sce > Sent 102655686381 bytes 386831397 pkt (dropped 377557, overlimits > 230324326 requeues 0) > backlog 0b 0p requeues 0 > memory used: 8389696b of 8Mb > capacity estimate: 8Mbit > min/max network layer size: 42 / 1514 > min/max overhead-adjusted size: 42 / 1514 > average network hdr offset: 14 > > Bulk Best Effort Video Voice > thresh 500Kbit 8Mbit 4Mbit 2Mbit > target 36.3ms 5.0ms 5.0ms 9.1ms > interval 131.3ms 100.0ms 100.0ms 104.1ms > pk_delay 17.5ms 4.1ms 1.7ms 2.0ms > av_delay 4.5ms 232us 82us 382us > sp_delay 14us 23us 6us 7us > backlog 0b 0b 0b 0b > pkts 165781 324229040 61032588 1781545 > bytes 45344666 98316181815 4518083349 291035866 > way_inds 42 18492140 7782656 34167 > way_miss 4289 6575635 77917 62650 > way_cols 0 340 0 0 > sce 0 0 0 0 > marks 7 19542 1 0 > drops 8 377528 15 6 > ack_drop 0 0 0 0 > sp_flows 0 2 0 1 > bk_flows 0 1 0 0 > un_flows 0 0 0 0 > max_len 1514 1514 1514 1514 > quantum 300 300 300 300 > > Ingress: > -------- > > qdisc cake 1: dev ifb4eth4 root refcnt 2 bandwidth 40Mbit diffserv4 > dual-dsthost nat nowash no-ack-filter split-gso rtt 100.0ms raw > overhead 0 fwmark 0xf no-sce > Sent 1081750548776 bytes 839670608 pkt (dropped 1616216, overlimits > 657093747 requeues 0) > backlog 0b 0p requeues 0 > memory used: 6027328b of 8Mb > capacity estimate: 40Mbit > min/max network layer size: 60 / 1514 > min/max overhead-adjusted size: 60 / 1514 > average network hdr offset: 14 > > Bulk Best Effort Video Voice > thresh 2500Kbit 40Mbit 20Mbit 10Mbit > target 7.3ms 5.0ms 5.0ms 5.0ms > interval 102.3ms 100.0ms 100.0ms 100.0ms > pk_delay 2.0ms 6.5ms 692us 1.1ms > av_delay 248us 2.6ms 88us 139us > sp_delay 12us 16us 7us 6us > backlog 0b 0b 0b 0b > pkts 29132 682090182 155330794 3836716 > bytes 25452377 850788764602 230875228776 2357107850 > way_inds 3 37939194 21198287 71120 > way_miss 9448 6525310 97141 51428 > way_cols 0 107 0 0 > sce 0 0 0 0 > marks 0 206496 203 30 > drops 30 1534636 80809 741 > ack_drop 0 0 0 0 > sp_flows 0 2 0 0 > bk_flows 0 1 0 0 > un_flows 0 0 0 0 > max_len 1484 1514 1514 1514 > quantum 300 1220 610 305 > > From dave.taht at gmail.com Thu Jun 24 09:33:30 2021 From: dave.taht at gmail.com (Dave Taht) Date: Thu, 24 Jun 2021 06:33:30 -0700 Subject: [Cake] Cake memory usage after 1TB In-Reply-To: References: <3d529e0b78ef8ea54021d8f8e93285e9a474cba1.camel@heistp.net> Message-ID: it would be interesting to turn the ack-filter on in this environment. On Thu, Jun 24, 2021 at 6:27 AM Pete Heist wrote: > > Hmm, since I see in the code that this could lead to drops, I've > restarted it and upped the limit to 16M. I haven't had reports of > widespread problems, but I'll keep an eye on this more closely. :) > > Pete > > On Thu, 2021-06-24 at 15:06 +0200, Pete Heist wrote: > > I'm using Cake on an EdgeRouter-X at a site that does around 50- > > 100GB/day, and we're quite happy with it so far. Egress and ingress > > stats are below after ~1TB download and ~100GB upload. > > > > I use "memlimit 8M", and noticed that it's reporting 8389696 bytes > > used > > on egress, which is at the maximum (slightly over actually). Around > > 6M > > is used on ingress. Should I then raise these limits, or is it normal > > for it to settle in around the maximum over time? > > > > Pete > > > > > > Egress: > > ------- > > > > qdisc cake 8008: dev eth4 root refcnt 2 bandwidth 8Mbit diffserv4 > > dual- > > srchost nat nowash no-ack-filter split-gso rtt 100.0ms raw overhead 0 > > fwmark 0xf no-sce > > Sent 102655686381 bytes 386831397 pkt (dropped 377557, overlimits > > 230324326 requeues 0) > > backlog 0b 0p requeues 0 > > memory used: 8389696b of 8Mb > > capacity estimate: 8Mbit > > min/max network layer size: 42 / 1514 > > min/max overhead-adjusted size: 42 / 1514 > > average network hdr offset: 14 > > > > Bulk Best Effort Video Voice > > thresh 500Kbit 8Mbit 4Mbit 2Mbit > > target 36.3ms 5.0ms 5.0ms 9.1ms > > interval 131.3ms 100.0ms 100.0ms 104.1ms > > pk_delay 17.5ms 4.1ms 1.7ms 2.0ms > > av_delay 4.5ms 232us 82us 382us > > sp_delay 14us 23us 6us 7us > > backlog 0b 0b 0b 0b > > pkts 165781 324229040 61032588 1781545 > > bytes 45344666 98316181815 4518083349 291035866 > > way_inds 42 18492140 7782656 34167 > > way_miss 4289 6575635 77917 62650 > > way_cols 0 340 0 0 > > sce 0 0 0 0 > > marks 7 19542 1 0 > > drops 8 377528 15 6 > > ack_drop 0 0 0 0 > > sp_flows 0 2 0 1 > > bk_flows 0 1 0 0 > > un_flows 0 0 0 0 > > max_len 1514 1514 1514 1514 > > quantum 300 300 300 300 > > > > Ingress: > > -------- > > > > qdisc cake 1: dev ifb4eth4 root refcnt 2 bandwidth 40Mbit diffserv4 > > dual-dsthost nat nowash no-ack-filter split-gso rtt 100.0ms raw > > overhead 0 fwmark 0xf no-sce > > Sent 1081750548776 bytes 839670608 pkt (dropped 1616216, overlimits > > 657093747 requeues 0) > > backlog 0b 0p requeues 0 > > memory used: 6027328b of 8Mb > > capacity estimate: 40Mbit > > min/max network layer size: 60 / 1514 > > min/max overhead-adjusted size: 60 / 1514 > > average network hdr offset: 14 > > > > Bulk Best Effort Video Voice > > thresh 2500Kbit 40Mbit 20Mbit 10Mbit > > target 7.3ms 5.0ms 5.0ms 5.0ms > > interval 102.3ms 100.0ms 100.0ms 100.0ms > > pk_delay 2.0ms 6.5ms 692us 1.1ms > > av_delay 248us 2.6ms 88us 139us > > sp_delay 12us 16us 7us 6us > > backlog 0b 0b 0b 0b > > pkts 29132 682090182 155330794 3836716 > > bytes 25452377 850788764602 230875228776 2357107850 > > way_inds 3 37939194 21198287 71120 > > way_miss 9448 6525310 97141 51428 > > way_cols 0 107 0 0 > > sce 0 0 0 0 > > marks 0 206496 203 30 > > drops 30 1534636 80809 741 > > ack_drop 0 0 0 0 > > sp_flows 0 2 0 0 > > bk_flows 0 1 0 0 > > un_flows 0 0 0 0 > > max_len 1484 1514 1514 1514 > > quantum 300 1220 610 305 > > > > > > > _______________________________________________ > Cake mailing list > Cake at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake -- Latest Podcast: https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/ Dave Täht CTO, TekLibre, LLC From toke at toke.dk Thu Jun 24 09:59:43 2021 From: toke at toke.dk (Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?=) Date: Thu, 24 Jun 2021 15:59:43 +0200 Subject: [Cake] Cake memory usage after 1TB In-Reply-To: <3d529e0b78ef8ea54021d8f8e93285e9a474cba1.camel@heistp.net> References: <3d529e0b78ef8ea54021d8f8e93285e9a474cba1.camel@heistp.net> Message-ID: <87h7hnmkhc.fsf@toke.dk> Pete Heist writes: > I'm using Cake on an EdgeRouter-X at a site that does around 50- > 100GB/day, and we're quite happy with it so far. Egress and ingress > stats are below after ~1TB download and ~100GB upload. > > I use "memlimit 8M", and noticed that it's reporting 8389696 bytes used > on egress, which is at the maximum (slightly over actually). Around 6M > is used on ingress. Should I then raise these limits, or is it normal > for it to settle in around the maximum over time? That stat is the maximum memory ever used, not the current usage (that is reported by "backlog 0b 0p"). So if your queue ever overflows it'll be at its maximum. -Toke From pete at heistp.net Thu Jun 24 11:00:59 2021 From: pete at heistp.net (Pete Heist) Date: Thu, 24 Jun 2021 17:00:59 +0200 Subject: [Cake] Cake memory usage after 1TB In-Reply-To: <87h7hnmkhc.fsf@toke.dk> References: <3d529e0b78ef8ea54021d8f8e93285e9a474cba1.camel@heistp.net> <87h7hnmkhc.fsf@toke.dk> Message-ID: On Thu, 2021-06-24 at 15:59 +0200, Toke Høiland-Jørgensen wrote: > Pete Heist writes: > > > I'm using Cake on an EdgeRouter-X at a site that does around 50- > > 100GB/day, and we're quite happy with it so far. Egress and ingress > > stats are below after ~1TB download and ~100GB upload. > > > > I use "memlimit 8M", and noticed that it's reporting 8389696 bytes > > used > > on egress, which is at the maximum (slightly over actually). Around > > 6M > > is used on ingress. Should I then raise these limits, or is it > > normal > > for it to settle in around the maximum over time? > > That stat is the maximum memory ever used, not the current usage > (that > is reported by "backlog 0b 0p"). So if your queue ever overflows > it'll > be at its maximum. Thanks, it's not inconceivable that it reached that at some point, if there were an unresponsive flow for example. After my bump to 16M, I'll get notified now if it reaches that, for curiosity's sake. Pete > > -Toke From moeller0 at gmx.de Thu Jun 24 11:07:23 2021 From: moeller0 at gmx.de (Sebastian Moeller) Date: Thu, 24 Jun 2021 17:07:23 +0200 Subject: [Cake] Cake memory usage after 1TB In-Reply-To: <87h7hnmkhc.fsf@toke.dk> References: <3d529e0b78ef8ea54021d8f8e93285e9a474cba1.camel@heistp.net> <87h7hnmkhc.fsf@toke.dk> Message-ID: <0341E253-B8CB-4976-ABC1-553C070E77F3@gmx.de> > On Jun 24, 2021, at 15:59, Toke Høiland-Jørgensen wrote: > > Pete Heist writes: > >> I'm using Cake on an EdgeRouter-X at a site that does around 50- >> 100GB/day, and we're quite happy with it so far. Egress and ingress >> stats are below after ~1TB download and ~100GB upload. >> >> I use "memlimit 8M", and noticed that it's reporting 8389696 bytes used >> on egress, which is at the maximum (slightly over actually). Around 6M >> is used on ingress. Should I then raise these limits, or is it normal >> for it to settle in around the maximum over time? > > That stat is the maximum memory ever used, not the current usage (that > is reported by "backlog 0b 0p"). So if your queue ever overflows it'll > be at its maximum. Is there an easy way to reset that, short of tearing down the qdisc set-up and building it up again? Regards Sebastian > > -Toke > _______________________________________________ > Cake mailing list > Cake at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake From toke at toke.dk Thu Jun 24 11:20:52 2021 From: toke at toke.dk (Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?=) Date: Thu, 24 Jun 2021 17:20:52 +0200 Subject: [Cake] Cake memory usage after 1TB In-Reply-To: <0341E253-B8CB-4976-ABC1-553C070E77F3@gmx.de> References: <3d529e0b78ef8ea54021d8f8e93285e9a474cba1.camel@heistp.net> <87h7hnmkhc.fsf@toke.dk> <0341E253-B8CB-4976-ABC1-553C070E77F3@gmx.de> Message-ID: <87bl7vmgq3.fsf@toke.dk> Sebastian Moeller writes: >> On Jun 24, 2021, at 15:59, Toke Høiland-Jørgensen wrote: >> >> Pete Heist writes: >> >>> I'm using Cake on an EdgeRouter-X at a site that does around 50- >>> 100GB/day, and we're quite happy with it so far. Egress and ingress >>> stats are below after ~1TB download and ~100GB upload. >>> >>> I use "memlimit 8M", and noticed that it's reporting 8389696 bytes used >>> on egress, which is at the maximum (slightly over actually). Around 6M >>> is used on ingress. Should I then raise these limits, or is it normal >>> for it to settle in around the maximum over time? >> >> That stat is the maximum memory ever used, not the current usage (that >> is reported by "backlog 0b 0p"). So if your queue ever overflows it'll >> be at its maximum. > > Is there an easy way to reset that, short of tearing down the > qdisc set-up and building it up again? Nope, 'fraid not :) -Toke From dave.taht at gmail.com Thu Jun 24 11:25:53 2021 From: dave.taht at gmail.com (Dave Taht) Date: Thu, 24 Jun 2021 08:25:53 -0700 Subject: [Cake] Cake memory usage after 1TB In-Reply-To: <87bl7vmgq3.fsf@toke.dk> References: <3d529e0b78ef8ea54021d8f8e93285e9a474cba1.camel@heistp.net> <87h7hnmkhc.fsf@toke.dk> <0341E253-B8CB-4976-ABC1-553C070E77F3@gmx.de> <87bl7vmgq3.fsf@toke.dk> Message-ID: extremely interested in ack-filtering results both ways. On Thu, Jun 24, 2021 at 8:21 AM Toke Høiland-Jørgensen via Cake wrote: > > Sebastian Moeller writes: > > >> On Jun 24, 2021, at 15:59, Toke Høiland-Jørgensen wrote: > >> > >> Pete Heist writes: > >> > >>> I'm using Cake on an EdgeRouter-X at a site that does around 50- > >>> 100GB/day, and we're quite happy with it so far. Egress and ingress > >>> stats are below after ~1TB download and ~100GB upload. > >>> > >>> I use "memlimit 8M", and noticed that it's reporting 8389696 bytes used > >>> on egress, which is at the maximum (slightly over actually). Around 6M > >>> is used on ingress. Should I then raise these limits, or is it normal > >>> for it to settle in around the maximum over time? > >> > >> That stat is the maximum memory ever used, not the current usage (that > >> is reported by "backlog 0b 0p"). So if your queue ever overflows it'll > >> be at its maximum. > > > > Is there an easy way to reset that, short of tearing down the > > qdisc set-up and building it up again? > > Nope, 'fraid not :) > > -Toke > _______________________________________________ > Cake mailing list > Cake at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake -- Latest Podcast: https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/ Dave Täht CTO, TekLibre, LLC From dave.taht at gmail.com Sat Jun 26 22:54:39 2021 From: dave.taht at gmail.com (Dave Taht) Date: Sat, 26 Jun 2021 19:54:39 -0700 Subject: [Cake] Fwd: ack filter rrul result at 200/10 In-Reply-To: References: Message-ID: I am curious if anyone has any tc -s qdisc show output of ack-filtering on both the up and down at any symmetry? ---------- Forwarded message --------- From: Dave Taht Date: Wed, Nov 15, 2017 at 9:22 PM Subject: Re: ack filter rrul result at 200/10 To: Cake List Cc: Jim Gettys and 200/10 looked reasonable too. -- Latest Podcast: https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/ Dave Täht CTO, TekLibre, LLC -------------- next part -------------- A non-text attachment was scrubbed... Name: rrul_be_-_ack-filter-200mbit-10mbit-really.png Type: image/png Size: 1026491 bytes Desc: not available URL: From ryan at mounce.com.au Sun Jun 27 02:30:24 2021 From: ryan at mounce.com.au (Ryan Mounce) Date: Sun, 27 Jun 2021 16:00:24 +0930 Subject: [Cake] Fwd: ack filter rrul result at 200/10 In-Reply-To: References: Message-ID: Hi Dave, EdgeRouter X running OpenWrt 19.07.4, shaping to 112/20 qdisc cake 1111: dev eth0.2 root refcnt 2 bandwidth 20Mbit diffserv3 dual-srchost nat wash ack-filter-aggressive split-gso rtt 100.0ms noatm overhead 26 mpu 64 fwmark 0x3 Sent 391625627749 bytes 493756837 pkt (dropped 4534914, overlimits 249653529 requeues 0) backlog 1216b 1p requeues 0 memory used: 4209984b of 4Mb capacity estimate: 20Mbit min/max network layer size: 28 / 1500 min/max overhead-adjusted size: 64 / 1526 average network hdr offset: 14 Bulk Best Effort Voice thresh 1250Kbit 20Mbit 5Mbit target 14.5ms 5.0ms 5.0ms interval 109.5ms 100.0ms 100.0ms pk_delay 6.4ms 6.1ms 4.3ms av_delay 1.5ms 668us 239us sp_delay 67us 22us 12us backlog 0b 1216b 0b pkts 260936279 231124444 6231029 bytes 313359244900 77984175519 690666701 way_inds 36626015 21720222 636374 way_miss 5483045 5979450 13296 way_cols 13 0 0 drops 35283 33392 108 marks 1635374 4611 1 ack_drop 139228 4326498 405 sp_flows 0 2 1 bk_flows 1 0 0 un_flows 0 0 0 max_len 67436 49038 1269 quantum 300 610 300 qdisc cake 8001: dev ibe0d2 root refcnt 2 bandwidth 112Mbit diffserv3 dual-dsthost nat nowash ingress no-ack-filter split-gso rtt 100.0ms noatm overhead 26 mpu 64 fwmark 0x3 Sent 622977131545 bytes 687017027 pkt (dropped 485937, overlimits 331101691 requeues 0) backlog 0b 0p requeues 0 memory used: 4052877b of 5600000b capacity estimate: 112Mbit min/max network layer size: 48 / 1500 min/max overhead-adjusted size: 74 / 1526 average network hdr offset: 14 Bulk Best Effort Voice thresh 7Mbit 112Mbit 28Mbit target 5.0ms 5.0ms 5.0ms interval 100.0ms 100.0ms 100.0ms pk_delay 965us 7.1ms 962us av_delay 40us 3.3ms 40us sp_delay 15us 16us 9us backlog 0b 0b 0b pkts 137420272 549998256 84436 bytes 25919208286 597744167438 8279109 way_inds 18794733 42309661 0 way_miss 4235643 9977445 5 way_cols 9 0 0 drops 2853 483084 0 marks 8811 1306562 0 ack_drop 0 0 0 sp_flows 1 1 1 bk_flows 0 1 0 un_flows 0 0 0 max_len 48448 68130 342 quantum 300 1514 854 On Sun, 27 Jun 2021 at 12:24, Dave Taht wrote: > I am curious if anyone has any tc -s qdisc show output of > ack-filtering on both the up and down at any symmetry? > > ---------- Forwarded message --------- > From: Dave Taht > Date: Wed, Nov 15, 2017 at 9:22 PM > Subject: Re: ack filter rrul result at 200/10 > To: Cake List > Cc: Jim Gettys > > > and 200/10 looked reasonable too. > > > -- > Latest Podcast: > https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/ > > Dave Täht CTO, TekLibre, LLC > _______________________________________________ > Cake mailing list > Cake at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.taht at gmail.com Wed Jun 30 20:12:05 2021 From: dave.taht at gmail.com (Dave Taht) Date: Wed, 30 Jun 2021 17:12:05 -0700 Subject: [Cake] Due Aug 2: Internet Quality workshop CFP for the internet architecture board Message-ID: The program committee members are *amazing*. Perhaps, finally, we can move the bar for the internet's quality metrics past endless, blind repetitions of speedtest. For complete details, please see: https://www.iab.org/activities/workshops/network-quality/ Submissions Due: Monday 2nd August 2021, midnight AOE (Anywhere On Earth) Invitations Issued by: Monday 16th August 2021 Workshop Date: This will be a virtual workshop, spread over three days: 1400-1800 UTC Tue 14th September 2021 1400-1800 UTC Wed 15th September 2021 1400-1800 UTC Thu 16th September 2021 Workshop co-chairs: Wes Hardaker, Evgeny Khorov, Omer Shapira The Program Committee members: Jari Arkko, Olivier Bonaventure, Vint Cerf, Stuart Cheshire, Sam Crowford, Nick Feamster, Jim Gettys, Toke Hoiland-Jorgensen, Geoff Huston, Cullen Jennings, Katarzyna Kosek-Szott, Mirja Kuehlewind, Jason Livingood, Matt Mathias, Randall Meyer, Kathleen Nichols, Christoph Paasch, Tommy Pauly, Greg White, Keith Winstein. Send Submissions to: network-quality-workshop-pc at iab.org. Position papers from academia, industry, the open source community and others that focus on measurements, experiences, observations and advice for the future are welcome. Papers that reflect experience based on deployed services are especially welcome. The organizers understand that specific actions taken by operators are unlikely to be discussed in detail, so papers discussing general categories of actions and issues without naming specific technologies, products, or other players in the ecosystem are expected. Papers should not focus on specific protocol solutions. The workshop will be by invitation only. Those wishing to attend should submit a position paper to the address above; it may take the form of an Internet-Draft. All inputs submitted and considered relevant will be published on the workshop website. The organisers will decide whom to invite based on the submissions received. Sessions will be organized according to content, and not every accepted submission or invited attendee will have an opportunity to present as the intent is to foster discussion and not simply to have a sequence of presentations. Position papers from those not planning to attend the virtual sessions themselves are also encouraged. A workshop report will be published afterwards. Overview: "We believe that one of the major factors behind this lack of progress is the popular perception that throughput is the often sole measure of the quality of Internet connectivity. With such narrow focus, people don’t consider questions such as: What is the latency under typical working conditions? How reliable is the connectivity across longer time periods? Does the network allow the use of a broad range of protocols? What services can be run by clients of the network? What kind of IPv4, NAT or IPv6 connectivity is offered, and are there firewalls? What security mechanisms are available for local services, such as DNS? To what degree are the privacy, confidentiality, integrity and authenticity of user communications guarded? Improving these aspects of network quality will likely depend on measurement and exposing metrics to all involved parties, including to end users in a meaningful way. Such measurements and exposure of the right metrics will allow service providers and network operators to focus on the aspects that impacts the users’ experience most and at the same time empowers users to choose the Internet service that will give them the best experience." -- Latest Podcast: https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/ Dave Täht CTO, TekLibre, LLC