From dos at scarff.id.au Sun Jan 3 21:11:24 2021 From: dos at scarff.id.au (Dean Scarff) Date: Mon, 04 Jan 2021 13:11:24 +1100 Subject: [Cake] [Bloat] New board that looks interesting In-Reply-To: References: <1586011622.632930657@apps.rackspace.com> Message-ID: Any stats on how much power it pulled during your tests and when idle? On Fri, 18 Dec 2020 15:48:46 -0800, Aaron Wood wrote: > I have, finally.  It's been running for a week or so, now. > > OpenWRT was an _adventure_.  The board is UEFI, not standard bios..  > And while it will merrily boot OpenWRT's non-uefi images off of USB, > it won't boot the non-UEFI setup from the internal storage (I'm using > the eMMC).  So _that_ was fun (and I made some dumb mistakes that > were especially fun to correct. > > But it's running OpenWRT 19.07 (and a UEFI bootloader before grub > that's from ToT OpenWRT). > > Anyway, I have cake running, 950Mbps ingress and 35Mbps egress (modem > is provisioned at 1.3G ingress, and a bit over 35Mbps egress). >  fq_codel was defaulted, in multi-queue mode.  While I'm using cake > on ingress, my local link hasn't been hitting the limiter very often: > >                 Tin 0 >   thresh        950Mbit >   target          1.5ms >   interval       30.0ms >   pk_delay         22us >   av_delay          9us >   sp_delay          2us >   backlog            0b >   pkts        243608193 >   bytes    250748364896 >   way_inds     13167720 >   way_miss      1245030 >   way_cols            0 >   drops            1075 >   marks             101 >   ack_drop            0 >   sp_flows            0 >   bk_flows            1 >   un_flows            0 >   max_len         69876 >   quantum          1514 > > Given that most of the hosts that I interact with are only about > 10-15ms away, I'm probably going to change the interval target to > better match that. > > Interestingly, while it has a pair of multiqueue NICs (i211s), the > igbe driver isn't configuring them for RSS.  Both output queues are > being used, but not the ingress queues: > > wan interface: > >      tx_queue_0_packets: 56635989 >      tx_queue_1_packets: 39777210 >      rx_queue_0_packets: 243646072 >      rx_queue_1_packets: 0 > > lan interface: > >      tx_queue_0_packets: 85047897 >      tx_queue_1_packets: 162004500 >      rx_queue_0_packets: 111174855 >      rx_queue_1_packets: 0 > > Since I have housemates that don't appreciate me messing with the > network during their meetings, I haven't gotten around to poking more > deeply at that (or at experimenting with running cake on two ingress > queues). > > That being said, I bench-tested this before I put it into operation > and was able to see 940Mbps of iperf goodput through cake and NAT...  > Took all of a core, though (and that core was still cold and > therefore > potentially able to boost to 2.5GHz).  I haven't determined how long > it will take to thermally throttle, and if bandwidth suffers as a > result. > > Pretty happy with it so far, though. From dave.taht at gmail.com Tue Jan 26 10:46:56 2021 From: dave.taht at gmail.com (Dave Taht) Date: Tue, 26 Jan 2021 07:46:56 -0800 Subject: [Cake] quantum configuration In-Reply-To: <87lfj95cbq.fsf@toke.dk> References: <87lfj95cbq.fsf@toke.dk> Message-ID: I have kind of thought we could scale the quantum much higher as we crack 200mbit to see if that helps on performance. On Fri, Jul 24, 2020 at 5:26 AM Toke Høiland-Jørgensen via Cake wrote: > > Luca Muscariello writes: > > > Is there a reason why in cake the quantum cannot be configured to a > > different value like in fq_codel? > > I think this was mostly to be as no-knob as possible; so the quantum is > auto-scaled with the tin bandwidths, instead of being configurable. > > Jonathan can probably expand on this... > > -Toke > _______________________________________________ > Cake mailing list > Cake at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake -- "For a successful technology, reality must take precedence over public relations, for Mother Nature cannot be fooled" - Richard Feynman dave at taht.net CTO, TekLibre, LLC Tel: 1-831-435-0729 From jmarks2 at gmail.com Tue Jan 26 19:21:58 2021 From: jmarks2 at gmail.com (jmarks2 at gmail.com) Date: Tue, 26 Jan 2021 18:21:58 -0600 Subject: [Cake] Trouble getting decent performance with CAKE Message-ID: <005701d6f442$6d620e90$48262bb0$@gmail.com> Dear CAKE experts, I write for your help in getting CAKE to perform on my router as I understand it should. I replaced my aging Asus home router with a Ubiquiti Edgerouter 4 specifically so I could take advantage of the port of CAKE to it. I am in the US, with VDSL2 service provided by AT&T at 100/20. I now have all unfiltered traffic delivered by the AT&T modem to the eth0 port on the router. My desktop runs Windows. So, to take advantage of flent to quantify CAKE performance, I stood up a Linux Mint VM and ran flent rrul tests before CAKE installation, and after. The first attached datafile, labeled qdisc pfifo fast, is the performance before CAKE. The other file, called 20/95 ack filtered bridged ptm, is my latest attempt with configuring CAKE flags. My major issue is with Ingress. Egress seems fine. For ingress, I see very little difference between performances with and without CAKE. And the overall throughput is far less than 100 mbit. The queuing disciplines on the router are: jeremymarks at ubnt:~$ sudo tc qdisc show qdisc cake 1: dev eth0 root refcnt 2 bandwidth 20Mbit besteffort dual-srchost nat nowash ack-filter split-gso rtt 100.0ms ptm overhead 22 no-sce qdisc ingress ffff: dev eth0 parent ffff:fff1 ---------------- qdisc pfifo_fast 0: dev imq0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc cake 2: dev ifb0 root refcnt 2 bandwidth 95Mbit besteffort dual-dsthost nat nowash ingress ack-filter split-gso rtt 100.0ms ptm overhead 22 no-sce I must be making a major mistake in configuring CAKE. I would be grateful for any help or advice in getting this amazing program to work as it should. Many thanks for any help. Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: rrul-2021-01-24T110301.456648.qdisc_pfifo_fast.flent.gz Type: application/x-gzip Size: 121758 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: rrul-2021-01-24T162210.973709.20_95_ack_filtered_bridged-ptm.flent.gz Type: application/x-gzip Size: 119863 bytes Desc: not available URL: From xnoreq at gmail.com Tue Jan 26 21:04:46 2021 From: xnoreq at gmail.com (xnor) Date: Wed, 27 Jan 2021 03:04:46 +0100 Subject: [Cake] Trouble getting decent performance with CAKE In-Reply-To: <005701d6f442$6d620e90$48262bb0$@gmail.com> References: <005701d6f442$6d620e90$48262bb0$@gmail.com> Message-ID: <21fe9efe-cc74-0b83-f659-4778ba54d420@gmail.com> Haven't looked at your measurements, but what's the issue? Is it throughput or latency? Because I have a 750 MHz single core mips router here that handles 100 Mbps... barely. Latency on ingress (and traffic shaping in general on ingress) sucks compared to egress. Why is there an imq0 device? Haven't those been obsolete for several years? What kernel are you running? I assume you have a filter with tc-action mirred egress set up, but just to be sure could you post the output of: $ tc filter show dev eth0 parent ffff: Also, there's little point in having the ack-filter on on the ingress cake. From dave.taht at gmail.com Wed Jan 27 00:19:19 2021 From: dave.taht at gmail.com (Dave Taht) Date: Tue, 26 Jan 2021 21:19:19 -0800 Subject: [Cake] Trouble getting decent performance with CAKE In-Reply-To: <005701d6f442$6d620e90$48262bb0$@gmail.com> References: <005701d6f442$6d620e90$48262bb0$@gmail.com> Message-ID: As much as I like to get the fustest with the mostest with the rrul test, it is often easier to test up and down separately. do : tc qdisc delete ifb0 root tc qdisc delete eth0 root I am not sure what imq is being used for.... Can you share the output of tc -s instead? a simple upload test would then be: (unless you made cake the default) flent -t pfifo_fast -H wherever --socket-stats -x --te=upload_streams=4 tcp_nup tc qdisc add dev eth0 root cake all_your_options flent -t cake -H wherever --socket-stats -x --te=upload_streams=4 tcp_nup the download setting up tc mirred I assume you are doing with sqm? Do that. flent -t -H wherever --socket-stats -x --te=download_streams=4 tcp_ndown then send tc -s It's entirely feasible your so-called 100mbit service is running at far far less than 100mbit in the first place. Another thing to check is if your modem is connecting to the device at 100Mbit or a gbit. SOME - not enough - dsl modems actually connect at 100Mbit and/or joy of joys excert hw flow control. 95Mbit is gonna be slightly too high in former case for cake to control things, try 92mbit, or less. On Tue, Jan 26, 2021 at 4:22 PM wrote: > > Dear CAKE experts, > > I write for your help in getting CAKE to perform on my router as I understand it should. > > > > I replaced my aging Asus home router with a Ubiquiti Edgerouter 4 specifically so I could take advantage of the port of CAKE to it. > > > > I am in the US, with VDSL2 service provided by AT&T at 100/20. I now have all unfiltered traffic delivered by the AT&T modem to the eth0 port on the router. > > > > My desktop runs Windows. So, to take advantage of flent to quantify CAKE performance, I stood up a Linux Mint VM and ran flent rrul tests before CAKE installation, and after. > > > > The first attached datafile, labeled qdisc pfifo fast, is the performance before CAKE. The other file, called 20/95 ack filtered bridged ptm, is my latest attempt with configuring CAKE flags. > > My major issue is with Ingress. Egress seems fine. > > For ingress, I see very little difference between performances with and without CAKE. And the overall throughput is far less than 100 mbit. > > > > The queuing disciplines on the router are: > > > > jeremymarks at ubnt:~$ sudo tc qdisc show > > > > qdisc cake 1: dev eth0 root refcnt 2 bandwidth 20Mbit besteffort dual-srchost nat nowash ack-filter split-gso rtt 100.0ms ptm overhead 22 no-sce > > qdisc ingress ffff: dev eth0 parent ffff:fff1 ---------------- > > qdisc pfifo_fast 0: dev imq0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 > > qdisc cake 2: dev ifb0 root refcnt 2 bandwidth 95Mbit besteffort dual-dsthost nat nowash ingress ack-filter split-gso rtt 100.0ms ptm overhead 22 no-sce > > > > I must be making a major mistake in configuring CAKE. > > > > I would be grateful for any help or advice in getting this amazing program to work as it should. > > > > Many thanks for any help. > > Jeremy > > > > > > _______________________________________________ > Cake mailing list > Cake at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake -- "For a successful technology, reality must take precedence over public relations, for Mother Nature cannot be fooled" - Richard Feynman dave at taht.net CTO, TekLibre, LLC Tel: 1-831-435-0729 From jmarks2 at gmail.com Wed Jan 27 12:49:29 2021 From: jmarks2 at gmail.com (jmarks2 at gmail.com) Date: Wed, 27 Jan 2021 11:49:29 -0600 Subject: [Cake] Trouble getting decent performance with CAKE In-Reply-To: References: <005701d6f442$6d620e90$48262bb0$@gmail.com> Message-ID: <008401d6f4d4$c39479a0$4abd6ce0$@gmail.com> Thanks so much for getting back to me. First, I am not so expert in these things, so please excuse my questions about providing the info you have requested and implementing your suggestions. 1. I don’t know what imq0 is used for, either. I don't have a script to run CAKE at system start because I haven't got it working well enough. So, at reboot, the out-of-the box qdiscs are: qdisc ingress ffff: dev eth0 parent ffff:fff1 ---------------- qdisc pfifo_fast 0: dev imq0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev ifb0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 You can see that the imq0 device is present at startup. A quick search suggests that Ubiquiti uses this for native QOS during egress. If it's not clear what it's doing and if it's interfering, I will happily remove it, if I knew how. The worst that happens is that the system crashes and I have to reboot. 2. At startup (so no CAKE), sudo tc qdisc -s show: qdisc ingress ffff: dev eth0 parent ffff:fff1 ---------------- Sent 2055477419 bytes 2270158 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc pfifo_fast 0: dev imq0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc pfifo_fast 0: dev ifb0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Sent 2087261885 bytes 2270207 pkt (dropped 0, overlimits 0 requeues 1) backlog 0b 0p requeues 1 3. I set up CAKE with the following commands: sudo tc qdisc add dev eth0 root handle 1:0 cake bandwidth...[switches] sudo set interfaces input ifb0 sudo set interfaces ethernet eth0 redirect ifb0 sudo tc qdisc add dev ifb0 root handle 2:0 cake bandwidth...[switches] Do these commands make CAKE the default, or can I choose a qdisc to use when running flent? If so, how do I do this? 4. Regarding your suggestion: "the download setting up tc mirred I assume you are doing with sqm? Do that." This is outside my knowledge, and I have not done anything like that. I have read an alternative (older?) way to set up CAKE for ingress: sudo ip link add name ifb4eth0 type ifb sudo tc qdisc del dev eth0 ingress sudo tc qdisc add dev eth0 handle ffff: ingress sudo tc qdisc del dev ifb4eth0 root sudo tc qdisc add dev ifb4eth0 root cake bandwidth... sudo ifconfig ifb4eth1 up sudo tc filter add dev eth1 parent ffff: protocol all prio 10 u32 match u32 0 0 flowid 1:1 action mirred egress redirect dev ifb4eth1 I suspect the first 6 lines of this code achieve the same thing as the first two lines of code that I used (see point 3, above). Am I right? Is the filter that is specified in the last line what you are suggesting I add? This is about as much as I can address with what you have suggested, without a super long email. Please feel free to correct my mistakes in understanding. If you have suggestions based on what I have written, I would so grateful. I am really looking forward to hearing what you think. This learning curve is a little steep. Many thanks. -----Original Message----- From: Dave Taht Sent: Tuesday, January 26, 2021 11:19 PM To: jmarks2 at gmail.com Cc: Cake List Subject: Re: [Cake] Trouble getting decent performance with CAKE As much as I like to get the fustest with the mostest with the rrul test, it is often easier to test up and down separately. do : tc qdisc delete ifb0 root tc qdisc delete eth0 root I am not sure what imq is being used for.... Can you share the output of tc -s instead? a simple upload test would then be: (unless you made cake the default) flent -t pfifo_fast -H wherever --socket-stats -x --te=upload_streams=4 tcp_nup tc qdisc add dev eth0 root cake all_your_options flent -t cake -H wherever --socket-stats -x --te=upload_streams=4 tcp_nup the download setting up tc mirred I assume you are doing with sqm? Do that. flent -t -H wherever --socket-stats -x --te=download_streams=4 tcp_ndown then send tc -s It's entirely feasible your so-called 100mbit service is running at far far less than 100mbit in the first place. Another thing to check is if your modem is connecting to the device at 100Mbit or a gbit. SOME - not enough - dsl modems actually connect at 100Mbit and/or joy of joys excert hw flow control. 95Mbit is gonna be slightly too high in former case for cake to control things, try 92mbit, or less. On Tue, Jan 26, 2021 at 4:22 PM wrote: > > Dear CAKE experts, > > I write for your help in getting CAKE to perform on my router as I understand it should. > > > > I replaced my aging Asus home router with a Ubiquiti Edgerouter 4 specifically so I could take advantage of the port of CAKE to it. > > > > I am in the US, with VDSL2 service provided by AT&T at 100/20. I now have all unfiltered traffic delivered by the AT&T modem to the eth0 port on the router. > > > > My desktop runs Windows. So, to take advantage of flent to quantify CAKE performance, I stood up a Linux Mint VM and ran flent rrul tests before CAKE installation, and after. > > > > The first attached datafile, labeled qdisc pfifo fast, is the performance before CAKE. The other file, called 20/95 ack filtered bridged ptm, is my latest attempt with configuring CAKE flags. > > My major issue is with Ingress. Egress seems fine. > > For ingress, I see very little difference between performances with and without CAKE. And the overall throughput is far less than 100 mbit. > > > > The queuing disciplines on the router are: > > > > jeremymarks at ubnt:~$ sudo tc qdisc show > > > > qdisc cake 1: dev eth0 root refcnt 2 bandwidth 20Mbit besteffort > dual-srchost nat nowash ack-filter split-gso rtt 100.0ms ptm overhead > 22 no-sce > > qdisc ingress ffff: dev eth0 parent ffff:fff1 ---------------- > > qdisc pfifo_fast 0: dev imq0 root refcnt 2 bands 3 priomap 1 2 2 2 1 > 2 0 0 1 1 1 1 1 1 1 1 > > qdisc cake 2: dev ifb0 root refcnt 2 bandwidth 95Mbit besteffort > dual-dsthost nat nowash ingress ack-filter split-gso rtt 100.0ms ptm > overhead 22 no-sce > > > > I must be making a major mistake in configuring CAKE. > > > > I would be grateful for any help or advice in getting this amazing program to work as it should. > > > > Many thanks for any help. > > Jeremy > > > > > > _______________________________________________ > Cake mailing list > Cake at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake -- "For a successful technology, reality must take precedence over public relations, for Mother Nature cannot be fooled" - Richard Feynman dave at taht.net CTO, TekLibre, LLC Tel: 1-831-435-0729 From dave.taht at gmail.com Wed Jan 27 12:51:19 2021 From: dave.taht at gmail.com (Dave Taht) Date: Wed, 27 Jan 2021 09:51:19 -0800 Subject: [Cake] Trouble getting decent performance with CAKE In-Reply-To: <008401d6f4d4$c39479a0$4abd6ce0$@gmail.com> References: <005701d6f442$6d620e90$48262bb0$@gmail.com> <008401d6f4d4$c39479a0$4abd6ce0$@gmail.com> Message-ID: You need the tc mirred to get stuff into ifb0. I don't know what the redirect command you are using does. tc -s qdisc will show stats for the cake instance if that was all you need on this platform. On Wed, Jan 27, 2021 at 9:49 AM wrote: > > Thanks so much for getting back to me. > First, I am not so expert in these things, so please excuse my questions about providing the info you have requested and implementing your suggestions. > > 1. I don’t know what imq0 is used for, either. > I don't have a script to run CAKE at system start because I haven't got it working well enough. So, at reboot, the out-of-the box qdiscs are: > > qdisc ingress ffff: dev eth0 parent ffff:fff1 ---------------- > qdisc pfifo_fast 0: dev imq0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 > qdisc pfifo_fast 0: dev ifb0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 > > You can see that the imq0 device is present at startup. A quick search suggests that Ubiquiti uses this for native QOS during egress. If it's not clear what it's doing and if it's interfering, I will happily remove it, if I knew how. The worst that happens is that the system crashes and I have to reboot. > > 2. At startup (so no CAKE), sudo tc qdisc -s show: > > qdisc ingress ffff: dev eth0 parent ffff:fff1 ---------------- > Sent 2055477419 bytes 2270158 pkt (dropped 0, overlimits 0 requeues 0) > backlog 0b 0p requeues 0 > > qdisc pfifo_fast 0: dev imq0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 > Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) > backlog 0b 0p requeues 0 > > qdisc pfifo_fast 0: dev ifb0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 > Sent 2087261885 bytes 2270207 pkt (dropped 0, overlimits 0 requeues 1) > backlog 0b 0p requeues 1 > > 3. I set up CAKE with the following commands: > sudo tc qdisc add dev eth0 root handle 1:0 cake bandwidth...[switches] > > sudo set interfaces input ifb0 > sudo set interfaces ethernet eth0 redirect ifb0 > sudo tc qdisc add dev ifb0 root handle 2:0 cake bandwidth...[switches] > > Do these commands make CAKE the default, or can I choose a qdisc to use when running flent? If so, how do I do this? > > 4. Regarding your suggestion: "the download setting up tc mirred I assume you are doing with sqm? Do that." > > This is outside my knowledge, and I have not done anything like that. > I have read an alternative (older?) way to set up CAKE for ingress: > sudo ip link add name ifb4eth0 type ifb > sudo tc qdisc del dev eth0 ingress > sudo tc qdisc add dev eth0 handle ffff: ingress > sudo tc qdisc del dev ifb4eth0 root > sudo tc qdisc add dev ifb4eth0 root cake bandwidth... > sudo ifconfig ifb4eth1 up > sudo tc filter add dev eth1 parent ffff: protocol all prio 10 u32 match u32 0 0 flowid 1:1 action mirred egress redirect dev ifb4eth1 > > I suspect the first 6 lines of this code achieve the same thing as the first two lines of code that I used (see point 3, above). Am I right? > Is the filter that is specified in the last line what you are suggesting I add? > > This is about as much as I can address with what you have suggested, without a super long email. > > Please feel free to correct my mistakes in understanding. > If you have suggestions based on what I have written, I would so grateful. I am really looking forward to hearing what you think. > This learning curve is a little steep. > > Many thanks. > > > -----Original Message----- > From: Dave Taht > Sent: Tuesday, January 26, 2021 11:19 PM > To: jmarks2 at gmail.com > Cc: Cake List > Subject: Re: [Cake] Trouble getting decent performance with CAKE > > As much as I like to get the fustest with the mostest with the rrul test, it is often easier to test up and down separately. > > do : > > tc qdisc delete ifb0 root > tc qdisc delete eth0 root > > I am not sure what imq is being used for.... > > Can you share the output of tc -s > > instead? > > a simple upload test would then be: > > (unless you made cake the default) > > flent -t pfifo_fast -H wherever --socket-stats -x --te=upload_streams=4 tcp_nup > > tc qdisc add dev eth0 root cake all_your_options > > flent -t cake -H wherever --socket-stats -x --te=upload_streams=4 tcp_nup > > the download setting up tc mirred I assume you are doing with sqm? Do that. > > flent -t -H wherever --socket-stats -x --te=download_streams=4 tcp_ndown > > then send tc -s > > It's entirely feasible your so-called 100mbit service is running at far far less than 100mbit in the first place. > > Another thing to check is if your modem is connecting to the device at 100Mbit or a gbit. SOME - not enough - dsl modems actually connect at 100Mbit and/or joy of joys excert hw flow control. > 95Mbit is gonna be slightly too high in former case for cake to control things, try 92mbit, or less. > > On Tue, Jan 26, 2021 at 4:22 PM wrote: > > > > Dear CAKE experts, > > > > I write for your help in getting CAKE to perform on my router as I understand it should. > > > > > > > > I replaced my aging Asus home router with a Ubiquiti Edgerouter 4 specifically so I could take advantage of the port of CAKE to it. > > > > > > > > I am in the US, with VDSL2 service provided by AT&T at 100/20. I now have all unfiltered traffic delivered by the AT&T modem to the eth0 port on the router. > > > > > > > > My desktop runs Windows. So, to take advantage of flent to quantify CAKE performance, I stood up a Linux Mint VM and ran flent rrul tests before CAKE installation, and after. > > > > > > > > The first attached datafile, labeled qdisc pfifo fast, is the performance before CAKE. The other file, called 20/95 ack filtered bridged ptm, is my latest attempt with configuring CAKE flags. > > > > My major issue is with Ingress. Egress seems fine. > > > > For ingress, I see very little difference between performances with and without CAKE. And the overall throughput is far less than 100 mbit. > > > > > > > > The queuing disciplines on the router are: > > > > > > > > jeremymarks at ubnt:~$ sudo tc qdisc show > > > > > > > > qdisc cake 1: dev eth0 root refcnt 2 bandwidth 20Mbit besteffort > > dual-srchost nat nowash ack-filter split-gso rtt 100.0ms ptm overhead > > 22 no-sce > > > > qdisc ingress ffff: dev eth0 parent ffff:fff1 ---------------- > > > > qdisc pfifo_fast 0: dev imq0 root refcnt 2 bands 3 priomap 1 2 2 2 1 > > 2 0 0 1 1 1 1 1 1 1 1 > > > > qdisc cake 2: dev ifb0 root refcnt 2 bandwidth 95Mbit besteffort > > dual-dsthost nat nowash ingress ack-filter split-gso rtt 100.0ms ptm > > overhead 22 no-sce > > > > > > > > I must be making a major mistake in configuring CAKE. > > > > > > > > I would be grateful for any help or advice in getting this amazing program to work as it should. > > > > > > > > Many thanks for any help. > > > > Jeremy > > > > > > > > > > > > _______________________________________________ > > Cake mailing list > > Cake at lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/cake > > > > -- > "For a successful technology, reality must take precedence over public relations, for Mother Nature cannot be fooled" - Richard Feynman > > dave at taht.net CTO, TekLibre, LLC Tel: 1-831-435-0729 > -- "For a successful technology, reality must take precedence over public relations, for Mother Nature cannot be fooled" - Richard Feynman dave at taht.net CTO, TekLibre, LLC Tel: 1-831-435-0729 From jmarks2 at gmail.com Wed Jan 27 13:25:43 2021 From: jmarks2 at gmail.com (Jeremy Marks) Date: Wed, 27 Jan 2021 12:25:43 -0600 Subject: [Cake] Trouble getting decent performance with CAKE In-Reply-To: References: Message-ID: I think the redirect command (in that alternate code block that I didn't use) is from older code, so it's no longer needed. I will reimplement CAKE, adding the mirred filter. Do you think I can safely remove the imq0 interface? The stats (thank you!) suggest to me that it's not being used. Removing may not be necessary anyway. After the bandwidth is not being consumed by others behind this router, I will run dsl reports and flent with and without CAKE, and report back. I really appreciate the help. On Jan 27, 2021, at 11:51 AM, Dave Taht wrote: You need the tc mirred to get stuff into ifb0. I don't know what the redirect command you are using does. tc -s qdisc will show stats for the cake instance if that was all you need on this platform. On Wed, Jan 27, 2021 at 9:49 AM wrote: > > Thanks so much for getting back to me. > First, I am not so expert in these things, so please excuse my questions about providing the info you have requested and implementing your suggestions. > > 1. I don’t know what imq0 is used for, either. > I don't have a script to run CAKE at system start because I haven't got it working well enough. So, at reboot, the out-of-the box qdiscs are: > > qdisc ingress ffff: dev eth0 parent ffff:fff1 ---------------- > qdisc pfifo_fast 0: dev imq0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 > qdisc pfifo_fast 0: dev ifb0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 > > You can see that the imq0 device is present at startup. A quick search suggests that Ubiquiti uses this for native QOS during egress. If it's not clear what it's doing and if it's interfering, I will happily remove it, if I knew how. The worst that happens is that the system crashes and I have to reboot. > > 2. At startup (so no CAKE), sudo tc qdisc -s show: > > qdisc ingress ffff: dev eth0 parent ffff:fff1 ---------------- > Sent 2055477419 bytes 2270158 pkt (dropped 0, overlimits 0 requeues 0) > backlog 0b 0p requeues 0 > > qdisc pfifo_fast 0: dev imq0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 > Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) > backlog 0b 0p requeues 0 > > qdisc pfifo_fast 0: dev ifb0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 > Sent 2087261885 bytes 2270207 pkt (dropped 0, overlimits 0 requeues 1) > backlog 0b 0p requeues 1 > > 3. I set up CAKE with the following commands: > sudo tc qdisc add dev eth0 root handle 1:0 cake bandwidth...[switches] > > sudo set interfaces input ifb0 > sudo set interfaces ethernet eth0 redirect ifb0 > sudo tc qdisc add dev ifb0 root handle 2:0 cake bandwidth...[switches] > > Do these commands make CAKE the default, or can I choose a qdisc to use when running flent? If so, how do I do this? > > 4. Regarding your suggestion: "the download setting up tc mirred I assume you are doing with sqm? Do that." > > This is outside my knowledge, and I have not done anything like that. > I have read an alternative (older?) way to set up CAKE for ingress: > sudo ip link add name ifb4eth0 type ifb > sudo tc qdisc del dev eth0 ingress > sudo tc qdisc add dev eth0 handle ffff: ingress > sudo tc qdisc del dev ifb4eth0 root > sudo tc qdisc add dev ifb4eth0 root cake bandwidth... > sudo ifconfig ifb4eth1 up > sudo tc filter add dev eth1 parent ffff: protocol all prio 10 u32 match u32 0 0 flowid 1:1 action mirred egress redirect dev ifb4eth1 > > I suspect the first 6 lines of this code achieve the same thing as the first two lines of code that I used (see point 3, above). Am I right? > Is the filter that is specified in the last line what you are suggesting I add? > > This is about as much as I can address with what you have suggested, without a super long email. > > Please feel free to correct my mistakes in understanding. > If you have suggestions based on what I have written, I would so grateful. I am really looking forward to hearing what you think. > This learning curve is a little steep. > > Many thanks. > > > -----Original Message----- > From: Dave Taht > Sent: Tuesday, January 26, 2021 11:19 PM > To: jmarks2 at gmail.com > Cc: Cake List > Subject: Re: [Cake] Trouble getting decent performance with CAKE > > As much as I like to get the fustest with the mostest with the rrul test, it is often easier to test up and down separately. > > do : > > tc qdisc delete ifb0 root > tc qdisc delete eth0 root > > I am not sure what imq is being used for.... > > Can you share the output of tc -s > > instead? > > a simple upload test would then be: > > (unless you made cake the default) > > flent -t pfifo_fast -H wherever --socket-stats -x --te=upload_streams=4 tcp_nup > > tc qdisc add dev eth0 root cake all_your_options > > flent -t cake -H wherever --socket-stats -x --te=upload_streams=4 tcp_nup > > the download setting up tc mirred I assume you are doing with sqm? Do that. > > flent -t -H wherever --socket-stats -x --te=download_streams=4 tcp_ndown > > then send tc -s > > It's entirely feasible your so-called 100mbit service is running at far far less than 100mbit in the first place. > > Another thing to check is if your modem is connecting to the device at 100Mbit or a gbit. SOME - not enough - dsl modems actually connect at 100Mbit and/or joy of joys excert hw flow control. > 95Mbit is gonna be slightly too high in former case for cake to control things, try 92mbit, or less. > > On Tue, Jan 26, 2021 at 4:22 PM wrote: >> >> Dear CAKE experts, >> >> I write for your help in getting CAKE to perform on my router as I understand it should. >> >> >> >> I replaced my aging Asus home router with a Ubiquiti Edgerouter 4 specifically so I could take advantage of the port of CAKE to it. >> >> >> >> I am in the US, with VDSL2 service provided by AT&T at 100/20. I now have all unfiltered traffic delivered by the AT&T modem to the eth0 port on the router. >> >> >> >> My desktop runs Windows. So, to take advantage of flent to quantify CAKE performance, I stood up a Linux Mint VM and ran flent rrul tests before CAKE installation, and after. >> >> >> >> The first attached datafile, labeled qdisc pfifo fast, is the performance before CAKE. The other file, called 20/95 ack filtered bridged ptm, is my latest attempt with configuring CAKE flags. >> >> My major issue is with Ingress. Egress seems fine. >> >> For ingress, I see very little difference between performances with and without CAKE. And the overall throughput is far less than 100 mbit. >> >> >> >> The queuing disciplines on the router are: >> >> >> >> jeremymarks at ubnt:~$ sudo tc qdisc show >> >> >> >> qdisc cake 1: dev eth0 root refcnt 2 bandwidth 20Mbit besteffort >> dual-srchost nat nowash ack-filter split-gso rtt 100.0ms ptm overhead >> 22 no-sce >> >> qdisc ingress ffff: dev eth0 parent ffff:fff1 ---------------- >> >> qdisc pfifo_fast 0: dev imq0 root refcnt 2 bands 3 priomap 1 2 2 2 1 >> 2 0 0 1 1 1 1 1 1 1 1 >> >> qdisc cake 2: dev ifb0 root refcnt 2 bandwidth 95Mbit besteffort >> dual-dsthost nat nowash ingress ack-filter split-gso rtt 100.0ms ptm >> overhead 22 no-sce >> >> >> >> I must be making a major mistake in configuring CAKE. >> >> >> >> I would be grateful for any help or advice in getting this amazing program to work as it should. >> >> >> >> Many thanks for any help. >> >> Jeremy >> >> >> >> >> >> _______________________________________________ >> Cake mailing list >> Cake at lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/cake > > > > -- > "For a successful technology, reality must take precedence over public relations, for Mother Nature cannot be fooled" - Richard Feynman > > dave at taht.net CTO, TekLibre, LLC Tel: 1-831-435-0729 > -- "For a successful technology, reality must take precedence over public relations, for Mother Nature cannot be fooled" - Richard Feynman dave at taht.net CTO, TekLibre, LLC Tel: 1-831-435-0729 From xnoreq at gmail.com Thu Jan 28 12:17:51 2021 From: xnoreq at gmail.com (xnor) Date: Thu, 28 Jan 2021 18:17:51 +0100 Subject: [Cake] Trouble getting decent performance with CAKE In-Reply-To: References: Message-ID: <556d6202-1719-297e-b950-38f7a66d04dd@gmail.com> > I think the redirect command (in that alternate code block that I didn't use) is from older code, so it's no longer needed. >> 3. I set up CAKE with the following commands: >>        sudo tc qdisc add dev eth0 root handle 1:0 cake bandwidth...[switches] >> >>        sudo set interfaces input ifb0 >>        sudo set interfaces ethernet eth0 redirect ifb0 >>        sudo tc qdisc add dev ifb0 root handle 2:0 cake bandwidth...[switches] What is that supposed to do? What is "set"? Which package does "set" come from? It seems very weird to have a command that is a shell built-in... >> 4. Regarding your suggestion: "the download setting up tc mirred I assume you are doing with sqm? Do that." >> >>     This is outside my knowledge, and I have not done anything like that. >>     I have read an alternative (older?) way to set up CAKE for ingress: >>        sudo ip link add name ifb4eth0 type ifb >>        sudo tc qdisc del dev eth0 ingress >>        sudo tc qdisc add dev eth0 handle ffff: ingress >>        sudo tc qdisc del dev ifb4eth0 root >>        sudo tc qdisc add dev ifb4eth0 root cake bandwidth... >>        sudo ifconfig ifb4eth1 up >>        sudo tc filter add dev eth1 parent ffff: protocol all prio 10 u32 match u32 0 0 flowid 1:1 action mirred egress redirect dev ifb4eth1 No, this is the *proper* way, but there are mistakes in your example. It adds the tc filter to eth1 instead of eth0. You probably also don't want the flowid. > Do you think I can safely remove the imq0 interface? The stats (thank you!) suggest to me that it's not being used. Removing may not be necessary anyway. As I said before, imq has been obsolete for a couple of years. Iirc it hasn't been shipped with the kernel for years. imq was used for QoS back in the day. It extended iptables to do the redirection. ifb is the "modern" replacement and works at a lower level, hence the need for the tc-mirred It looks to me like the linux distribution on your router is doing some nonstandard things and possibly runs custom QoS scripts that could conflict with your setup. If that is the case you should ask the manufacturer for support. From xnoreq at gmail.com Thu Jan 28 17:00:28 2021 From: xnoreq at gmail.com (xnor) Date: Thu, 28 Jan 2021 23:00:28 +0100 Subject: [Cake] Trouble getting decent performance with CAKE In-Reply-To: <00ba01d6f5a1$450d07f0$cf2717d0$@gmail.com> References: <556d6202-1719-297e-b950-38f7a66d04dd@gmail.com> <00ba01d6f5a1$450d07f0$cf2717d0$@gmail.com> Message-ID: <48924c59-ec54-696f-d20e-4768a5d16b49@gmail.com> > Thank you for following up. > > I think the best way to answer your questions is to point to the site where the code for CAKE for the Ubiquiti Edgerouter, and the EdgeOS system, is available, along with instructions on how to install and implement as a sticky at the top. > > You will also see that there are 29 pages of discussion with the developer, dating from 4 years ago, the latest from this week. In general, this port and the associated directions on installation scripts have been well accepted. > > https://community.ui.com/questions/Cake-and-FQ-PIE-compiled-for-the-EdgeRouter-devices/fc1ff27c-f321-4344-8737-fcc755cae8a2?page=1 > > I would be very interested to learn your thoughts. > Many thanks. I see that ubiquiti has their own configuration mode/utility [1] and "set interfaces .." is one of the accepted configuration commands. What it does under the hood is a matter of reading the manual or even looking at the code, but I guess it results in running pretty much the same tc commands that were posted before. If the imq device stays unused then you can simply ignore it, or blacklist the module to not even load it on boot. (Please don't ask me how to do this on EdgeOS.) That aside, I still don't understand what your issue is. [1] https://help.ui.com/hc/en-us/articles/204960094-EdgeRouter-Configuration-and-Operational-Mode PS: Please send your responses also to the mailing list, thanks. From dave.taht at gmail.com Fri Jan 29 06:50:02 2021 From: dave.taht at gmail.com (Dave Taht) Date: Fri, 29 Jan 2021 03:50:02 -0800 Subject: [Cake] cake rant on eero Message-ID: https://www.reddit.com/r/eero/comments/l41p1a/sqm_officially_doesnt_work_on_low_bandwidth/ -- "For a successful technology, reality must take precedence over public relations, for Mother Nature cannot be fooled" - Richard Feynman dave at taht.net CTO, TekLibre, LLC Tel: 1-831-435-0729