From dave.taht at gmail.com Sat Jun 3 04:03:54 2023 From: dave.taht at gmail.com (Dave Taht) Date: Sat, 3 Jun 2023 02:03:54 -0600 Subject: [Bloat] receive window bug fix Message-ID: these folk do good work, and I loved the graphs https://blog.cloudflare.com/unbounded-memory-usage-by-tcp-for-receive-buffers-and-how-we-fixed-it/ -- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos From j.w.r.dexter at gmail.com Sat Jun 3 09:56:26 2023 From: j.w.r.dexter at gmail.com (John D) Date: Sat, 3 Jun 2023 14:56:26 +0100 Subject: [Bloat] SQM tuning question Message-ID: Hi, On the website it says the following: *CoDel is a novel “no knobs”, “just works”, “handles variable bandwidth and RTT”, and simple AQM algorithm.* - *It is parameterless — no knobs are required for operators, users, or implementers to adjust.* - *It treats good queue and bad queue differently - that is, it keeps the delays low while permitting bursts of traffic.* - *It controls delay, while insensitive to round-trip delays, link rates, and traffic loads.* - *It adapts to dynamically changing link rates with no negative impact on utilization.* But everywhere I have read about about hardware which implements SQM (including the bufferbloat website) it describes the need to tune based on actual internet connection speed. These seem to conflict especially that "handles variable bandwidth" bit. Have I misunderstood or do the algorithms used in modern hardware just not provide this part typically? My connection performance is quite variable and I'm worried about crippling SQM to the lowest speed seen. Thanks for any help. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chromatix99 at gmail.com Sat Jun 3 11:44:44 2023 From: chromatix99 at gmail.com (Jonathan Morton) Date: Sat, 3 Jun 2023 18:44:44 +0300 Subject: [Bloat] SQM tuning question In-Reply-To: References: Message-ID: <39DED14A-AACF-4C45-9834-C295F92E8800@gmail.com> > On 3 Jun, 2023, at 4:56 pm, John D via Bloat wrote: > > On the website it says the following: > > CoDel is a novel “no knobs”, “just works”, “handles variable bandwidth and RTT”, and simple AQM algorithm. > > • It is parameterless — no knobs are required for operators, users, or implementers to adjust. > • It treats good queue and bad queue differently - that is, it keeps the delays low while permitting bursts of traffic. > • It controls delay, while insensitive to round-trip delays, link rates, and traffic loads. > • It adapts to dynamically changing link rates with no negative impact on utilization. > > But everywhere I have read about about hardware which implements SQM (including the bufferbloat website) it describes the need to tune based on actual internet connection speed. > These seem to conflict especially that "handles variable bandwidth" bit. Have I misunderstood or do the algorithms used in modern hardware just not provide this part typically? My connection performance is quite variable and I'm worried about crippling SQM to the lowest speed seen. SQM in practice requires three components: 1: Flow isolation, so that different flows don't affect each others' latency and are delivered fairly; 2: Active Queue Management (AQM) to signal flows to slow down transmissions when link capacity is exceeded; 3: Bandwidth shaping to match the queue to the available capacity. CoDel is, in itself, only the AQM component. It does indeed work pretty well with no additional tuning - but only in combination with the other two components, or when applied directly to the actual bottleneck. Unfortunately in most consumer internet links, the actual bottleneck is inaccessible for this purpose. Thus an artificial bottleneck must be introduced, at which SQM is applied. The most convenient tool for applying all three SQM components at once is Cake. This includes implementations of advanced flow isolation, CoDel AQM, and a deficit-mode bandwidth shaper. All you really need to do is to tell it how much bandwidth you have in each direction, minus a small margin to ensure it becomes the actual bottleneck and can exert the necessary control. When your available bandwidth varies over time, that can be inconvenient. There are methods, however, of observing how available capacity tends to change over time (typically on diurnal and weekly patterns, if the variations are due to congestion in the ISP backhaul or peering) and scheduling adjustments on that basis. If you have more information on your situation, we might be able to give more detailed advice. - Jonathan Morton From shiva at sewingwitch.com Sat Jun 3 12:54:34 2023 From: shiva at sewingwitch.com (Kenneth Porter) Date: Sat, 03 Jun 2023 09:54:34 -0700 Subject: [Bloat] SQM tuning question In-Reply-To: <39DED14A-AACF-4C45-9834-C295F92E8800@gmail.com> References: <39DED14A-AACF-4C45-9834-C295F92E8800@gmail.com> Message-ID: <642118391FA06B4E63C6FA57@[10.96.7.39]> --On Saturday, June 03, 2023 7:44 PM +0300 Jonathan Morton via Bloat wrote: > When your available bandwidth varies over time, that can be inconvenient. > There are methods, however, of observing how available capacity tends to > change over time (typically on diurnal and weekly patterns, if the > variations are due to congestion in the ISP backhaul or peering) and > scheduling adjustments on that basis. If you have more information on > your situation, we might be able to give more detailed advice. Are there any good solutions for regularly recomputing the bandwidth on a consumer link? I'm running the Linux SQM scripts to launch cake on a recent Debian system and CoDel on a CentOS 7 gateway. I set the bandwidth manually based on periodic manual speed tests. I'd love to automate that. From j.w.r.dexter at gmail.com Sat Jun 3 13:17:19 2023 From: j.w.r.dexter at gmail.com (John D) Date: Sat, 3 Jun 2023 18:17:19 +0100 Subject: [Bloat] SQM tuning question In-Reply-To: <39DED14A-AACF-4C45-9834-C295F92E8800@gmail.com> References: <39DED14A-AACF-4C45-9834-C295F92E8800@gmail.com> Message-ID: Thanks for the detail. It makes sense but it kind of feels like in some (maybe many) cases the router could know the internet link performance. Particularly home router-modems often monitor this already. Maybe that's just not exposed in any standardised way? I'm guessing if I was into openwrt I could maybe do something, but I prefer just to find something off the shelf with half decent SQM... If "auto configuration" isn't a feature then that answers my question and I can get on choosing the best option. On Sat, Jun 3, 2023, 16:44 Jonathan Morton wrote: > > On 3 Jun, 2023, at 4:56 pm, John D via Bloat < > bloat at lists.bufferbloat.net> wrote: > > > > On the website it says the following: > > > > CoDel is a novel “no knobs”, “just works”, “handles variable bandwidth > and RTT”, and simple AQM algorithm. > > > > • It is parameterless — no knobs are required for operators, > users, or implementers to adjust. > > • It treats good queue and bad queue differently - that is, it > keeps the delays low while permitting bursts of traffic. > > • It controls delay, while insensitive to round-trip delays, link > rates, and traffic loads. > > • It adapts to dynamically changing link rates with no negative > impact on utilization. > > > > But everywhere I have read about about hardware which implements SQM > (including the bufferbloat website) it describes the need to tune based on > actual internet connection speed. > > These seem to conflict especially that "handles variable bandwidth" bit. > Have I misunderstood or do the algorithms used in modern hardware just not > provide this part typically? My connection performance is quite variable > and I'm worried about crippling SQM to the lowest speed seen. > > SQM in practice requires three components: > > 1: Flow isolation, so that different flows don't affect each others' > latency and are delivered fairly; > > 2: Active Queue Management (AQM) to signal flows to slow down > transmissions when link capacity is exceeded; > > 3: Bandwidth shaping to match the queue to the available capacity. > > CoDel is, in itself, only the AQM component. It does indeed work pretty > well with no additional tuning - but only in combination with the other two > components, or when applied directly to the actual bottleneck. > Unfortunately in most consumer internet links, the actual bottleneck is > inaccessible for this purpose. Thus an artificial bottleneck must be > introduced, at which SQM is applied. > > The most convenient tool for applying all three SQM components at once is > Cake. This includes implementations of advanced flow isolation, CoDel AQM, > and a deficit-mode bandwidth shaper. All you really need to do is to tell > it how much bandwidth you have in each direction, minus a small margin to > ensure it becomes the actual bottleneck and can exert the necessary control. > > When your available bandwidth varies over time, that can be inconvenient. > There are methods, however, of observing how available capacity tends to > change over time (typically on diurnal and weekly patterns, if the > variations are due to congestion in the ISP backhaul or peering) and > scheduling adjustments on that basis. If you have more information on your > situation, we might be able to give more detailed advice. > > - Jonathan Morton -------------- next part -------------- An HTML attachment was scrubbed... URL: From woody77 at gmail.com Sat Jun 3 14:04:56 2023 From: woody77 at gmail.com (Aaron Wood) Date: Sat, 3 Jun 2023 11:04:56 -0700 Subject: [Bloat] SQM tuning question In-Reply-To: References: <39DED14A-AACF-4C45-9834-C295F92E8800@gmail.com> Message-ID: I’ve found that _usually_ I can set cake’s bandwidth limits to 90-95% of the advertised bandwidth, and everything “just works”. So long as you’re routinely able to achieve the bandwidth, it tends to work. I’ve found in my testing over the years (I’ve been a user of fq_codel since 2013) that limiting the upstream bandwidth is the most important one to do. Downstream bandwidth limits, especially if you have well above 100Mbps downstream aren’t as critical, but it still can make a big difference if you routinely saturate your downstream connection. It’s especially important to manage the upstream bandwidth when the connection is asymmetric. For instance my 1Gbps Comcast internet service only has a 35Mbps upstream limit, which is just a bit more than you need for the acks that go back upstream in response to 1Gbps of downstream traffic. But if your connection quality suffers with time of day (congestion on shared local ISP infrastructure), or due to weather (radio links can degrade during the rain), then that can be a bit more annoying. But as I said, I’ve found with my Comcast connection that I can just set my bandwidth limits to 90-95% of advertised and it just works. The other thing to look into is DNS. Most of the weird pauses that I see are, DNS related. I e switches to using a local DNS cache/lookup-forwarding server, and use DNS over HTTPS to parallel requests to both cloudflare and google’s HTTPS DNS proxies. That seems to be more stable (long-lived connections with faster recovery from a dropped packed than UDP has). (More continued below) On Sat, Jun 3, 2023 at 10:17 AM John D via Bloat < bloat at lists.bufferbloat.net> wrote: > Thanks for the detail. It makes sense but it kind of feels like in some > (maybe many) cases the router could know the internet link performance. > Particularly home router-modems often monitor this already. Maybe that's > just not exposed in any standardised way? > There are two issues I’ve seen here: 1) there’s not a standardized way (or even usually any api) for getting the provisioned rate, or the current link speed, from the modem portion of things. 2) local congestion at the ISP can degrade things below the provisioned rates, and that’s even harder to detect or have available. If your upstream connection is DOCSIS 3.1, you should have the PIE AQM running in the modem, which _should_ help considerably. But at least at my cable headend, while downstream runs DOCSIS 3.1, upstream is only DOCSIS 3.0, and definitely isn’t using PIE. The PIE AQM, running on the modem itself has the advantage of running on what is almost always the bottle neck for upstream traffic in my experience: the provisioned upstream bandwidth in the modem. I'm guessing if I was into openwrt I could maybe do something, but I prefer > just to find something off the shelf with half decent SQM... If "auto > configuration" isn't a feature then that answers my question and I can get > on choosing the best option. > I’m not aware of any decent “off the shelf” solutions that can track bandwidth correctly, aside from DOCSIS 3.1 modems (which still requires the cable company to provision it correctly to enable it) But, as I said, if your connection is stable, you can get the majority of the benefits just by setting it up once and leaving it be. I’ve only tweaked things 1-2 times a year, for the last three years, and that was only because I was moving. Once I had setup a router that could run cake at my line rates, I’ve just left it there and it’s been fine. Especially since for me, WiFi is my usual bottleneck (my APs only do about 250-300Mbps) -Aaron > On Sat, Jun 3, 2023, 16:44 Jonathan Morton wrote: > >> > On 3 Jun, 2023, at 4:56 pm, John D via Bloat < >> bloat at lists.bufferbloat.net> wrote: >> > >> > On the website it says the following: >> > >> > CoDel is a novel “no knobs”, “just works”, “handles variable bandwidth >> and RTT”, and simple AQM algorithm. >> > >> > • It is parameterless — no knobs are required for operators, >> users, or implementers to adjust. >> > • It treats good queue and bad queue differently - that is, it >> keeps the delays low while permitting bursts of traffic. >> > • It controls delay, while insensitive to round-trip delays, link >> rates, and traffic loads. >> > • It adapts to dynamically changing link rates with no negative >> impact on utilization. >> > >> > But everywhere I have read about about hardware which implements SQM >> (including the bufferbloat website) it describes the need to tune based on >> actual internet connection speed. >> > These seem to conflict especially that "handles variable bandwidth" >> bit. Have I misunderstood or do the algorithms used in modern hardware just >> not provide this part typically? My connection performance is quite >> variable and I'm worried about crippling SQM to the lowest speed seen. >> >> SQM in practice requires three components: >> >> 1: Flow isolation, so that different flows don't affect each others' >> latency and are delivered fairly; >> >> 2: Active Queue Management (AQM) to signal flows to slow down >> transmissions when link capacity is exceeded; >> >> 3: Bandwidth shaping to match the queue to the available capacity. >> >> CoDel is, in itself, only the AQM component. It does indeed work pretty >> well with no additional tuning - but only in combination with the other two >> components, or when applied directly to the actual bottleneck. >> Unfortunately in most consumer internet links, the actual bottleneck is >> inaccessible for this purpose. Thus an artificial bottleneck must be >> introduced, at which SQM is applied. >> >> The most convenient tool for applying all three SQM components at once is >> Cake. This includes implementations of advanced flow isolation, CoDel AQM, >> and a deficit-mode bandwidth shaper. All you really need to do is to tell >> it how much bandwidth you have in each direction, minus a small margin to >> ensure it becomes the actual bottleneck and can exert the necessary control. >> >> When your available bandwidth varies over time, that can be >> inconvenient. There are methods, however, of observing how available >> capacity tends to change over time (typically on diurnal and weekly >> patterns, if the variations are due to congestion in the ISP backhaul or >> peering) and scheduling adjustments on that basis. If you have more >> information on your situation, we might be able to give more detailed >> advice. >> >> - Jonathan Morton > > _______________________________________________ > Bloat mailing list > Bloat at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat > -- - Sent from my iPhone. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rjmcmahon at rjmcmahon.com Sat Jun 3 14:56:12 2023 From: rjmcmahon at rjmcmahon.com (rjmcmahon) Date: Sat, 03 Jun 2023 11:56:12 -0700 Subject: [Bloat] [Rpm] receive window bug fix In-Reply-To: References: Message-ID: <709fd354372c370233f4b4eb33bab216@rjmcmahon.com> > these folk do good work, and I loved the graphs > > https://blog.cloudflare.com/unbounded-memory-usage-by-tcp-for-receive-buffers-and-how-we-fixed-it/ Very cool. Thanks for sharing. I've been considering adding stress tests to iperf 2. Looks like Cloudfare has at least two Small reads & writes with short delay to stress receive window processing per At the sending host, run a TCP program with an infinite loop, sending 1500B packets, with a 1 ms delay between each send. At the receiving host, run a TCP program with an infinite loop, reading 1B at a time, with a 1 ms delay between each read. And then, rx buffer limit tests, from https://blog.cloudflare.com/optimizing-tcp-for-high-throughput-and-low-latency/ reads as fast as it can, for five seconds this is called fast mode, opens up the window calculates 5% of the high watermark of the bytes reader during any previous one second for each second of the next 15 seconds: this is called slow mode reads that 5% number of bytes, then stops reading sleeps for the remainder of that particular second most of the second consists of no reading at all steps 1-3 are repeated in a loop three times, so the entire run is 60 seconds This has the effect of highlighting any issues in the handling of packets when the buffers repeatedly hit the limit. Curious about any other traffic scenarios driven by socket read/write behaviors that could be useful. Or any others that might apply to WiFi aggregation. Then, if there is a way to generalize these types of send/read/delay graphs with a parametric command line? Bob From woody77 at gmail.com Sat Jun 3 14:20:43 2023 From: woody77 at gmail.com (Aaron Wood) Date: Sat, 3 Jun 2023 11:20:43 -0700 Subject: [Bloat] [Rpm] receive window bug fix In-Reply-To: References: Message-ID: This is good work! I love reading their posts on scale like this. It’s wild to me that the Linux kernel has (apparently) never implemented shrinking the receive window, or handling the case of userspace starting a large transfer and then just not ever reading it… the latter is less surprising, I guess, because that’s an application bug that you probably would catch separately, and would be focused on fixing in the application layer… -Aaron On Sat, Jun 3, 2023 at 1:04 AM Dave Taht via Rpm wrote: > these folk do good work, and I loved the graphs > > > https://blog.cloudflare.com/unbounded-memory-usage-by-tcp-for-receive-buffers-and-how-we-fixed-it/ > > -- > Podcast: > https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ > Dave Täht CSO, LibreQos > _______________________________________________ > Rpm mailing list > Rpm at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/rpm > -- - Sent from my iPhone. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rjmcmahon at rjmcmahon.com Sat Jun 3 15:15:31 2023 From: rjmcmahon at rjmcmahon.com (rjmcmahon) Date: Sat, 03 Jun 2023 12:15:31 -0700 Subject: [Bloat] [Rpm] receive window bug fix In-Reply-To: References: Message-ID: <6968461fa2076430aa8d379709488c5a@rjmcmahon.com> I think better tooling can help and I am always interested in suggestions on what to add to iperf 2 for better coverages. I've thought it good for iperf 2 to support some sort of graph which drives socket read/write/delays vs a simplistic pattern of AFAP. It for sure stresses things differently, even in drivers. I've seen huge delays in some 10G drivers where some UDP packets seem to get stuck in queues and where the e2e latency is driven by the socket write rates vs the network delays. This is most obvious using burst patterns where the last packet of a latency burst is coupled to the first packet of the subsequent burst. The coupling between the syscalls to network performance is nonobvious and sometimes hard to believe. We've been adding more "traffic profile" knobs for socket testing and have much of the latency metrics incorporated. Most don't use these. They seem to be hard to generalize. Cloudflare seems to have crafted specific tests after obtaining knowledge of causality. Bob PS. As a side note, I'm now being asked how to generate "AI loads" into switch fabrics, though there it probably won't be based upon socket syscalls but maybe using io_urings - not sure. > This is good work! I love reading their posts on scale like this. > > It’s wild to me that the Linux kernel has (apparently) never > implemented shrinking the receive window, or handling the case of > userspace starting a large transfer and then just not ever reading > it… the latter is less surprising, I guess, because that’s an > application bug that you probably would catch separately, and would be > focused on fixing in the application layer… > > -Aaron > > On Sat, Jun 3, 2023 at 1:04 AM Dave Taht via Rpm > wrote: > >> these folk do good work, and I loved the graphs >> >> > https://blog.cloudflare.com/unbounded-memory-usage-by-tcp-for-receive-buffers-and-how-we-fixed-it/ >> >> -- >> Podcast: >> > https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ >> Dave Täht CSO, LibreQos >> _______________________________________________ >> Rpm mailing list >> Rpm at lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/rpm > -- > - Sent from my iPhone. > _______________________________________________ > Rpm mailing list > Rpm at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/rpm From dave.taht at gmail.com Sat Jun 3 17:23:24 2023 From: dave.taht at gmail.com (Dave Taht) Date: Sat, 3 Jun 2023 15:23:24 -0600 Subject: [Bloat] decoding what digital equity means? Message-ID: after reading this, I still have no idea wtf they are talking about. Does it have IPv6? https://www.benton.org/visions-digital-equity -- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos From dave.taht at gmail.com Sun Jun 4 04:14:52 2023 From: dave.taht at gmail.com (Dave Taht) Date: Sun, 4 Jun 2023 02:14:52 -0600 Subject: [Bloat] Fwd: [iccrg] I-D Action: draft-irtf-iccrg-rledbat-04.txt In-Reply-To: <168586571145.14252.2019070009409533947@ietfa.amsl.com> References: <168586571145.14252.2019070009409533947@ietfa.amsl.com> Message-ID: ---------- Forwarded message --------- From: Date: Sun, Jun 4, 2023 at 2:02 AM Subject: [iccrg] I-D Action: draft-irtf-iccrg-rledbat-04.txt To: Cc: A New Internet-Draft is available from the on-line Internet-Drafts directories. This Internet-Draft is a work item of the Internet Congestion Control (ICCRG) RG of the IRTF. Title : rLEDBAT: receiver-driven Low Extra Delay Background Transport for TCP Authors : Marcelo Bagnulo Alberto Garcia-Martinez Gabriel Montenegro Praveen Balasubramanian Filename : draft-irtf-iccrg-rledbat-04.txt Pages : 15 Date : 2023-06-04 Abstract: This document specifies the rLEDBAT, a set of mechanisms that enable the execution of a less-than-best-effort congestion control algorithm for TCP at the receiver end. The IETF datatracker status page for this Internet-Draft is: https://datatracker.ietf.org/doc/draft-irtf-iccrg-rledbat/ There is also an htmlized version available at: https://datatracker.ietf.org/doc/html/draft-irtf-iccrg-rledbat-04 A diff from the previous version is available at: https://author-tools.ietf.org/iddiff?url2=draft-irtf-iccrg-rledbat-04 Internet-Drafts are also available by rsync at rsync.ietf.org::internet-drafts _______________________________________________ iccrg mailing list iccrg at irtf.org https://www.irtf.org/mailman/listinfo/iccrg -- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos From j.w.r.dexter at gmail.com Sun Jun 4 04:48:21 2023 From: j.w.r.dexter at gmail.com (John D) Date: Sun, 4 Jun 2023 09:48:21 +0100 Subject: [Bloat] Dual WAN home router with decent SQM? Message-ID: I want to set up a new home router with SQM support to alleviate (mostly upload) bloat. However I would also like dual ISP support, more for backup/failover than load sharing. I can see affordable multi wan "load balancer" routers eg TP-link, and affordable SQM routers such as ubiquity Edgerouter, but I'm struggling to find a single device providing both. Just after any advice on a) am I better having two separate components b) if so any suggestions which product/brand? Two devices means more configuration but possibly more flexible and cheaper. My internet speed is looking to max out around 100/20mbps so I don't need Gbit performance. Thanks for any help. -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at lochnair.net Sun Jun 4 09:07:35 2023 From: me at lochnair.net (Nils Andreas Svee) Date: Sun, 04 Jun 2023 15:07:35 +0200 Subject: [Bloat] Dual WAN home router with decent SQM? In-Reply-To: References: Message-ID: Hi The EdgeRouter line does have load balancing as well as SQM (htb + fq_codel), so one of those should cover your needs. Of note, CAKE is not built-in to the firmware if you care about it, but support can be compiled With your current bandwidth, I'd guess an EdgeRouter Lite would to weak, but the EdgeRouter X or more powerful would do the job well. Keep in mind though, that as far as I can tell, the EdgeMax line is basically on life support, and only receives some security fixes once in a blue moon. Mikrotik's RouterOS devices supports load balancing / failover and SQM I believe, so there might be an option for you there, but as I've never used them personally, I can't comment beyond that. Best Regards Nils On Sun, Jun 4, 2023, at 10:48, John D via Bloat wrote: > I want to set up a new home router with SQM support to alleviate (mostly upload) bloat. > However I would also like dual ISP support, more for backup/failover than load sharing. > I can see affordable multi wan "load balancer" routers eg TP-link, and affordable SQM routers such as ubiquity Edgerouter, but I'm struggling to find a single device providing both. > Just after any advice on a) am I better having two separate components b) if so any suggestions which product/brand? > Two devices means more configuration but possibly more flexible and cheaper. > > My internet speed is looking to max out around 100/20mbps so I don't need Gbit performance. Thanks for any help. > _______________________________________________ > Bloat mailing list > Bloat at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat > From j.w.r.dexter at gmail.com Sun Jun 4 09:57:18 2023 From: j.w.r.dexter at gmail.com (John D) Date: Sun, 4 Jun 2023 14:57:18 +0100 Subject: [Bloat] Dual WAN home router with decent SQM? In-Reply-To: References: Message-ID: I might be missing the very obvious but I'm struggling to see which ER models support multiple WAN when I look on their own website. Unless ALL ports support WAN... if the ER-X can do this already then that was what I was looking at on the router side anyway. On Sun, Jun 4, 2023, 14:07 Nils Andreas Svee wrote: > Hi > > The EdgeRouter line does have load balancing as well as SQM (htb + > fq_codel), > so one of those should cover your needs. > > Of note, CAKE is not built-in to the firmware if you care about it, > but support can be compiled > > With your current bandwidth, I'd guess an EdgeRouter Lite would to weak, > but the EdgeRouter X or more powerful would do the job well. > > Keep in mind though, that as far as I can tell, the EdgeMax line is > basically on life support, > and only receives some security fixes once in a blue moon. > > Mikrotik's RouterOS devices supports load balancing / failover and SQM I > believe, > so there might be an option for you there, but as I've never used them > personally, > I can't comment beyond that. > > Best Regards > Nils > > On Sun, Jun 4, 2023, at 10:48, John D via Bloat wrote: > > I want to set up a new home router with SQM support to alleviate (mostly > upload) bloat. > > However I would also like dual ISP support, more for backup/failover > than load sharing. > > I can see affordable multi wan "load balancer" routers eg TP-link, and > affordable SQM routers such as ubiquity Edgerouter, but I'm struggling to > find a single device providing both. > > Just after any advice on a) am I better having two separate components > b) if so any suggestions which product/brand? > > Two devices means more configuration but possibly more flexible and > cheaper. > > > > My internet speed is looking to max out around 100/20mbps so I don't > need Gbit performance. Thanks for any help. > > _______________________________________________ > > Bloat mailing list > > Bloat at lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/bloat > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at lochnair.net Sun Jun 4 10:22:07 2023 From: me at lochnair.net (Nils Andreas Svee) Date: Sun, 04 Jun 2023 16:22:07 +0200 Subject: [Bloat] Dual WAN home router with decent SQM? In-Reply-To: References: Message-ID: <7f8b3f52-c947-471b-9530-4adcba84cef2@app.fastmail.com> The wizards for quick setup makes some assumptions about which ports is WAN/LAN IIRC, but you're free to use the ports for whatever purpose you wish So yes, all the models can be used with load balancing / failover Best Regards Nils On Sun, Jun 4, 2023, at 15:57, John D wrote: > I might be missing the very obvious but I'm struggling to see which ER models support multiple WAN when I look on their own website. Unless ALL ports support WAN... if the ER-X can do this already then that was what I was looking at on the router side anyway. > > On Sun, Jun 4, 2023, 14:07 Nils Andreas Svee wrote: >> Hi >> >> The EdgeRouter line does have load balancing as well as SQM (htb + fq_codel), >> so one of those should cover your needs. >> >> Of note, CAKE is not built-in to the firmware if you care about it, >> but support can be compiled >> >> With your current bandwidth, I'd guess an EdgeRouter Lite would to weak, >> but the EdgeRouter X or more powerful would do the job well. >> >> Keep in mind though, that as far as I can tell, the EdgeMax line is basically on life support, >> and only receives some security fixes once in a blue moon. >> >> Mikrotik's RouterOS devices supports load balancing / failover and SQM I believe, >> so there might be an option for you there, but as I've never used them personally, >> I can't comment beyond that. >> >> Best Regards >> Nils >> >> On Sun, Jun 4, 2023, at 10:48, John D via Bloat wrote: >> > I want to set up a new home router with SQM support to alleviate (mostly upload) bloat. >> > However I would also like dual ISP support, more for backup/failover than load sharing. >> > I can see affordable multi wan "load balancer" routers eg TP-link, and affordable SQM routers such as ubiquity Edgerouter, but I'm struggling to find a single device providing both. >> > Just after any advice on a) am I better having two separate components b) if so any suggestions which product/brand? >> > Two devices means more configuration but possibly more flexible and cheaper. >> > >> > My internet speed is looking to max out around 100/20mbps so I don't need Gbit performance. Thanks for any help. >> > _______________________________________________ >> > Bloat mailing list >> > Bloat at lists.bufferbloat.net >> > https://lists.bufferbloat.net/listinfo/bloat >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at lang.hm Sun Jun 4 10:35:16 2023 From: david at lang.hm (David Lang) Date: Sun, 4 Jun 2023 07:35:16 -0700 (PDT) Subject: [Bloat] Dual WAN home router with decent SQM? In-Reply-To: References: Message-ID: <58238rpr-80q1-3p5s-849q-604ns5659s7r@ynat.uz> any router that you can run OpenWRT on will do the job look at the mwan3 package to mange the multiple ISPs at 100Mb, you don't need an especially beefy router. David Lang On Sun, 4 Jun 2023, John D via Bloat wrote: > Date: Sun, 4 Jun 2023 09:48:21 +0100 > From: John D via Bloat > Reply-To: John D > To: bloat at lists.bufferbloat.net > Subject: [Bloat] Dual WAN home router with decent SQM? > > I want to set up a new home router with SQM support to alleviate (mostly > upload) bloat. > However I would also like dual ISP support, more for backup/failover than > load sharing. > I can see affordable multi wan "load balancer" routers eg TP-link, and > affordable SQM routers such as ubiquity Edgerouter, but I'm struggling to > find a single device providing both. > Just after any advice on a) am I better having two separate components b) > if so any suggestions which product/brand? > Two devices means more configuration but possibly more flexible and cheaper. > > My internet speed is looking to max out around 100/20mbps so I don't need > Gbit performance. Thanks for any help. > -------------- next part -------------- _______________________________________________ Bloat mailing list Bloat at lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat From stephen at networkplumber.org Sun Jun 4 12:22:51 2023 From: stephen at networkplumber.org (Stephen Hemminger) Date: Sun, 4 Jun 2023 09:22:51 -0700 Subject: [Bloat] decoding what digital equity means? In-Reply-To: References: Message-ID: <20230604092251.1a6fc50a@hermes.local> On Sat, 3 Jun 2023 15:23:24 -0600 Dave Taht via Bloat wrote: > after reading this, I still have no idea wtf they are talking about. > Does it have IPv6? > > https://www.benton.org/visions-digital-equity > Digital equity is about making equivalent bandwidth available to under served communities. https://diversity.social/digital-equity/ It looks like a broad brush statement that lower income and older people have less computers and smartphones. That vision statement is typical political meta statement with no direct action or policy. From moeller0 at gmx.de Sun Jun 4 14:48:48 2023 From: moeller0 at gmx.de (Sebastian Moeller) Date: Sun, 4 Jun 2023 20:48:48 +0200 Subject: [Bloat] SQM tuning question In-Reply-To: <642118391FA06B4E63C6FA57@[10.96.7.39]> References: <39DED14A-AACF-4C45-9834-C295F92E8800@gmail.com> <642118391FA06B4E63C6FA57@[10.96.7.39]> Message-ID: Have a look at: cake-autorate: https://github.com/lynxthecat/cake-autorate/blob/master/cake-autorate.sh lua sqm-autorate: https://github.com/sqm-autorate/sqm-autorate perl sqm-autorate: https://github.com/tievolu/sqm-autorate all three use active latency probes to detect an increase in delay and interpret that as increase in queueing delay which happens if the offered load exceeds the available capacity, and all three will, by slightly different heuristics, adjust the traffic shaper so that latency under loads stays (mostly) within acceptable (configurable) limits. Note the "organic" traffic over the link is measured and used as "load" so no cyclic saturating speedtest is necessary. I think that evenroute's iqrouter uses a different approach in that they periodically perform speedtests and then set the traffic shaper in accordance to that result. However the three different *-autorates should be runnable on essentially every linux distribution, while for evenroute's implementation one would need to buy their hardware IIRC. Best Regards Sebastian > On Jun 3, 2023, at 18:54, Kenneth Porter via Bloat wrote: > > --On Saturday, June 03, 2023 7:44 PM +0300 Jonathan Morton via Bloat wrote: > >> When your available bandwidth varies over time, that can be inconvenient. >> There are methods, however, of observing how available capacity tends to >> change over time (typically on diurnal and weekly patterns, if the >> variations are due to congestion in the ISP backhaul or peering) and >> scheduling adjustments on that basis. If you have more information on >> your situation, we might be able to give more detailed advice. > > Are there any good solutions for regularly recomputing the bandwidth on a consumer link? I'm running the Linux SQM scripts to launch cake on a recent Debian system and CoDel on a CentOS 7 gateway. I set the bandwidth manually based on periodic manual speed tests. I'd love to automate that. > > _______________________________________________ > Bloat mailing list > Bloat at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat From moeller0 at gmx.de Sun Jun 4 14:56:12 2023 From: moeller0 at gmx.de (Sebastian Moeller) Date: Sun, 4 Jun 2023 20:56:12 +0200 Subject: [Bloat] SQM tuning question In-Reply-To: References: <39DED14A-AACF-4C45-9834-C295F92E8800@gmail.com> Message-ID: <724593C4-0427-45EB-A12B-D1C372CFE780@gmx.de> Hi John, > On Jun 3, 2023, at 19:17, John D via Bloat wrote: > > Thanks for the detail. It makes sense but it kind of feels like in some (maybe many) cases the router could know the internet link performance. Sometimes they do, sometimes they do not, and sometimes the number the router might now is well above the contractual limit, e.g. my ISP for some time configured my VDSL2 link to sync at ~100/40 Mbps and restrict my capacity at an upstream device (BNG) to my contracted ~50/10 Mbps, so knowing the link speed will only give you an upper bound for some link technologies. > Particularly home router-modems often monitor this already. Maybe that's just not exposed in any standardised way? Yes and no, all the link technologies I looked at (DSL/DOCSIS/PON) have some channel between the ISP side equipment (DSLAM/CMTS/OLT) and the equipment on the user side (CPE: DSL-/cable-/pon-modem) by which the ISP gear can query information from the CPE; but these three seem to be using three different approaches, and more importantly, just because the head-end can query these, does not mean that devices in the home network can do so as well... Plus, as mentioned above, the reported "sync" capacity might not be the capacity required to configure sqm's traffic shaper. > I'm guessing if I was into openwrt I could maybe do something, but I prefer just to find something off the shelf with half decent SQM... If "auto configuration" isn't a feature then that answers my question and I can get on choosing the best option. There are the work-in-progress *-autorate approaches mentioned in my earlier post... Kind Regards Sebastian > > On Sat, Jun 3, 2023, 16:44 Jonathan Morton wrote: > > On 3 Jun, 2023, at 4:56 pm, John D via Bloat wrote: > > > > On the website it says the following: > > > > CoDel is a novel “no knobs”, “just works”, “handles variable bandwidth and RTT”, and simple AQM algorithm. > > > > • It is parameterless — no knobs are required for operators, users, or implementers to adjust. > > • It treats good queue and bad queue differently - that is, it keeps the delays low while permitting bursts of traffic. > > • It controls delay, while insensitive to round-trip delays, link rates, and traffic loads. > > • It adapts to dynamically changing link rates with no negative impact on utilization. > > > > But everywhere I have read about about hardware which implements SQM (including the bufferbloat website) it describes the need to tune based on actual internet connection speed. > > These seem to conflict especially that "handles variable bandwidth" bit. Have I misunderstood or do the algorithms used in modern hardware just not provide this part typically? My connection performance is quite variable and I'm worried about crippling SQM to the lowest speed seen. > > SQM in practice requires three components: > > 1: Flow isolation, so that different flows don't affect each others' latency and are delivered fairly; > > 2: Active Queue Management (AQM) to signal flows to slow down transmissions when link capacity is exceeded; > > 3: Bandwidth shaping to match the queue to the available capacity. > > CoDel is, in itself, only the AQM component. It does indeed work pretty well with no additional tuning - but only in combination with the other two components, or when applied directly to the actual bottleneck. Unfortunately in most consumer internet links, the actual bottleneck is inaccessible for this purpose. Thus an artificial bottleneck must be introduced, at which SQM is applied. > > The most convenient tool for applying all three SQM components at once is Cake. This includes implementations of advanced flow isolation, CoDel AQM, and a deficit-mode bandwidth shaper. All you really need to do is to tell it how much bandwidth you have in each direction, minus a small margin to ensure it becomes the actual bottleneck and can exert the necessary control. > > When your available bandwidth varies over time, that can be inconvenient. There are methods, however, of observing how available capacity tends to change over time (typically on diurnal and weekly patterns, if the variations are due to congestion in the ISP backhaul or peering) and scheduling adjustments on that basis. If you have more information on your situation, we might be able to give more detailed advice. > > - Jonathan Morton > _______________________________________________ > Bloat mailing list > Bloat at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat From moeller0 at gmx.de Sun Jun 4 15:55:41 2023 From: moeller0 at gmx.de (Sebastian Moeller) Date: Sun, 4 Jun 2023 21:55:41 +0200 Subject: [Bloat] SQM tuning question In-Reply-To: References: <39DED14A-AACF-4C45-9834-C295F92E8800@gmail.com> Message-ID: <63272858-46FF-493E-9A81-F6152DD13E8D@gmx.de> Hi Aaron, > On Jun 3, 2023, at 20:04, Aaron Wood via Bloat wrote: > > I’ve found that _usually_ I can set cake’s bandwidth limits to 90-95% of the advertised bandwidth, and everything “just works”. So long as you’re routinely able to achieve the bandwidth, it tends to work. > > I’ve found in my testing over the years (I’ve been a user of fq_codel since 2013) that limiting the upstream bandwidth is the most important one to do. Downstream bandwidth limits, especially if you have well above 100Mbps downstream aren’t as critical, but it still can make a big difference if you routinely saturate your downstream connection. > > It’s especially important to manage the upstream bandwidth when the connection is asymmetric. For instance my 1Gbps Comcast internet service only has a 35Mbps upstream limit, which is just a bit more than you need for the acks that go back upstream in response to 1Gbps of downstream traffic. > > But if your connection quality suffers with time of day (congestion on shared local ISP infrastructure), or due to weather (radio links can degrade during the rain), then that can be a bit more annoying. > > But as I said, I’ve found with my Comcast connection that I can just set my bandwidth limits to 90-95% of advertised and it just works. > > The other thing to look into is DNS. Most of the weird pauses that I see are, DNS related. I e switches to using a local DNS cache/lookup-forwarding server, and use DNS over HTTPS to parallel requests to both cloudflare and google’s HTTPS DNS proxies. That seems to be more stable (long-lived connections with faster recovery from a dropped packed than UDP has). > > (More continued below) > > On Sat, Jun 3, 2023 at 10:17 AM John D via Bloat wrote: > Thanks for the detail. It makes sense but it kind of feels like in some (maybe many) cases the router could know the internet link performance. Particularly home router-modems often monitor this already. Maybe that's just not exposed in any standardised way? > > There are two issues I’ve seen here: > > 1) there’s not a standardized way (or even usually any api) for getting the provisioned rate, or the current link speed, from the modem portion of things. > > 2) local congestion at the ISP can degrade things below the provisioned rates, and that’s even harder to detect or have available. > > If your upstream connection is DOCSIS 3.1, you should have the PIE AQM running in the modem, which _should_ help considerably. But at least at my cable headend, while downstream runs DOCSIS 3.1, upstream is only DOCSIS 3.0, and definitely isn’t using PIE. > > The PIE AQM, running on the modem itself has the advantage of running on what is almost always the bottle neck for upstream traffic in my experience: the provisioned upstream bandwidth in the modem. [SM] Even better, the cable modem requests mini-slots from the scheduler and really has control over the true upstream queue, so even in an overloaded segment (where mini-slot scheduling gets slow) the PIE instance sees the relevant queueing delay to do its thing even if the current rate is well below the "provisioned" rate (which is just a maximum peak rate anyway). Now this will only help for congestion in the DOCSIS segment, as you said that is likely to be the most likely cause for upstream delay anyways... > > I'm guessing if I was into openwrt I could maybe do something, but I prefer just to find something off the shelf with half decent SQM... If "auto configuration" isn't a feature then that answers my question and I can get on choosing the best option. > > I’m not aware of any decent “off the shelf” solutions that can track bandwidth correctly, aside from DOCSIS 3.1 modems (which still requires the cable company to provision it correctly to enable it) > > But, as I said, if your connection is stable, you can get the majority of the benefits just by setting it up once and leaving it be. > > I’ve only tweaked things 1-2 times a year, for the last three years, and that was only because I was moving. Once I had setup a router that could run cake at my line rates, I’ve just left it there and it’s been fine. > > Especially since for me, WiFi is my usual bottleneck (my APs only do about 250-300Mbps) > > -Aaron > > > On Sat, Jun 3, 2023, 16:44 Jonathan Morton wrote: > > On 3 Jun, 2023, at 4:56 pm, John D via Bloat wrote: > > > > On the website it says the following: > > > > CoDel is a novel “no knobs”, “just works”, “handles variable bandwidth and RTT”, and simple AQM algorithm. > > > > • It is parameterless — no knobs are required for operators, users, or implementers to adjust. > > • It treats good queue and bad queue differently - that is, it keeps the delays low while permitting bursts of traffic. > > • It controls delay, while insensitive to round-trip delays, link rates, and traffic loads. > > • It adapts to dynamically changing link rates with no negative impact on utilization. > > > > But everywhere I have read about about hardware which implements SQM (including the bufferbloat website) it describes the need to tune based on actual internet connection speed. > > These seem to conflict especially that "handles variable bandwidth" bit. Have I misunderstood or do the algorithms used in modern hardware just not provide this part typically? My connection performance is quite variable and I'm worried about crippling SQM to the lowest speed seen. > > SQM in practice requires three components: > > 1: Flow isolation, so that different flows don't affect each others' latency and are delivered fairly; > > 2: Active Queue Management (AQM) to signal flows to slow down transmissions when link capacity is exceeded; > > 3: Bandwidth shaping to match the queue to the available capacity. > > CoDel is, in itself, only the AQM component. It does indeed work pretty well with no additional tuning - but only in combination with the other two components, or when applied directly to the actual bottleneck. Unfortunately in most consumer internet links, the actual bottleneck is inaccessible for this purpose. Thus an artificial bottleneck must be introduced, at which SQM is applied. > > The most convenient tool for applying all three SQM components at once is Cake. This includes implementations of advanced flow isolation, CoDel AQM, and a deficit-mode bandwidth shaper. All you really need to do is to tell it how much bandwidth you have in each direction, minus a small margin to ensure it becomes the actual bottleneck and can exert the necessary control. > > When your available bandwidth varies over time, that can be inconvenient. There are methods, however, of observing how available capacity tends to change over time (typically on diurnal and weekly patterns, if the variations are due to congestion in the ISP backhaul or peering) and scheduling adjustments on that basis. If you have more information on your situation, we might be able to give more detailed advice. > > - Jonathan Morton > _______________________________________________ > Bloat mailing list > Bloat at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat > -- > - Sent from my iPhone. > _______________________________________________ > Bloat mailing list > Bloat at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat From dave.taht at gmail.com Sun Jun 4 20:44:42 2023 From: dave.taht at gmail.com (Dave Taht) Date: Sun, 4 Jun 2023 18:44:42 -0600 Subject: [Bloat] a new mechanism for congestion control Message-ID: sayeth nvidia. Anyone got a clue as to how? https://www.zdnet.com/article/nvidia-unveils-new-kind-of-ethernet-for-ai-grace-hopper-superchip-in-full-production/ -- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos From nathan at nathan.io Mon Jun 5 08:01:55 2023 From: nathan at nathan.io (Nathan Owens) Date: Mon, 5 Jun 2023 06:01:55 -0600 Subject: [Bloat] a new mechanism for congestion control In-Reply-To: References: Message-ID: There’s a little bit of detail here: https://developer.nvidia.com/blog/turbocharging-ai-workloads-with-nvidia-spectrum-x-networking-platform/ They run some sort congestion control on the BlueField-3 chip all synchronized across the fabric or something. On Sun, Jun 4, 2023 at 6:44 PM Dave Taht via Bloat < bloat at lists.bufferbloat.net> wrote: > sayeth nvidia. Anyone got a clue as to how? > > > https://www.zdnet.com/article/nvidia-unveils-new-kind-of-ethernet-for-ai-grace-hopper-superchip-in-full-production/ > > -- > Podcast: > https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ > Dave Täht CSO, LibreQos > _______________________________________________ > Bloat mailing list > Bloat at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat > -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.a.cornejo at gmail.com Mon Jun 5 12:06:43 2023 From: luis.a.cornejo at gmail.com (Luis A. Cornejo) Date: Mon, 5 Jun 2023 11:06:43 -0500 Subject: [Bloat] Dual WAN home router with decent SQM? In-Reply-To: <58238rpr-80q1-3p5s-849q-604ns5659s7r@ynat.uz> References: <58238rpr-80q1-3p5s-849q-604ns5659s7r@ynat.uz> Message-ID: I’ve done the multi wan with mwan3 and it works fine for load balance or failover. But if you are trying to improve your upload then the best you can get is one upload vs the other. You won’t get both uploads to the same stream, that is bonding, which is a little harder to do. -Luis On Sun, Jun 4, 2023 at 9:35 AM David Lang via Bloat < bloat at lists.bufferbloat.net> wrote: > any router that you can run OpenWRT on will do the job > look at the mwan3 package to mange the multiple ISPs > > at 100Mb, you don't need an especially beefy router. > > David Lang > > On Sun, 4 Jun 2023, John D > via Bloat wrote: > > > Date: Sun, 4 Jun 2023 09:48:21 +0100 > > From: John D via Bloat > > Reply-To: John D > > To: bloat at lists.bufferbloat.net > > Subject: [Bloat] Dual WAN home router with decent SQM? > > > > I want to set up a new home router with SQM support to alleviate (mostly > > upload) bloat. > > However I would also like dual ISP support, more for backup/failover than > > load sharing. > > I can see affordable multi wan "load balancer" routers eg TP-link, and > > affordable SQM routers such as ubiquity Edgerouter, but I'm struggling to > > find a single device providing both. > > Just after any advice on a) am I better having two separate components b) > > if so any suggestions which product/brand? > > Two devices means more configuration but possibly more flexible and > cheaper. > > > > My internet speed is looking to max out around 100/20mbps so I don't need > > Gbit performance. Thanks for any help. > >_______________________________________________ > Bloat mailing list > Bloat at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat > _______________________________________________ > Bloat mailing list > Bloat at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at lang.hm Mon Jun 5 12:20:28 2023 From: david at lang.hm (David Lang) Date: Mon, 5 Jun 2023 09:20:28 -0700 (PDT) Subject: [Bloat] Dual WAN home router with decent SQM? In-Reply-To: References: <58238rpr-80q1-3p5s-849q-604ns5659s7r@ynat.uz> Message-ID: you aren't going to bond two differnt ISPs for faster upload. you may setup tunnels over each of the ISPs and bond them together. Even with a full BGP link, any given connection is very unlikely to use both ISPs. David Lang On Mon, 5 Jun 2023, Luis A. Cornejo wrote: > I’ve done the multi wan with mwan3 and it works fine for load balance or > failover. > > But if you are trying to improve your upload then the best you can get is > one upload vs the other. You won’t get both uploads to the same stream, > that is bonding, which is a little harder to do. > > -Luis > > On Sun, Jun 4, 2023 at 9:35 AM David Lang via Bloat < > bloat at lists.bufferbloat.net> wrote: > >> any router that you can run OpenWRT on will do the job >> look at the mwan3 package to mange the multiple ISPs >> >> at 100Mb, you don't need an especially beefy router. >> >> David Lang >> >> On Sun, 4 Jun 2023, John D >> via Bloat wrote: >> >>> Date: Sun, 4 Jun 2023 09:48:21 +0100 >>> From: John D via Bloat >>> Reply-To: John D >>> To: bloat at lists.bufferbloat.net >>> Subject: [Bloat] Dual WAN home router with decent SQM? >>> >>> I want to set up a new home router with SQM support to alleviate (mostly >>> upload) bloat. >>> However I would also like dual ISP support, more for backup/failover than >>> load sharing. >>> I can see affordable multi wan "load balancer" routers eg TP-link, and >>> affordable SQM routers such as ubiquity Edgerouter, but I'm struggling to >>> find a single device providing both. >>> Just after any advice on a) am I better having two separate components b) >>> if so any suggestions which product/brand? >>> Two devices means more configuration but possibly more flexible and >> cheaper. >>> >>> My internet speed is looking to max out around 100/20mbps so I don't need >>> Gbit performance. Thanks for any help. >>> _______________________________________________ >> Bloat mailing list >> Bloat at lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/bloat >> _______________________________________________ >> Bloat mailing list >> Bloat at lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/bloat >> > From dave.taht at gmail.com Mon Jun 5 12:24:36 2023 From: dave.taht at gmail.com (Dave Taht) Date: Mon, 5 Jun 2023 10:24:36 -0600 Subject: [Bloat] Dual WAN home router with decent SQM? In-Reply-To: References: <58238rpr-80q1-3p5s-849q-604ns5659s7r@ynat.uz> Message-ID: we invented "source specific routing", for ipv6, (now called SADR) during the cerowrt project and it has been part of openwrt ever since. Two or more ipv6 upstreams will distribute their /subnet to the downstream. Regrettably, few protocols or end user applications attempt to leverage these disparate address spaces, although multipath tcp and quic have become more common. On Mon, Jun 5, 2023 at 10:20 AM David Lang via Bloat wrote: > > you aren't going to bond two differnt ISPs for faster upload. > > you may setup tunnels over each of the ISPs and bond them together. > > Even with a full BGP link, any given connection is very unlikely to use both > ISPs. > > David Lang > > On Mon, 5 Jun 2023, Luis A. Cornejo wrote: > > > I’ve done the multi wan with mwan3 and it works fine for load balance or > > failover. > > > > But if you are trying to improve your upload then the best you can get is > > one upload vs the other. You won’t get both uploads to the same stream, > > that is bonding, which is a little harder to do. > > > > -Luis > > > > On Sun, Jun 4, 2023 at 9:35 AM David Lang via Bloat < > > bloat at lists.bufferbloat.net> wrote: > > > >> any router that you can run OpenWRT on will do the job > >> look at the mwan3 package to mange the multiple ISPs > >> > >> at 100Mb, you don't need an especially beefy router. > >> > >> David Lang > >> > >> On Sun, 4 Jun 2023, John D > >> via Bloat wrote: > >> > >>> Date: Sun, 4 Jun 2023 09:48:21 +0100 > >>> From: John D via Bloat > >>> Reply-To: John D > >>> To: bloat at lists.bufferbloat.net > >>> Subject: [Bloat] Dual WAN home router with decent SQM? > >>> > >>> I want to set up a new home router with SQM support to alleviate (mostly > >>> upload) bloat. > >>> However I would also like dual ISP support, more for backup/failover than > >>> load sharing. > >>> I can see affordable multi wan "load balancer" routers eg TP-link, and > >>> affordable SQM routers such as ubiquity Edgerouter, but I'm struggling to > >>> find a single device providing both. > >>> Just after any advice on a) am I better having two separate components b) > >>> if so any suggestions which product/brand? > >>> Two devices means more configuration but possibly more flexible and > >> cheaper. > >>> > >>> My internet speed is looking to max out around 100/20mbps so I don't need > >>> Gbit performance. Thanks for any help. > >>> _______________________________________________ > >> Bloat mailing list > >> Bloat at lists.bufferbloat.net > >> https://lists.bufferbloat.net/listinfo/bloat > >> _______________________________________________ > >> Bloat mailing list > >> Bloat at lists.bufferbloat.net > >> https://lists.bufferbloat.net/listinfo/bloat > >> > >_______________________________________________ > Bloat mailing list > Bloat at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat -- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos From j.w.r.dexter at gmail.com Mon Jun 5 14:25:47 2023 From: j.w.r.dexter at gmail.com (John D) Date: Mon, 5 Jun 2023 19:25:47 +0100 Subject: [Bloat] Dual WAN home router with decent SQM? In-Reply-To: References: <58238rpr-80q1-3p5s-849q-604ns5659s7r@ynat.uz> Message-ID: My scenario is upgrading from DSL (35/8mbps) to StarLink (150/15mbps) so I really only need a failover - keep the DSL as it's cheap. My question though is how quickly will failover occur? I work over RDP & Teams all day and I'm unclear if failover should give me near seamless service, or kicks in after things already stopped working. (Ps: I'm unsure of etiquette on this list, should I reply all or reply back to the list only every time) On Mon, Jun 5, 2023, 17:06 Luis A. Cornejo wrote: > I’ve done the multi wan with mwan3 and it works fine for load balance or > failover. > > But if you are trying to improve your upload then the best you can get is > one upload vs the other. You won’t get both uploads to the same stream, > that is bonding, which is a little harder to do. > > -Luis > > On Sun, Jun 4, 2023 at 9:35 AM David Lang via Bloat < > bloat at lists.bufferbloat.net> wrote: > >> any router that you can run OpenWRT on will do the job >> look at the mwan3 package to mange the multiple ISPs >> >> at 100Mb, you don't need an especially beefy router. >> >> David Lang >> >> On Sun, 4 Jun 2023, John D >> via Bloat wrote: >> >> > Date: Sun, 4 Jun 2023 09:48:21 +0100 >> > From: John D via Bloat >> > Reply-To: John D >> > To: bloat at lists.bufferbloat.net >> > Subject: [Bloat] Dual WAN home router with decent SQM? >> > >> > I want to set up a new home router with SQM support to alleviate (mostly >> > upload) bloat. >> > However I would also like dual ISP support, more for backup/failover >> than >> > load sharing. >> > I can see affordable multi wan "load balancer" routers eg TP-link, and >> > affordable SQM routers such as ubiquity Edgerouter, but I'm struggling >> to >> > find a single device providing both. >> > Just after any advice on a) am I better having two separate components >> b) >> > if so any suggestions which product/brand? >> > Two devices means more configuration but possibly more flexible and >> cheaper. >> > >> > My internet speed is looking to max out around 100/20mbps so I don't >> need >> > Gbit performance. Thanks for any help. >> >_______________________________________________ >> Bloat mailing list >> Bloat at lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/bloat >> _______________________________________________ >> Bloat mailing list >> Bloat at lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/bloat >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at lang.hm Mon Jun 5 14:53:10 2023 From: david at lang.hm (David Lang) Date: Mon, 5 Jun 2023 11:53:10 -0700 (PDT) Subject: [Bloat] Dual WAN home router with decent SQM? In-Reply-To: References: <58238rpr-80q1-3p5s-849q-604ns5659s7r@ynat.uz> Message-ID: On Mon, 5 Jun 2023, John D wrote: > My scenario is upgrading from DSL (35/8mbps) to StarLink (150/15mbps) so I > really only need a failover - keep the DSL as it's cheap. > > My question though is how quickly will failover occur? I work over RDP & > Teams all day and I'm unclear if failover should give me near seamless > service, or kicks in after things already stopped working. it's not seamless, for two reasons. 1. it takes time to detect that the connectivity is down if it's the local wire breaking, your router can detect the loss of the link, but if it's a problem further up in the ISP, you can only detect it by sending a ping and having it not respond. It's common to send such pings every minute or so, which limits how quickly you can detect a failure. It's also common to wait for a couple failures so that you don't fail over due to a single dropped packet. (as noted below, failovers are not going to be transparent, so a short outage is preferrable to a false failover) 2. zoom/teams/RDP/etc will need to see their connection fail and reconnect. Since you have two ISPs and don't have BGP across them, you have a different IP address on each ISP, so when you shift from one to the other, existing connections have to fail and when they are re-established, they show up to the other end as the new IP. It's not something that you want to have happen during a meeting, and it won't cover for the few-second outages that Starlink sometimes has, but it does cover longer outages. now, you can set things up so that you have a VPN over each of the connections to a VM you run on a hosted service somewhere and failover from one VPN leg to the other and the remote end will not realize this due to the NAT being done on your remote VM. But that's a lot more work to setup and has it's own set of 'interesting' problems. David Lang P.S. reply-to-all is common. > (Ps: I'm unsure of etiquette on this list, should I reply all or reply back > to the list only every time) > > On Mon, Jun 5, 2023, 17:06 Luis A. Cornejo wrote: > >> I’ve done the multi wan with mwan3 and it works fine for load balance or >> failover. >> >> But if you are trying to improve your upload then the best you can get is >> one upload vs the other. You won’t get both uploads to the same stream, >> that is bonding, which is a little harder to do. >> >> -Luis >> >> On Sun, Jun 4, 2023 at 9:35 AM David Lang via Bloat < >> bloat at lists.bufferbloat.net> wrote: >> >>> any router that you can run OpenWRT on will do the job >>> look at the mwan3 package to mange the multiple ISPs >>> >>> at 100Mb, you don't need an especially beefy router. >>> >>> David Lang >>> >>> On Sun, 4 Jun 2023, John D >>> via Bloat wrote: >>> >>>> Date: Sun, 4 Jun 2023 09:48:21 +0100 >>>> From: John D via Bloat >>>> Reply-To: John D >>>> To: bloat at lists.bufferbloat.net >>>> Subject: [Bloat] Dual WAN home router with decent SQM? >>>> >>>> I want to set up a new home router with SQM support to alleviate (mostly >>>> upload) bloat. >>>> However I would also like dual ISP support, more for backup/failover >>> than >>>> load sharing. >>>> I can see affordable multi wan "load balancer" routers eg TP-link, and >>>> affordable SQM routers such as ubiquity Edgerouter, but I'm struggling >>> to >>>> find a single device providing both. >>>> Just after any advice on a) am I better having two separate components >>> b) >>>> if so any suggestions which product/brand? >>>> Two devices means more configuration but possibly more flexible and >>> cheaper. >>>> >>>> My internet speed is looking to max out around 100/20mbps so I don't >>> need >>>> Gbit performance. Thanks for any help. >>>> _______________________________________________ >>> Bloat mailing list >>> Bloat at lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/bloat >>> _______________________________________________ >>> Bloat mailing list >>> Bloat at lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/bloat >>> >> > From dave.taht at gmail.com Tue Jun 6 08:21:05 2023 From: dave.taht at gmail.com (Dave Taht) Date: Tue, 6 Jun 2023 06:21:05 -0600 Subject: [Bloat] data growth report in the eu Message-ID: https://www.linkedin.com/feed/update/urn:li:activity:7071576961424203777/ -- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos From dave.taht at gmail.com Sun Jun 11 11:15:34 2023 From: dave.taht at gmail.com (Dave Taht) Date: Sun, 11 Jun 2023 09:15:34 -0600 Subject: [Bloat] explaining orbits, gravity and thrust etc, in spanish Message-ID: I often get a bit down about the rise of tiktok and the relative decline in the use of the internet to educate and inform. This morning, my landlord (in nicaragua)'s 16 year old son, was watching this: https://www.youtube.com/watch?v=KFNuC2I93T4 The visuals were often cheesy, but pretty accurate... and I learned more technical terms in Spanish in one session than I ever knew before. And we both dug it! I think I'll try to introduce him to the idea of an O'Neil cylinder... -- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos From Jason_Livingood at comcast.com Fri Jun 16 12:45:31 2023 From: Jason_Livingood at comcast.com (Livingood, Jason) Date: Fri, 16 Jun 2023 16:45:31 +0000 Subject: [Bloat] Inform: Volunteer for Comcast Field Trials Message-ID: FYI that today we (Comcast) have announced the start of low latency networking (L4S) field trials. If you are a customer and would like to volunteer, please visit this page. For more info, there is a blog post that just went up at https://corporate.comcast.com/stories/comcast-kicks-off-industrys-first-low-latency-docsis-field-trials We anticipate testing with several different cable modems and a range of applications that are marking. We plan to share detailed results of the trial at IETF-118 in November. Any app developers interested in working with us can either email me direction or Low-Latency-Partner-Interest at Comcast.com. Thanks! Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.taht at gmail.com Fri Jun 16 20:50:54 2023 From: dave.taht at gmail.com (Dave Taht) Date: Fri, 16 Jun 2023 18:50:54 -0600 Subject: [Bloat] first l4s trial signups Message-ID: https://www.linkedin.com/feed/update/urn:li:activity:7075510291198795776/ -- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos From dpreed at deepplum.com Sat Jun 24 14:41:52 2023 From: dpreed at deepplum.com (David P. Reed) Date: Sat, 24 Jun 2023 14:41:52 -0400 (EDT) Subject: [Bloat] Two questions re high speed congestion management and datagram protocols Message-ID: <1687632112.01524513@apps.rackspace.com> I was recently looking at congestion control algorithms - endpoint-based ones - that would deal with very low level, very low latency requirements in datacenters that use high speed switch fabrics. (note: congestion control in such datacenters is a very, very real issue, especially since some layer 2 switches are overbuffered by a huge factor, so they don't signal congestion while building very long queues at 40-400 Gb/sec) I came across this work from MIT CSAIL [ https://ccp-project.github.io/ccp-guide/ ]( https://ccp-project.github.io/ccp-guide/ ) I also was looking back to DCCP as a useful way to get a UDP that handled congestion without engaging the higher layers, and preserving the other flexibility of UDP. Anyone here have any experience with looking at the performance of these, especially w.r.t. Cake, which operates at the IP layer and thus takes direct advantage of IP congestion signalling? Does libreqos look like it might help? [as the guy most associated with the creation of UDP, this remains an area of great interest personally, and is also presumably of relevance to QUIC... I don't know if either of these things are supported in systems other than Linux, but in server datacenters and my home lab, Linux networking is all that matters] - David -------------- next part -------------- An HTML attachment was scrubbed... URL: From moeller0 at gmx.de Sat Jun 24 17:00:46 2023 From: moeller0 at gmx.de (Sebastian Moeller) Date: Sat, 24 Jun 2023 23:00:46 +0200 Subject: [Bloat] Anybody has contacts at Dropbox? Message-ID: <24D1A566-DFE9-4FDF-86A2-CFB30F20FCB2@gmx.de> Dear All, I started to look at ECN action on my homelink and I do see the expected ECT(0) and CE marks (mostly under Linix, macos, inspite of what sysctl seems to imply does not use ECN for me any more). BUt I also see the following ECT(1) marked traffic roughly every 5 seconds: root at turris:/srv/persistent/captures# tcpdump -i pppoe-wan -v -n '(ip6 and (ip6[0:2] & 0x30) >> 4 == 1)' or '(ip and (ip[1] & 0x3) == 1)' # ECT(1) tcpdump: listening on pppoe-wan, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes 22:53:07.470573 IP (tos 0x1,ECT(1), ttl 47, id 42039, offset 0, flags [DF], proto TCP (6), length 296) 162.125.21.3.443 > 77.3.0.223.51272: Flags [P.], cksum 0x0a63 (correct), seq 2630111900:2630112144, ack 2551484650, win 130, options [nop,nop,TS val 808373969 ecr 2543738360], length 244 22:53:08.371720 IP (tos 0x21,ECT(1), ttl 47, id 64288, offset 0, flags [DF], proto TCP (6), length 170) 162.125.21.3.443 > 77.3.0.223.51276: Flags [P.], cksum 0x06c6 (correct), seq 141475331:141475449, ack 2804301808, win 130, options [nop,nop,TS val 3764736866 ecr 3088513970], length 118 22:53:22.260844 IP (tos 0x1,ECT(1), ttl 47, id 42042, offset 0, flags [DF], proto TCP (6), length 173) 162.125.21.3.443 > 77.3.0.223.51272: Flags [P.], cksum 0xfa7c (correct), seq 244:365, ack 415, win 130, options [nop,nop,TS val 808388758 ecr 2543769161], length 121 22:53:28.499273 IP (tos 0x1,ECT(1), ttl 48, id 12519, offset 0, flags [DF], proto TCP (6), length 170) 162.125.21.3.443 > 77.3.0.223.51275: Flags [P.], cksum 0xac16 (correct), seq 2923091763:2923091881, ack 2176795132, win 130, options [nop,nop,TS val 4027691853 ecr 2152611355], length 118 22:53:34.021366 IP (tos 0x1,ECT(1), ttl 48, id 24318, offset 0, flags [DF], proto TCP (6), length 943) 162.125.21.2.443 > 77.3.0.223.51273: Flags [P.], cksum 0x3498 (correct), seq 4110227536:4110228427, ack 4169408519, win 130, options [nop,nop,TS val 561393037 ecr 811888120], length 891 22:53:38.862554 IP (tos 0x1,ECT(1), ttl 48, id 46323, offset 0, flags [DF], proto TCP (6), length 284) 162.125.21.2.443 > 77.3.0.223.51274: Flags [P.], cksum 0x7bc6 (correct), seq 3336805019:3336805251, ack 1852591643, win 130, options [nop,nop,TS val 723450354 ecr 2627633601], length 232 Whois tells me 162.125.21.[2|3] belong to dropbox (I have the dropbox client installed on that machine) So what is going on here? Is dropbox using ECT(1) for a rfc3168 style response? Is dropbox silently already using an L4S-style CC for their TCP? Is dropbox aiming for DSCP decimal 1 (aka LE) and actually setting TOS decimal 1 instead? Was the assumption that ECT(1) is bot used simply wrong? Anyway, if anybody from Dropbox is subscribed, or anybody here has contacts into dropbox, this seems like something worth looking into to get it fixed? Regards Sebastian From chromatix99 at gmail.com Sat Jun 24 19:04:00 2023 From: chromatix99 at gmail.com (Jonathan Morton) Date: Sun, 25 Jun 2023 02:04:00 +0300 Subject: [Bloat] [Cake] Anybody has contacts at Dropbox? In-Reply-To: <24D1A566-DFE9-4FDF-86A2-CFB30F20FCB2@gmx.de> References: <24D1A566-DFE9-4FDF-86A2-CFB30F20FCB2@gmx.de> Message-ID: <46B3DF93-48F1-429C-B9E0-41683CBBE1E8@gmail.com> > On 25 Jun, 2023, at 12:00 am, Sebastian Moeller via Cake wrote: > > Is dropbox silently already using an L4S-style CC for their TCP? It should be possible to distinguish this by looking at the three-way handshake at the start of the connection. This will show a different set of TCP flags and ECN field values depending on whether RFC-3168 or AccECN is being attempted. Without AccECN, you won't have functioning L4S on a TCP stream. But I think it is more likely that it's a misapplied DSCP. - Jonathan Morton From moeller0 at gmx.de Sun Jun 25 04:43:07 2023 From: moeller0 at gmx.de (Sebastian Moeller) Date: Sun, 25 Jun 2023 10:43:07 +0200 Subject: [Bloat] [Cake] Anybody has contacts at Dropbox? In-Reply-To: <46B3DF93-48F1-429C-B9E0-41683CBBE1E8@gmail.com> References: <24D1A566-DFE9-4FDF-86A2-CFB30F20FCB2@gmx.de> <46B3DF93-48F1-429C-B9E0-41683CBBE1E8@gmail.com> Message-ID: <27C08679-D769-4FA6-B088-F8FD138AAA73@gmx.de> Hi Jonathan, > On Jun 25, 2023, at 01:04, Jonathan Morton wrote: > >> On 25 Jun, 2023, at 12:00 am, Sebastian Moeller via Cake wrote: >> >> Is dropbox silently already using an L4S-style CC for their TCP? > > It should be possible to distinguish this by looking at the three-way handshake at the start of the connection. [SM] Great idea, just need to google how to restart the dropbox client to force a new handshake. > This will show a different set of TCP flags and ECN field values depending on whether RFC-3168 or AccECN is being attempted. Without AccECN, you won't have functioning L4S on a TCP stream. > > But I think it is more likely that it's a misapplied DSCP. [SM] Probably... Thanks & Best Regards Sebastian > > - Jonathan Morton From stephen at networkplumber.org Sun Jun 25 14:51:13 2023 From: stephen at networkplumber.org (Stephen Hemminger) Date: Sun, 25 Jun 2023 11:51:13 -0700 Subject: [Bloat] Two questions re high speed congestion management and datagram protocols In-Reply-To: <1687632112.01524513@apps.rackspace.com> References: <1687632112.01524513@apps.rackspace.com> Message-ID: <20230625115113.2291f7d6@hermes.local> On Sat, 24 Jun 2023 14:41:52 -0400 (EDT) "David P. Reed via Bloat" wrote: > I also was looking back to DCCP as a useful way to get a UDP that handled congestion without engaging the higher layers, and preserving the other flexibility of UDP. DCCP never got widely used, and Linux is on the path of deprecating it. From dpreed at deepplum.com Mon Jun 26 23:41:59 2023 From: dpreed at deepplum.com (David P. Reed) Date: Mon, 26 Jun 2023 23:41:59 -0400 (EDT) Subject: [Bloat] Two questions re high speed congestion management anddatagram protocols Message-ID: <1687837319.447910185@mobile.rackspace.com> Sorry for top posting, but ... Bigger question: Why would DCCP be deprecated by Linux kernel? Who makes that decision? Who argues against it? It's a pretty good approach to properly congestion controlling many non-TCP communications protocols that might be implemented on UDP and lack good congestion control otherwise. E.g. QUIC or RTP (for non CBR traffic) or various RPC-style protocols. Larry Peterson recently wrote a piece asking why RPC wasn't well supported in distributed computing even after almost 50 years. Lack of Conception Control that works is a big issue. QUIC ain't it. QUiC is a HTTP replacement for REST protocol sementics. So why discard a good thing that works? -----Original Message----- From: "Stephen Hemminger" Sent: Sun, Jun 25, 2023 at 2:51 pm To: "David P. Reed via Bloat" Cc: "David P. Reed via Bloat" , "Cake List" Subject: Re: [Bloat] Two questions re high speed congestion management anddatagram protocols On Sat, 24 Jun 2023 14:41:52 -0400 (EDT) "David P. Reed via Bloat" wrote: > I also was looking back to DCCP as a useful way to get a UDP that handled congestion without engaging the higher layers, and preserving the other flexibility of UDP. DCCP never got widely used, and Linux is on the path of deprecating it. From stephen at networkplumber.org Tue Jun 27 10:56:28 2023 From: stephen at networkplumber.org (Stephen Hemminger) Date: Tue, 27 Jun 2023 07:56:28 -0700 Subject: [Bloat] Two questions re high speed congestion management anddatagram protocols In-Reply-To: <1687837319.447910185@mobile.rackspace.com> References: <1687837319.447910185@mobile.rackspace.com> Message-ID: <20230627075628.1d8d3b7d@hermes.local> On Mon, 26 Jun 2023 23:41:59 -0400 (EDT) "David P. Reed" wrote: > Sorry for top posting, but ... Bigger question: > Why would DCCP be deprecated by Linux kernel? > Who makes that decision? Who argues against it? No one uses it, and unused protocols are targeted by hackers. And there are few tests and no maintainer. From david at lang.hm Tue Jun 27 15:47:01 2023 From: david at lang.hm (David Lang) Date: Tue, 27 Jun 2023 12:47:01 -0700 (PDT) Subject: [Bloat] Two questions re high speed congestion management anddatagram protocols In-Reply-To: <1687837319.447910185@mobile.rackspace.com> References: <1687837319.447910185@mobile.rackspace.com> Message-ID: <4np8q9q1-on99-rn94-5s1p-55643oo11037@ynat.uz> On Mon, 26 Jun 2023, David P. Reed via Bloat wrote: > Sorry for top posting, but ... Bigger question: > Why would DCCP be deprecated by Linux kernel? > Who makes that decision? Who argues against it? Linus or the networking maintaners make the decision. Usually things get pulled from the kernel because there are updates that need to be made to the code (to match changes elsewhere in the kernel or because of security issues) and there isn't a maintainer who works on the code in a resonable time. This means that the maintainers for the general code area (in this case networking maintainers) will need to do extra work in an area they aren't that interested in (and, especially in the case of hardware, may not have the ability to test). They do some of it, especially if it's commonly used, but eventually either another maintainer steps up, or it goes away David Lang From stephen at networkplumber.org Tue Jun 27 18:49:31 2023 From: stephen at networkplumber.org (Stephen Hemminger) Date: Tue, 27 Jun 2023 15:49:31 -0700 Subject: [Bloat] Two questions re high speed congestion management anddatagram protocols In-Reply-To: <4np8q9q1-on99-rn94-5s1p-55643oo11037@ynat.uz> References: <1687837319.447910185@mobile.rackspace.com> <4np8q9q1-on99-rn94-5s1p-55643oo11037@ynat.uz> Message-ID: <20230627154931.71aff7ca@hermes.local> On Tue, 27 Jun 2023 12:47:01 -0700 (PDT) David Lang wrote: > On Mon, 26 Jun 2023, David P. Reed via Bloat wrote: > > > Sorry for top posting, but ... Bigger question: > > Why would DCCP be deprecated by Linux kernel? > > Who makes that decision? Who argues against it? > > Linus or the networking maintaners make the decision. > > Usually things get pulled from the kernel because there are updates that need to > be made to the code (to match changes elsewhere in the kernel or because of > security issues) and there isn't a maintainer who works on the code in a > resonable time. This means that the maintainers for the general code area (in > this case networking maintainers) will need to do extra work in an area they > aren't that interested in (and, especially in the case of hardware, may not have > the ability to test). They do some of it, especially if it's commonly used, but > eventually either another maintainer steps up, or it goes away > > David Lang See https://patchwork.kernel.org/project/netdevbpf/patch/20230614194705.90673-3-kuniyu at amazon.com/ From dave.taht at gmail.com Tue Jun 27 22:03:39 2023 From: dave.taht at gmail.com (Dave Taht) Date: Tue, 27 Jun 2023 20:03:39 -0600 Subject: [Bloat] [Cake] Two questions re high speed congestion management anddatagram protocols In-Reply-To: <20230627154931.71aff7ca@hermes.local> References: <1687837319.447910185@mobile.rackspace.com> <4np8q9q1-on99-rn94-5s1p-55643oo11037@ynat.uz> <20230627154931.71aff7ca@hermes.local> Message-ID: https://datatracker.ietf.org/doc/charter-ietf-ccwg/ is a new wg intended to poke into these issues On Tue, Jun 27, 2023 at 4:49 PM Stephen Hemminger via Cake wrote: > > On Tue, 27 Jun 2023 12:47:01 -0700 (PDT) > David Lang wrote: > > > On Mon, 26 Jun 2023, David P. Reed via Bloat wrote: > > > > > Sorry for top posting, but ... Bigger question: > > > Why would DCCP be deprecated by Linux kernel? > > > Who makes that decision? Who argues against it? > > > > Linus or the networking maintaners make the decision. > > > > Usually things get pulled from the kernel because there are updates that need to > > be made to the code (to match changes elsewhere in the kernel or because of > > security issues) and there isn't a maintainer who works on the code in a > > resonable time. This means that the maintainers for the general code area (in > > this case networking maintainers) will need to do extra work in an area they > > aren't that interested in (and, especially in the case of hardware, may not have > > the ability to test). They do some of it, especially if it's commonly used, but > > eventually either another maintainer steps up, or it goes away > > > > David Lang > > See https://patchwork.kernel.org/project/netdevbpf/patch/20230614194705.90673-3-kuniyu at amazon.com/ > _______________________________________________ > Cake mailing list > Cake at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake -- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos From auerswal at unix-ag.uni-kl.de Wed Jun 28 05:43:13 2023 From: auerswal at unix-ag.uni-kl.de (Erik Auerswald) Date: Wed, 28 Jun 2023 11:43:13 +0200 Subject: [Bloat] Two questions re high speed congestion management anddatagram protocols In-Reply-To: <20230627075628.1d8d3b7d@hermes.local> References: <1687837319.447910185@mobile.rackspace.com> <20230627075628.1d8d3b7d@hermes.local> Message-ID: <20230628094313.GA18471@unix-ag.uni-kl.de> Hi, On Tue, Jun 27, 2023 at 07:56:28AM -0700, Stephen Hemminger via Bloat wrote: > On Mon, 26 Jun 2023 23:41:59 -0400 (EDT) > "David P. Reed" wrote: > > > Sorry for top posting, but ... Bigger question: > > Why would DCCP be deprecated by Linux kernel? > > Who makes that decision? Who argues against it? > > No one uses it, and unused protocols are targeted by hackers. > And there are few tests and no maintainer. Perhaps the telephony (3GPP) people use it, there even seems to be some work on extending DCCP with multi-path capabilities: . [That's all I know about this.] Thanks, Erik -- A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable. -- Leslie Lamport From dpreed at deepplum.com Wed Jun 28 10:32:32 2023 From: dpreed at deepplum.com (David P. Reed) Date: Wed, 28 Jun 2023 10:32:32 -0400 (EDT) Subject: [Bloat] [Cake] Two questions re high speed congestionmanagement anddatagram protocols Message-ID: <1687962752.39077378@mobile.rackspace.com> How to find a kernel maintainer to care about DCCP, seems to be the question for Linux. I am tempted... Not to get involved with IETF "barriers" (what a mess, given the folks in IETF who resisted in AQM, I wouldn't last a minute), but to keep DCCP support alive. The barrier here is getting accepted as a Linux maintainer, which is a different issue entirely, looking at my last two experiences with submitting simple bug fixes to the kernel, which were nightmares. I don't have the commitment to become accepted as a maintainer. But it seems good to maintain DCCP, despite its lack of popularity as an IETF standard. It does deal with CC in a way that simplifies use of UDP for serious work. (One such nightmare can be seen in LKML... Search for dpreed at deepplum.com patch emails. I tried hard, was worn down, then gave up, since I found a way to avoid the bug, in virtualization code on x86, and gave up on getting it fixed after a year. Life is too short. The prior one was almost 20 years ago when Alan Cox showed his asshole side, insulting me and HP. Alan is, thank goodness, gone now from Linus's inside circle) -----Original Message----- From: "Dave Taht via Cake" Sent: Tue, Jun 27, 2023 at 10:03 pm To: "Stephen Hemminger" Cc: "Stephen Hemminger" , cake at lists.bufferbloat.net, bloat at lists.bufferbloat.net Subject: Re: [Cake] [Bloat] Two questions re high speed congestionmanagement anddatagram protocols https://datatracker.ietf.org/doc/charter-ietf-ccwg/ is a new wg intended to poke into these issues On Tue, Jun 27, 2023 at 4:49 PM Stephen Hemminger via Cake wrote: > > On Tue, 27 Jun 2023 12:47:01 -0700 (PDT) > David Lang wrote: > > > On Mon, 26 Jun 2023, David P. Reed via Bloat wrote: > > > > > Sorry for top posting, but ... Bigger question: > > > Why would DCCP be deprecated by Linux kernel? > > > Who makes that decision? Who argues against it? > > > > Linus or the networking maintaners make the decision. > > > > Usually things get pulled from the kernel because there are updates that need to > > be made to the code (to match changes elsewhere in the kernel or because of > > security issues) and there isn't a maintainer who works on the code in a > > resonable time. This means that the maintainers for the general code area (in > > this case networking maintainers) will need to do extra work in an area they > > aren't that interested in (and, especially in the case of hardware, may not have > > the ability to test). They do some of it, especially if it's commonly used, but > > eventually either another maintainer steps up, or it goes away > > > > David Lang > > See https://patchwork.kernel.org/project/netdevbpf/patch/20230614194705.90673-3-kuniyu at amazon.com/ > _______________________________________________ > Cake mailing list > Cake at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake -- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos _______________________________________________ Cake mailing list Cake at lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake From toke at toke.dk Wed Jun 28 17:53:51 2023 From: toke at toke.dk (Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?=) Date: Wed, 28 Jun 2023 23:53:51 +0200 Subject: [Bloat] [Cake] Two questions re high speed congestionmanagement anddatagram protocols In-Reply-To: <1687962752.39077378@mobile.rackspace.com> References: <1687962752.39077378@mobile.rackspace.com> Message-ID: <877crngsnk.fsf@toke.dk> "David P. Reed via Bloat" writes: > (One such nightmare can be seen in LKML... Search for > dpreed at deepplum.com patch emails. I tried hard, was worn down, then > gave up, since I found a way to avoid the bug, in virtualization code > on x86, and gave up on getting it fixed after a year. Life is too > short. Went looking, since I think it's important to learn from such process failures (and you're certainly not the first to opine that getting patches into the kernel is challenging). I'm assuming you're referring to this series, right? https://lore.kernel.org/all/20200704203809.76391-4-dpreed at deepplum.com/ Which, to me, looks like it was pretty close to being accepted; another revision would probably have made the cut... ...In fact it seems those patches were later resurrected by Sean as part of this series, six months later: https://lore.kernel.org/all/20201231002702.2223707-1-seanjc at google.com/ One of the patches retained your authorship and made it into the kernel in this commit: https://git.kernel.org/torvalds/c/53666664a305 So wouldn't necessarily call that a complete failure :) Seems the main process issue you hit here was a subsystem that was too resource constrained on the review side, which does sadly happen. The kernel process tends to heavily favour convenience of reviewers for the same reason (which is one of the reasons it can be off-putting to outsiders, so it's a bit of a chicken and egg situation...) -Toke From Jason_Livingood at comcast.com Thu Jun 29 10:12:13 2023 From: Jason_Livingood at comcast.com (Livingood, Jason) Date: Thu, 29 Jun 2023 14:12:13 +0000 Subject: [Bloat] Testing Offer - Comcast L4S Trials Message-ID: As noted recently, we (Comcast) will soon begin L4S testing with customers. We’re working on our test plans now. So my offer here is to let me know if: 1 – You have test probes you’d like us to deploy. We are working on a plan to send out (1) University of Chicago NetMicroscope probes and (2) RIPE Atlas probes. 2 – You have web-based tests you’d like people to run, or have one-click install client for Mac or Windows. (Most testers will not be technical enough to run CLI-based tests, the the exception of the Apple network responsiveness test.) Thx! If this is of interest, please ping me 1:1 off-list. Jason Current list (1.a) Ookla web-based (1.b) Ookla client (1.c) M-Labs NDT (1.d) Fastly web-based (1.e) Netflix web-based (1.f) Waveform web-based (1.g) DSL Reports web-based (1.h) Apple Network Responsiveness test -------------- next part -------------- An HTML attachment was scrubbed... URL: From moeller0 at gmx.de Thu Jun 29 10:32:25 2023 From: moeller0 at gmx.de (Sebastian Moeller) Date: Thu, 29 Jun 2023 16:32:25 +0200 Subject: [Bloat] Testing Offer - Comcast L4S Trials In-Reply-To: References: Message-ID: While I have no test that I can volunteer, let alone a test that supports L4S. I want to take the opportunity to share my tcpdump invocations that can show ECN in action on the IP-level: # ECN: IPv4 tcpdump -i pppoe-wan -v -n 'ip and (ip[1] & 0x3) != 0' # NOT Not-ECT tcpdump -i pppoe-wan -v -n 'ip and (ip[1] & 0x3) == 0' # Not-ECT tcpdump -i pppoe-wan -v -n 'ip and (ip[1] & 0x3) == 1' # ECT(1) tcpdump -i pppoe-wan -v -n 'ip and (ip[1] & 0x3) == 2' # ECT(0) tcpdump -i pppoe-wan -v -n 'ip and (ip[1] & 0x3) == 3' # CE # ECN: IPv6 tcpdump -i pppoe-wan -v -n 'ip6 and (ip6[0:2] & 0x30) >> 4 != 0' # NOT Not-ECT tcpdump -i pppoe-wan -v -n 'ip6 and (ip6[0:2] & 0x30) >> 4 == 0' # Not-ECT tcpdump -i pppoe-wan -v -n 'ip6 and (ip6[0:2] & 0x30) >> 4 == 1' # ECT(1) tcpdump -i pppoe-wan -v -n 'ip6 and (ip6[0:2] & 0x30) >> 4 == 2' # ECT(0) tcpdump -i pppoe-wan -v -n 'ip6 and (ip6[0:2] & 0x30) >> 4 == 3' # CE # ECN IPv4/6 tcpdump -i pppoe-wan -v -n '(ip6 and (ip6[0:2] & 0x30) >> 4 != 0)' or '(ip and (ip[1] & 0x3) != 0)' # NOT Not-ECT tcpdump -i pppoe-wan -v -n '(ip6 and (ip6[0:2] & 0x30) >> 4 == 1)' or '(ip and (ip[1] & 0x3) == 1)' # ECT(1) tcpdump -i pppoe-wan -v -n '(ip6 and (ip6[0:2] & 0x30) >> 4 == 2)' or '(ip and (ip[1] & 0x3) == 2)' # ECT(0) tcpdump -i pppoe-wan -v -n '(ip6 and (ip6[0:2] & 0x30) >> 4 == 3)' or '(ip and (ip[1] & 0x3) == 3)' # CE Just replace "-i pppoe-wan" with your own network interface of interest, this seems to work on both recent inux and older macos (12.6.7). Before L4S roll-out the test for ECT(1) can be helpful to confirm that there is no background ECT(1) usage that might (or might not) cause issues with L4S. > On Jun 29, 2023, at 16:12, Livingood, Jason via Bloat wrote: > > As noted recently, we (Comcast) will soon begin L4S testing with customers. We’re working on our test plans now. So my offer here is to let me know if: > 1 – You have test probes you’d like us to deploy. We are working on a plan to send out (1) University of Chicago NetMicroscope probes and (2) RIPE Atlas probes. > 2 – You have web-based tests you’d like people to run, or have one-click install client for Mac or Windows. (Most testers will not be technical enough to run CLI-based tests, the the exception of the Apple network responsiveness test.) > > Thx! If this is of interest, please ping me 1:1 off-list. > Jason > > Current list > (1.a) Ookla web-based > (1.b) Ookla client > (1.c) M-Labs NDT > (1.d) Fastly web-based > (1.e) Netflix web-based > (1.f) Waveform web-based > (1.g) DSL Reports web-based > (1.h) Apple Network Responsiveness test [SM] Not mine to offer, but I think speed.cloudflare.com also might be interesting as it seems to follow the modern "let's not ignore latency under load" paradigm as well as the translate the results into terms suitable for casual users, while also reporting the hard numbers for those interested. What seems not available right now is a public test that can selectively exercise both L4S queues independently. Regards Sebastian > > _______________________________________________ > Bloat mailing list > Bloat at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat From dave.taht at gmail.com Thu Jun 29 11:22:14 2023 From: dave.taht at gmail.com (Dave Taht) Date: Thu, 29 Jun 2023 09:22:14 -0600 Subject: [Bloat] If only we could cross the chasm Message-ID: on all the intertubes... https://preseem.com/2023/06/how-to-reduce-latency-for-isps-and-subscribers/ While I do embrace the need for a middlebox on the down, the home wifi really needs fq_codel running directly on it too, and cake on the up... I really wish somehow, someday, I could sit across a six pack of beer from a spacex engineer with a cross compiler for the dishy... I am really tickled that the majority of devices I have encountered on my trip to nicaragua are either running fq_codel or are sanely buffered in the first place. -- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos From DGoncz at replikon.net Thu Jun 22 06:57:56 2023 From: DGoncz at replikon.net (Douglas Goncz A.A.S. M.E.T. 1990) Date: Thu, 22 Jun 2023 06:57:56 -0400 Subject: [Bloat] MiM attacks in the L4S low latency networking environment Message-ID: Theorem: If N bots cooperate in A man in the middle attack or a denial of service attack in an L4S environment, with 1 point of entry into each susceptible buffer, WITH A generalized fixed estimated probability of success 0 < P.S.1 < 1 = (1 - P.F.1) THEN including interactions between the bots the Generalized probability of success is (about): P.S.N ~= 1 - ( (1-P.S.1) ^ (N^2 - N) ). Reframed for sci-fi physics research: If 1 atom fissions in a specific amount of time with probability P.S.1 above, Do N tightly coupled atoms fission with probability P.S.N in the same amount of time? Douglas REPLIKON Research 783774974 BCC: Lynch PhD, Teri, Jason, and MIT81 -------------- next part -------------- An HTML attachment was scrubbed... URL: From DGoncz at replikon.net Fri Jun 23 02:57:13 2023 From: DGoncz at replikon.net (Douglas Goncz A.A.S. M.E.T. 1990) Date: Fri, 23 Jun 2023 02:57:13 -0400 Subject: [Bloat] MiM attacks in the L4S low latency networking environment In-Reply-To: References: Message-ID: Moderators please note the most recent digest contained some information about l4s testing. I was stimulated by progress in the l4s area to consider whether reduction in latency would affect susceptibility of buffers to MIM and other attacks I received a note from automatic robot somewhere saying that I posted to a members only list using an unsubscribed address I repeat the address used for this communication dgoncz at replikon.net Is it possible that a capitalization error has caused the robot to think this is an unsubscribed address? For example using various email clients the mail might be sent from DGoncz at Replikon.net ...., Douglas On Thu, Jun 22, 2023, 6:57 AM Douglas Goncz A.A.S. M.E.T. 1990 < DGoncz at replikon.net> wrote: > Theorem: > > If N bots cooperate in > > A man in the middle attack or a denial of service attack in an L4S > environment, with 1 point of entry into each susceptible buffer, > > WITH > > A generalized fixed estimated probability of success 0 < P.S.1 < 1 = (1 - > P.F.1) > > THEN including interactions between the bots the > > Generalized probability of success is (about): > > P.S.N ~= 1 - ( (1-P.S.1) ^ (N^2 - N) ). > > Reframed for sci-fi physics research: > > If 1 atom fissions in a specific amount of time with probability P.S.1 > above, > > Do N tightly coupled atoms fission with probability P.S.N in the same > amount of time? > > Douglas > REPLIKON Research 783774974 > > BCC: > Lynch PhD, > Teri, > Jason, and > MIT81 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at iconia.com Thu Jun 29 17:46:51 2023 From: geoff at iconia.com (the keyboard of geoff goodfellow) Date: Thu, 29 Jun 2023 14:46:51 -0700 Subject: [Bloat] [Starlink] If only we could cross the chasm In-Reply-To: References: Message-ID: Paging Nathan Owens! :D *"After nearly 5 great years, today was my last day at Netflix. Excited to get started at SpaceX (Starlink) in 2 weeks!* https://twitter.com/VirtuallyNathan/status/1662256501220671494 On Thu, Jun 29, 2023 at 8:22 AM Dave Taht via Starlink < starlink at lists.bufferbloat.net> wrote: > on all the intertubes... > > > https://preseem.com/2023/06/how-to-reduce-latency-for-isps-and-subscribers/ > > While I do embrace the need for a middlebox on the down, the home wifi > really needs fq_codel running directly on it too, and cake on the > up... I really wish somehow, someday, I could sit across a six pack of > beer from a spacex engineer with a cross compiler for the dishy... > > I am really tickled that the majority of devices I have encountered on > my trip to nicaragua are either running fq_codel or are sanely > buffered in the first place. > > -- > Podcast: > https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ > Dave Täht CSO, LibreQos > _______________________________________________ > Starlink mailing list > Starlink at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/starlink > > -- Geoff.Goodfellow at iconia.com living as The Truth is True -------------- next part -------------- An HTML attachment was scrubbed... URL: From shiva at sewingwitch.com Thu Jun 29 19:05:36 2023 From: shiva at sewingwitch.com (Kenneth Porter) Date: Thu, 29 Jun 2023 16:05:36 -0700 Subject: [Bloat] Does SQM always require you to tell it your ISP up/down speed? In-Reply-To: References: Message-ID: Didn't you just ask this question? There were several replies including links to auto-tuning scripts. From dave.taht at gmail.com Fri Jun 30 11:46:34 2023 From: dave.taht at gmail.com (Dave Taht) Date: Fri, 30 Jun 2023 09:46:34 -0600 Subject: [Bloat] Instant Messaging Meets Video Conferencing: Studying the Performance of IM Video Calls Message-ID: This paper does a really good job of measuring the impacts of tcp cross traffic, including BBRv2, against the videoconferencing subsystems in signal, telegram, and whatsapp. https://tma.ifip.org/2023/wp-content/uploads/sites/12/2023/06/tma2023-final37.pdf Naturally fq_codel shines. The paper also introduces a new means of measuring videoconferenceQoE, called BRISQUE. About my only kvetch is they didn't try cake, which has a different shaper, drops less packets, has less collisions, as well as diffserv support. Seeing BBRv2 stats and impact was very interesting also. -- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos