* [LibreQoS] Fwd: A quick report from the WISPA conference [not found] ` <CAA93jw4f701R+4B538jF1+qAW=cUgP35EmWy8VZG-1h=w8woOA@mail.gmail.com> @ 2022-11-12 15:11 ` Dave Taht 2022-11-12 15:36 ` dan 0 siblings, 1 reply; 10+ messages in thread From: Dave Taht @ 2022-11-12 15:11 UTC (permalink / raw) To: libreqos this report predates the libreqos list... ---------- Forwarded message --------- From: Dave Taht <dave.taht@gmail.com> Date: Mon, Oct 17, 2022 at 8:15 PM Subject: A quick report from the WISPA conference To: Sina Khanifar <sina@waveform.com> Cc: Cake List <cake@lists.bufferbloat.net>, Make-Wifi-fast <make-wifi-fast@lists.bufferbloat.net>, Rpm <rpm@lists.bufferbloat.net>, Stuart Cheshire <cheshire@apple.com>, bloat <bloat@lists.bufferbloat.net> On Mon, Oct 17, 2022 at 7:51 PM Sina Khanifar <sina@waveform.com> wrote: > > Positive or negative, I can claim a bit of credit for this video :). We've been working with LTT on a few projects and we pitched them on doing something around bufferbloat. We've seen more traffic to our Waveforn test than ever before, which has been fun! Thank you. Great job with that video! And waveform has become the goto site for many now. I can't help but wonder tho... are you collecting any statistics, over time, as to how much better the problem is getting? And any chance they could do something similar explaining wifi? ... I was just at WISPA conference week before last. Preseem's booth (fq_codel) was always packed. Vilo living had put cake in their wifi 6 product. A keynote speaker had deployed it and talked about it with waveform results on the big screen (2k people there). A large wireless vendor demo'd privately to me their flent results before/after cake on their next-gen radios... and people dissed tarana without me prompting for their bad bufferbloat... and the best thing of all that happened to me was... besides getting a hug from a young lady (megan) who'd salvaged her schooling in alaska using sqm - I walked up to the paraqum booth (another large QoE middlebox maker centered more in india) and asked. "So... do y'all have fq_codel yet?" And they smiled and said: "No, we have something better... we've got cake." "Cake? What's that?" - I said, innocently. They then stepped me through their 200Gbps (!!) product, which uses a bunch of offloads, and can track rtt down to a ms with the intel ethernet card they were using. They'd modifed cake to provide 16 (?) levels of service, and were running under dpdk (I am not sure if cake was). It was a great, convincing pitch... ... then I told 'em who I was. There's a video of the in-both concert after. ... The downside to me (and the subject of my talk) was that in nearly every person I talked to, fq_codel was viewed as a means to better subscriber bandwidth plan enforcement (which is admittedly the market that preseem pioneered) and it was not understood that I'd got involved in this whole thing because I'd wanted an algorithm to deal with "rain fade", running directly on the radios. People wanted to use the statistics on the radios to drive the plan enforcement better (which is an ok approach, I guess), and for 10+ I'd been whinging about the... physics. So I ranted about rfc7567 a lot and begged people now putting routerOS 7.2 and later out there (mikrotik is huge in this market), to kill their fifos and sfqs at the native rates of the interfaces... and watch their network improve that way also. I think one more wispa conference will be a clean sweep of everyone in the fixed wireless market to not only adopt these algorithms for plan enforcement, but even more directly on the radios and more CPE. I also picked up enough consulting business to keep me busy the rest of this year, and possibly more than I can handle (anybody looking?) I wonder what will happen at a fiber conference? > On Mon, Oct 17, 2022 at 7:45 PM Dave Taht via Bloat <bloat@lists.bufferbloat.net> wrote: >> >> On Mon, Oct 17, 2022 at 5:02 PM Stuart Cheshire <cheshire@apple.com> wrote: >> > >> > On 9 Oct 2022, at 06:14, Dave Taht via Make-wifi-fast <make-wifi-fast@lists.bufferbloat.net> wrote: >> > >> > > This was so massively well done, I cried. Does anyone know how to get in touch with the ifxit folk? >> > > >> > > https://www.youtube.com/watch?v=UICh3ScfNWI >> > >> > I’m surprised that you liked this video. It seems to me that it repeats all the standard misinformation. The analogy they use is the standard terrible example of waiting in a long line at a grocery store, and the “solution” is letting certain traffic “jump the line, angering everyone behind them”. >> >> Accuracy be damned. The analogy to common experience resonates more. >> >> > >> > Some quotes from the video: >> > >> > > it would be so much more efficient for them to let you skip the line and just check out, especially since you’re in a hurry, but they’re rudely refusing >> >> I think the person with the cheetos pulling out a gun and shooting >> everyone in front of him (AQM) would not go down well. >> >> > > to go back to our grocery store analogy this would be like if a worker saw you standing at the back ... and either let you skip to the front of the line or opens up an express lane just for you >> >> Actually that analogy is fairly close to fair queuing. The multiple >> checker analogy is one of the most common analogies in queue theory >> itself. >> >> > >> > The video describes the problem of bufferbloat, and then describes the same failed solution that hasn’t worked for the last three decades. >> >> Hmm? It establishes the scenario, explains the problem *quickly*, >> disses gamer routers for not getting it right.. *points to an >> accurate test*, and then to the ideas and products that *actually >> work* with "smart queueing", with a screenshot of the most common >> (eero's optimize for gaming and videoconferencing), and fq_codel and >> cake *by name*, and points folk at the best known solution available, >> openwrt. >> >> Bing, baddabang, boom. Also the comments were revealing. A goodly >> percentage already knew the problem, more than a few were inspired to >> take the test, >> there was a whole bunch of "Aha!" success stories and 360k views, >> which is more people than we've ever been able to reach in for >> example, a nanog conference. >> >> I loved that folk taking the test actually had quite a few A results, >> without having had to do anything. At least some ISPs are getting it >> more right now! >> >> At this point I think gamers in particular know what "brands" we've >> tried to establish - "Smart queues", "SQM", "OpenWrt", fq_codel and >> now "cake" are "good" things to have, and are stimulating demand by >> asking for them, It's certainly working out better and better for >> evenroute, firewalla, ubnt and others, and I saw an uptick in >> questions about this on various user forums. >> >> I even like that there's a backlash now of people saying "fixing >> bufferbloat doesn't solve everything" - >> >> > Describing the obvious simple-minded (wrong) solution that any normal person would think of based on their personal human experience waiting in grocery stores and airports, is not describing the solution to bufferbloat. The solution to bufferbloat is not that if you are privileged then you get to “skip to the front of the line”. The solution to bufferbloat is that there is no line! >> >> I like the idea of a guru floating above a grocery cart with a better >> string of explanations, explaining >> >> - "no, grasshopper, the solution to bufferbloat is no line... at all". >> >> > >> > With grocery stores and airports people’s arrivals are independent and not controlled. There is no way for a grocery store or airport to generate backpressure to tell people to wait at home when a queue begins to form. The key to solving bufferbloat is generating timely backpressure to prevent the queue forming in the first place, not accepting a huge queue and then deciding who deserves special treatment to get better service than all the other peons who still have to wait in a long queue, just like before. >> >> I am not huge on the word "backpressure" here. Needs to signal the >> other side to slow down, is more accurate. So might say timely >> signalling rather than timely backpressure? >> >> Other feedback I got was that the video was too smarmy (I agree), >> different audiences than gamers need different forms of outreach... >> >> but to me, winning the gamers has always been one of the most >> important things, as they make a lot of buying decisions, and they >> benefit the most for >> fq and packet prioritization as we do today in gamer routers and in >> cake + qosify. >> >> maybe that gets in the way of more serious markets. Certainly I would >> like another video explaining what goes wrong with videoconferencing. >> >> >> >> >> >> >> > >> > Stuart Cheshire >> > >> >> >> -- >> This song goes out to all the folk that thought Stadia would work: >> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz >> Dave Täht CEO, TekLibre, LLC >> _______________________________________________ >> Bloat mailing list >> Bloat@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/bloat -- This song goes out to all the folk that thought Stadia would work: https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz Dave Täht CEO, TekLibre, LLC -- This song goes out to all the folk that thought Stadia would work: https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz Dave Täht CEO, TekLibre, LLC ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [LibreQoS] Fwd: A quick report from the WISPA conference 2022-11-12 15:11 ` [LibreQoS] Fwd: A quick report from the WISPA conference Dave Taht @ 2022-11-12 15:36 ` dan 2022-11-12 16:09 ` Robert Chacón 0 siblings, 1 reply; 10+ messages in thread From: dan @ 2022-11-12 15:36 UTC (permalink / raw) To: Dave Taht; +Cc: libreqos [-- Attachment #1: Type: text/plain, Size: 12096 bytes --] I think there's a thin grey line between what other vendors (paraqum, bequant/cambium) and libreQoS is trying to do. The first two are REALLY pushing this selecting narrowing of traffic and they'll go straight after streaming for their demos. Most people that drop in the cambium QoE appliance turn the knobs on Netflix down and then praise the 40% savings on their network. Same with PQ. I think this is fundamentally wrong. This is the entire reason Net Neutrality keeps popping up. Why does the ISP get to say "You only need 25Mbps for your netflix". That's double wrong if they offer their own video services. It's not about improving the customer's experience, it's about spending less on upstream bandwidth at the customer's expense. As soon as they say 'pays for itself in bandwidth savings' you know what the product is primarily for. This is why I like the preseem and libreqos model of just using a 'stock' cake and just finessing the data coming in to keep the pipe from clogging. Who cares if someone pulls 90M bursts of apple TV+ on their 100M pipe. Only dial that back so other requests they make get through cleanly. preseem is going more towards wireless monitoring etc and kinda letting shaping be a secondary.... if they were doing more libreqos like stuff I'd just stick with preseem. libreQoS may end up being the best and most net neutral product you can get. The only current disadvantage (ignoring DPI) is lack of TCP accelleration that PQ and Bequant/cambium have. I should add that 3 states already have laws on the books that straight up make PQ and Cambium's solution illegal because they target certain services for throttling. My state (Montana) has the same rule except only when working with a government contract. Libreqos is actually compliant (as is preseem). On Sat, Nov 12, 2022 at 8:11 AM Dave Taht via LibreQoS < libreqos@lists.bufferbloat.net> wrote: > this report predates the libreqos list... > > ---------- Forwarded message --------- > From: Dave Taht <dave.taht@gmail.com> > Date: Mon, Oct 17, 2022 at 8:15 PM > Subject: A quick report from the WISPA conference > To: Sina Khanifar <sina@waveform.com> > Cc: Cake List <cake@lists.bufferbloat.net>, Make-Wifi-fast > <make-wifi-fast@lists.bufferbloat.net>, Rpm > <rpm@lists.bufferbloat.net>, Stuart Cheshire <cheshire@apple.com>, > bloat <bloat@lists.bufferbloat.net> > > > On Mon, Oct 17, 2022 at 7:51 PM Sina Khanifar <sina@waveform.com> wrote: > > > > Positive or negative, I can claim a bit of credit for this video :). > We've been working with LTT on a few projects and we pitched them on doing > something around bufferbloat. We've seen more traffic to our Waveforn test > than ever before, which has been fun! > > Thank you. Great job with that video! And waveform has become the goto > site for many now. > > I can't help but wonder tho... are you collecting any statistics, over > time, as to how much better the problem is getting? > > And any chance they could do something similar explaining wifi? > > ... > > I was just at WISPA conference week before last. Preseem's booth > (fq_codel) was always packed. Vilo living had put cake in their wifi 6 > product. A > keynote speaker had deployed it and talked about it with waveform > results on the big screen (2k people there). A large wireless vendor > demo'd privately to me their flent results before/after cake on their > next-gen radios... and people dissed tarana without me prompting for > their bad bufferbloat... and the best thing of all that happened to me > was... besides getting a hug from a young lady (megan) who'd salvaged > her schooling in alaska using sqm - I walked up to the paraqum booth > (another large QoE middlebox maker centered more in india) and asked. > > "So... do y'all have fq_codel yet?" > > And they smiled and said: "No, we have something better... we've got cake." > > "Cake? What's that?" - I said, innocently. > > They then stepped me through their 200Gbps (!!) product, which uses a > bunch of offloads, and can track rtt down to a ms with the intel > ethernet card they were using. They'd modifed cake to provide 16 (?) > levels of service, and were running under dpdk (I am not sure if cake > was). It was a great, convincing pitch... > > ... then I told 'em who I was. There's a video of the in-both concert > after. > > ... > > The downside to me (and the subject of my talk) was that in nearly > every person I talked to, fq_codel was viewed as a means to better > subscriber bandwidth plan enforcement (which is admittedly the market > that preseem pioneered) and it was not understood that I'd got > involved in this whole thing because I'd wanted an algorithm to deal > with "rain fade", running directly on the radios. People wanted to use > the statistics on the radios to drive the plan enforcement better > (which is an ok approach, I guess), and for 10+ I'd been whinging > about the... physics. > > So I ranted about rfc7567 a lot and begged people now putting routerOS > 7.2 and later out there (mikrotik is huge in this market), to kill > their fifos and sfqs at the native rates of the interfaces... and > watch their network improve that way also. > > I think one more wispa conference will be a clean sweep of everyone in > the fixed wireless market to not only adopt these algorithms for plan > enforcement, but even more directly on the radios and more CPE. > > I also picked up enough consulting business to keep me busy the rest > of this year, and possibly more than I can handle (anybody looking?) > > I wonder what will happen at a fiber conference? > > > On Mon, Oct 17, 2022 at 7:45 PM Dave Taht via Bloat < > bloat@lists.bufferbloat.net> wrote: > >> > >> On Mon, Oct 17, 2022 at 5:02 PM Stuart Cheshire <cheshire@apple.com> > wrote: > >> > > >> > On 9 Oct 2022, at 06:14, Dave Taht via Make-wifi-fast < > make-wifi-fast@lists.bufferbloat.net> wrote: > >> > > >> > > This was so massively well done, I cried. Does anyone know how to > get in touch with the ifxit folk? > >> > > > >> > > https://www.youtube.com/watch?v=UICh3ScfNWI > >> > > >> > I’m surprised that you liked this video. It seems to me that it > repeats all the standard misinformation. The analogy they use is the > standard terrible example of waiting in a long line at a grocery store, and > the “solution” is letting certain traffic “jump the line, angering everyone > behind them”. > >> > >> Accuracy be damned. The analogy to common experience resonates more. > >> > >> > > >> > Some quotes from the video: > >> > > >> > > it would be so much more efficient for them to let you skip the > line and just check out, especially since you’re in a hurry, but they’re > rudely refusing > >> > >> I think the person with the cheetos pulling out a gun and shooting > >> everyone in front of him (AQM) would not go down well. > >> > >> > > to go back to our grocery store analogy this would be like if a > worker saw you standing at the back ... and either let you skip to the > front of the line or opens up an express lane just for you > >> > >> Actually that analogy is fairly close to fair queuing. The multiple > >> checker analogy is one of the most common analogies in queue theory > >> itself. > >> > >> > > >> > The video describes the problem of bufferbloat, and then describes > the same failed solution that hasn’t worked for the last three decades. > >> > >> Hmm? It establishes the scenario, explains the problem *quickly*, > >> disses gamer routers for not getting it right.. *points to an > >> accurate test*, and then to the ideas and products that *actually > >> work* with "smart queueing", with a screenshot of the most common > >> (eero's optimize for gaming and videoconferencing), and fq_codel and > >> cake *by name*, and points folk at the best known solution available, > >> openwrt. > >> > >> Bing, baddabang, boom. Also the comments were revealing. A goodly > >> percentage already knew the problem, more than a few were inspired to > >> take the test, > >> there was a whole bunch of "Aha!" success stories and 360k views, > >> which is more people than we've ever been able to reach in for > >> example, a nanog conference. > >> > >> I loved that folk taking the test actually had quite a few A results, > >> without having had to do anything. At least some ISPs are getting it > >> more right now! > >> > >> At this point I think gamers in particular know what "brands" we've > >> tried to establish - "Smart queues", "SQM", "OpenWrt", fq_codel and > >> now "cake" are "good" things to have, and are stimulating demand by > >> asking for them, It's certainly working out better and better for > >> evenroute, firewalla, ubnt and others, and I saw an uptick in > >> questions about this on various user forums. > >> > >> I even like that there's a backlash now of people saying "fixing > >> bufferbloat doesn't solve everything" - > >> > >> > Describing the obvious simple-minded (wrong) solution that any > normal person would think of based on their personal human experience > waiting in grocery stores and airports, is not describing the solution to > bufferbloat. The solution to bufferbloat is not that if you are privileged > then you get to “skip to the front of the line”. The solution to > bufferbloat is that there is no line! > >> > >> I like the idea of a guru floating above a grocery cart with a better > >> string of explanations, explaining > >> > >> - "no, grasshopper, the solution to bufferbloat is no line... at > all". > >> > >> > > >> > With grocery stores and airports people’s arrivals are independent > and not controlled. There is no way for a grocery store or airport to > generate backpressure to tell people to wait at home when a queue begins to > form. The key to solving bufferbloat is generating timely backpressure to > prevent the queue forming in the first place, not accepting a huge queue > and then deciding who deserves special treatment to get better service than > all the other peons who still have to wait in a long queue, just like > before. > >> > >> I am not huge on the word "backpressure" here. Needs to signal the > >> other side to slow down, is more accurate. So might say timely > >> signalling rather than timely backpressure? > >> > >> Other feedback I got was that the video was too smarmy (I agree), > >> different audiences than gamers need different forms of outreach... > >> > >> but to me, winning the gamers has always been one of the most > >> important things, as they make a lot of buying decisions, and they > >> benefit the most for > >> fq and packet prioritization as we do today in gamer routers and in > >> cake + qosify. > >> > >> maybe that gets in the way of more serious markets. Certainly I would > >> like another video explaining what goes wrong with videoconferencing. > >> > >> > >> > >> > >> > >> > >> > > >> > Stuart Cheshire > >> > > >> > >> > >> -- > >> This song goes out to all the folk that thought Stadia would work: > >> > https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz > >> Dave Täht CEO, TekLibre, LLC > >> _______________________________________________ > >> Bloat mailing list > >> Bloat@lists.bufferbloat.net > >> https://lists.bufferbloat.net/listinfo/bloat > > > > -- > This song goes out to all the folk that thought Stadia would work: > > https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz > Dave Täht CEO, TekLibre, LLC > > > -- > This song goes out to all the folk that thought Stadia would work: > > https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz > Dave Täht CEO, TekLibre, LLC > _______________________________________________ > LibreQoS mailing list > LibreQoS@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/libreqos > [-- Attachment #2: Type: text/html, Size: 15295 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [LibreQoS] Fwd: A quick report from the WISPA conference 2022-11-12 15:36 ` dan @ 2022-11-12 16:09 ` Robert Chacón 2022-11-12 16:15 ` dan 2022-11-12 17:19 ` Herbert Wolverson 0 siblings, 2 replies; 10+ messages in thread From: Robert Chacón @ 2022-11-12 16:09 UTC (permalink / raw) To: dan; +Cc: Dave Taht, libreqos [-- Attachment #1: Type: text/plain, Size: 15221 bytes --] Thank you, Dan. I completely agree with you 100% there. It's a risky way to save a tiny amount of money on bandwidth costs. Comcast paid out $16 million in a settlement <https://arstechnica.com/tech-policy/2010/04/comcast-owes-p2p-users-16-yes-they-should-take-it/?comments=1&post=20329187> over using Sandvine's DPI in this way. Since then, Comcast has switched from DPI to fair queueing with DOCSIS-pie <https://www.reddit.com/r/eero/comments/rc8e2j/getting_a_bufferbloat_grade_with_comcast_aqm/> . California's SB-822 <https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=201720180SB822&fbclid=IwAR2LmNyy5ZkbtzayIyNerJ-Q_OUwK4G0H4aYv4XZwVNz-QF41d5v0BiST-E> will be used as a template by many other states going forward on Net Neutrality. <https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=201720180SB822&fbclid=IwAR2LmNyy5ZkbtzayIyNerJ-Q_OUwK4G0H4aYv4XZwVNz-QF41d5v0BiST-E> SB-822 even clarified that "Reasonable Network Management" needs to be "as application-agnostic as possible". These DPI approaches like Cambium's will not hold up in court the way Fair Queuing (Preseem, LibreQoS) based solutions will. We can point to years of public research and RFCs when defending a Fair Queuing QoE approach. They can't so easily defend using a product that advertised its big "scale down 4k to save money" knob or "accelerate their speeds only when they run speed tests" button. It's concerning to me that these QoE vendors are not disclosing "hey, don't use this in NN states like California please". The WISP industry will be hurt long-term by these products with consumers viewing us as an inferior technology that limits their usage in intrusive ways. Like you said, let's let them use their plan however they want. It doesn't hurt us. But things like capping end-users at HD will hurt us long-term in lost sales. Regarding TCP acceleration, I'm leaning toward the argument made by Dan from Preseem <https://youtu.be/3r4FWGKho3c?t=1879> that QUIC adoption is growing pretty quickly and we may not want to mess with TCP at the expense of UDP given the rise of QUIC. Cloudlfare says HTTP/3 and QUIC are already at 28% and growing <https://radar.cloudflare.com/adoption-and-usage?range=28d>. One point he brought up is that if we use TCP acceleration, and a middlebox has to be rebooted or something - all client TCP sessions will break when OSPF switches paths around the box. On Sat, Nov 12, 2022 at 8:36 AM dan via LibreQoS < libreqos@lists.bufferbloat.net> wrote: > I think there's a thin grey line between what other vendors (paraqum, > bequant/cambium) and libreQoS is trying to do. The first two are REALLY > pushing this selecting narrowing of traffic and they'll go straight after > streaming for their demos. Most people that drop in the cambium QoE > appliance turn the knobs on Netflix down and then praise the 40% savings on > their network. Same with PQ. I think this is fundamentally wrong. This > is the entire reason Net Neutrality keeps popping up. Why does the ISP get > to say "You only need 25Mbps for your netflix". That's double wrong if > they offer their own video services. It's not about improving the > customer's experience, it's about spending less on upstream bandwidth at > the customer's expense. As soon as they say 'pays for itself in bandwidth > savings' you know what the product is primarily for. > > This is why I like the preseem and libreqos model of just using a 'stock' > cake and just finessing the data coming in to keep the pipe from clogging. > Who cares if someone pulls 90M bursts of apple TV+ on their 100M pipe. > Only dial that back so other requests they make get through cleanly. > > preseem is going more towards wireless monitoring etc and kinda letting > shaping be a secondary.... if they were doing more libreqos like stuff I'd > just stick with preseem. > > libreQoS may end up being the best and most net neutral product you can > get. The only current disadvantage (ignoring DPI) is lack of TCP > accelleration that PQ and Bequant/cambium have. > > I should add that 3 states already have laws on the books that straight up > make PQ and Cambium's solution illegal because they target certain services > for throttling. My state (Montana) has the same rule except only when > working with a government contract. Libreqos is actually compliant (as is > preseem). > > On Sat, Nov 12, 2022 at 8:11 AM Dave Taht via LibreQoS < > libreqos@lists.bufferbloat.net> wrote: > >> this report predates the libreqos list... >> >> ---------- Forwarded message --------- >> From: Dave Taht <dave.taht@gmail.com> >> Date: Mon, Oct 17, 2022 at 8:15 PM >> Subject: A quick report from the WISPA conference >> To: Sina Khanifar <sina@waveform.com> >> Cc: Cake List <cake@lists.bufferbloat.net>, Make-Wifi-fast >> <make-wifi-fast@lists.bufferbloat.net>, Rpm >> <rpm@lists.bufferbloat.net>, Stuart Cheshire <cheshire@apple.com>, >> bloat <bloat@lists.bufferbloat.net> >> >> >> On Mon, Oct 17, 2022 at 7:51 PM Sina Khanifar <sina@waveform.com> wrote: >> > >> > Positive or negative, I can claim a bit of credit for this video :). >> We've been working with LTT on a few projects and we pitched them on doing >> something around bufferbloat. We've seen more traffic to our Waveforn test >> than ever before, which has been fun! >> >> Thank you. Great job with that video! And waveform has become the goto >> site for many now. >> >> I can't help but wonder tho... are you collecting any statistics, over >> time, as to how much better the problem is getting? >> >> And any chance they could do something similar explaining wifi? >> >> ... >> >> I was just at WISPA conference week before last. Preseem's booth >> (fq_codel) was always packed. Vilo living had put cake in their wifi 6 >> product. A >> keynote speaker had deployed it and talked about it with waveform >> results on the big screen (2k people there). A large wireless vendor >> demo'd privately to me their flent results before/after cake on their >> next-gen radios... and people dissed tarana without me prompting for >> their bad bufferbloat... and the best thing of all that happened to me >> was... besides getting a hug from a young lady (megan) who'd salvaged >> her schooling in alaska using sqm - I walked up to the paraqum booth >> (another large QoE middlebox maker centered more in india) and asked. >> >> "So... do y'all have fq_codel yet?" >> >> And they smiled and said: "No, we have something better... we've got >> cake." >> >> "Cake? What's that?" - I said, innocently. >> >> They then stepped me through their 200Gbps (!!) product, which uses a >> bunch of offloads, and can track rtt down to a ms with the intel >> ethernet card they were using. They'd modifed cake to provide 16 (?) >> levels of service, and were running under dpdk (I am not sure if cake >> was). It was a great, convincing pitch... >> >> ... then I told 'em who I was. There's a video of the in-both concert >> after. >> >> ... >> >> The downside to me (and the subject of my talk) was that in nearly >> every person I talked to, fq_codel was viewed as a means to better >> subscriber bandwidth plan enforcement (which is admittedly the market >> that preseem pioneered) and it was not understood that I'd got >> involved in this whole thing because I'd wanted an algorithm to deal >> with "rain fade", running directly on the radios. People wanted to use >> the statistics on the radios to drive the plan enforcement better >> (which is an ok approach, I guess), and for 10+ I'd been whinging >> about the... physics. >> >> So I ranted about rfc7567 a lot and begged people now putting routerOS >> 7.2 and later out there (mikrotik is huge in this market), to kill >> their fifos and sfqs at the native rates of the interfaces... and >> watch their network improve that way also. >> >> I think one more wispa conference will be a clean sweep of everyone in >> the fixed wireless market to not only adopt these algorithms for plan >> enforcement, but even more directly on the radios and more CPE. >> >> I also picked up enough consulting business to keep me busy the rest >> of this year, and possibly more than I can handle (anybody looking?) >> >> I wonder what will happen at a fiber conference? >> >> > On Mon, Oct 17, 2022 at 7:45 PM Dave Taht via Bloat < >> bloat@lists.bufferbloat.net> wrote: >> >> >> >> On Mon, Oct 17, 2022 at 5:02 PM Stuart Cheshire <cheshire@apple.com> >> wrote: >> >> > >> >> > On 9 Oct 2022, at 06:14, Dave Taht via Make-wifi-fast < >> make-wifi-fast@lists.bufferbloat.net> wrote: >> >> > >> >> > > This was so massively well done, I cried. Does anyone know how to >> get in touch with the ifxit folk? >> >> > > >> >> > > https://www.youtube.com/watch?v=UICh3ScfNWI >> >> > >> >> > I’m surprised that you liked this video. It seems to me that it >> repeats all the standard misinformation. The analogy they use is the >> standard terrible example of waiting in a long line at a grocery store, and >> the “solution” is letting certain traffic “jump the line, angering everyone >> behind them”. >> >> >> >> Accuracy be damned. The analogy to common experience resonates more. >> >> >> >> > >> >> > Some quotes from the video: >> >> > >> >> > > it would be so much more efficient for them to let you skip the >> line and just check out, especially since you’re in a hurry, but they’re >> rudely refusing >> >> >> >> I think the person with the cheetos pulling out a gun and shooting >> >> everyone in front of him (AQM) would not go down well. >> >> >> >> > > to go back to our grocery store analogy this would be like if a >> worker saw you standing at the back ... and either let you skip to the >> front of the line or opens up an express lane just for you >> >> >> >> Actually that analogy is fairly close to fair queuing. The multiple >> >> checker analogy is one of the most common analogies in queue theory >> >> itself. >> >> >> >> > >> >> > The video describes the problem of bufferbloat, and then describes >> the same failed solution that hasn’t worked for the last three decades. >> >> >> >> Hmm? It establishes the scenario, explains the problem *quickly*, >> >> disses gamer routers for not getting it right.. *points to an >> >> accurate test*, and then to the ideas and products that *actually >> >> work* with "smart queueing", with a screenshot of the most common >> >> (eero's optimize for gaming and videoconferencing), and fq_codel and >> >> cake *by name*, and points folk at the best known solution available, >> >> openwrt. >> >> >> >> Bing, baddabang, boom. Also the comments were revealing. A goodly >> >> percentage already knew the problem, more than a few were inspired to >> >> take the test, >> >> there was a whole bunch of "Aha!" success stories and 360k views, >> >> which is more people than we've ever been able to reach in for >> >> example, a nanog conference. >> >> >> >> I loved that folk taking the test actually had quite a few A results, >> >> without having had to do anything. At least some ISPs are getting it >> >> more right now! >> >> >> >> At this point I think gamers in particular know what "brands" we've >> >> tried to establish - "Smart queues", "SQM", "OpenWrt", fq_codel and >> >> now "cake" are "good" things to have, and are stimulating demand by >> >> asking for them, It's certainly working out better and better for >> >> evenroute, firewalla, ubnt and others, and I saw an uptick in >> >> questions about this on various user forums. >> >> >> >> I even like that there's a backlash now of people saying "fixing >> >> bufferbloat doesn't solve everything" - >> >> >> >> > Describing the obvious simple-minded (wrong) solution that any >> normal person would think of based on their personal human experience >> waiting in grocery stores and airports, is not describing the solution to >> bufferbloat. The solution to bufferbloat is not that if you are privileged >> then you get to “skip to the front of the line”. The solution to >> bufferbloat is that there is no line! >> >> >> >> I like the idea of a guru floating above a grocery cart with a better >> >> string of explanations, explaining >> >> >> >> - "no, grasshopper, the solution to bufferbloat is no line... at >> all". >> >> >> >> > >> >> > With grocery stores and airports people’s arrivals are independent >> and not controlled. There is no way for a grocery store or airport to >> generate backpressure to tell people to wait at home when a queue begins to >> form. The key to solving bufferbloat is generating timely backpressure to >> prevent the queue forming in the first place, not accepting a huge queue >> and then deciding who deserves special treatment to get better service than >> all the other peons who still have to wait in a long queue, just like >> before. >> >> >> >> I am not huge on the word "backpressure" here. Needs to signal the >> >> other side to slow down, is more accurate. So might say timely >> >> signalling rather than timely backpressure? >> >> >> >> Other feedback I got was that the video was too smarmy (I agree), >> >> different audiences than gamers need different forms of outreach... >> >> >> >> but to me, winning the gamers has always been one of the most >> >> important things, as they make a lot of buying decisions, and they >> >> benefit the most for >> >> fq and packet prioritization as we do today in gamer routers and in >> >> cake + qosify. >> >> >> >> maybe that gets in the way of more serious markets. Certainly I would >> >> like another video explaining what goes wrong with videoconferencing. >> >> >> >> >> >> >> >> >> >> >> >> >> >> > >> >> > Stuart Cheshire >> >> > >> >> >> >> >> >> -- >> >> This song goes out to all the folk that thought Stadia would work: >> >> >> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz >> >> Dave Täht CEO, TekLibre, LLC >> >> _______________________________________________ >> >> Bloat mailing list >> >> Bloat@lists.bufferbloat.net >> >> https://lists.bufferbloat.net/listinfo/bloat >> >> >> >> -- >> This song goes out to all the folk that thought Stadia would work: >> >> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz >> Dave Täht CEO, TekLibre, LLC >> >> >> -- >> This song goes out to all the folk that thought Stadia would work: >> >> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz >> Dave Täht CEO, TekLibre, LLC >> _______________________________________________ >> LibreQoS mailing list >> LibreQoS@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/libreqos >> > _______________________________________________ > LibreQoS mailing list > LibreQoS@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/libreqos > -- Robert Chacón CEO | JackRabbit Wireless LLC <http://jackrabbitwireless.com> Dev | LibreQoS.io [-- Attachment #2: Type: text/html, Size: 19750 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [LibreQoS] Fwd: A quick report from the WISPA conference 2022-11-12 16:09 ` Robert Chacón @ 2022-11-12 16:15 ` dan 2022-11-12 17:19 ` Herbert Wolverson 1 sibling, 0 replies; 10+ messages in thread From: dan @ 2022-11-12 16:15 UTC (permalink / raw) To: Robert Chacón; +Cc: Dave Taht, libreqos [-- Attachment #1: Type: text/plain, Size: 794 bytes --] On Sat, Nov 12, 2022 at 9:09 AM Robert Chacón < robert.chacon@jackrabbitwireless.com> wrote: > > > Regarding TCP acceleration, I'm leaning toward the argument made by Dan > from Preseem <https://youtu.be/3r4FWGKho3c?t=1879> that QUIC adoption is > growing pretty quickly and we may not want to mess with TCP at the expense > of UDP given the rise of QUIC. > Cloudlfare says HTTP/3 and QUIC are already at 28% and growing > <https://radar.cloudflare.com/adoption-and-usage?range=28d>. > One point he brought up is that if we use TCP acceleration, and a > middlebox has to be rebooted or something - all client TCP sessions will > break when OSPF switches paths around the box. > > > I'll buy that. TCP accelleration is a *TODAY* fix with a decreasing gain as time goes on. [-- Attachment #2: Type: text/html, Size: 1493 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [LibreQoS] Fwd: A quick report from the WISPA conference 2022-11-12 16:09 ` Robert Chacón 2022-11-12 16:15 ` dan @ 2022-11-12 17:19 ` Herbert Wolverson 2022-11-12 17:39 ` dan 1 sibling, 1 reply; 10+ messages in thread From: Herbert Wolverson @ 2022-11-12 17:19 UTC (permalink / raw) Cc: libreqos [-- Attachment #1: Type: text/plain, Size: 18395 bytes --] Net neutrality is complex (I'm generally in favor of the concept). There's nothing stopping an ISP from having rules that video streaming be limited - but ALL video streaming has to be equally limited. So ComCast could use Sandvine, but only without any vendor-specific limitations. Same for Cambium. Limit streams that look like video, and you're fine. With that said, it runs into something I say a lot: don't penalize your customers for using your product. If you're buying a 100mbit/s pipe, hooking it to your fancy 4k streaming system - it's entirely reasonable to expect sharp 4k video. In fact, your shaping emphasis should be on keeping their video nice and watchable even though someone just decided to download the newest Call of Duty, update their xbox, and update the firmware on their smart fridge. Likewise, if someone really wants the smallest 5 mbit connection you once offered - because they only ever use it to check email (we actually had a customer complain when we offered them a free upgrade from a 5mbit/s plan, years ago!) - you want to do your best to make it a usable tiny plan for them. Fair queuing (along with a bit of user education; we eventually convinced the customer that "no change in fees" actually meant not charging them more money) helps with all of that. It can win customers, too. We have a couple of business customers who switched us from their backup to their primary because "our service felt snappier". We had a business go with us because their 24x7 lobby streaming video looked better on our (smaller) connection, and a couple of fraternities who likewise decided to use us because their overkill 4k video setup looked better (even with the 20+ Xboxes we could see on their network). QUIC makes me chuckle, because it's *exactly* what we were doing in gamedev land in the late 90s to make deathmatches run smoothly. (Seriously, QuakeWorld and the original Unreal Tournament had the most amazing network code; they basically implemented what is now known as "reliable UDP" to incredible results). With that said, I think we've got a remarkable amount of wiggle-room to make things better (and it's not at all shoddy right now!). XDP is amazing as-is, and is improving fast. (I'm currently reading https://github.com/ilejn/xdpbridge - an older project that doesn't seem to have gained much traction, but I'm wondering if it couldn't provide some seriously fast bridge offloading; I'll have to setup a better test environment to find out) On Sat, Nov 12, 2022 at 10:09 AM Robert Chacón via LibreQoS < libreqos@lists.bufferbloat.net> wrote: > Thank you, Dan. > I completely agree with you 100% there. It's a risky way to save a tiny > amount of money on bandwidth costs. > Comcast paid out $16 million in a settlement > <https://arstechnica.com/tech-policy/2010/04/comcast-owes-p2p-users-16-yes-they-should-take-it/?comments=1&post=20329187> > over using Sandvine's DPI in this way. > Since then, Comcast has switched from DPI to fair queueing with DOCSIS-pie > <https://www.reddit.com/r/eero/comments/rc8e2j/getting_a_bufferbloat_grade_with_comcast_aqm/> > . > > California's SB-822 > <https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=201720180SB822&fbclid=IwAR2LmNyy5ZkbtzayIyNerJ-Q_OUwK4G0H4aYv4XZwVNz-QF41d5v0BiST-E> > will be used as a template by many other states going forward on Net > Neutrality. > <https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=201720180SB822&fbclid=IwAR2LmNyy5ZkbtzayIyNerJ-Q_OUwK4G0H4aYv4XZwVNz-QF41d5v0BiST-E> > SB-822 even clarified that "Reasonable Network Management" needs to be "as > application-agnostic as possible". > These DPI approaches like Cambium's will not hold up in court the way Fair > Queuing (Preseem, LibreQoS) based solutions will. > We can point to years of public research and RFCs when defending a Fair > Queuing QoE approach. > They can't so easily defend using a product that advertised its big "scale > down 4k to save money" knob or "accelerate their speeds only when they run > speed tests" button. > It's concerning to me that these QoE vendors are not disclosing "hey, > don't use this in NN states like California please". > The WISP industry will be hurt long-term by these products with consumers > viewing us as an inferior technology that limits their usage in intrusive > ways. > Like you said, let's let them use their plan however they want. It doesn't > hurt us. But things like capping end-users at HD will hurt us long-term in > lost sales. > > Regarding TCP acceleration, I'm leaning toward the argument made by Dan > from Preseem <https://youtu.be/3r4FWGKho3c?t=1879> that QUIC adoption is > growing pretty quickly and we may not want to mess with TCP at the expense > of UDP given the rise of QUIC. > Cloudlfare says HTTP/3 and QUIC are already at 28% and growing > <https://radar.cloudflare.com/adoption-and-usage?range=28d>. > One point he brought up is that if we use TCP acceleration, and a > middlebox has to be rebooted or something - all client TCP sessions will > break when OSPF switches paths around the box. > > On Sat, Nov 12, 2022 at 8:36 AM dan via LibreQoS < > libreqos@lists.bufferbloat.net> wrote: > >> I think there's a thin grey line between what other vendors (paraqum, >> bequant/cambium) and libreQoS is trying to do. The first two are REALLY >> pushing this selecting narrowing of traffic and they'll go straight after >> streaming for their demos. Most people that drop in the cambium QoE >> appliance turn the knobs on Netflix down and then praise the 40% savings on >> their network. Same with PQ. I think this is fundamentally wrong. This >> is the entire reason Net Neutrality keeps popping up. Why does the ISP get >> to say "You only need 25Mbps for your netflix". That's double wrong if >> they offer their own video services. It's not about improving the >> customer's experience, it's about spending less on upstream bandwidth at >> the customer's expense. As soon as they say 'pays for itself in bandwidth >> savings' you know what the product is primarily for. >> >> This is why I like the preseem and libreqos model of just using a 'stock' >> cake and just finessing the data coming in to keep the pipe from clogging. >> Who cares if someone pulls 90M bursts of apple TV+ on their 100M pipe. >> Only dial that back so other requests they make get through cleanly. >> >> preseem is going more towards wireless monitoring etc and kinda letting >> shaping be a secondary.... if they were doing more libreqos like stuff I'd >> just stick with preseem. >> >> libreQoS may end up being the best and most net neutral product you can >> get. The only current disadvantage (ignoring DPI) is lack of TCP >> accelleration that PQ and Bequant/cambium have. >> >> I should add that 3 states already have laws on the books that straight >> up make PQ and Cambium's solution illegal because they target certain >> services for throttling. My state (Montana) has the same rule except only >> when working with a government contract. Libreqos is actually compliant >> (as is preseem). >> >> On Sat, Nov 12, 2022 at 8:11 AM Dave Taht via LibreQoS < >> libreqos@lists.bufferbloat.net> wrote: >> >>> this report predates the libreqos list... >>> >>> ---------- Forwarded message --------- >>> From: Dave Taht <dave.taht@gmail.com> >>> Date: Mon, Oct 17, 2022 at 8:15 PM >>> Subject: A quick report from the WISPA conference >>> To: Sina Khanifar <sina@waveform.com> >>> Cc: Cake List <cake@lists.bufferbloat.net>, Make-Wifi-fast >>> <make-wifi-fast@lists.bufferbloat.net>, Rpm >>> <rpm@lists.bufferbloat.net>, Stuart Cheshire <cheshire@apple.com>, >>> bloat <bloat@lists.bufferbloat.net> >>> >>> >>> On Mon, Oct 17, 2022 at 7:51 PM Sina Khanifar <sina@waveform.com> wrote: >>> > >>> > Positive or negative, I can claim a bit of credit for this video :). >>> We've been working with LTT on a few projects and we pitched them on doing >>> something around bufferbloat. We've seen more traffic to our Waveforn test >>> than ever before, which has been fun! >>> >>> Thank you. Great job with that video! And waveform has become the goto >>> site for many now. >>> >>> I can't help but wonder tho... are you collecting any statistics, over >>> time, as to how much better the problem is getting? >>> >>> And any chance they could do something similar explaining wifi? >>> >>> ... >>> >>> I was just at WISPA conference week before last. Preseem's booth >>> (fq_codel) was always packed. Vilo living had put cake in their wifi 6 >>> product. A >>> keynote speaker had deployed it and talked about it with waveform >>> results on the big screen (2k people there). A large wireless vendor >>> demo'd privately to me their flent results before/after cake on their >>> next-gen radios... and people dissed tarana without me prompting for >>> their bad bufferbloat... and the best thing of all that happened to me >>> was... besides getting a hug from a young lady (megan) who'd salvaged >>> her schooling in alaska using sqm - I walked up to the paraqum booth >>> (another large QoE middlebox maker centered more in india) and asked. >>> >>> "So... do y'all have fq_codel yet?" >>> >>> And they smiled and said: "No, we have something better... we've got >>> cake." >>> >>> "Cake? What's that?" - I said, innocently. >>> >>> They then stepped me through their 200Gbps (!!) product, which uses a >>> bunch of offloads, and can track rtt down to a ms with the intel >>> ethernet card they were using. They'd modifed cake to provide 16 (?) >>> levels of service, and were running under dpdk (I am not sure if cake >>> was). It was a great, convincing pitch... >>> >>> ... then I told 'em who I was. There's a video of the in-both concert >>> after. >>> >>> ... >>> >>> The downside to me (and the subject of my talk) was that in nearly >>> every person I talked to, fq_codel was viewed as a means to better >>> subscriber bandwidth plan enforcement (which is admittedly the market >>> that preseem pioneered) and it was not understood that I'd got >>> involved in this whole thing because I'd wanted an algorithm to deal >>> with "rain fade", running directly on the radios. People wanted to use >>> the statistics on the radios to drive the plan enforcement better >>> (which is an ok approach, I guess), and for 10+ I'd been whinging >>> about the... physics. >>> >>> So I ranted about rfc7567 a lot and begged people now putting routerOS >>> 7.2 and later out there (mikrotik is huge in this market), to kill >>> their fifos and sfqs at the native rates of the interfaces... and >>> watch their network improve that way also. >>> >>> I think one more wispa conference will be a clean sweep of everyone in >>> the fixed wireless market to not only adopt these algorithms for plan >>> enforcement, but even more directly on the radios and more CPE. >>> >>> I also picked up enough consulting business to keep me busy the rest >>> of this year, and possibly more than I can handle (anybody looking?) >>> >>> I wonder what will happen at a fiber conference? >>> >>> > On Mon, Oct 17, 2022 at 7:45 PM Dave Taht via Bloat < >>> bloat@lists.bufferbloat.net> wrote: >>> >> >>> >> On Mon, Oct 17, 2022 at 5:02 PM Stuart Cheshire <cheshire@apple.com> >>> wrote: >>> >> > >>> >> > On 9 Oct 2022, at 06:14, Dave Taht via Make-wifi-fast < >>> make-wifi-fast@lists.bufferbloat.net> wrote: >>> >> > >>> >> > > This was so massively well done, I cried. Does anyone know how to >>> get in touch with the ifxit folk? >>> >> > > >>> >> > > https://www.youtube.com/watch?v=UICh3ScfNWI >>> >> > >>> >> > I’m surprised that you liked this video. It seems to me that it >>> repeats all the standard misinformation. The analogy they use is the >>> standard terrible example of waiting in a long line at a grocery store, and >>> the “solution” is letting certain traffic “jump the line, angering everyone >>> behind them”. >>> >> >>> >> Accuracy be damned. The analogy to common experience resonates more. >>> >> >>> >> > >>> >> > Some quotes from the video: >>> >> > >>> >> > > it would be so much more efficient for them to let you skip the >>> line and just check out, especially since you’re in a hurry, but they’re >>> rudely refusing >>> >> >>> >> I think the person with the cheetos pulling out a gun and shooting >>> >> everyone in front of him (AQM) would not go down well. >>> >> >>> >> > > to go back to our grocery store analogy this would be like if a >>> worker saw you standing at the back ... and either let you skip to the >>> front of the line or opens up an express lane just for you >>> >> >>> >> Actually that analogy is fairly close to fair queuing. The multiple >>> >> checker analogy is one of the most common analogies in queue theory >>> >> itself. >>> >> >>> >> > >>> >> > The video describes the problem of bufferbloat, and then describes >>> the same failed solution that hasn’t worked for the last three decades. >>> >> >>> >> Hmm? It establishes the scenario, explains the problem *quickly*, >>> >> disses gamer routers for not getting it right.. *points to an >>> >> accurate test*, and then to the ideas and products that *actually >>> >> work* with "smart queueing", with a screenshot of the most common >>> >> (eero's optimize for gaming and videoconferencing), and fq_codel and >>> >> cake *by name*, and points folk at the best known solution available, >>> >> openwrt. >>> >> >>> >> Bing, baddabang, boom. Also the comments were revealing. A goodly >>> >> percentage already knew the problem, more than a few were inspired to >>> >> take the test, >>> >> there was a whole bunch of "Aha!" success stories and 360k views, >>> >> which is more people than we've ever been able to reach in for >>> >> example, a nanog conference. >>> >> >>> >> I loved that folk taking the test actually had quite a few A results, >>> >> without having had to do anything. At least some ISPs are getting it >>> >> more right now! >>> >> >>> >> At this point I think gamers in particular know what "brands" we've >>> >> tried to establish - "Smart queues", "SQM", "OpenWrt", fq_codel and >>> >> now "cake" are "good" things to have, and are stimulating demand by >>> >> asking for them, It's certainly working out better and better for >>> >> evenroute, firewalla, ubnt and others, and I saw an uptick in >>> >> questions about this on various user forums. >>> >> >>> >> I even like that there's a backlash now of people saying "fixing >>> >> bufferbloat doesn't solve everything" - >>> >> >>> >> > Describing the obvious simple-minded (wrong) solution that any >>> normal person would think of based on their personal human experience >>> waiting in grocery stores and airports, is not describing the solution to >>> bufferbloat. The solution to bufferbloat is not that if you are privileged >>> then you get to “skip to the front of the line”. The solution to >>> bufferbloat is that there is no line! >>> >> >>> >> I like the idea of a guru floating above a grocery cart with a better >>> >> string of explanations, explaining >>> >> >>> >> - "no, grasshopper, the solution to bufferbloat is no line... at >>> all". >>> >> >>> >> > >>> >> > With grocery stores and airports people’s arrivals are independent >>> and not controlled. There is no way for a grocery store or airport to >>> generate backpressure to tell people to wait at home when a queue begins to >>> form. The key to solving bufferbloat is generating timely backpressure to >>> prevent the queue forming in the first place, not accepting a huge queue >>> and then deciding who deserves special treatment to get better service than >>> all the other peons who still have to wait in a long queue, just like >>> before. >>> >> >>> >> I am not huge on the word "backpressure" here. Needs to signal the >>> >> other side to slow down, is more accurate. So might say timely >>> >> signalling rather than timely backpressure? >>> >> >>> >> Other feedback I got was that the video was too smarmy (I agree), >>> >> different audiences than gamers need different forms of outreach... >>> >> >>> >> but to me, winning the gamers has always been one of the most >>> >> important things, as they make a lot of buying decisions, and they >>> >> benefit the most for >>> >> fq and packet prioritization as we do today in gamer routers and in >>> >> cake + qosify. >>> >> >>> >> maybe that gets in the way of more serious markets. Certainly I would >>> >> like another video explaining what goes wrong with videoconferencing. >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> > >>> >> > Stuart Cheshire >>> >> > >>> >> >>> >> >>> >> -- >>> >> This song goes out to all the folk that thought Stadia would work: >>> >> >>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz >>> >> Dave Täht CEO, TekLibre, LLC >>> >> _______________________________________________ >>> >> Bloat mailing list >>> >> Bloat@lists.bufferbloat.net >>> >> https://lists.bufferbloat.net/listinfo/bloat >>> >>> >>> >>> -- >>> This song goes out to all the folk that thought Stadia would work: >>> >>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz >>> Dave Täht CEO, TekLibre, LLC >>> >>> >>> -- >>> This song goes out to all the folk that thought Stadia would work: >>> >>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz >>> Dave Täht CEO, TekLibre, LLC >>> _______________________________________________ >>> LibreQoS mailing list >>> LibreQoS@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/libreqos >>> >> _______________________________________________ >> LibreQoS mailing list >> LibreQoS@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/libreqos >> > > > -- > Robert Chacón > CEO | JackRabbit Wireless LLC <http://jackrabbitwireless.com> > Dev | LibreQoS.io > > _______________________________________________ > LibreQoS mailing list > LibreQoS@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/libreqos > [-- Attachment #2: Type: text/html, Size: 23194 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [LibreQoS] Fwd: A quick report from the WISPA conference 2022-11-12 17:19 ` Herbert Wolverson @ 2022-11-12 17:39 ` dan 2022-11-12 19:30 ` [LibreQoS] Various aspects of net neutrality Dave Taht 2022-11-13 6:13 ` [LibreQoS] Fwd: A quick report from the WISPA conference Hayden Simon 0 siblings, 2 replies; 10+ messages in thread From: dan @ 2022-11-12 17:39 UTC (permalink / raw) To: Herbert Wolverson; +Cc: libreqos [-- Attachment #1: Type: text/plain, Size: 20336 bytes --] I have to argue with you on 'equally limited' streaming being ok. That allows for 'internet-like' service but no/poor streaming, a service that clamps down really hard on streaming and then offers you an upgrade to make it work. Neutrality doesn't mean 'kind of' neutral. If it's called 'Internet' then no specific service or even type of service should be limited for the provider's direct benefit or whims. I can see no positive case for a provider limiting any service/type below the customers quoted service. The provider gets to save money, but since they offer an inferior product that is essentially false advertised and customers have no way to know this then capitalist tools like consumer choice can't work. Consumers don't know they can get a better suited service, they don't show demand in a market, that prevents competition from having an opportunity, etc. Most people have just 1 fast internet option. Does a customer that can't figure out why their new Roku 4K on their new 4K TV refuses to stream 4K Netflix on their 100M internet service say "Netflix sucks!". or Roku sucks or Sony sucks or whatever? So this spills over into making other services take the blame for bad connections. On Sat, Nov 12, 2022 at 10:19 AM Herbert Wolverson via LibreQoS < libreqos@lists.bufferbloat.net> wrote: > Net neutrality is complex (I'm generally in favor of the concept). There's > nothing stopping an ISP from having rules that video streaming be limited - > but ALL video streaming has to be equally limited. So ComCast could use > Sandvine, but only without any vendor-specific limitations. Same for > Cambium. Limit streams that look like video, and you're fine. With that > said, it runs into something I say a lot: don't penalize your customers for > using your product. If you're buying a 100mbit/s pipe, hooking it to your > fancy 4k streaming system - it's entirely reasonable to expect sharp 4k > video. In fact, your shaping emphasis should be on keeping their video nice > and watchable even though someone just decided to download the newest Call > of Duty, update their xbox, and update the firmware on their smart fridge. > Likewise, if someone really wants the smallest 5 mbit connection you once > offered - because they only ever use it to check email (we actually had a > customer complain when we offered them a free upgrade from a 5mbit/s plan, > years ago!) - you want to do your best to make it a usable tiny plan for > them. Fair queuing (along with a bit of user education; we eventually > convinced the customer that "no change in fees" actually meant not charging > them more money) helps with all of that. > > It can win customers, too. We have a couple of business customers who > switched us from their backup to their primary because "our service felt > snappier". We had a business go with us because their 24x7 lobby streaming > video looked better on our (smaller) connection, and a couple of > fraternities who likewise decided to use us because their overkill 4k video > setup looked better (even with the 20+ Xboxes we could see on their > network). > > QUIC makes me chuckle, because it's *exactly* what we were doing in > gamedev land in the late 90s to make deathmatches run smoothly. (Seriously, > QuakeWorld and the original Unreal Tournament had the most amazing network > code; they basically implemented what is now known as "reliable UDP" to > incredible results). > > With that said, I think we've got a remarkable amount of wiggle-room to > make things better (and it's not at all shoddy right now!). XDP is amazing > as-is, and is improving fast. (I'm currently reading > https://github.com/ilejn/xdpbridge - an older project that doesn't seem > to have gained much traction, but I'm wondering if it couldn't provide some > seriously fast bridge offloading; I'll have to setup a better test > environment to find out) > > On Sat, Nov 12, 2022 at 10:09 AM Robert Chacón via LibreQoS < > libreqos@lists.bufferbloat.net> wrote: > >> Thank you, Dan. >> I completely agree with you 100% there. It's a risky way to save a tiny >> amount of money on bandwidth costs. >> Comcast paid out $16 million in a settlement >> <https://arstechnica.com/tech-policy/2010/04/comcast-owes-p2p-users-16-yes-they-should-take-it/?comments=1&post=20329187> >> over using Sandvine's DPI in this way. >> Since then, Comcast has switched from DPI to fair queueing with >> DOCSIS-pie >> <https://www.reddit.com/r/eero/comments/rc8e2j/getting_a_bufferbloat_grade_with_comcast_aqm/> >> . >> >> California's SB-822 >> <https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=201720180SB822&fbclid=IwAR2LmNyy5ZkbtzayIyNerJ-Q_OUwK4G0H4aYv4XZwVNz-QF41d5v0BiST-E> >> will be used as a template by many other states going forward on Net >> Neutrality. >> <https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=201720180SB822&fbclid=IwAR2LmNyy5ZkbtzayIyNerJ-Q_OUwK4G0H4aYv4XZwVNz-QF41d5v0BiST-E> >> SB-822 even clarified that "Reasonable Network Management" needs to be >> "as application-agnostic as possible". >> These DPI approaches like Cambium's will not hold up in court the way >> Fair Queuing (Preseem, LibreQoS) based solutions will. >> We can point to years of public research and RFCs when defending a Fair >> Queuing QoE approach. >> They can't so easily defend using a product that advertised its big >> "scale down 4k to save money" knob or "accelerate their speeds only when >> they run speed tests" button. >> It's concerning to me that these QoE vendors are not disclosing "hey, >> don't use this in NN states like California please". >> The WISP industry will be hurt long-term by these products with consumers >> viewing us as an inferior technology that limits their usage in intrusive >> ways. >> Like you said, let's let them use their plan however they want. It >> doesn't hurt us. But things like capping end-users at HD will hurt us >> long-term in lost sales. >> >> Regarding TCP acceleration, I'm leaning toward the argument made by Dan >> from Preseem <https://youtu.be/3r4FWGKho3c?t=1879> that QUIC adoption is >> growing pretty quickly and we may not want to mess with TCP at the expense >> of UDP given the rise of QUIC. >> Cloudlfare says HTTP/3 and QUIC are already at 28% and growing >> <https://radar.cloudflare.com/adoption-and-usage?range=28d>. >> One point he brought up is that if we use TCP acceleration, and a >> middlebox has to be rebooted or something - all client TCP sessions will >> break when OSPF switches paths around the box. >> >> On Sat, Nov 12, 2022 at 8:36 AM dan via LibreQoS < >> libreqos@lists.bufferbloat.net> wrote: >> >>> I think there's a thin grey line between what other vendors (paraqum, >>> bequant/cambium) and libreQoS is trying to do. The first two are REALLY >>> pushing this selecting narrowing of traffic and they'll go straight after >>> streaming for their demos. Most people that drop in the cambium QoE >>> appliance turn the knobs on Netflix down and then praise the 40% savings on >>> their network. Same with PQ. I think this is fundamentally wrong. This >>> is the entire reason Net Neutrality keeps popping up. Why does the ISP get >>> to say "You only need 25Mbps for your netflix". That's double wrong if >>> they offer their own video services. It's not about improving the >>> customer's experience, it's about spending less on upstream bandwidth at >>> the customer's expense. As soon as they say 'pays for itself in bandwidth >>> savings' you know what the product is primarily for. >>> >>> This is why I like the preseem and libreqos model of just using a >>> 'stock' cake and just finessing the data coming in to keep the pipe from >>> clogging. Who cares if someone pulls 90M bursts of apple TV+ on their 100M >>> pipe. Only dial that back so other requests they make get through cleanly. >>> >>> preseem is going more towards wireless monitoring etc and kinda letting >>> shaping be a secondary.... if they were doing more libreqos like stuff I'd >>> just stick with preseem. >>> >>> libreQoS may end up being the best and most net neutral product you can >>> get. The only current disadvantage (ignoring DPI) is lack of TCP >>> accelleration that PQ and Bequant/cambium have. >>> >>> I should add that 3 states already have laws on the books that straight >>> up make PQ and Cambium's solution illegal because they target certain >>> services for throttling. My state (Montana) has the same rule except only >>> when working with a government contract. Libreqos is actually compliant >>> (as is preseem). >>> >>> On Sat, Nov 12, 2022 at 8:11 AM Dave Taht via LibreQoS < >>> libreqos@lists.bufferbloat.net> wrote: >>> >>>> this report predates the libreqos list... >>>> >>>> ---------- Forwarded message --------- >>>> From: Dave Taht <dave.taht@gmail.com> >>>> Date: Mon, Oct 17, 2022 at 8:15 PM >>>> Subject: A quick report from the WISPA conference >>>> To: Sina Khanifar <sina@waveform.com> >>>> Cc: Cake List <cake@lists.bufferbloat.net>, Make-Wifi-fast >>>> <make-wifi-fast@lists.bufferbloat.net>, Rpm >>>> <rpm@lists.bufferbloat.net>, Stuart Cheshire <cheshire@apple.com>, >>>> bloat <bloat@lists.bufferbloat.net> >>>> >>>> >>>> On Mon, Oct 17, 2022 at 7:51 PM Sina Khanifar <sina@waveform.com> >>>> wrote: >>>> > >>>> > Positive or negative, I can claim a bit of credit for this video :). >>>> We've been working with LTT on a few projects and we pitched them on doing >>>> something around bufferbloat. We've seen more traffic to our Waveforn test >>>> than ever before, which has been fun! >>>> >>>> Thank you. Great job with that video! And waveform has become the goto >>>> site for many now. >>>> >>>> I can't help but wonder tho... are you collecting any statistics, over >>>> time, as to how much better the problem is getting? >>>> >>>> And any chance they could do something similar explaining wifi? >>>> >>>> ... >>>> >>>> I was just at WISPA conference week before last. Preseem's booth >>>> (fq_codel) was always packed. Vilo living had put cake in their wifi 6 >>>> product. A >>>> keynote speaker had deployed it and talked about it with waveform >>>> results on the big screen (2k people there). A large wireless vendor >>>> demo'd privately to me their flent results before/after cake on their >>>> next-gen radios... and people dissed tarana without me prompting for >>>> their bad bufferbloat... and the best thing of all that happened to me >>>> was... besides getting a hug from a young lady (megan) who'd salvaged >>>> her schooling in alaska using sqm - I walked up to the paraqum booth >>>> (another large QoE middlebox maker centered more in india) and asked. >>>> >>>> "So... do y'all have fq_codel yet?" >>>> >>>> And they smiled and said: "No, we have something better... we've got >>>> cake." >>>> >>>> "Cake? What's that?" - I said, innocently. >>>> >>>> They then stepped me through their 200Gbps (!!) product, which uses a >>>> bunch of offloads, and can track rtt down to a ms with the intel >>>> ethernet card they were using. They'd modifed cake to provide 16 (?) >>>> levels of service, and were running under dpdk (I am not sure if cake >>>> was). It was a great, convincing pitch... >>>> >>>> ... then I told 'em who I was. There's a video of the in-both concert >>>> after. >>>> >>>> ... >>>> >>>> The downside to me (and the subject of my talk) was that in nearly >>>> every person I talked to, fq_codel was viewed as a means to better >>>> subscriber bandwidth plan enforcement (which is admittedly the market >>>> that preseem pioneered) and it was not understood that I'd got >>>> involved in this whole thing because I'd wanted an algorithm to deal >>>> with "rain fade", running directly on the radios. People wanted to use >>>> the statistics on the radios to drive the plan enforcement better >>>> (which is an ok approach, I guess), and for 10+ I'd been whinging >>>> about the... physics. >>>> >>>> So I ranted about rfc7567 a lot and begged people now putting routerOS >>>> 7.2 and later out there (mikrotik is huge in this market), to kill >>>> their fifos and sfqs at the native rates of the interfaces... and >>>> watch their network improve that way also. >>>> >>>> I think one more wispa conference will be a clean sweep of everyone in >>>> the fixed wireless market to not only adopt these algorithms for plan >>>> enforcement, but even more directly on the radios and more CPE. >>>> >>>> I also picked up enough consulting business to keep me busy the rest >>>> of this year, and possibly more than I can handle (anybody looking?) >>>> >>>> I wonder what will happen at a fiber conference? >>>> >>>> > On Mon, Oct 17, 2022 at 7:45 PM Dave Taht via Bloat < >>>> bloat@lists.bufferbloat.net> wrote: >>>> >> >>>> >> On Mon, Oct 17, 2022 at 5:02 PM Stuart Cheshire <cheshire@apple.com> >>>> wrote: >>>> >> > >>>> >> > On 9 Oct 2022, at 06:14, Dave Taht via Make-wifi-fast < >>>> make-wifi-fast@lists.bufferbloat.net> wrote: >>>> >> > >>>> >> > > This was so massively well done, I cried. Does anyone know how >>>> to get in touch with the ifxit folk? >>>> >> > > >>>> >> > > https://www.youtube.com/watch?v=UICh3ScfNWI >>>> >> > >>>> >> > I’m surprised that you liked this video. It seems to me that it >>>> repeats all the standard misinformation. The analogy they use is the >>>> standard terrible example of waiting in a long line at a grocery store, and >>>> the “solution” is letting certain traffic “jump the line, angering everyone >>>> behind them”. >>>> >> >>>> >> Accuracy be damned. The analogy to common experience resonates more. >>>> >> >>>> >> > >>>> >> > Some quotes from the video: >>>> >> > >>>> >> > > it would be so much more efficient for them to let you skip the >>>> line and just check out, especially since you’re in a hurry, but they’re >>>> rudely refusing >>>> >> >>>> >> I think the person with the cheetos pulling out a gun and shooting >>>> >> everyone in front of him (AQM) would not go down well. >>>> >> >>>> >> > > to go back to our grocery store analogy this would be like if a >>>> worker saw you standing at the back ... and either let you skip to the >>>> front of the line or opens up an express lane just for you >>>> >> >>>> >> Actually that analogy is fairly close to fair queuing. The multiple >>>> >> checker analogy is one of the most common analogies in queue theory >>>> >> itself. >>>> >> >>>> >> > >>>> >> > The video describes the problem of bufferbloat, and then describes >>>> the same failed solution that hasn’t worked for the last three decades. >>>> >> >>>> >> Hmm? It establishes the scenario, explains the problem *quickly*, >>>> >> disses gamer routers for not getting it right.. *points to an >>>> >> accurate test*, and then to the ideas and products that *actually >>>> >> work* with "smart queueing", with a screenshot of the most common >>>> >> (eero's optimize for gaming and videoconferencing), and fq_codel and >>>> >> cake *by name*, and points folk at the best known solution available, >>>> >> openwrt. >>>> >> >>>> >> Bing, baddabang, boom. Also the comments were revealing. A goodly >>>> >> percentage already knew the problem, more than a few were inspired to >>>> >> take the test, >>>> >> there was a whole bunch of "Aha!" success stories and 360k views, >>>> >> which is more people than we've ever been able to reach in for >>>> >> example, a nanog conference. >>>> >> >>>> >> I loved that folk taking the test actually had quite a few A results, >>>> >> without having had to do anything. At least some ISPs are getting it >>>> >> more right now! >>>> >> >>>> >> At this point I think gamers in particular know what "brands" we've >>>> >> tried to establish - "Smart queues", "SQM", "OpenWrt", fq_codel and >>>> >> now "cake" are "good" things to have, and are stimulating demand by >>>> >> asking for them, It's certainly working out better and better for >>>> >> evenroute, firewalla, ubnt and others, and I saw an uptick in >>>> >> questions about this on various user forums. >>>> >> >>>> >> I even like that there's a backlash now of people saying "fixing >>>> >> bufferbloat doesn't solve everything" - >>>> >> >>>> >> > Describing the obvious simple-minded (wrong) solution that any >>>> normal person would think of based on their personal human experience >>>> waiting in grocery stores and airports, is not describing the solution to >>>> bufferbloat. The solution to bufferbloat is not that if you are privileged >>>> then you get to “skip to the front of the line”. The solution to >>>> bufferbloat is that there is no line! >>>> >> >>>> >> I like the idea of a guru floating above a grocery cart with a better >>>> >> string of explanations, explaining >>>> >> >>>> >> - "no, grasshopper, the solution to bufferbloat is no line... at >>>> all". >>>> >> >>>> >> > >>>> >> > With grocery stores and airports people’s arrivals are independent >>>> and not controlled. There is no way for a grocery store or airport to >>>> generate backpressure to tell people to wait at home when a queue begins to >>>> form. The key to solving bufferbloat is generating timely backpressure to >>>> prevent the queue forming in the first place, not accepting a huge queue >>>> and then deciding who deserves special treatment to get better service than >>>> all the other peons who still have to wait in a long queue, just like >>>> before. >>>> >> >>>> >> I am not huge on the word "backpressure" here. Needs to signal the >>>> >> other side to slow down, is more accurate. So might say timely >>>> >> signalling rather than timely backpressure? >>>> >> >>>> >> Other feedback I got was that the video was too smarmy (I agree), >>>> >> different audiences than gamers need different forms of outreach... >>>> >> >>>> >> but to me, winning the gamers has always been one of the most >>>> >> important things, as they make a lot of buying decisions, and they >>>> >> benefit the most for >>>> >> fq and packet prioritization as we do today in gamer routers and in >>>> >> cake + qosify. >>>> >> >>>> >> maybe that gets in the way of more serious markets. Certainly I would >>>> >> like another video explaining what goes wrong with videoconferencing. >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> > >>>> >> > Stuart Cheshire >>>> >> > >>>> >> >>>> >> >>>> >> -- >>>> >> This song goes out to all the folk that thought Stadia would work: >>>> >> >>>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz >>>> >> Dave Täht CEO, TekLibre, LLC >>>> >> _______________________________________________ >>>> >> Bloat mailing list >>>> >> Bloat@lists.bufferbloat.net >>>> >> https://lists.bufferbloat.net/listinfo/bloat >>>> >>>> >>>> >>>> -- >>>> This song goes out to all the folk that thought Stadia would work: >>>> >>>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz >>>> Dave Täht CEO, TekLibre, LLC >>>> >>>> >>>> -- >>>> This song goes out to all the folk that thought Stadia would work: >>>> >>>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz >>>> Dave Täht CEO, TekLibre, LLC >>>> _______________________________________________ >>>> LibreQoS mailing list >>>> LibreQoS@lists.bufferbloat.net >>>> https://lists.bufferbloat.net/listinfo/libreqos >>>> >>> _______________________________________________ >>> LibreQoS mailing list >>> LibreQoS@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/libreqos >>> >> >> >> -- >> Robert Chacón >> CEO | JackRabbit Wireless LLC <http://jackrabbitwireless.com> >> Dev | LibreQoS.io >> >> _______________________________________________ >> LibreQoS mailing list >> LibreQoS@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/libreqos >> > _______________________________________________ > LibreQoS mailing list > LibreQoS@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/libreqos > [-- Attachment #2: Type: text/html, Size: 25272 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* [LibreQoS] Various aspects of net neutrality 2022-11-12 17:39 ` dan @ 2022-11-12 19:30 ` Dave Taht 2022-11-12 19:58 ` dan 2022-11-13 6:13 ` [LibreQoS] Fwd: A quick report from the WISPA conference Hayden Simon 1 sibling, 1 reply; 10+ messages in thread From: Dave Taht @ 2022-11-12 19:30 UTC (permalink / raw) To: dan; +Cc: Herbert Wolverson, libreqos, Sascha Meinrath Changing the subject line. This is a very nuanced topic and I would rather enjoy the day AFK! To stir the pot, tho: I have had a cable industry insider claim that fq_codel violated the california code you cite. Wrote about it here: https://blog.cerowrt.org/post/net_neutrality_customers/ And taking on the other side: https://blog.cerowrt.org/post/net_neutrality_isps/ I have asked multiple lawyers about it since, and the rules need to be clarified, here. The level of doublethink in the cable/telco industry about NN has to be experienced to be believed. Just spent 6 years losing a fight about it on l4s and related, so now they can hide behind the IETFs skirts now. I didn't make friends with any side that way, and of my relationship from that period, with the eff and the ACLU, I don't know... but I did write a nice piece of rhetoric: "fq codel (now IETF standard RFC8290) is a uniquely “American” algorithm. It gives the “little guy” - the little packet, the first packets in a new connection to anywhere, a little boost until the flow achieves parity with other flows from other sources, with minimal buffering. This means that all network traffic gets treated equally - faster. Isn’t that what you want in a network neutral framework? DNS, gaming traffic, voip, videoconferencing, and the first packets of any new flow, to anywhere, get a small boost. That’s it. Big flows - from anybody - from netflix to google to comcast - all achieve parity, with minimal delay and buffering, at a wide variety of real round trip times." Just to throw another bit of controversial thought before I gaily depart for a nice lunch... I tended to support WISPA's non-defense of NN in multiple respects. Title II, and additional regulation seemed worse, then, than having some mildly chastized ISPs remain free to do whatever they wanted. With my brutal experiences with politics starting with the clipper chip, the CDA, SOPA, network neutrality, and of late the NTIA broadband funds act.... My basic conclusion was this: There must be a middle path, and it starts with less lawyers being in charge at the FCC and extends to, in general, less politicians on any side of the debate, making policy and slogans that don't line up with reality. "For a successful technology .... " - richard feynman. On Sat, Nov 12, 2022 at 9:39 AM dan via LibreQoS <libreqos@lists.bufferbloat.net> wrote: > > I have to argue with you on 'equally limited' streaming being ok. That allows for 'internet-like' service but no/poor streaming, a service that clamps down really hard on streaming and then offers you an upgrade to make it work. Neutrality doesn't mean 'kind of' neutral. If it's called 'Internet' then no specific service or even type of service should be limited for the provider's direct benefit or whims. I can see no positive case for a provider limiting any service/type below the customers quoted service. The provider gets to save money, but since they offer an inferior product that is essentially false advertised and customers have no way to know this then capitalist tools like consumer choice can't work. Consumers don't know they can get a better suited service, they don't show demand in a market, that prevents competition from having an opportunity, etc. Most people have just 1 fast internet option. > > Does a customer that can't figure out why their new Roku 4K on their new 4K TV refuses to stream 4K Netflix on their 100M internet service say "Netflix sucks!". or Roku sucks or Sony sucks or whatever? So this spills over into making other services take the blame for bad connections. > > > > > On Sat, Nov 12, 2022 at 10:19 AM Herbert Wolverson via LibreQoS <libreqos@lists.bufferbloat.net> wrote: >> >> Net neutrality is complex (I'm generally in favor of the concept). There's nothing stopping an ISP from having rules that video streaming be limited - but ALL video streaming has to be equally limited. So ComCast could use Sandvine, but only without any vendor-specific limitations. Same for Cambium. Limit streams that look like video, and you're fine. With that said, it runs into something I say a lot: don't penalize your customers for using your product. If you're buying a 100mbit/s pipe, hooking it to your fancy 4k streaming system - it's entirely reasonable to expect sharp 4k video. In fact, your shaping emphasis should be on keeping their video nice and watchable even though someone just decided to download the newest Call of Duty, update their xbox, and update the firmware on their smart fridge. Likewise, if someone really wants the smallest 5 mbit connection you once offered - because they only ever use it to check email (we actually had a customer complain when we offered them a free upgrade from a 5mbit/s plan, years ago!) - you want to do your best to make it a usable tiny plan for them. Fair queuing (along with a bit of user education; we eventually convinced the customer that "no change in fees" actually meant not charging them more money) helps with all of that. >> >> It can win customers, too. We have a couple of business customers who switched us from their backup to their primary because "our service felt snappier". We had a business go with us because their 24x7 lobby streaming video looked better on our (smaller) connection, and a couple of fraternities who likewise decided to use us because their overkill 4k video setup looked better (even with the 20+ Xboxes we could see on their network). >> >> QUIC makes me chuckle, because it's exactly what we were doing in gamedev land in the late 90s to make deathmatches run smoothly. (Seriously, QuakeWorld and the original Unreal Tournament had the most amazing network code; they basically implemented what is now known as "reliable UDP" to incredible results). >> >> With that said, I think we've got a remarkable amount of wiggle-room to make things better (and it's not at all shoddy right now!). XDP is amazing as-is, and is improving fast. (I'm currently reading https://github.com/ilejn/xdpbridge - an older project that doesn't seem to have gained much traction, but I'm wondering if it couldn't provide some seriously fast bridge offloading; I'll have to setup a better test environment to find out) >> >> On Sat, Nov 12, 2022 at 10:09 AM Robert Chacón via LibreQoS <libreqos@lists.bufferbloat.net> wrote: >>> >>> Thank you, Dan. >>> I completely agree with you 100% there. It's a risky way to save a tiny amount of money on bandwidth costs. >>> Comcast paid out $16 million in a settlement over using Sandvine's DPI in this way. >>> Since then, Comcast has switched from DPI to fair queueing with DOCSIS-pie. >>> >>> California's SB-822 will be used as a template by many other states going forward on Net Neutrality. >>> SB-822 even clarified that "Reasonable Network Management" needs to be "as application-agnostic as possible". >>> These DPI approaches like Cambium's will not hold up in court the way Fair Queuing (Preseem, LibreQoS) based solutions will. >>> We can point to years of public research and RFCs when defending a Fair Queuing QoE approach. >>> They can't so easily defend using a product that advertised its big "scale down 4k to save money" knob or "accelerate their speeds only when they run speed tests" button. >>> It's concerning to me that these QoE vendors are not disclosing "hey, don't use this in NN states like California please". >>> The WISP industry will be hurt long-term by these products with consumers viewing us as an inferior technology that limits their usage in intrusive ways. >>> Like you said, let's let them use their plan however they want. It doesn't hurt us. But things like capping end-users at HD will hurt us long-term in lost sales. >>> >>> Regarding TCP acceleration, I'm leaning toward the argument made by Dan from Preseem that QUIC adoption is growing pretty quickly and we may not want to mess with TCP at the expense of UDP given the rise of QUIC. >>> Cloudlfare says HTTP/3 and QUIC are already at 28% and growing. >>> One point he brought up is that if we use TCP acceleration, and a middlebox has to be rebooted or something - all client TCP sessions will break when OSPF switches paths around the box. >>> >>> On Sat, Nov 12, 2022 at 8:36 AM dan via LibreQoS <libreqos@lists.bufferbloat.net> wrote: >>>> >>>> I think there's a thin grey line between what other vendors (paraqum, bequant/cambium) and libreQoS is trying to do. The first two are REALLY pushing this selecting narrowing of traffic and they'll go straight after streaming for their demos. Most people that drop in the cambium QoE appliance turn the knobs on Netflix down and then praise the 40% savings on their network. Same with PQ. I think this is fundamentally wrong. This is the entire reason Net Neutrality keeps popping up. Why does the ISP get to say "You only need 25Mbps for your netflix". That's double wrong if they offer their own video services. It's not about improving the customer's experience, it's about spending less on upstream bandwidth at the customer's expense. As soon as they say 'pays for itself in bandwidth savings' you know what the product is primarily for. >>>> >>>> This is why I like the preseem and libreqos model of just using a 'stock' cake and just finessing the data coming in to keep the pipe from clogging. Who cares if someone pulls 90M bursts of apple TV+ on their 100M pipe. Only dial that back so other requests they make get through cleanly. >>>> >>>> preseem is going more towards wireless monitoring etc and kinda letting shaping be a secondary.... if they were doing more libreqos like stuff I'd just stick with preseem. >>>> >>>> libreQoS may end up being the best and most net neutral product you can get. The only current disadvantage (ignoring DPI) is lack of TCP accelleration that PQ and Bequant/cambium have. >>>> >>>> I should add that 3 states already have laws on the books that straight up make PQ and Cambium's solution illegal because they target certain services for throttling. My state (Montana) has the same rule except only when working with a government contract. Libreqos is actually compliant (as is preseem). >>>> >>>> On Sat, Nov 12, 2022 at 8:11 AM Dave Taht via LibreQoS <libreqos@lists.bufferbloat.net> wrote: >>>>> >>>>> this report predates the libreqos list... >>>>> >>>>> ---------- Forwarded message --------- >>>>> From: Dave Taht <dave.taht@gmail.com> >>>>> Date: Mon, Oct 17, 2022 at 8:15 PM >>>>> Subject: A quick report from the WISPA conference >>>>> To: Sina Khanifar <sina@waveform.com> >>>>> Cc: Cake List <cake@lists.bufferbloat.net>, Make-Wifi-fast >>>>> <make-wifi-fast@lists.bufferbloat.net>, Rpm >>>>> <rpm@lists.bufferbloat.net>, Stuart Cheshire <cheshire@apple.com>, >>>>> bloat <bloat@lists.bufferbloat.net> >>>>> >>>>> >>>>> On Mon, Oct 17, 2022 at 7:51 PM Sina Khanifar <sina@waveform.com> wrote: >>>>> > >>>>> > Positive or negative, I can claim a bit of credit for this video :). We've been working with LTT on a few projects and we pitched them on doing something around bufferbloat. We've seen more traffic to our Waveforn test than ever before, which has been fun! >>>>> >>>>> Thank you. Great job with that video! And waveform has become the goto >>>>> site for many now. >>>>> >>>>> I can't help but wonder tho... are you collecting any statistics, over >>>>> time, as to how much better the problem is getting? >>>>> >>>>> And any chance they could do something similar explaining wifi? >>>>> >>>>> ... >>>>> >>>>> I was just at WISPA conference week before last. Preseem's booth >>>>> (fq_codel) was always packed. Vilo living had put cake in their wifi 6 >>>>> product. A >>>>> keynote speaker had deployed it and talked about it with waveform >>>>> results on the big screen (2k people there). A large wireless vendor >>>>> demo'd privately to me their flent results before/after cake on their >>>>> next-gen radios... and people dissed tarana without me prompting for >>>>> their bad bufferbloat... and the best thing of all that happened to me >>>>> was... besides getting a hug from a young lady (megan) who'd salvaged >>>>> her schooling in alaska using sqm - I walked up to the paraqum booth >>>>> (another large QoE middlebox maker centered more in india) and asked. >>>>> >>>>> "So... do y'all have fq_codel yet?" >>>>> >>>>> And they smiled and said: "No, we have something better... we've got cake." >>>>> >>>>> "Cake? What's that?" - I said, innocently. >>>>> >>>>> They then stepped me through their 200Gbps (!!) product, which uses a >>>>> bunch of offloads, and can track rtt down to a ms with the intel >>>>> ethernet card they were using. They'd modifed cake to provide 16 (?) >>>>> levels of service, and were running under dpdk (I am not sure if cake >>>>> was). It was a great, convincing pitch... >>>>> >>>>> ... then I told 'em who I was. There's a video of the in-both concert after. >>>>> >>>>> ... >>>>> >>>>> The downside to me (and the subject of my talk) was that in nearly >>>>> every person I talked to, fq_codel was viewed as a means to better >>>>> subscriber bandwidth plan enforcement (which is admittedly the market >>>>> that preseem pioneered) and it was not understood that I'd got >>>>> involved in this whole thing because I'd wanted an algorithm to deal >>>>> with "rain fade", running directly on the radios. People wanted to use >>>>> the statistics on the radios to drive the plan enforcement better >>>>> (which is an ok approach, I guess), and for 10+ I'd been whinging >>>>> about the... physics. >>>>> >>>>> So I ranted about rfc7567 a lot and begged people now putting routerOS >>>>> 7.2 and later out there (mikrotik is huge in this market), to kill >>>>> their fifos and sfqs at the native rates of the interfaces... and >>>>> watch their network improve that way also. >>>>> >>>>> I think one more wispa conference will be a clean sweep of everyone in >>>>> the fixed wireless market to not only adopt these algorithms for plan >>>>> enforcement, but even more directly on the radios and more CPE. >>>>> >>>>> I also picked up enough consulting business to keep me busy the rest >>>>> of this year, and possibly more than I can handle (anybody looking?) >>>>> >>>>> I wonder what will happen at a fiber conference? >>>>> >>>>> > On Mon, Oct 17, 2022 at 7:45 PM Dave Taht via Bloat <bloat@lists.bufferbloat.net> wrote: >>>>> >> >>>>> >> On Mon, Oct 17, 2022 at 5:02 PM Stuart Cheshire <cheshire@apple.com> wrote: >>>>> >> > >>>>> >> > On 9 Oct 2022, at 06:14, Dave Taht via Make-wifi-fast <make-wifi-fast@lists.bufferbloat.net> wrote: >>>>> >> > >>>>> >> > > This was so massively well done, I cried. Does anyone know how to get in touch with the ifxit folk? >>>>> >> > > >>>>> >> > > https://www.youtube.com/watch?v=UICh3ScfNWI >>>>> >> > >>>>> >> > I’m surprised that you liked this video. It seems to me that it repeats all the standard misinformation. The analogy they use is the standard terrible example of waiting in a long line at a grocery store, and the “solution” is letting certain traffic “jump the line, angering everyone behind them”. >>>>> >> >>>>> >> Accuracy be damned. The analogy to common experience resonates more. >>>>> >> >>>>> >> > >>>>> >> > Some quotes from the video: >>>>> >> > >>>>> >> > > it would be so much more efficient for them to let you skip the line and just check out, especially since you’re in a hurry, but they’re rudely refusing >>>>> >> >>>>> >> I think the person with the cheetos pulling out a gun and shooting >>>>> >> everyone in front of him (AQM) would not go down well. >>>>> >> >>>>> >> > > to go back to our grocery store analogy this would be like if a worker saw you standing at the back ... and either let you skip to the front of the line or opens up an express lane just for you >>>>> >> >>>>> >> Actually that analogy is fairly close to fair queuing. The multiple >>>>> >> checker analogy is one of the most common analogies in queue theory >>>>> >> itself. >>>>> >> >>>>> >> > >>>>> >> > The video describes the problem of bufferbloat, and then describes the same failed solution that hasn’t worked for the last three decades. >>>>> >> >>>>> >> Hmm? It establishes the scenario, explains the problem *quickly*, >>>>> >> disses gamer routers for not getting it right.. *points to an >>>>> >> accurate test*, and then to the ideas and products that *actually >>>>> >> work* with "smart queueing", with a screenshot of the most common >>>>> >> (eero's optimize for gaming and videoconferencing), and fq_codel and >>>>> >> cake *by name*, and points folk at the best known solution available, >>>>> >> openwrt. >>>>> >> >>>>> >> Bing, baddabang, boom. Also the comments were revealing. A goodly >>>>> >> percentage already knew the problem, more than a few were inspired to >>>>> >> take the test, >>>>> >> there was a whole bunch of "Aha!" success stories and 360k views, >>>>> >> which is more people than we've ever been able to reach in for >>>>> >> example, a nanog conference. >>>>> >> >>>>> >> I loved that folk taking the test actually had quite a few A results, >>>>> >> without having had to do anything. At least some ISPs are getting it >>>>> >> more right now! >>>>> >> >>>>> >> At this point I think gamers in particular know what "brands" we've >>>>> >> tried to establish - "Smart queues", "SQM", "OpenWrt", fq_codel and >>>>> >> now "cake" are "good" things to have, and are stimulating demand by >>>>> >> asking for them, It's certainly working out better and better for >>>>> >> evenroute, firewalla, ubnt and others, and I saw an uptick in >>>>> >> questions about this on various user forums. >>>>> >> >>>>> >> I even like that there's a backlash now of people saying "fixing >>>>> >> bufferbloat doesn't solve everything" - >>>>> >> >>>>> >> > Describing the obvious simple-minded (wrong) solution that any normal person would think of based on their personal human experience waiting in grocery stores and airports, is not describing the solution to bufferbloat. The solution to bufferbloat is not that if you are privileged then you get to “skip to the front of the line”. The solution to bufferbloat is that there is no line! >>>>> >> >>>>> >> I like the idea of a guru floating above a grocery cart with a better >>>>> >> string of explanations, explaining >>>>> >> >>>>> >> - "no, grasshopper, the solution to bufferbloat is no line... at all". >>>>> >> >>>>> >> > >>>>> >> > With grocery stores and airports people’s arrivals are independent and not controlled. There is no way for a grocery store or airport to generate backpressure to tell people to wait at home when a queue begins to form. The key to solving bufferbloat is generating timely backpressure to prevent the queue forming in the first place, not accepting a huge queue and then deciding who deserves special treatment to get better service than all the other peons who still have to wait in a long queue, just like before. >>>>> >> >>>>> >> I am not huge on the word "backpressure" here. Needs to signal the >>>>> >> other side to slow down, is more accurate. So might say timely >>>>> >> signalling rather than timely backpressure? >>>>> >> >>>>> >> Other feedback I got was that the video was too smarmy (I agree), >>>>> >> different audiences than gamers need different forms of outreach... >>>>> >> >>>>> >> but to me, winning the gamers has always been one of the most >>>>> >> important things, as they make a lot of buying decisions, and they >>>>> >> benefit the most for >>>>> >> fq and packet prioritization as we do today in gamer routers and in >>>>> >> cake + qosify. >>>>> >> >>>>> >> maybe that gets in the way of more serious markets. Certainly I would >>>>> >> like another video explaining what goes wrong with videoconferencing. >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> > >>>>> >> > Stuart Cheshire >>>>> >> > >>>>> >> >>>>> >> >>>>> >> -- >>>>> >> This song goes out to all the folk that thought Stadia would work: >>>>> >> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz >>>>> >> Dave Täht CEO, TekLibre, LLC >>>>> >> _______________________________________________ >>>>> >> Bloat mailing list >>>>> >> Bloat@lists.bufferbloat.net >>>>> >> https://lists.bufferbloat.net/listinfo/bloat >>>>> >>>>> >>>>> >>>>> -- >>>>> This song goes out to all the folk that thought Stadia would work: >>>>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz >>>>> Dave Täht CEO, TekLibre, LLC >>>>> >>>>> >>>>> -- >>>>> This song goes out to all the folk that thought Stadia would work: >>>>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz >>>>> Dave Täht CEO, TekLibre, LLC >>>>> _______________________________________________ >>>>> LibreQoS mailing list >>>>> LibreQoS@lists.bufferbloat.net >>>>> https://lists.bufferbloat.net/listinfo/libreqos >>>> >>>> _______________________________________________ >>>> LibreQoS mailing list >>>> LibreQoS@lists.bufferbloat.net >>>> https://lists.bufferbloat.net/listinfo/libreqos >>> >>> >>> >>> -- >>> Robert Chacón >>> CEO | JackRabbit Wireless LLC >>> Dev | LibreQoS.io >>> >>> _______________________________________________ >>> LibreQoS mailing list >>> LibreQoS@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/libreqos >> >> _______________________________________________ >> LibreQoS mailing list >> LibreQoS@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/libreqos > > _______________________________________________ > LibreQoS mailing list > LibreQoS@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/libreqos -- This song goes out to all the folk that thought Stadia would work: https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz Dave Täht CEO, TekLibre, LLC ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [LibreQoS] Various aspects of net neutrality 2022-11-12 19:30 ` [LibreQoS] Various aspects of net neutrality Dave Taht @ 2022-11-12 19:58 ` dan 2022-11-14 13:45 ` Sascha Meinrath 0 siblings, 1 reply; 10+ messages in thread From: dan @ 2022-11-12 19:58 UTC (permalink / raw) To: Dave Taht; +Cc: Herbert Wolverson, libreqos, Sascha Meinrath [-- Attachment #1: Type: text/plain, Size: 25527 bytes --] I think the argument you site is a difficult one, it's not without merits but it's also likely the losing position ultimately. There's no clear cut "you took something away from the consumer" bit in there that would trigger most consumer protection triggers. ANY legislation that doesn't have a clear case for public benefit can be endlessly attacked and special exemptions to it made on every win against it. Makes it something most lawmakers cringe at the thought of dealing with for an issue that isn't in the headlines ie doesn't help them get (or lose) votes. I think the middle ground is actually really simple. Opt-In or Opt-Out. Hook up new service and in the installation checklist there's a "we offer connection enhancements free of charge, can I enable that for you?". All consumer protections are out when the customer agrees. Even Cambium/Bequant is perfectly legal in California if the customer opts-in to it (I don't believe you can do opt-out in California's language). Then in libreQOS or whatever just have an option to use a fifo instead of fq_codel or cake. Most people will take that without question, some might ask what it is and it's pretty easy to explain that it makes sure all their stuff gets a fair share of the connection instead of the Xbox downloads causing Netflix to buffer. fifo buffers hurt the consumer, not the operator, and we can always just turn it on when they call in "would you like to try the enhanced service to see if that makes things better?" We've actually considered offering service plans based around it. I was looking at both PQ and Cambium pretty hard and in that would offer 'NN' packages and then 'Enhanced' with a number of bulletpoints for customers. ie, NN is a 100x20 bare connection (fifo...) while the same price could get a 200x100 enhanced (AQM) that would cap streaming lower and elevate gaming etc. All in the open and opt-in via the purchase of the 'enhancement' on the service. Ultimately, the lack of top level shapers as table stakes is what got me into libreqos in the first place. On Sat, Nov 12, 2022 at 12:31 PM Dave Taht <dave.taht@gmail.com> wrote: > Changing the subject line. This is a very nuanced topic and I would > rather enjoy the day AFK! To stir the pot, tho: > > I have had a cable industry insider claim that fq_codel violated the > california code you cite. Wrote about it here: > > https://blog.cerowrt.org/post/net_neutrality_customers/ > > And taking on the other side: > > https://blog.cerowrt.org/post/net_neutrality_isps/ > > I have asked multiple lawyers about it since, and the rules need to be > clarified, here. The level of doublethink in the cable/telco industry > about NN has to be experienced to be believed. Just spent 6 years > losing a fight about it on l4s and related, so now they can hide > behind the IETFs skirts now. > > I didn't make friends with any side that way, and of my relationship > from that period, with the eff and the ACLU, I don't know... but I did > write a nice piece of rhetoric: > > "fq codel (now IETF standard RFC8290) is a uniquely “American” > algorithm. It gives the “little guy” - the little packet, the first > packets in a new connection to anywhere, a little boost until the flow > achieves parity with other flows from other sources, with minimal > buffering. This means that all network traffic gets treated equally - > faster. Isn’t that what you want in a network neutral framework? DNS, > gaming traffic, voip, videoconferencing, and the first packets of any > new flow, to anywhere, get a small boost. That’s it. Big flows - from > anybody - from netflix to google to comcast - all achieve parity, with > minimal delay and buffering, at a wide variety of real round trip > times." > > Just to throw another bit of controversial thought before I gaily > depart for a nice lunch... I tended to support WISPA's non-defense of > NN > in multiple respects. Title II, and additional regulation seemed > worse, then, than having some mildly chastized ISPs remain free to do > whatever they wanted. With my brutal experiences with politics > starting with the clipper chip, the CDA, SOPA, network neutrality, and > of late the NTIA broadband funds act.... > > My basic conclusion was this: > > There must be a middle path, and it starts with less lawyers being in > charge at the FCC and extends to, in general, less politicians on any > side of the debate, making policy and slogans that don't line up with > reality. > > "For a successful technology .... " - richard feynman. > > On Sat, Nov 12, 2022 at 9:39 AM dan via LibreQoS > <libreqos@lists.bufferbloat.net> wrote: > > > > I have to argue with you on 'equally limited' streaming being ok. That > allows for 'internet-like' service but no/poor streaming, a service that > clamps down really hard on streaming and then offers you an upgrade to make > it work. Neutrality doesn't mean 'kind of' neutral. If it's called > 'Internet' then no specific service or even type of service should be > limited for the provider's direct benefit or whims. I can see no positive > case for a provider limiting any service/type below the customers quoted > service. The provider gets to save money, but since they offer an inferior > product that is essentially false advertised and customers have no way to > know this then capitalist tools like consumer choice can't work. Consumers > don't know they can get a better suited service, they don't show demand in > a market, that prevents competition from having an opportunity, etc. Most > people have just 1 fast internet option. > > > > Does a customer that can't figure out why their new Roku 4K on their new > 4K TV refuses to stream 4K Netflix on their 100M internet service say > "Netflix sucks!". or Roku sucks or Sony sucks or whatever? So this spills > over into making other services take the blame for bad connections. > > > > > > > > > > On Sat, Nov 12, 2022 at 10:19 AM Herbert Wolverson via LibreQoS < > libreqos@lists.bufferbloat.net> wrote: > >> > >> Net neutrality is complex (I'm generally in favor of the concept). > There's nothing stopping an ISP from having rules that video streaming be > limited - but ALL video streaming has to be equally limited. So ComCast > could use Sandvine, but only without any vendor-specific limitations. Same > for Cambium. Limit streams that look like video, and you're fine. With that > said, it runs into something I say a lot: don't penalize your customers for > using your product. If you're buying a 100mbit/s pipe, hooking it to your > fancy 4k streaming system - it's entirely reasonable to expect sharp 4k > video. In fact, your shaping emphasis should be on keeping their video nice > and watchable even though someone just decided to download the newest Call > of Duty, update their xbox, and update the firmware on their smart fridge. > Likewise, if someone really wants the smallest 5 mbit connection you once > offered - because they only ever use it to check email (we actually had a > customer complain when we offered them a free upgrade from a 5mbit/s plan, > years ago!) - you want to do your best to make it a usable tiny plan for > them. Fair queuing (along with a bit of user education; we eventually > convinced the customer that "no change in fees" actually meant not charging > them more money) helps with all of that. > >> > >> It can win customers, too. We have a couple of business customers who > switched us from their backup to their primary because "our service felt > snappier". We had a business go with us because their 24x7 lobby streaming > video looked better on our (smaller) connection, and a couple of > fraternities who likewise decided to use us because their overkill 4k video > setup looked better (even with the 20+ Xboxes we could see on their > network). > >> > >> QUIC makes me chuckle, because it's exactly what we were doing in > gamedev land in the late 90s to make deathmatches run smoothly. (Seriously, > QuakeWorld and the original Unreal Tournament had the most amazing network > code; they basically implemented what is now known as "reliable UDP" to > incredible results). > >> > >> With that said, I think we've got a remarkable amount of wiggle-room to > make things better (and it's not at all shoddy right now!). XDP is amazing > as-is, and is improving fast. (I'm currently reading > https://github.com/ilejn/xdpbridge - an older project that doesn't seem > to have gained much traction, but I'm wondering if it couldn't provide some > seriously fast bridge offloading; I'll have to setup a better test > environment to find out) > >> > >> On Sat, Nov 12, 2022 at 10:09 AM Robert Chacón via LibreQoS < > libreqos@lists.bufferbloat.net> wrote: > >>> > >>> Thank you, Dan. > >>> I completely agree with you 100% there. It's a risky way to save a > tiny amount of money on bandwidth costs. > >>> Comcast paid out $16 million in a settlement over using Sandvine's DPI > in this way. > >>> Since then, Comcast has switched from DPI to fair queueing with > DOCSIS-pie. > >>> > >>> California's SB-822 will be used as a template by many other states > going forward on Net Neutrality. > >>> SB-822 even clarified that "Reasonable Network Management" needs to be > "as application-agnostic as possible". > >>> These DPI approaches like Cambium's will not hold up in court the way > Fair Queuing (Preseem, LibreQoS) based solutions will. > >>> We can point to years of public research and RFCs when defending a > Fair Queuing QoE approach. > >>> They can't so easily defend using a product that advertised its big > "scale down 4k to save money" knob or "accelerate their speeds only when > they run speed tests" button. > >>> It's concerning to me that these QoE vendors are not disclosing "hey, > don't use this in NN states like California please". > >>> The WISP industry will be hurt long-term by these products with > consumers viewing us as an inferior technology that limits their usage in > intrusive ways. > >>> Like you said, let's let them use their plan however they want. It > doesn't hurt us. But things like capping end-users at HD will hurt us > long-term in lost sales. > >>> > >>> Regarding TCP acceleration, I'm leaning toward the argument made by > Dan from Preseem that QUIC adoption is growing pretty quickly and we may > not want to mess with TCP at the expense of UDP given the rise of QUIC. > >>> Cloudlfare says HTTP/3 and QUIC are already at 28% and growing. > >>> One point he brought up is that if we use TCP acceleration, and a > middlebox has to be rebooted or something - all client TCP sessions will > break when OSPF switches paths around the box. > >>> > >>> On Sat, Nov 12, 2022 at 8:36 AM dan via LibreQoS < > libreqos@lists.bufferbloat.net> wrote: > >>>> > >>>> I think there's a thin grey line between what other vendors (paraqum, > bequant/cambium) and libreQoS is trying to do. The first two are REALLY > pushing this selecting narrowing of traffic and they'll go straight after > streaming for their demos. Most people that drop in the cambium QoE > appliance turn the knobs on Netflix down and then praise the 40% savings on > their network. Same with PQ. I think this is fundamentally wrong. This > is the entire reason Net Neutrality keeps popping up. Why does the ISP get > to say "You only need 25Mbps for your netflix". That's double wrong if > they offer their own video services. It's not about improving the > customer's experience, it's about spending less on upstream bandwidth at > the customer's expense. As soon as they say 'pays for itself in bandwidth > savings' you know what the product is primarily for. > >>>> > >>>> This is why I like the preseem and libreqos model of just using a > 'stock' cake and just finessing the data coming in to keep the pipe from > clogging. Who cares if someone pulls 90M bursts of apple TV+ on their 100M > pipe. Only dial that back so other requests they make get through cleanly. > >>>> > >>>> preseem is going more towards wireless monitoring etc and kinda > letting shaping be a secondary.... if they were doing more libreqos like > stuff I'd just stick with preseem. > >>>> > >>>> libreQoS may end up being the best and most net neutral product you > can get. The only current disadvantage (ignoring DPI) is lack of TCP > accelleration that PQ and Bequant/cambium have. > >>>> > >>>> I should add that 3 states already have laws on the books that > straight up make PQ and Cambium's solution illegal because they target > certain services for throttling. My state (Montana) has the same rule > except only when working with a government contract. Libreqos is actually > compliant (as is preseem). > >>>> > >>>> On Sat, Nov 12, 2022 at 8:11 AM Dave Taht via LibreQoS < > libreqos@lists.bufferbloat.net> wrote: > >>>>> > >>>>> this report predates the libreqos list... > >>>>> > >>>>> ---------- Forwarded message --------- > >>>>> From: Dave Taht <dave.taht@gmail.com> > >>>>> Date: Mon, Oct 17, 2022 at 8:15 PM > >>>>> Subject: A quick report from the WISPA conference > >>>>> To: Sina Khanifar <sina@waveform.com> > >>>>> Cc: Cake List <cake@lists.bufferbloat.net>, Make-Wifi-fast > >>>>> <make-wifi-fast@lists.bufferbloat.net>, Rpm > >>>>> <rpm@lists.bufferbloat.net>, Stuart Cheshire <cheshire@apple.com>, > >>>>> bloat <bloat@lists.bufferbloat.net> > >>>>> > >>>>> > >>>>> On Mon, Oct 17, 2022 at 7:51 PM Sina Khanifar <sina@waveform.com> > wrote: > >>>>> > > >>>>> > Positive or negative, I can claim a bit of credit for this video > :). We've been working with LTT on a few projects and we pitched them on > doing something around bufferbloat. We've seen more traffic to our Waveforn > test than ever before, which has been fun! > >>>>> > >>>>> Thank you. Great job with that video! And waveform has become the > goto > >>>>> site for many now. > >>>>> > >>>>> I can't help but wonder tho... are you collecting any statistics, > over > >>>>> time, as to how much better the problem is getting? > >>>>> > >>>>> And any chance they could do something similar explaining wifi? > >>>>> > >>>>> ... > >>>>> > >>>>> I was just at WISPA conference week before last. Preseem's booth > >>>>> (fq_codel) was always packed. Vilo living had put cake in their wifi > 6 > >>>>> product. A > >>>>> keynote speaker had deployed it and talked about it with waveform > >>>>> results on the big screen (2k people there). A large wireless vendor > >>>>> demo'd privately to me their flent results before/after cake on their > >>>>> next-gen radios... and people dissed tarana without me prompting for > >>>>> their bad bufferbloat... and the best thing of all that happened to > me > >>>>> was... besides getting a hug from a young lady (megan) who'd salvaged > >>>>> her schooling in alaska using sqm - I walked up to the paraqum booth > >>>>> (another large QoE middlebox maker centered more in india) and asked. > >>>>> > >>>>> "So... do y'all have fq_codel yet?" > >>>>> > >>>>> And they smiled and said: "No, we have something better... we've got > cake." > >>>>> > >>>>> "Cake? What's that?" - I said, innocently. > >>>>> > >>>>> They then stepped me through their 200Gbps (!!) product, which uses a > >>>>> bunch of offloads, and can track rtt down to a ms with the intel > >>>>> ethernet card they were using. They'd modifed cake to provide 16 (?) > >>>>> levels of service, and were running under dpdk (I am not sure if cake > >>>>> was). It was a great, convincing pitch... > >>>>> > >>>>> ... then I told 'em who I was. There's a video of the in-both > concert after. > >>>>> > >>>>> ... > >>>>> > >>>>> The downside to me (and the subject of my talk) was that in nearly > >>>>> every person I talked to, fq_codel was viewed as a means to better > >>>>> subscriber bandwidth plan enforcement (which is admittedly the market > >>>>> that preseem pioneered) and it was not understood that I'd got > >>>>> involved in this whole thing because I'd wanted an algorithm to deal > >>>>> with "rain fade", running directly on the radios. People wanted to > use > >>>>> the statistics on the radios to drive the plan enforcement better > >>>>> (which is an ok approach, I guess), and for 10+ I'd been whinging > >>>>> about the... physics. > >>>>> > >>>>> So I ranted about rfc7567 a lot and begged people now putting > routerOS > >>>>> 7.2 and later out there (mikrotik is huge in this market), to kill > >>>>> their fifos and sfqs at the native rates of the interfaces... and > >>>>> watch their network improve that way also. > >>>>> > >>>>> I think one more wispa conference will be a clean sweep of everyone > in > >>>>> the fixed wireless market to not only adopt these algorithms for plan > >>>>> enforcement, but even more directly on the radios and more CPE. > >>>>> > >>>>> I also picked up enough consulting business to keep me busy the rest > >>>>> of this year, and possibly more than I can handle (anybody looking?) > >>>>> > >>>>> I wonder what will happen at a fiber conference? > >>>>> > >>>>> > On Mon, Oct 17, 2022 at 7:45 PM Dave Taht via Bloat < > bloat@lists.bufferbloat.net> wrote: > >>>>> >> > >>>>> >> On Mon, Oct 17, 2022 at 5:02 PM Stuart Cheshire < > cheshire@apple.com> wrote: > >>>>> >> > > >>>>> >> > On 9 Oct 2022, at 06:14, Dave Taht via Make-wifi-fast < > make-wifi-fast@lists.bufferbloat.net> wrote: > >>>>> >> > > >>>>> >> > > This was so massively well done, I cried. Does anyone know > how to get in touch with the ifxit folk? > >>>>> >> > > > >>>>> >> > > https://www.youtube.com/watch?v=UICh3ScfNWI > >>>>> >> > > >>>>> >> > I’m surprised that you liked this video. It seems to me that it > repeats all the standard misinformation. The analogy they use is the > standard terrible example of waiting in a long line at a grocery store, and > the “solution” is letting certain traffic “jump the line, angering everyone > behind them”. > >>>>> >> > >>>>> >> Accuracy be damned. The analogy to common experience resonates > more. > >>>>> >> > >>>>> >> > > >>>>> >> > Some quotes from the video: > >>>>> >> > > >>>>> >> > > it would be so much more efficient for them to let you skip > the line and just check out, especially since you’re in a hurry, but > they’re rudely refusing > >>>>> >> > >>>>> >> I think the person with the cheetos pulling out a gun and shooting > >>>>> >> everyone in front of him (AQM) would not go down well. > >>>>> >> > >>>>> >> > > to go back to our grocery store analogy this would be like if > a worker saw you standing at the back ... and either let you skip to the > front of the line or opens up an express lane just for you > >>>>> >> > >>>>> >> Actually that analogy is fairly close to fair queuing. The > multiple > >>>>> >> checker analogy is one of the most common analogies in queue > theory > >>>>> >> itself. > >>>>> >> > >>>>> >> > > >>>>> >> > The video describes the problem of bufferbloat, and then > describes the same failed solution that hasn’t worked for the last three > decades. > >>>>> >> > >>>>> >> Hmm? It establishes the scenario, explains the problem *quickly*, > >>>>> >> disses gamer routers for not getting it right.. *points to an > >>>>> >> accurate test*, and then to the ideas and products that *actually > >>>>> >> work* with "smart queueing", with a screenshot of the most common > >>>>> >> (eero's optimize for gaming and videoconferencing), and fq_codel > and > >>>>> >> cake *by name*, and points folk at the best known solution > available, > >>>>> >> openwrt. > >>>>> >> > >>>>> >> Bing, baddabang, boom. Also the comments were revealing. A goodly > >>>>> >> percentage already knew the problem, more than a few were > inspired to > >>>>> >> take the test, > >>>>> >> there was a whole bunch of "Aha!" success stories and 360k views, > >>>>> >> which is more people than we've ever been able to reach in for > >>>>> >> example, a nanog conference. > >>>>> >> > >>>>> >> I loved that folk taking the test actually had quite a few A > results, > >>>>> >> without having had to do anything. At least some ISPs are getting > it > >>>>> >> more right now! > >>>>> >> > >>>>> >> At this point I think gamers in particular know what "brands" > we've > >>>>> >> tried to establish - "Smart queues", "SQM", "OpenWrt", fq_codel > and > >>>>> >> now "cake" are "good" things to have, and are stimulating demand > by > >>>>> >> asking for them, It's certainly working out better and better > for > >>>>> >> evenroute, firewalla, ubnt and others, and I saw an uptick in > >>>>> >> questions about this on various user forums. > >>>>> >> > >>>>> >> I even like that there's a backlash now of people saying "fixing > >>>>> >> bufferbloat doesn't solve everything" - > >>>>> >> > >>>>> >> > Describing the obvious simple-minded (wrong) solution that any > normal person would think of based on their personal human experience > waiting in grocery stores and airports, is not describing the solution to > bufferbloat. The solution to bufferbloat is not that if you are privileged > then you get to “skip to the front of the line”. The solution to > bufferbloat is that there is no line! > >>>>> >> > >>>>> >> I like the idea of a guru floating above a grocery cart with a > better > >>>>> >> string of explanations, explaining > >>>>> >> > >>>>> >> - "no, grasshopper, the solution to bufferbloat is no line... > at all". > >>>>> >> > >>>>> >> > > >>>>> >> > With grocery stores and airports people’s arrivals are > independent and not controlled. There is no way for a grocery store or > airport to generate backpressure to tell people to wait at home when a > queue begins to form. The key to solving bufferbloat is generating timely > backpressure to prevent the queue forming in the first place, not accepting > a huge queue and then deciding who deserves special treatment to get better > service than all the other peons who still have to wait in a long queue, > just like before. > >>>>> >> > >>>>> >> I am not huge on the word "backpressure" here. Needs to signal the > >>>>> >> other side to slow down, is more accurate. So might say timely > >>>>> >> signalling rather than timely backpressure? > >>>>> >> > >>>>> >> Other feedback I got was that the video was too smarmy (I agree), > >>>>> >> different audiences than gamers need different forms of > outreach... > >>>>> >> > >>>>> >> but to me, winning the gamers has always been one of the most > >>>>> >> important things, as they make a lot of buying decisions, and they > >>>>> >> benefit the most for > >>>>> >> fq and packet prioritization as we do today in gamer routers and > in > >>>>> >> cake + qosify. > >>>>> >> > >>>>> >> maybe that gets in the way of more serious markets. Certainly I > would > >>>>> >> like another video explaining what goes wrong with > videoconferencing. > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > > >>>>> >> > Stuart Cheshire > >>>>> >> > > >>>>> >> > >>>>> >> > >>>>> >> -- > >>>>> >> This song goes out to all the folk that thought Stadia would work: > >>>>> >> > https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz > >>>>> >> Dave Täht CEO, TekLibre, LLC > >>>>> >> _______________________________________________ > >>>>> >> Bloat mailing list > >>>>> >> Bloat@lists.bufferbloat.net > >>>>> >> https://lists.bufferbloat.net/listinfo/bloat > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> This song goes out to all the folk that thought Stadia would work: > >>>>> > https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz > >>>>> Dave Täht CEO, TekLibre, LLC > >>>>> > >>>>> > >>>>> -- > >>>>> This song goes out to all the folk that thought Stadia would work: > >>>>> > https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz > >>>>> Dave Täht CEO, TekLibre, LLC > >>>>> _______________________________________________ > >>>>> LibreQoS mailing list > >>>>> LibreQoS@lists.bufferbloat.net > >>>>> https://lists.bufferbloat.net/listinfo/libreqos > >>>> > >>>> _______________________________________________ > >>>> LibreQoS mailing list > >>>> LibreQoS@lists.bufferbloat.net > >>>> https://lists.bufferbloat.net/listinfo/libreqos > >>> > >>> > >>> > >>> -- > >>> Robert Chacón > >>> CEO | JackRabbit Wireless LLC > >>> Dev | LibreQoS.io > >>> > >>> _______________________________________________ > >>> LibreQoS mailing list > >>> LibreQoS@lists.bufferbloat.net > >>> https://lists.bufferbloat.net/listinfo/libreqos > >> > >> _______________________________________________ > >> LibreQoS mailing list > >> LibreQoS@lists.bufferbloat.net > >> https://lists.bufferbloat.net/listinfo/libreqos > > > > _______________________________________________ > > LibreQoS mailing list > > LibreQoS@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/libreqos > > > > -- > This song goes out to all the folk that thought Stadia would work: > > https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz > Dave Täht CEO, TekLibre, LLC > [-- Attachment #2: Type: text/html, Size: 34093 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [LibreQoS] Various aspects of net neutrality 2022-11-12 19:58 ` dan @ 2022-11-14 13:45 ` Sascha Meinrath 0 siblings, 0 replies; 10+ messages in thread From: Sascha Meinrath @ 2022-11-14 13:45 UTC (permalink / raw) To: dan, Dave Taht; +Cc: Herbert Wolverson, libreqos Hi all, I've had an ongoing, multi-year conversation on the utility of AQM & etc. and how one could implement it without running afoul of Network Neutrality concerns with various technologists (spanning industry to the public interest sector). As an avid Network Neutrality advocate, I can see how these technologies can be implemented without violating NN (and with salubrious outcomes for all). And it is also the case that without transparency, accountability, and oversight, these same technologies could also become a new battlefront for a variety of prioritization schemes that would prove harmful to certain Internet users. I, for one, would very much like to see bufferbloat addressed -- and I also think there's a lot of myth built up around what NN advocates are concerned about. All in all, there is certainly a way to implement these technologies while addressing NN concerns. This isn't a NN vs. AQM debate -- it's a fundamental issue of transparency of the undergirding technological operationalization and its impact on end-user agency. --Sascha On 11/12/22 14:58, dan wrote: > I think the argument you site is a difficult one, it's not without merits but > it's also likely the losing position ultimately. There's no clear cut "you took > something away from the consumer" bit in there that would trigger most consumer > protection triggers. ANY legislation that doesn't have a clear case for public > benefit can be endlessly attacked and special exemptions to it made on every win > against it. Makes it something most lawmakers cringe at the thought of dealing > with for an issue that isn't in the headlines ie doesn't help them get (or lose) > votes. > > I think the middle ground is actually really simple. Opt-In or Opt-Out. Hook > up new service and in the installation checklist there's a "we offer connection > enhancements free of charge, can I enable that for you?". All consumer > protections are out when the customer agrees. Even Cambium/Bequant is perfectly > legal in California if the customer opts-in to it (I don't believe you can do > opt-out in California's language). Then in libreQOS or whatever just have an > option to use a fifo instead of fq_codel or cake. Most people will take that > without question, some might ask what it is and it's pretty easy to explain that > it makes sure all their stuff gets a fair share of the connection instead of the > Xbox downloads causing Netflix to buffer. fifo buffers hurt the consumer, not > the operator, and we can always just turn it on when they call in "would you > like to try the enhanced service to see if that makes things better?" > > We've actually considered offering service plans based around it. I was looking > at both PQ and Cambium pretty hard and in that would offer 'NN' packages and > then 'Enhanced' with a number of bulletpoints for customers. ie, NN is a 100x20 > bare connection (fifo...) while the same price could get a 200x100 enhanced > (AQM) that would cap streaming lower and elevate gaming etc. All in the open > and opt-in via the purchase of the 'enhancement' on the service. > > Ultimately, the lack of top level shapers as table stakes is what got me into > libreqos in the first place. > > On Sat, Nov 12, 2022 at 12:31 PM Dave Taht <dave.taht@gmail.com > <mailto:dave.taht@gmail.com>> wrote: > > Changing the subject line. This is a very nuanced topic and I would > rather enjoy the day AFK! To stir the pot, tho: > > I have had a cable industry insider claim that fq_codel violated the > california code you cite. Wrote about it here: > > https://blog.cerowrt.org/post/net_neutrality_customers/ > <https://blog.cerowrt.org/post/net_neutrality_customers/> > > And taking on the other side: > > https://blog.cerowrt.org/post/net_neutrality_isps/ > <https://blog.cerowrt.org/post/net_neutrality_isps/> > > I have asked multiple lawyers about it since, and the rules need to be > clarified, here. The level of doublethink in the cable/telco industry > about NN has to be experienced to be believed. Just spent 6 years > losing a fight about it on l4s and related, so now they can hide > behind the IETFs skirts now. > > I didn't make friends with any side that way, and of my relationship > from that period, with the eff and the ACLU, I don't know... but I did > write a nice piece of rhetoric: > > "fq codel (now IETF standard RFC8290) is a uniquely “American” > algorithm. It gives the “little guy” - the little packet, the first > packets in a new connection to anywhere, a little boost until the flow > achieves parity with other flows from other sources, with minimal > buffering. This means that all network traffic gets treated equally - > faster. Isn’t that what you want in a network neutral framework? DNS, > gaming traffic, voip, videoconferencing, and the first packets of any > new flow, to anywhere, get a small boost. That’s it. Big flows - from > anybody - from netflix to google to comcast - all achieve parity, with > minimal delay and buffering, at a wide variety of real round trip > times." > > Just to throw another bit of controversial thought before I gaily > depart for a nice lunch... I tended to support WISPA's non-defense of > NN > in multiple respects. Title II, and additional regulation seemed > worse, then, than having some mildly chastized ISPs remain free to do > whatever they wanted. With my brutal experiences with politics > starting with the clipper chip, the CDA, SOPA, network neutrality, and > of late the NTIA broadband funds act.... > > My basic conclusion was this: > > There must be a middle path, and it starts with less lawyers being in > charge at the FCC and extends to, in general, less politicians on any > side of the debate, making policy and slogans that don't line up with > reality. > > "For a successful technology .... " - richard feynman. > > On Sat, Nov 12, 2022 at 9:39 AM dan via LibreQoS > <libreqos@lists.bufferbloat.net <mailto:libreqos@lists.bufferbloat.net>> wrote: > > > > I have to argue with you on 'equally limited' streaming being ok. That > allows for 'internet-like' service but no/poor streaming, a service that > clamps down really hard on streaming and then offers you an upgrade to make > it work. Neutrality doesn't mean 'kind of' neutral. If it's called > 'Internet' then no specific service or even type of service should be > limited for the provider's direct benefit or whims. I can see no positive > case for a provider limiting any service/type below the customers quoted > service. The provider gets to save money, but since they offer an inferior > product that is essentially false advertised and customers have no way to > know this then capitalist tools like consumer choice can't work. Consumers > don't know they can get a better suited service, they don't show demand in a > market, that prevents competition from having an opportunity, etc. Most > people have just 1 fast internet option. > > > > Does a customer that can't figure out why their new Roku 4K on their new > 4K TV refuses to stream 4K Netflix on their 100M internet service say > "Netflix sucks!". or Roku sucks or Sony sucks or whatever? So this spills > over into making other services take the blame for bad connections. > > > > > > > > > > On Sat, Nov 12, 2022 at 10:19 AM Herbert Wolverson via LibreQoS > <libreqos@lists.bufferbloat.net <mailto:libreqos@lists.bufferbloat.net>> wrote: > >> > >> Net neutrality is complex (I'm generally in favor of the concept). > There's nothing stopping an ISP from having rules that video streaming be > limited - but ALL video streaming has to be equally limited. So ComCast > could use Sandvine, but only without any vendor-specific limitations. Same > for Cambium. Limit streams that look like video, and you're fine. With that > said, it runs into something I say a lot: don't penalize your customers for > using your product. If you're buying a 100mbit/s pipe, hooking it to your > fancy 4k streaming system - it's entirely reasonable to expect sharp 4k > video. In fact, your shaping emphasis should be on keeping their video nice > and watchable even though someone just decided to download the newest Call > of Duty, update their xbox, and update the firmware on their smart fridge. > Likewise, if someone really wants the smallest 5 mbit connection you once > offered - because they only ever use it to check email (we actually had a > customer complain when we offered them a free upgrade from a 5mbit/s plan, > years ago!) - you want to do your best to make it a usable tiny plan for > them. Fair queuing (along with a bit of user education; we eventually > convinced the customer that "no change in fees" actually meant not charging > them more money) helps with all of that. > >> > >> It can win customers, too. We have a couple of business customers who > switched us from their backup to their primary because "our service felt > snappier". We had a business go with us because their 24x7 lobby streaming > video looked better on our (smaller) connection, and a couple of > fraternities who likewise decided to use us because their overkill 4k video > setup looked better (even with the 20+ Xboxes we could see on their network). > >> > >> QUIC makes me chuckle, because it's exactly what we were doing in > gamedev land in the late 90s to make deathmatches run smoothly. (Seriously, > QuakeWorld and the original Unreal Tournament had the most amazing network > code; they basically implemented what is now known as "reliable UDP" to > incredible results). > >> > >> With that said, I think we've got a remarkable amount of wiggle-room to > make things better (and it's not at all shoddy right now!). XDP is amazing > as-is, and is improving fast. (I'm currently reading > https://github.com/ilejn/xdpbridge <https://github.com/ilejn/xdpbridge> - an > older project that doesn't seem to have gained much traction, but I'm > wondering if it couldn't provide some seriously fast bridge offloading; I'll > have to setup a better test environment to find out) > >> > >> On Sat, Nov 12, 2022 at 10:09 AM Robert Chacón via LibreQoS > <libreqos@lists.bufferbloat.net <mailto:libreqos@lists.bufferbloat.net>> wrote: > >>> > >>> Thank you, Dan. > >>> I completely agree with you 100% there. It's a risky way to save a tiny > amount of money on bandwidth costs. > >>> Comcast paid out $16 million in a settlement over using Sandvine's DPI > in this way. > >>> Since then, Comcast has switched from DPI to fair queueing with DOCSIS-pie. > >>> > >>> California's SB-822 will be used as a template by many other states > going forward on Net Neutrality. > >>> SB-822 even clarified that "Reasonable Network Management" needs to be > "as application-agnostic as possible". > >>> These DPI approaches like Cambium's will not hold up in court the way > Fair Queuing (Preseem, LibreQoS) based solutions will. > >>> We can point to years of public research and RFCs when defending a Fair > Queuing QoE approach. > >>> They can't so easily defend using a product that advertised its big > "scale down 4k to save money" knob or "accelerate their speeds only when > they run speed tests" button. > >>> It's concerning to me that these QoE vendors are not disclosing "hey, > don't use this in NN states like California please". > >>> The WISP industry will be hurt long-term by these products with > consumers viewing us as an inferior technology that limits their usage in > intrusive ways. > >>> Like you said, let's let them use their plan however they want. It > doesn't hurt us. But things like capping end-users at HD will hurt us > long-term in lost sales. > >>> > >>> Regarding TCP acceleration, I'm leaning toward the argument made by Dan > from Preseem that QUIC adoption is growing pretty quickly and we may not > want to mess with TCP at the expense of UDP given the rise of QUIC. > >>> Cloudlfare says HTTP/3 and QUIC are already at 28% and growing. > >>> One point he brought up is that if we use TCP acceleration, and a > middlebox has to be rebooted or something - all client TCP sessions will > break when OSPF switches paths around the box. > >>> > >>> On Sat, Nov 12, 2022 at 8:36 AM dan via LibreQoS > <libreqos@lists.bufferbloat.net <mailto:libreqos@lists.bufferbloat.net>> wrote: > >>>> > >>>> I think there's a thin grey line between what other vendors (paraqum, > bequant/cambium) and libreQoS is trying to do. The first two are REALLY > pushing this selecting narrowing of traffic and they'll go straight after > streaming for their demos. Most people that drop in the cambium QoE > appliance turn the knobs on Netflix down and then praise the 40% savings on > their network. Same with PQ. I think this is fundamentally wrong. This > is the entire reason Net Neutrality keeps popping up. Why does the ISP get > to say "You only need 25Mbps for your netflix". That's double wrong if they > offer their own video services. It's not about improving the customer's > experience, it's about spending less on upstream bandwidth at the customer's > expense. As soon as they say 'pays for itself in bandwidth savings' you > know what the product is primarily for. > >>>> > >>>> This is why I like the preseem and libreqos model of just using a > 'stock' cake and just finessing the data coming in to keep the pipe from > clogging. Who cares if someone pulls 90M bursts of apple TV+ on their 100M > pipe. Only dial that back so other requests they make get through cleanly. > >>>> > >>>> preseem is going more towards wireless monitoring etc and kinda > letting shaping be a secondary.... if they were doing more libreqos like > stuff I'd just stick with preseem. > >>>> > >>>> libreQoS may end up being the best and most net neutral product you > can get. The only current disadvantage (ignoring DPI) is lack of TCP > accelleration that PQ and Bequant/cambium have. > >>>> > >>>> I should add that 3 states already have laws on the books that > straight up make PQ and Cambium's solution illegal because they target > certain services for throttling. My state (Montana) has the same rule > except only when working with a government contract. Libreqos is actually > compliant (as is preseem). > >>>> > >>>> On Sat, Nov 12, 2022 at 8:11 AM Dave Taht via LibreQoS > <libreqos@lists.bufferbloat.net <mailto:libreqos@lists.bufferbloat.net>> wrote: > >>>>> > >>>>> this report predates the libreqos list... > >>>>> > >>>>> ---------- Forwarded message --------- > >>>>> From: Dave Taht <dave.taht@gmail.com <mailto:dave.taht@gmail.com>> > >>>>> Date: Mon, Oct 17, 2022 at 8:15 PM > >>>>> Subject: A quick report from the WISPA conference > >>>>> To: Sina Khanifar <sina@waveform.com <mailto:sina@waveform.com>> > >>>>> Cc: Cake List <cake@lists.bufferbloat.net > <mailto:cake@lists.bufferbloat.net>>, Make-Wifi-fast > >>>>> <make-wifi-fast@lists.bufferbloat.net > <mailto:make-wifi-fast@lists.bufferbloat.net>>, Rpm > >>>>> <rpm@lists.bufferbloat.net <mailto:rpm@lists.bufferbloat.net>>, > Stuart Cheshire <cheshire@apple.com <mailto:cheshire@apple.com>>, > >>>>> bloat <bloat@lists.bufferbloat.net <mailto:bloat@lists.bufferbloat.net>> > >>>>> > >>>>> > >>>>> On Mon, Oct 17, 2022 at 7:51 PM Sina Khanifar <sina@waveform.com > <mailto:sina@waveform.com>> wrote: > >>>>> > > >>>>> > Positive or negative, I can claim a bit of credit for this video > :). We've been working with LTT on a few projects and we pitched them on > doing something around bufferbloat. We've seen more traffic to our Waveforn > test than ever before, which has been fun! > >>>>> > >>>>> Thank you. Great job with that video! And waveform has become the goto > >>>>> site for many now. > >>>>> > >>>>> I can't help but wonder tho... are you collecting any statistics, over > >>>>> time, as to how much better the problem is getting? > >>>>> > >>>>> And any chance they could do something similar explaining wifi? > >>>>> > >>>>> ... > >>>>> > >>>>> I was just at WISPA conference week before last. Preseem's booth > >>>>> (fq_codel) was always packed. Vilo living had put cake in their wifi 6 > >>>>> product. A > >>>>> keynote speaker had deployed it and talked about it with waveform > >>>>> results on the big screen (2k people there). A large wireless vendor > >>>>> demo'd privately to me their flent results before/after cake on their > >>>>> next-gen radios... and people dissed tarana without me prompting for > >>>>> their bad bufferbloat... and the best thing of all that happened to me > >>>>> was... besides getting a hug from a young lady (megan) who'd salvaged > >>>>> her schooling in alaska using sqm - I walked up to the paraqum booth > >>>>> (another large QoE middlebox maker centered more in india) and asked. > >>>>> > >>>>> "So... do y'all have fq_codel yet?" > >>>>> > >>>>> And they smiled and said: "No, we have something better... we've got > cake." > >>>>> > >>>>> "Cake? What's that?" - I said, innocently. > >>>>> > >>>>> They then stepped me through their 200Gbps (!!) product, which uses a > >>>>> bunch of offloads, and can track rtt down to a ms with the intel > >>>>> ethernet card they were using. They'd modifed cake to provide 16 (?) > >>>>> levels of service, and were running under dpdk (I am not sure if cake > >>>>> was). It was a great, convincing pitch... > >>>>> > >>>>> ... then I told 'em who I was. There's a video of the in-both concert > after. > >>>>> > >>>>> ... > >>>>> > >>>>> The downside to me (and the subject of my talk) was that in nearly > >>>>> every person I talked to, fq_codel was viewed as a means to better > >>>>> subscriber bandwidth plan enforcement (which is admittedly the market > >>>>> that preseem pioneered) and it was not understood that I'd got > >>>>> involved in this whole thing because I'd wanted an algorithm to deal > >>>>> with "rain fade", running directly on the radios. People wanted to use > >>>>> the statistics on the radios to drive the plan enforcement better > >>>>> (which is an ok approach, I guess), and for 10+ I'd been whinging > >>>>> about the... physics. > >>>>> > >>>>> So I ranted about rfc7567 a lot and begged people now putting routerOS > >>>>> 7.2 and later out there (mikrotik is huge in this market), to kill > >>>>> their fifos and sfqs at the native rates of the interfaces... and > >>>>> watch their network improve that way also. > >>>>> > >>>>> I think one more wispa conference will be a clean sweep of everyone in > >>>>> the fixed wireless market to not only adopt these algorithms for plan > >>>>> enforcement, but even more directly on the radios and more CPE. > >>>>> > >>>>> I also picked up enough consulting business to keep me busy the rest > >>>>> of this year, and possibly more than I can handle (anybody looking?) > >>>>> > >>>>> I wonder what will happen at a fiber conference? > >>>>> > >>>>> > On Mon, Oct 17, 2022 at 7:45 PM Dave Taht via Bloat > <bloat@lists.bufferbloat.net <mailto:bloat@lists.bufferbloat.net>> wrote: > >>>>> >> > >>>>> >> On Mon, Oct 17, 2022 at 5:02 PM Stuart Cheshire > <cheshire@apple.com <mailto:cheshire@apple.com>> wrote: > >>>>> >> > > >>>>> >> > On 9 Oct 2022, at 06:14, Dave Taht via Make-wifi-fast > <make-wifi-fast@lists.bufferbloat.net > <mailto:make-wifi-fast@lists.bufferbloat.net>> wrote: > >>>>> >> > > >>>>> >> > > This was so massively well done, I cried. Does anyone know how > to get in touch with the ifxit folk? > >>>>> >> > > > >>>>> >> > > https://www.youtube.com/watch?v=UICh3ScfNWI > <https://www.youtube.com/watch?v=UICh3ScfNWI> > >>>>> >> > > >>>>> >> > I’m surprised that you liked this video. It seems to me that it > repeats all the standard misinformation. The analogy they use is the > standard terrible example of waiting in a long line at a grocery store, and > the “solution” is letting certain traffic “jump the line, angering everyone > behind them”. > >>>>> >> > >>>>> >> Accuracy be damned. The analogy to common experience resonates more. > >>>>> >> > >>>>> >> > > >>>>> >> > Some quotes from the video: > >>>>> >> > > >>>>> >> > > it would be so much more efficient for them to let you skip > the line and just check out, especially since you’re in a hurry, but they’re > rudely refusing > >>>>> >> > >>>>> >> I think the person with the cheetos pulling out a gun and shooting > >>>>> >> everyone in front of him (AQM) would not go down well. > >>>>> >> > >>>>> >> > > to go back to our grocery store analogy this would be like if > a worker saw you standing at the back ... and either let you skip to the > front of the line or opens up an express lane just for you > >>>>> >> > >>>>> >> Actually that analogy is fairly close to fair queuing. The multiple > >>>>> >> checker analogy is one of the most common analogies in queue theory > >>>>> >> itself. > >>>>> >> > >>>>> >> > > >>>>> >> > The video describes the problem of bufferbloat, and then > describes the same failed solution that hasn’t worked for the last three > decades. > >>>>> >> > >>>>> >> Hmm? It establishes the scenario, explains the problem *quickly*, > >>>>> >> disses gamer routers for not getting it right.. *points to an > >>>>> >> accurate test*, and then to the ideas and products that *actually > >>>>> >> work* with "smart queueing", with a screenshot of the most common > >>>>> >> (eero's optimize for gaming and videoconferencing), and fq_codel and > >>>>> >> cake *by name*, and points folk at the best known solution available, > >>>>> >> openwrt. > >>>>> >> > >>>>> >> Bing, baddabang, boom. Also the comments were revealing. A goodly > >>>>> >> percentage already knew the problem, more than a few were inspired to > >>>>> >> take the test, > >>>>> >> there was a whole bunch of "Aha!" success stories and 360k views, > >>>>> >> which is more people than we've ever been able to reach in for > >>>>> >> example, a nanog conference. > >>>>> >> > >>>>> >> I loved that folk taking the test actually had quite a few A results, > >>>>> >> without having had to do anything. At least some ISPs are getting it > >>>>> >> more right now! > >>>>> >> > >>>>> >> At this point I think gamers in particular know what "brands" we've > >>>>> >> tried to establish - "Smart queues", "SQM", "OpenWrt", fq_codel and > >>>>> >> now "cake" are "good" things to have, and are stimulating demand by > >>>>> >> asking for them, It's certainly working out better and better for > >>>>> >> evenroute, firewalla, ubnt and others, and I saw an uptick in > >>>>> >> questions about this on various user forums. > >>>>> >> > >>>>> >> I even like that there's a backlash now of people saying "fixing > >>>>> >> bufferbloat doesn't solve everything" - > >>>>> >> > >>>>> >> > Describing the obvious simple-minded (wrong) solution that any > normal person would think of based on their personal human experience > waiting in grocery stores and airports, is not describing the solution to > bufferbloat. The solution to bufferbloat is not that if you are privileged > then you get to “skip to the front of the line”. The solution to bufferbloat > is that there is no line! > >>>>> >> > >>>>> >> I like the idea of a guru floating above a grocery cart with a better > >>>>> >> string of explanations, explaining > >>>>> >> > >>>>> >> - "no, grasshopper, the solution to bufferbloat is no line... > at all". > >>>>> >> > >>>>> >> > > >>>>> >> > With grocery stores and airports people’s arrivals are > independent and not controlled. There is no way for a grocery store or > airport to generate backpressure to tell people to wait at home when a queue > begins to form. The key to solving bufferbloat is generating timely > backpressure to prevent the queue forming in the first place, not accepting > a huge queue and then deciding who deserves special treatment to get better > service than all the other peons who still have to wait in a long queue, > just like before. > >>>>> >> > >>>>> >> I am not huge on the word "backpressure" here. Needs to signal the > >>>>> >> other side to slow down, is more accurate. So might say timely > >>>>> >> signalling rather than timely backpressure? > >>>>> >> > >>>>> >> Other feedback I got was that the video was too smarmy (I agree), > >>>>> >> different audiences than gamers need different forms of outreach... > >>>>> >> > >>>>> >> but to me, winning the gamers has always been one of the most > >>>>> >> important things, as they make a lot of buying decisions, and they > >>>>> >> benefit the most for > >>>>> >> fq and packet prioritization as we do today in gamer routers and in > >>>>> >> cake + qosify. > >>>>> >> > >>>>> >> maybe that gets in the way of more serious markets. Certainly I would > >>>>> >> like another video explaining what goes wrong with videoconferencing. > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > > >>>>> >> > Stuart Cheshire > >>>>> >> > > >>>>> >> > >>>>> >> > >>>>> >> -- > >>>>> >> This song goes out to all the folk that thought Stadia would work: > >>>>> >> > https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz <https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz> > >>>>> >> Dave Täht CEO, TekLibre, LLC > >>>>> >> _______________________________________________ > >>>>> >> Bloat mailing list > >>>>> >> Bloat@lists.bufferbloat.net <mailto:Bloat@lists.bufferbloat.net> > >>>>> >> https://lists.bufferbloat.net/listinfo/bloat > <https://lists.bufferbloat.net/listinfo/bloat> > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> This song goes out to all the folk that thought Stadia would work: > >>>>> > https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz <https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz> > >>>>> Dave Täht CEO, TekLibre, LLC > >>>>> > >>>>> > >>>>> -- > >>>>> This song goes out to all the folk that thought Stadia would work: > >>>>> > https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz <https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz> > >>>>> Dave Täht CEO, TekLibre, LLC > >>>>> _______________________________________________ > >>>>> LibreQoS mailing list > >>>>> LibreQoS@lists.bufferbloat.net <mailto:LibreQoS@lists.bufferbloat.net> > >>>>> https://lists.bufferbloat.net/listinfo/libreqos > <https://lists.bufferbloat.net/listinfo/libreqos> > >>>> > >>>> _______________________________________________ > >>>> LibreQoS mailing list > >>>> LibreQoS@lists.bufferbloat.net <mailto:LibreQoS@lists.bufferbloat.net> > >>>> https://lists.bufferbloat.net/listinfo/libreqos > <https://lists.bufferbloat.net/listinfo/libreqos> > >>> > >>> > >>> > >>> -- > >>> Robert Chacón > >>> CEO | JackRabbit Wireless LLC > >>> Dev | LibreQoS.io > >>> > >>> _______________________________________________ > >>> LibreQoS mailing list > >>> LibreQoS@lists.bufferbloat.net <mailto:LibreQoS@lists.bufferbloat.net> > >>> https://lists.bufferbloat.net/listinfo/libreqos > <https://lists.bufferbloat.net/listinfo/libreqos> > >> > >> _______________________________________________ > >> LibreQoS mailing list > >> LibreQoS@lists.bufferbloat.net <mailto:LibreQoS@lists.bufferbloat.net> > >> https://lists.bufferbloat.net/listinfo/libreqos > <https://lists.bufferbloat.net/listinfo/libreqos> > > > > _______________________________________________ > > LibreQoS mailing list > > LibreQoS@lists.bufferbloat.net <mailto:LibreQoS@lists.bufferbloat.net> > > https://lists.bufferbloat.net/listinfo/libreqos > <https://lists.bufferbloat.net/listinfo/libreqos> > > > > -- > This song goes out to all the folk that thought Stadia would work: > https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz <https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz> > Dave Täht CEO, TekLibre, LLC > -- --Sascha Meinrath Director, X-Lab Palmer Chair in Telecommunications Penn State University ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [LibreQoS] Fwd: A quick report from the WISPA conference 2022-11-12 17:39 ` dan 2022-11-12 19:30 ` [LibreQoS] Various aspects of net neutrality Dave Taht @ 2022-11-13 6:13 ` Hayden Simon 1 sibling, 0 replies; 10+ messages in thread From: Hayden Simon @ 2022-11-13 6:13 UTC (permalink / raw) To: dan, Herbert Wolverson; +Cc: libreqos [-- Attachment #1: Type: text/plain, Size: 20617 bytes --] This list has been fantastic with respect to understanding some of the development choices made. The concerns being discussed in this thread are a non-issue in NZ (perhaps off the back of laws like yours; and privacy ones from the EU). In any event, if what the consumer wants to do doesn’t work – they simply vote with their wallet – I guess you could say we’re all kept fairly honest. In NZ, Gigabit (fibre) “unlimited data” to the home in any major town is ubiquitous and priced ~ $55/month USD. Rurally, unlimited FWA at (atleast) 50mbit down / 10ish up is available from all the major telco’s at a small premium. Yay for tiny populations?..... HAYDEN SIMON UBER GROUP LIMITED MANAGING DIRECTOR ‑ SUPREME OVERLORD [http://uber.co.nz/wp-content/uploads/signatures/uber-facebook-icon.png]<https://www.facebook.com/UberGroup?_rdr=p> [http://uber.co.nz/wp-content/uploads/signatures/uber-twitter-icon.png]<https://twitter.com/ubergroupltd> E: h@uber.nz<mailto:h@uber.nz> M: 021 0707 014<tel:021%200707%20014> W: www.uber.nz<http://www.uber.nz/> 53 PORT ROAD | PO BOX 5083 | WHANGAREI | NEW ZEALAND From: LibreQoS <libreqos-bounces@lists.bufferbloat.net> On Behalf Of dan via LibreQoS Sent: Sunday, 13 November 2022 6:39 am To: Herbert Wolverson <herberticus@gmail.com> Cc: libreqos <libreqos@lists.bufferbloat.net> Subject: Re: [LibreQoS] Fwd: A quick report from the WISPA conference I have to argue with you on 'equally limited' streaming being ok. That allows for 'internet-like' service but no/poor streaming, a service that clamps down really hard on streaming and then offers you an upgrade to make it work. Neutrality doesn't mean 'kind of' neutral. If it's called 'Internet' then no specific service or even type of service should be limited for the provider's direct benefit or whims. I can see no positive case for a provider limiting any service/type below the customers quoted service. The provider gets to save money, but since they offer an inferior product that is essentially false advertised and customers have no way to know this then capitalist tools like consumer choice can't work. Consumers don't know they can get a better suited service, they don't show demand in a market, that prevents competition from having an opportunity, etc. Most people have just 1 fast internet option. Does a customer that can't figure out why their new Roku 4K on their new 4K TV refuses to stream 4K Netflix on their 100M internet service say "Netflix sucks!". or Roku sucks or Sony sucks or whatever? So this spills over into making other services take the blame for bad connections. On Sat, Nov 12, 2022 at 10:19 AM Herbert Wolverson via LibreQoS <libreqos@lists.bufferbloat.net<mailto:libreqos@lists.bufferbloat.net>> wrote: Net neutrality is complex (I'm generally in favor of the concept). There's nothing stopping an ISP from having rules that video streaming be limited - but ALL video streaming has to be equally limited. So ComCast could use Sandvine, but only without any vendor-specific limitations. Same for Cambium. Limit streams that look like video, and you're fine. With that said, it runs into something I say a lot: don't penalize your customers for using your product. If you're buying a 100mbit/s pipe, hooking it to your fancy 4k streaming system - it's entirely reasonable to expect sharp 4k video. In fact, your shaping emphasis should be on keeping their video nice and watchable even though someone just decided to download the newest Call of Duty, update their xbox, and update the firmware on their smart fridge. Likewise, if someone really wants the smallest 5 mbit connection you once offered - because they only ever use it to check email (we actually had a customer complain when we offered them a free upgrade from a 5mbit/s plan, years ago!) - you want to do your best to make it a usable tiny plan for them. Fair queuing (along with a bit of user education; we eventually convinced the customer that "no change in fees" actually meant not charging them more money) helps with all of that. It can win customers, too. We have a couple of business customers who switched us from their backup to their primary because "our service felt snappier". We had a business go with us because their 24x7 lobby streaming video looked better on our (smaller) connection, and a couple of fraternities who likewise decided to use us because their overkill 4k video setup looked better (even with the 20+ Xboxes we could see on their network). QUIC makes me chuckle, because it's exactly what we were doing in gamedev land in the late 90s to make deathmatches run smoothly. (Seriously, QuakeWorld and the original Unreal Tournament had the most amazing network code; they basically implemented what is now known as "reliable UDP" to incredible results). With that said, I think we've got a remarkable amount of wiggle-room to make things better (and it's not at all shoddy right now!). XDP is amazing as-is, and is improving fast. (I'm currently reading https://github.com/ilejn/xdpbridge - an older project that doesn't seem to have gained much traction, but I'm wondering if it couldn't provide some seriously fast bridge offloading; I'll have to setup a better test environment to find out) On Sat, Nov 12, 2022 at 10:09 AM Robert Chacón via LibreQoS <libreqos@lists.bufferbloat.net<mailto:libreqos@lists.bufferbloat.net>> wrote: Thank you, Dan. I completely agree with you 100% there. It's a risky way to save a tiny amount of money on bandwidth costs. Comcast paid out $16 million in a settlement<https://arstechnica.com/tech-policy/2010/04/comcast-owes-p2p-users-16-yes-they-should-take-it/?comments=1&post=20329187> over using Sandvine's DPI in this way. Since then, Comcast has switched from DPI to fair queueing with DOCSIS-pie<https://www.reddit.com/r/eero/comments/rc8e2j/getting_a_bufferbloat_grade_with_comcast_aqm/>. California's SB-822<https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=201720180SB822&fbclid=IwAR2LmNyy5ZkbtzayIyNerJ-Q_OUwK4G0H4aYv4XZwVNz-QF41d5v0BiST-E> will be used as a template by many other states going forward on Net Neutrality. SB-822 even clarified that "Reasonable Network Management" needs to be "as application-agnostic as possible". These DPI approaches like Cambium's will not hold up in court the way Fair Queuing (Preseem, LibreQoS) based solutions will. We can point to years of public research and RFCs when defending a Fair Queuing QoE approach. They can't so easily defend using a product that advertised its big "scale down 4k to save money" knob or "accelerate their speeds only when they run speed tests" button. It's concerning to me that these QoE vendors are not disclosing "hey, don't use this in NN states like California please". The WISP industry will be hurt long-term by these products with consumers viewing us as an inferior technology that limits their usage in intrusive ways. Like you said, let's let them use their plan however they want. It doesn't hurt us. But things like capping end-users at HD will hurt us long-term in lost sales. Regarding TCP acceleration, I'm leaning toward the argument made by Dan from Preseem<https://youtu.be/3r4FWGKho3c?t=1879> that QUIC adoption is growing pretty quickly and we may not want to mess with TCP at the expense of UDP given the rise of QUIC. Cloudlfare says HTTP/3 and QUIC are already at 28% and growing<https://radar.cloudflare.com/adoption-and-usage?range=28d>. One point he brought up is that if we use TCP acceleration, and a middlebox has to be rebooted or something - all client TCP sessions will break when OSPF switches paths around the box. On Sat, Nov 12, 2022 at 8:36 AM dan via LibreQoS <libreqos@lists.bufferbloat.net<mailto:libreqos@lists.bufferbloat.net>> wrote: I think there's a thin grey line between what other vendors (paraqum, bequant/cambium) and libreQoS is trying to do. The first two are REALLY pushing this selecting narrowing of traffic and they'll go straight after streaming for their demos. Most people that drop in the cambium QoE appliance turn the knobs on Netflix down and then praise the 40% savings on their network. Same with PQ. I think this is fundamentally wrong. This is the entire reason Net Neutrality keeps popping up. Why does the ISP get to say "You only need 25Mbps for your netflix". That's double wrong if they offer their own video services. It's not about improving the customer's experience, it's about spending less on upstream bandwidth at the customer's expense. As soon as they say 'pays for itself in bandwidth savings' you know what the product is primarily for. This is why I like the preseem and libreqos model of just using a 'stock' cake and just finessing the data coming in to keep the pipe from clogging. Who cares if someone pulls 90M bursts of apple TV+ on their 100M pipe. Only dial that back so other requests they make get through cleanly. preseem is going more towards wireless monitoring etc and kinda letting shaping be a secondary.... if they were doing more libreqos like stuff I'd just stick with preseem. libreQoS may end up being the best and most net neutral product you can get. The only current disadvantage (ignoring DPI) is lack of TCP accelleration that PQ and Bequant/cambium have. I should add that 3 states already have laws on the books that straight up make PQ and Cambium's solution illegal because they target certain services for throttling. My state (Montana) has the same rule except only when working with a government contract. Libreqos is actually compliant (as is preseem). On Sat, Nov 12, 2022 at 8:11 AM Dave Taht via LibreQoS <libreqos@lists.bufferbloat.net<mailto:libreqos@lists.bufferbloat.net>> wrote: this report predates the libreqos list... ---------- Forwarded message --------- From: Dave Taht <dave.taht@gmail.com<mailto:dave.taht@gmail.com>> Date: Mon, Oct 17, 2022 at 8:15 PM Subject: A quick report from the WISPA conference To: Sina Khanifar <sina@waveform.com<mailto:sina@waveform.com>> Cc: Cake List <cake@lists.bufferbloat.net<mailto:cake@lists.bufferbloat.net>>, Make-Wifi-fast <make-wifi-fast@lists.bufferbloat.net<mailto:make-wifi-fast@lists.bufferbloat.net>>, Rpm <rpm@lists.bufferbloat.net<mailto:rpm@lists.bufferbloat.net>>, Stuart Cheshire <cheshire@apple.com<mailto:cheshire@apple.com>>, bloat <bloat@lists.bufferbloat.net<mailto:bloat@lists.bufferbloat.net>> On Mon, Oct 17, 2022 at 7:51 PM Sina Khanifar <sina@waveform.com<mailto:sina@waveform.com>> wrote: > > Positive or negative, I can claim a bit of credit for this video :). We've been working with LTT on a few projects and we pitched them on doing something around bufferbloat. We've seen more traffic to our Waveforn test than ever before, which has been fun! Thank you. Great job with that video! And waveform has become the goto site for many now. I can't help but wonder tho... are you collecting any statistics, over time, as to how much better the problem is getting? And any chance they could do something similar explaining wifi? ... I was just at WISPA conference week before last. Preseem's booth (fq_codel) was always packed. Vilo living had put cake in their wifi 6 product. A keynote speaker had deployed it and talked about it with waveform results on the big screen (2k people there). A large wireless vendor demo'd privately to me their flent results before/after cake on their next-gen radios... and people dissed tarana without me prompting for their bad bufferbloat... and the best thing of all that happened to me was... besides getting a hug from a young lady (megan) who'd salvaged her schooling in alaska using sqm - I walked up to the paraqum booth (another large QoE middlebox maker centered more in india) and asked. "So... do y'all have fq_codel yet?" And they smiled and said: "No, we have something better... we've got cake." "Cake? What's that?" - I said, innocently. They then stepped me through their 200Gbps (!!) product, which uses a bunch of offloads, and can track rtt down to a ms with the intel ethernet card they were using. They'd modifed cake to provide 16 (?) levels of service, and were running under dpdk (I am not sure if cake was). It was a great, convincing pitch... ... then I told 'em who I was. There's a video of the in-both concert after. ... The downside to me (and the subject of my talk) was that in nearly every person I talked to, fq_codel was viewed as a means to better subscriber bandwidth plan enforcement (which is admittedly the market that preseem pioneered) and it was not understood that I'd got involved in this whole thing because I'd wanted an algorithm to deal with "rain fade", running directly on the radios. People wanted to use the statistics on the radios to drive the plan enforcement better (which is an ok approach, I guess), and for 10+ I'd been whinging about the... physics. So I ranted about rfc7567 a lot and begged people now putting routerOS 7.2 and later out there (mikrotik is huge in this market), to kill their fifos and sfqs at the native rates of the interfaces... and watch their network improve that way also. I think one more wispa conference will be a clean sweep of everyone in the fixed wireless market to not only adopt these algorithms for plan enforcement, but even more directly on the radios and more CPE. I also picked up enough consulting business to keep me busy the rest of this year, and possibly more than I can handle (anybody looking?) I wonder what will happen at a fiber conference? > On Mon, Oct 17, 2022 at 7:45 PM Dave Taht via Bloat <bloat@lists.bufferbloat.net<mailto:bloat@lists.bufferbloat.net>> wrote: >> >> On Mon, Oct 17, 2022 at 5:02 PM Stuart Cheshire <cheshire@apple.com<mailto:cheshire@apple.com>> wrote: >> > >> > On 9 Oct 2022, at 06:14, Dave Taht via Make-wifi-fast <make-wifi-fast@lists.bufferbloat.net<mailto:make-wifi-fast@lists.bufferbloat.net>> wrote: >> > >> > > This was so massively well done, I cried. Does anyone know how to get in touch with the ifxit folk? >> > > >> > > https://www.youtube.com/watch?v=UICh3ScfNWI >> > >> > I’m surprised that you liked this video. It seems to me that it repeats all the standard misinformation. The analogy they use is the standard terrible example of waiting in a long line at a grocery store, and the “solution” is letting certain traffic “jump the line, angering everyone behind them”. >> >> Accuracy be damned. The analogy to common experience resonates more. >> >> > >> > Some quotes from the video: >> > >> > > it would be so much more efficient for them to let you skip the line and just check out, especially since you’re in a hurry, but they’re rudely refusing >> >> I think the person with the cheetos pulling out a gun and shooting >> everyone in front of him (AQM) would not go down well. >> >> > > to go back to our grocery store analogy this would be like if a worker saw you standing at the back ... and either let you skip to the front of the line or opens up an express lane just for you >> >> Actually that analogy is fairly close to fair queuing. The multiple >> checker analogy is one of the most common analogies in queue theory >> itself. >> >> > >> > The video describes the problem of bufferbloat, and then describes the same failed solution that hasn’t worked for the last three decades. >> >> Hmm? It establishes the scenario, explains the problem *quickly*, >> disses gamer routers for not getting it right.. *points to an >> accurate test*, and then to the ideas and products that *actually >> work* with "smart queueing", with a screenshot of the most common >> (eero's optimize for gaming and videoconferencing), and fq_codel and >> cake *by name*, and points folk at the best known solution available, >> openwrt. >> >> Bing, baddabang, boom. Also the comments were revealing. A goodly >> percentage already knew the problem, more than a few were inspired to >> take the test, >> there was a whole bunch of "Aha!" success stories and 360k views, >> which is more people than we've ever been able to reach in for >> example, a nanog conference. >> >> I loved that folk taking the test actually had quite a few A results, >> without having had to do anything. At least some ISPs are getting it >> more right now! >> >> At this point I think gamers in particular know what "brands" we've >> tried to establish - "Smart queues", "SQM", "OpenWrt", fq_codel and >> now "cake" are "good" things to have, and are stimulating demand by >> asking for them, It's certainly working out better and better for >> evenroute, firewalla, ubnt and others, and I saw an uptick in >> questions about this on various user forums. >> >> I even like that there's a backlash now of people saying "fixing >> bufferbloat doesn't solve everything" - >> >> > Describing the obvious simple-minded (wrong) solution that any normal person would think of based on their personal human experience waiting in grocery stores and airports, is not describing the solution to bufferbloat. The solution to bufferbloat is not that if you are privileged then you get to “skip to the front of the line”. The solution to bufferbloat is that there is no line! >> >> I like the idea of a guru floating above a grocery cart with a better >> string of explanations, explaining >> >> - "no, grasshopper, the solution to bufferbloat is no line... at all". >> >> > >> > With grocery stores and airports people’s arrivals are independent and not controlled. There is no way for a grocery store or airport to generate backpressure to tell people to wait at home when a queue begins to form. The key to solving bufferbloat is generating timely backpressure to prevent the queue forming in the first place, not accepting a huge queue and then deciding who deserves special treatment to get better service than all the other peons who still have to wait in a long queue, just like before. >> >> I am not huge on the word "backpressure" here. Needs to signal the >> other side to slow down, is more accurate. So might say timely >> signalling rather than timely backpressure? >> >> Other feedback I got was that the video was too smarmy (I agree), >> different audiences than gamers need different forms of outreach... >> >> but to me, winning the gamers has always been one of the most >> important things, as they make a lot of buying decisions, and they >> benefit the most for >> fq and packet prioritization as we do today in gamer routers and in >> cake + qosify. >> >> maybe that gets in the way of more serious markets. Certainly I would >> like another video explaining what goes wrong with videoconferencing. >> >> >> >> >> >> >> > >> > Stuart Cheshire >> > >> >> >> -- >> This song goes out to all the folk that thought Stadia would work: >> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz >> Dave Täht CEO, TekLibre, LLC >> _______________________________________________ >> Bloat mailing list >> Bloat@lists.bufferbloat.net<mailto:Bloat@lists.bufferbloat.net> >> https://lists.bufferbloat.net/listinfo/bloat -- This song goes out to all the folk that thought Stadia would work: https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz Dave Täht CEO, TekLibre, LLC -- This song goes out to all the folk that thought Stadia would work: https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz Dave Täht CEO, TekLibre, LLC _______________________________________________ LibreQoS mailing list LibreQoS@lists.bufferbloat.net<mailto:LibreQoS@lists.bufferbloat.net> https://lists.bufferbloat.net/listinfo/libreqos _______________________________________________ LibreQoS mailing list LibreQoS@lists.bufferbloat.net<mailto:LibreQoS@lists.bufferbloat.net> https://lists.bufferbloat.net/listinfo/libreqos -- Robert Chacón CEO | JackRabbit Wireless LLC<http://jackrabbitwireless.com> Dev | LibreQoS.io<http://LibreQoS.io> _______________________________________________ LibreQoS mailing list LibreQoS@lists.bufferbloat.net<mailto:LibreQoS@lists.bufferbloat.net> https://lists.bufferbloat.net/listinfo/libreqos _______________________________________________ LibreQoS mailing list LibreQoS@lists.bufferbloat.net<mailto:LibreQoS@lists.bufferbloat.net> https://lists.bufferbloat.net/listinfo/libreqos [-- Attachment #2: Type: text/html, Size: 37467 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2022-11-14 13:45 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <CAA93jw77h=ztEOzyADriH2PnswUDQiyNvBdsuFi+K5EexpoxUQ@mail.gmail.com>
[not found] ` <938D9D45-DADA-4291-BD8A-84E4257CEE49@apple.com>
[not found] ` <CAA93jw4KOkgdfT2LunCtPYPjXL+=OtTrouJgPjM7U1bHKtErnw@mail.gmail.com>
[not found] ` <CACTgmGpgDjWF4d_+Kga4CL4vxb-YQ91Lu1U6Zt5vca0EGSwQ2w@mail.gmail.com>
[not found] ` <CAA93jw4f701R+4B538jF1+qAW=cUgP35EmWy8VZG-1h=w8woOA@mail.gmail.com>
2022-11-12 15:11 ` [LibreQoS] Fwd: A quick report from the WISPA conference Dave Taht
2022-11-12 15:36 ` dan
2022-11-12 16:09 ` Robert Chacón
2022-11-12 16:15 ` dan
2022-11-12 17:19 ` Herbert Wolverson
2022-11-12 17:39 ` dan
2022-11-12 19:30 ` [LibreQoS] Various aspects of net neutrality Dave Taht
2022-11-12 19:58 ` dan
2022-11-14 13:45 ` Sascha Meinrath
2022-11-13 6:13 ` [LibreQoS] Fwd: A quick report from the WISPA conference Hayden Simon
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox