[LibreQoS] Fwd: A quick report from the WISPA conference

Sat Nov 12 12:39:03 EST 2022

I have to argue with you on 'equally limited' streaming being ok.  That
allows for 'internet-like' service but no/poor streaming, a service that
clamps down really hard on streaming and then offers you an upgrade to make
it work.  Neutrality doesn't mean 'kind of' neutral.  If it's called
'Internet' then no specific service or even type of service should be
limited for the provider's direct benefit or whims.  I can see no positive
case for a provider limiting any service/type below the customers quoted
service.  The provider gets to save money, but since they offer an inferior
product that is essentially false advertised and customers have no way to
know this then capitalist tools like consumer choice can't work.  Consumers
don't know they can get a better suited service, they don't show demand in
a market, that prevents competition from having an opportunity, etc.   Most
people have just 1 fast internet option.

Does a customer that can't figure out why their new Roku 4K on their new 4K
TV refuses to stream 4K Netflix on their 100M internet service say "Netflix
sucks!".  or Roku sucks or Sony sucks or whatever?  So this spills over
into making other services take the blame for bad connections.

On Sat, Nov 12, 2022 at 10:19 AM Herbert Wolverson via LibreQoS <
libreqos at lists.bufferbloat.net> wrote:

> Net neutrality is complex (I'm generally in favor of the concept). There's
> nothing stopping an ISP from having rules that video streaming be limited -
> but ALL video streaming has to be equally limited. So ComCast could use
> Sandvine, but only without any vendor-specific limitations. Same for
> Cambium. Limit streams that look like video, and you're fine. With that
> said, it runs into something I say a lot: don't penalize your customers for
> using your product. If you're buying a 100mbit/s pipe, hooking it to your
> fancy 4k streaming system - it's entirely reasonable to expect sharp 4k
> video. In fact, your shaping emphasis should be on keeping their video nice
> and watchable even though someone just decided to download the newest Call
> of Duty, update their xbox, and update the firmware on their smart fridge.
> Likewise, if someone really wants the smallest 5 mbit connection you once
> offered - because they only ever use it to check email (we actually had a
> customer complain when we offered them a free upgrade from a 5mbit/s plan,
> years ago!) - you want to do your best to make it a usable tiny plan for
> them. Fair queuing (along with a bit of user education; we eventually
> convinced the customer that "no change in fees" actually meant not charging
> them more money) helps with all of that.
>
> It can win customers, too. We have a couple of business customers who
> switched us from their backup to their primary because "our service felt
> snappier". We had a business go with us because their 24x7 lobby streaming
> video looked better on our (smaller) connection, and a couple of
> fraternities who likewise decided to use us because their overkill 4k video
> setup looked better (even with the 20+ Xboxes we could see on their
> network).
>
> QUIC makes me chuckle, because it's *exactly* what we were doing in
> gamedev land in the late 90s to make deathmatches run smoothly. (Seriously,
> QuakeWorld and the original Unreal Tournament had the most amazing network
> code; they basically implemented what is now known as "reliable UDP" to
> incredible results).
>
> With that said, I think we've got a remarkable amount of wiggle-room to
> make things better (and it's not at all shoddy right now!). XDP is amazing
> as-is, and is improving fast. (I'm currently reading
> https://github.com/ilejn/xdpbridge - an older project that doesn't seem
> to have gained much traction, but I'm wondering if it couldn't provide some
> seriously fast bridge offloading; I'll have to setup a better test
> environment to find out)
>
> On Sat, Nov 12, 2022 at 10:09 AM Robert Chacón via LibreQoS <
> libreqos at lists.bufferbloat.net> wrote:
>
>> Thank you, Dan.
>> I completely agree with you 100% there. It's a risky way to save a tiny
>> amount of money on bandwidth costs.
>> Comcast paid out $16 million in a settlement
>> <https://arstechnica.com/tech-policy/2010/04/comcast-owes-p2p-users-16-yes-they-should-take-it/?comments=1&post=20329187>
>> over using Sandvine's DPI in this way.
>> Since then, Comcast has switched from DPI to fair queueing with
>> DOCSIS-pie
>> <https://www.reddit.com/r/eero/comments/rc8e2j/getting_a_bufferbloat_grade_with_comcast_aqm/>
>> .
>>
>> California's SB-822
>> <https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=201720180SB822&fbclid=IwAR2LmNyy5ZkbtzayIyNerJ-Q_OUwK4G0H4aYv4XZwVNz-QF41d5v0BiST-E>
>> will be used as a template by many other states going forward on Net
>> Neutrality.
>> <https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=201720180SB822&fbclid=IwAR2LmNyy5ZkbtzayIyNerJ-Q_OUwK4G0H4aYv4XZwVNz-QF41d5v0BiST-E>
>> SB-822 even clarified that "Reasonable Network Management" needs to be
>> "as application-agnostic as possible".
>> These DPI approaches like Cambium's will not hold up in court the way
>> Fair Queuing (Preseem, LibreQoS) based solutions will.
>> We can point to years of public research and RFCs when defending a Fair
>> Queuing QoE approach.
>> They can't so easily defend using a product that advertised its big
>> "scale down 4k to save money" knob or "accelerate their speeds only when
>> they run speed tests" button.
>> It's concerning to me that these QoE vendors are not disclosing "hey,
>> don't use this in NN states like California please".
>> The WISP industry will be hurt long-term by these products with consumers
>> viewing us as an inferior technology that limits their usage in intrusive
>> ways.
>> Like you said, let's let them use their plan however they want. It
>> doesn't hurt us. But things like capping end-users at HD will hurt us
>> long-term in lost sales.
>>
>> Regarding TCP acceleration, I'm leaning toward the argument made by Dan
>> from Preseem <https://youtu.be/3r4FWGKho3c?t=1879> that QUIC adoption is
>> growing pretty quickly and we may not want to mess with TCP at the expense
>> of UDP given the rise of QUIC.
>> Cloudlfare says HTTP/3 and QUIC are already at 28% and growing
>> <https://radar.cloudflare.com/adoption-and-usage?range=28d>.
>> One point he brought up is that if we use TCP acceleration, and a
>> middlebox has to be rebooted or something - all client TCP sessions will
>> break when OSPF switches paths around the box.
>>
>> On Sat, Nov 12, 2022 at 8:36 AM dan via LibreQoS <
>> libreqos at lists.bufferbloat.net> wrote:
>>
>>> I think there's a thin grey line between what other vendors (paraqum,
>>> bequant/cambium) and libreQoS is trying to do.  The first two are REALLY
>>> pushing this selecting narrowing of traffic and they'll go straight after
>>> streaming for their demos.  Most people that drop in the cambium QoE
>>> appliance turn the knobs on Netflix down and then praise the 40% savings on
>>> their network.  Same with PQ.   I think this is fundamentally wrong.  This
>>> is the entire reason Net Neutrality keeps popping up.  Why does the ISP get
>>> to say "You only need 25Mbps for your netflix".  That's double wrong if
>>> they offer their own video services.  It's not about improving the
>>> customer's experience, it's about spending less on upstream bandwidth at
>>> the customer's expense.  As soon as they say 'pays for itself in bandwidth
>>> savings' you know what the product is primarily for.
>>>
>>> This is why I like the preseem and libreqos model of just using a
>>> 'stock' cake and just finessing the data coming in to keep the pipe from
>>> clogging.  Who cares if someone pulls 90M bursts of apple TV+ on their 100M
>>> pipe.  Only dial that back so other requests they make get through cleanly.
>>>
>>> preseem is going more towards wireless monitoring etc and kinda letting
>>> shaping be a secondary.... if they were doing more libreqos like stuff I'd
>>> just stick with preseem.
>>>
>>> libreQoS may end up being the best and most net neutral product you can
>>> get.  The only current disadvantage (ignoring DPI) is lack of TCP
>>> accelleration that PQ and Bequant/cambium have.
>>>
>>> I should add that 3 states already have laws on the books that straight
>>> up make PQ and Cambium's solution illegal because they target certain
>>> services for throttling.  My state (Montana) has the same rule except only
>>> when working with a government contract.  Libreqos is actually compliant
>>> (as is preseem).
>>>
>>> On Sat, Nov 12, 2022 at 8:11 AM Dave Taht via LibreQoS <
>>> libreqos at lists.bufferbloat.net> wrote:
>>>
>>>> this report predates the libreqos list...
>>>>
>>>> ---------- Forwarded message ---------
>>>> From: Dave Taht <dave.taht at gmail.com>
>>>> Date: Mon, Oct 17, 2022 at 8:15 PM
>>>> Subject: A quick report from the WISPA conference
>>>> To: Sina Khanifar <sina at waveform.com>
>>>> Cc: Cake List <cake at lists.bufferbloat.net>, Make-Wifi-fast
>>>> <make-wifi-fast at lists.bufferbloat.net>, Rpm
>>>> <rpm at lists.bufferbloat.net>, Stuart Cheshire <cheshire at apple.com>,
>>>> bloat <bloat at lists.bufferbloat.net>
>>>>
>>>>
>>>> On Mon, Oct 17, 2022 at 7:51 PM Sina Khanifar <sina at waveform.com>
>>>> wrote:
>>>> >
>>>> > Positive or negative, I can claim a bit of credit for this video :).
>>>> We've been working with LTT on a few projects and we pitched them on doing
>>>> something around bufferbloat. We've seen more traffic to our Waveforn test
>>>> than ever before, which has been fun!
>>>>
>>>> Thank you. Great job with that video! And waveform has become the goto
>>>> site for many now.
>>>>
>>>> I can't help but wonder tho... are you collecting any statistics, over
>>>> time, as to how much better the problem is getting?
>>>>
>>>> And any chance they could do something similar explaining wifi?
>>>>
>>>> ...
>>>>
>>>> I was just at WISPA conference week before last. Preseem's booth
>>>> (fq_codel) was always packed. Vilo living had put cake in their wifi 6
>>>> product. A
>>>> keynote speaker had deployed it and talked about it with waveform
>>>> results on the big screen (2k people there). A large wireless vendor
>>>> demo'd privately to me their flent results before/after cake on their
>>>> next-gen radios... and people dissed tarana without me prompting for
>>>> their bad bufferbloat... and the best thing of all that happened to me
>>>> was... besides getting a hug from a young lady (megan) who'd salvaged
>>>> her schooling in alaska using sqm - I walked up to the paraqum booth
>>>> (another large QoE middlebox maker centered more in india) and asked.
>>>>
>>>> "So... do y'all have fq_codel yet?"
>>>>
>>>> And they smiled and said: "No, we have something better... we've got
>>>> cake."
>>>>
>>>> "Cake? What's that?" - I said, innocently.
>>>>
>>>> They then stepped me through their 200Gbps (!!) product, which uses a
>>>> bunch of offloads, and can track rtt down to a ms with the intel
>>>> ethernet card they were using. They'd modifed cake to provide 16 (?)
>>>> levels of service, and were running under dpdk (I am not sure if cake
>>>> was). It was a great, convincing pitch...
>>>>
>>>> ... then I told 'em who I was. There's a video of the in-both concert
>>>> after.
>>>>
>>>> ...
>>>>
>>>> The downside to me (and the subject of my talk) was that in nearly
>>>> every person I talked to, fq_codel was viewed as a means to better
>>>> subscriber bandwidth plan enforcement (which is admittedly the market
>>>> that preseem pioneered) and it was not understood that I'd got
>>>> involved in this whole thing because I'd wanted an algorithm to deal
>>>> with "rain fade", running directly on the radios. People wanted to use
>>>> the statistics on the radios to drive the plan enforcement better
>>>> (which is an ok approach, I guess), and for 10+ I'd been whinging
>>>> about the... physics.
>>>>
>>>> So I ranted about rfc7567 a lot and begged people now putting routerOS
>>>> 7.2 and later out there (mikrotik is huge in this market), to kill
>>>> their fifos and sfqs at the native rates of the interfaces... and
>>>> watch their network improve that way also.
>>>>
>>>> I think one more wispa conference will be a clean sweep of everyone in
>>>> the fixed wireless market to not only adopt these algorithms for plan
>>>> enforcement, but even more directly on the radios and more CPE.
>>>>
>>>> I also picked up enough consulting business to keep me busy the rest
>>>> of this year, and possibly more than I can handle (anybody looking?)
>>>>
>>>> I wonder what will happen at a fiber conference?
>>>>
>>>> > On Mon, Oct 17, 2022 at 7:45 PM Dave Taht via Bloat <
>>>> bloat at lists.bufferbloat.net> wrote:
>>>> >>
>>>> >> On Mon, Oct 17, 2022 at 5:02 PM Stuart Cheshire <cheshire at apple.com>
>>>> wrote:
>>>> >> >
>>>> >> > On 9 Oct 2022, at 06:14, Dave Taht via Make-wifi-fast <
>>>> make-wifi-fast at lists.bufferbloat.net> wrote:
>>>> >> >
>>>> >> > > This was so massively well done, I cried. Does anyone know how
>>>> to get in touch with the ifxit folk?
>>>> >> > >
>>>> >> > > https://www.youtube.com/watch?v=UICh3ScfNWI
>>>> >> >
>>>> >> > I’m surprised that you liked this video. It seems to me that it
>>>> repeats all the standard misinformation. The analogy they use is the
>>>> standard terrible example of waiting in a long line at a grocery store, and
>>>> the “solution” is letting certain traffic “jump the line, angering everyone
>>>> behind them”.
>>>> >>
>>>> >> Accuracy be damned. The analogy to common experience resonates more.
>>>> >>
>>>> >> >
>>>> >> > Some quotes from the video:
>>>> >> >
>>>> >> > > it would be so much more efficient for them to let you skip the
>>>> line and just check out, especially since you’re in a hurry, but they’re
>>>> rudely refusing
>>>> >>
>>>> >> I think the person with the cheetos pulling out a gun and shooting
>>>> >> everyone in front of him (AQM) would not go down well.
>>>> >>
>>>> >> > > to go back to our grocery store analogy this would be like if a
>>>> worker saw you standing at the back ... and either let you skip to the
>>>> front of the line or opens up an express lane just for you
>>>> >>
>>>> >> Actually that analogy is fairly close to fair queuing. The multiple
>>>> >> checker analogy is one of the most common analogies in queue theory
>>>> >> itself.
>>>> >>
>>>> >> >
>>>> >> > The video describes the problem of bufferbloat, and then describes
>>>> the same failed solution that hasn’t worked for the last three decades.
>>>> >>
>>>> >> Hmm? It establishes the scenario, explains the problem *quickly*,
>>>> >> disses gamer routers for not getting it right..  *points to an
>>>> >> accurate test*, and then to the ideas and products that *actually
>>>> >> work* with "smart queueing", with a screenshot of the most common
>>>> >> (eero's optimize for gaming and videoconferencing), and fq_codel and
>>>> >> cake *by name*, and points folk at the best known solution available,
>>>> >> openwrt.
>>>> >>
>>>> >> Bing, baddabang, boom. Also the comments were revealing. A goodly
>>>> >> percentage already knew the problem, more than a few were inspired to
>>>> >> take the test,
>>>> >> there was a whole bunch of "Aha!" success stories and 360k views,
>>>> >> which is more people than we've ever been able to reach in for
>>>> >> example, a nanog conference.
>>>> >>
>>>> >> I loved that folk taking the test actually had quite a few A results,
>>>> >> without having had to do anything. At least some ISPs are getting it
>>>> >> more right now!
>>>> >>
>>>> >> At this point I think gamers in particular know what "brands" we've
>>>> >> tried to establish - "Smart queues", "SQM", "OpenWrt", fq_codel and
>>>> >> now "cake" are "good" things to have, and are stimulating demand by
>>>> >> asking for them,   It's certainly working out better and better for
>>>> >> evenroute, firewalla, ubnt and others, and I saw an uptick in
>>>> >> questions about this on various user forums.
>>>> >>
>>>> >> I even like that there's a backlash now of people saying "fixing
>>>> >> bufferbloat doesn't solve everything" -
>>>> >>
>>>> >> >  Describing the obvious simple-minded (wrong) solution that any
>>>> normal person would think of based on their personal human experience
>>>> waiting in grocery stores and airports, is not describing the solution to
>>>> bufferbloat. The solution to bufferbloat is not that if you are privileged
>>>> then you get to “skip to the front of the line”. The solution to
>>>> bufferbloat is that there is no line!
>>>> >>
>>>> >> I like the idea of a guru floating above a grocery cart with a better
>>>> >> string of explanations, explaining
>>>> >>
>>>> >>    - "no, grasshopper, the solution to bufferbloat is no line... at
>>>> all".
>>>> >>
>>>> >> >
>>>> >> > With grocery stores and airports people’s arrivals are independent
>>>> and not controlled. There is no way for a grocery store or airport to
>>>> generate backpressure to tell people to wait at home when a queue begins to
>>>> form. The key to solving bufferbloat is generating timely backpressure to
>>>> prevent the queue forming in the first place, not accepting a huge queue
>>>> and then deciding who deserves special treatment to get better service than
>>>> all the other peons who still have to wait in a long queue, just like
>>>> before.
>>>> >>
>>>> >> I am not huge on the word "backpressure" here. Needs to signal the
>>>> >> other side to slow down, is more accurate. So might say timely
>>>> >> signalling rather than timely backpressure?
>>>> >>
>>>> >> Other feedback I got  was that the video was too smarmy (I agree),
>>>> >> different audiences than gamers need different forms of outreach...
>>>> >>
>>>> >> but to me, winning the gamers has always been one of the most
>>>> >> important things, as they make a lot of buying decisions, and they
>>>> >> benefit the most for
>>>> >> fq and packet prioritization as we do today in gamer routers and in
>>>> >> cake + qosify.
>>>> >>
>>>> >> maybe that gets in the way of more serious markets. Certainly I would
>>>> >> like another video explaining what goes wrong with videoconferencing.
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> >
>>>> >> > Stuart Cheshire
>>>> >> >
>>>> >>
>>>> >>
>>>> >> --
>>>> >> This song goes out to all the folk that thought Stadia would work:
>>>> >>
>>>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
>>>> >> Dave Täht CEO, TekLibre, LLC
>>>> >> _______________________________________________
>>>> >> Bloat mailing list
>>>> >> Bloat at lists.bufferbloat.net
>>>> >> https://lists.bufferbloat.net/listinfo/bloat
>>>>
>>>>
>>>>
>>>> --
>>>> This song goes out to all the folk that thought Stadia would work:
>>>>
>>>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
>>>> Dave Täht CEO, TekLibre, LLC
>>>>
>>>>
>>>> --
>>>> This song goes out to all the folk that thought Stadia would work:
>>>>
>>>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
>>>> Dave Täht CEO, TekLibre, LLC
>>>> _______________________________________________
>>>> LibreQoS mailing list
>>>> LibreQoS at lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/libreqos
>>>>
>>> _______________________________________________
>>> LibreQoS mailing list
>>> LibreQoS at lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/libreqos
>>>
>>
>>
>> --
>> Robert Chacón
>> CEO | JackRabbit Wireless LLC <http://jackrabbitwireless.com>
>> Dev | LibreQoS.io
>>
>> _______________________________________________
>> LibreQoS mailing list
>> LibreQoS at lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/libreqos
>>
> _______________________________________________
> LibreQoS mailing list
> LibreQoS at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/libreqos
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/libreqos/attachments/20221112/99af8e6e/attachment-0001.html>