[Bloat] A quick report from the WISPA conference
Sina Khanifar
sina at waveform.com
Thu Oct 20 01:15:47 EDT 2022
Hi Sebastian,
>
> [SM] Just an observation, using Safari I see large maximal delays (like a
> small group of samples far out to the right of the bulk) for both down-
> and upload that essentially disappear when I switch to firefox. Now I tend
> to have a ton of tabs open in Safari while I only open firefox for
> dedicated use-cases with a few tabs at most, so I do not intend to throw
> shade on Safari here; my point is more browsers can and do affect the
> reported latency numbers, of you want to be able to test this, maybe ask
> users to use the OS browser (safari, edge, konqueror ;) ) as well as
> firefox and chrome so you can directly compare across browsers?
>
I believe this is because we use the WebTiming APIs to get a more accurate latency numbers, but the API isn't fully supported on Safari. As such, latency measurements in Safari are much less accurate than in Firefox and Chrome.
>
> traceroute/mtr albeit not sure how well this approach works from inside
> the browser, can you e.g. control TTL and do you receive error messages
> via ICMP?
>
Unfortunately traceroutes via the browser don't really work :(. And I don't believe we can control TTL or see ICMP error messages either, though I haven't dug into this very deeply.
>
>
>
> Over in the OpenWrt forum we often see that server performance with
> iperf2/3 or netperf on a router is not all that representative for its
> routing performance. What do you expect to deduce from upload/download to
> the router? (I might misunderstand your point by a mile, if so please
> elaborate)
>
>
>
The goal would be to test the "local" latency, throughput, and bufferbloat between the user's device and the router, and then compare this with the latency, throughput, and bufferbloat when DL/ULing to a remote server.
This would reveal whether the dominant source of increase in latency under load is at the router's WAN interface or somewhere between the router and the user (e.g. WiFi, ethernet, powerline, Moca devices, PtP connections, etc).
Being able to test the user-to-router leg of the connection would be helpful more broadly beyond just bufferbloat. I often want to diagnose whether my connection issues or speed drops are happening due to an issue with my modem (and more generally the WAN connection) or if it's an issue with my wifi connection.
I guess I don't quite understand this part though: "iperf2/3 or netperf on a router is not all that representative for its routing performance." What exactly do you mean here?
>
> Most recent discussion moved over to https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379
>
>
>
Thanks! I have a lot of catching up to do on that thread, and some of it is definitely above my pay grade :).
>
> I think this ideally would be solved at the 3GPPP level
>
>
Agreed. I wonder if there's anything we can do to encourage them to pay attention to this.
Best regards,
Sina.
On Tue, Oct 18, 2022 at 12:04 PM, Sebastian Moeller < moeller0 at gmx.de > wrote:
>
>
>
> Hi Sina,
>
>
>
>
> On 18 October 2022 19:17:16 CEST, Sina Khanifar via Bloat < bloat@ lists. bufferbloat.
> net ( bloat at lists.bufferbloat.net ) > wrote:
>
>
>
>>
>>>
>>>
>>> I can't help but wonder tho... are you collecting any statistics, over
>>> time, as to how much better the problem is getting?
>>>
>>>
>>>
>>
>>
>>
>> We are collecting anonymized data, but we haven't analyzed it yet. If we
>> get a bit of time we'll look at that hopefully.
>>
>>
>>
>
>
>
> [SM] Just an observation, using Safari I see large maximal delays (like a
> small group of samples far out to the right of the bulk) for both down-
> and upload that essentially disappear when I switch to firefox. Now I tend
> to have a ton of tabs open in Safari while I only open firefox for
> dedicated use-cases with a few tabs at most, so I do not intend to throw
> shade on Safari here; my point is more browsers can and do affect the
> reported latency numbers, of you want to be able to test this, maybe ask
> users to use the OS browser (safari, edge, konqueror ;) ) as well as
> firefox and chrome so you can directly compare across browsers?
>
>
>
>>
>>>
>>>
>>> And any chance they could do something similar explaining wifi?
>>>
>>>
>>>
>>
>>
>>
>> I'm actually not exactly sure what mitigations exist for WiFi at the
>> moment - is there something I can read?
>>
>>
>>
>>
>> On this note: when we were building our test one of the things we really
>> wished existed was a standardized way to test latency and throughput to
>> routers.
>>
>>
>>
>
>
>
> [SM] traceroute/mtr albeit not sure how well this approach works from
> inside the browser, can you e.g. control TTL and do you receive error
> messages via ICMP?
>
>
>
>
> It would be super helpful if there was a standard in consumer routers that
> allowed users to both ping and fetch 0kB fils from their routers, and also
> run download/upload tests.
>
>
>
>
> [SM] I think I see where you are coming from here. Over in the OpenWrt
> forum we often see that server performance with iperf2/3 or netperf on a
> router is not all that representative for its routing performance. What do
> you expect to deduce from upload/download to the router? (I might
> misunderstand your point by a mile, if so please elaborate)
>
>
>
>
> Regards
> Sebastian
>
>
>>
>>>
>>>
>>> I think one more wispa conference will be a clean sweep of everyone in the
>>> fixed wireless market to not only adopt these algorithms for plan
>>> enforcement, but even more directly on the radios and more CPE.
>>>
>>>
>>>
>>
>>
>>
>> T-Mobile has signed up 1m+ people to their new Home Internet over 5G, and
>> all of them have really meaningful bufferbloat issues. I've been pointing
>> folks who reach out to this thread ( https:/ / forum. openwrt. org/ t/ cake-w-adaptive-bandwidth-historic/
>> 108848 (
>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth-historic/108848 ) )
>> about cake-autorate and sqm-autorate, but ideally it would be fixed at a
>> network level, just not sure how to apply pressure (I'm in contact with
>> the T-Mobile Home Internet team, but I think this is above their heads).
>>
>>
>>
>>
>> On Mon, Oct 17, 2022 at 8:15 PM, Dave Taht < dave. taht@ gmail. com (
>> dave.taht at gmail.com ) > wrote:
>>
>>
>>
>>>
>>>
>>> On Mon, Oct 17 , 2022 at 7:51 PM Sina Khanifar < sina@ waveform. com ( sina@
>>> waveform. com ( sina at waveform.com ) ) > wrote:
>>>
>>>
>>>
>>>>
>>>>
>>>> Positive or negative, I can claim a bit of credit for this video :). We've
>>>> been working with LTT on a few projects and we pitched them on doing
>>>> something around bufferbloat. We've seen more traffic to our Waveforn test
>>>> than ever before, which has been fun!
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> Thank you. Great job with that video! And waveform has become the goto
>>> site for many now.
>>>
>>>
>>>
>>>
>>> I can't help but wonder tho... are you collecting any statistics, over
>>> time, as to how much better the problem is getting?
>>>
>>>
>>>
>>>
>>> And any chance they could do something similar explaining wifi?
>>>
>>>
>>>
>>>
>>> ...
>>>
>>>
>>>
>>>
>>> I was just at WISPA conference week before last. Preseem's booth
>>> (fq_codel) was always packed. Vilo living had put cake in their wifi 6
>>> product. A
>>> keynote speaker had deployed it and talked about it with waveform results
>>> on the big screen (2k people there). A large wireless vendor demo'd
>>> privately to me their flent results before/after cake on their next-gen
>>> radios... and people dissed tarana without me prompting for their bad
>>> bufferbloat... and the best thing of all that happened to me was...
>>> besides getting a hug from a young lady (megan) who'd salvaged her
>>> schooling in alaska using sqm - I walked up to the paraqum booth
>>> (another large QoE middlebox maker centered more in india) and asked.
>>>
>>>
>>>
>>> "So... do y'all have fq_codel yet?"
>>>
>>>
>>>
>>>
>>> And they smiled and said: "No, we have something better... we've got
>>> cake."
>>>
>>>
>>>
>>>
>>> "Cake? What's that?" - I said, innocently.
>>>
>>>
>>>
>>>
>>> They then stepped me through their 200Gbps (!!) product, which uses a
>>> bunch of offloads, and can track rtt down to a ms with the intel ethernet
>>> card they were using. They'd modifed cake to provide 16 (?) levels of
>>> service, and were running under dpdk (I am not sure if cake was). It was a
>>> great, convincing pitch...
>>>
>>>
>>>
>>>
>>> ... then I told 'em who I was. There's a video of the in-both concert
>>> after.
>>>
>>>
>>>
>>>
>>> ...
>>>
>>>
>>>
>>>
>>> The downside to me (and the subject of my talk) was that in nearly every
>>> person I talked to, fq_codel was viewed as a means to better subscriber
>>> bandwidth plan enforcement (which is admittedly the market that preseem
>>> pioneered) and it was not understood that I'd got involved in this whole
>>> thing because I'd wanted an algorithm to deal with "rain fade", running
>>> directly on the radios. People wanted to use the statistics on the radios
>>> to drive the plan enforcement better
>>> (which is an ok approach, I guess), and for 10+ I'd been whinging about
>>> the... physics.
>>>
>>>
>>>
>>> So I ranted about rfc7567 a lot and begged people now putting routerOS
>>> 7.2 and later out there (mikrotik is huge in this market), to kill their
>>> fifos and sfqs at the native rates of the interfaces... and watch their
>>> network improve that way also.
>>>
>>>
>>>
>>> I think one more wispa conference will be a clean sweep of everyone in the
>>> fixed wireless market to not only adopt these algorithms for plan
>>> enforcement, but even more directly on the radios and more CPE.
>>>
>>>
>>>
>>>
>>> I also picked up enough consulting business to keep me busy the rest of
>>> this year, and possibly more than I can handle (anybody looking?)
>>>
>>>
>>>
>>>
>>> I wonder what will happen at a fiber conference?
>>>
>>>
>>>
>>>>
>>>>
>>>> On Mon, Oct 17 , 2022 at 7:45 PM Dave Taht via Bloat < bloat@ lists.
>>>> bufferbloat. net ( bloat@ lists. bufferbloat. net (
>>>> bloat at lists.bufferbloat.net ) ) > wrote:
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>> On Mon, Oct 17 , 2022 at 5:02 PM Stuart Cheshire < cheshire@ apple. com ( cheshire@
>>>>> apple. com ( cheshire at apple.com ) ) > wrote:
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> On 9 Oct 2022 , at 06:14, Dave Taht via Make-wifi-fast < make-wifi-fast@
>>>>>> lists. bufferbloat. net ( make-wifi-fast@ lists. bufferbloat. net (
>>>>>> make-wifi-fast at lists.bufferbloat.net ) ) > wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> This was so massively well done, I cried. Does anyone know how to get in
>>>>>>> touch with the ifxit folk?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> https:/ / www. youtube. com/ watch?v=UICh3ScfNWI ( https:/ / www. youtube.
>>>>>>> com/ watch?v=UICh3ScfNWI ( https://www.youtube.com/watch?v=UICh3ScfNWI ) )
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> I’m surprised that you liked this video. It seems to me that it repeats
>>>>>> all the standard misinformation. The analogy they use is the standard
>>>>>> terrible example of waiting in a long line at a grocery store, and the
>>>>>> “solution” is letting certain traffic “jump the line, angering everyone
>>>>>> behind them”.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Accuracy be damned. The analogy to common experience resonates more.
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> Some quotes from the video:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> it would be so much more efficient for them to let you skip the line and
>>>>>>> just check out, especially since you’re in a hurry, but they’re rudely
>>>>>>> refusing
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I think the person with the cheetos pulling out a gun and shooting
>>>>> everyone in front of him (AQM) would not go down well.
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> to go back to our grocery store analogy this would be like if a worker saw
>>>>>>> you standing at the back ... and either let you skip to the front of the
>>>>>>> line or opens up an express lane just for you
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Actually that analogy is fairly close to fair queuing. The multiple
>>>>> checker analogy is one of the most common analogies in queue theory
>>>>> itself.
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> The video describes the problem of bufferbloat, and then describes the
>>>>>> same failed solution that hasn’t worked for the last three decades.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Hmm? It establishes the scenario, explains the problem *quickly*, disses
>>>>> gamer routers for not getting it right.. *points to an accurate test*, and
>>>>> then to the ideas and products that *actually work* with "smart queueing",
>>>>> with a screenshot of the most common
>>>>> (eero's optimize for gaming and videoconferencing), and fq_codel and cake
>>>>> *by name*, and points folk at the best known solution available, openwrt.
>>>>>
>>>>>
>>>>>
>>>>> Bing, baddabang, boom. Also the comments were revealing. A goodly
>>>>> percentage already knew the problem, more than a few were inspired to take
>>>>> the test,
>>>>> there was a whole bunch of "Aha!" success stories and 360k views, which is
>>>>> more people than we've ever been able to reach in for example, a nanog
>>>>> conference.
>>>>>
>>>>>
>>>>>
>>>>> I loved that folk taking the test actually had quite a few A results,
>>>>> without having had to do anything. At least some ISPs are getting it more
>>>>> right now!
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> At this point I think gamers in particular know what "brands" we've tried
>>>>> to establish - "Smart queues", "SQM", "OpenWrt", fq_codel and now "cake"
>>>>> are "good" things to have, and are stimulating demand by asking for them,
>>>>> It's certainly working out better and better for evenroute, firewalla,
>>>>> ubnt and others, and I saw an uptick in questions about this on various
>>>>> user forums.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I even like that there's a backlash now of people saying "fixing
>>>>> bufferbloat doesn't solve everything" -
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> Describing the obvious simple-minded (wrong) solution that any normal
>>>>>> person would think of based on their personal human experience waiting in
>>>>>> grocery stores and airports, is not describing the solution to
>>>>>> bufferbloat. The solution to bufferbloat is not that if you are privileged
>>>>>> then you get to “skip to the front of the line”. The solution to
>>>>>> bufferbloat is that there is no line!
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I like the idea of a guru floating above a grocery cart with a better
>>>>> string of explanations, explaining
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> - "no, grasshopper, the solution to bufferbloat is no line... at all".
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> With grocery stores and airports people’s arrivals are independent and not
>>>>>> controlled. There is no way for a grocery store or airport to generate
>>>>>> backpressure to tell people to wait at home when a queue begins to form.
>>>>>> The key to solving bufferbloat is generating timely backpressure to
>>>>>> prevent the queue forming in the first place, not accepting a huge queue
>>>>>> and then deciding who deserves special treatment to get better service
>>>>>> than all the other peons who still have to wait in a long queue, just like
>>>>>> before.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I am not huge on the word "backpressure" here. Needs to signal the other
>>>>> side to slow down, is more accurate. So might say timely signalling rather
>>>>> than timely backpressure?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Other feedback I got was that the video was too smarmy (I agree),
>>>>> different audiences than gamers need different forms of outreach...
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> but to me, winning the gamers has always been one of the most important
>>>>> things, as they make a lot of buying decisions, and they benefit the most
>>>>> for
>>>>> fq and packet prioritization as we do today in gamer routers and in cake +
>>>>> qosify.
>>>>>
>>>>>
>>>>>
>>>>> maybe that gets in the way of more serious markets. Certainly I would like
>>>>> another video explaining what goes wrong with videoconferencing.
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> Stuart Cheshire
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> This song goes out to all the folk that thought Stadia would work: https:/
>>>>>
>>>>> / www. linkedin. com/ posts/
>>>>> dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
>>>>> (
>>>>> https:/ / www. linkedin. com/ posts/ dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
>>>>> (
>>>>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
>>>>> )
>>>>> ) Dave Täht CEO, TekLibre, LLC
>>>>> _______________________________________________
>>>>> Bloat mailing list
>>>>> Bloat@ lists. bufferbloat. net ( Bloat at lists.bufferbloat.net ) https://lists.bufferbloat.net/listinfo/bloat
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> This song goes out to all the folk that thought Stadia would work: https:/
>>>
>>> / www. linkedin. com/ posts/
>>> dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
>>> (
>>> https:/ / www. linkedin. com/ posts/ dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
>>> (
>>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
>>> )
>>> ) Dave Täht CEO, TekLibre, LLC
>>>
>>>
>>
>>
>
>
>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/bloat/attachments/20221020/2fbaf228/attachment-0001.html>
More information about the Bloat
mailing list