<html><head></head><body><div><div><div class=""><div class=""><div class=""><div class=""><div class="">Hi Sebastian,<br/></div><div class=""><br/></div><blockquote class="">[SM] Just an observation, using Safari I see large maximal delays (like a small group of samples far out to the right of the bulk) for both down- and upload that essentially disappear when I switch to firefox.
Now I tend to have a ton of tabs open in Safari while I only open firefox for dedicated use-cases with a few tabs at most, so I do not intend to throw shade on Safari here; my point is more browsers can and do affect the reported latency numbers, of you want to be able to test this, maybe ask users to use the OS browser (safari, edge, konqueror ;) ) as well as firefox and chrome so you can directly compare across browsers?<br/></blockquote><div class=""><br/></div></div><div class="">I believe this is because we use the WebTiming APIs to get a more accurate latency numbers, but the API isn't fully supported on Safari. As such, latency measurements in Safari are much less accurate than in Firefox and Chrome. <br/></div><div class=""><br/></div><div class=""><blockquote class="">traceroute/mtr albeit not sure how well this approach works from inside the browser, can you e.g. control TTL and do you receive error messages via ICMP?<br/></blockquote></div><div class=""><br/></div><div class="">Unfortunately traceroutes via the browser don't really work :(. And I don't believe we can control TTL or see ICMP error messages either, though I haven't dug into this very deeply.<br/></div><div class=""><br/></div><div class=""><blockquote class=""><p class="">Over in the OpenWrt forum we often see that server performance with iperf2/3 or netperf on a router is not all that representative for its routing performance.
What do you expect to deduce from upload/download to the router? (I might misunderstand your point by a mile, if so please elaborate) <br/></p></blockquote></div><div class=""><br/></div><div class="">The goal would be to test the "local" latency, throughput, and bufferbloat between the user's device and the router, and then compare this with the latency, throughput, and bufferbloat when DL/ULing to a remote server.<br/></div><div class=""><br/></div><div class="">This would reveal whether <span style="color: unset;" class="">the dominant source of increase in latency under load is at the router's WAN interface or somewhere between the router and the user (e.g. WiFi, ethernet, powerline, Moca devices, PtP connections, etc).</span><br/></div><div class=""><br/></div><div class="">Being able to test the user-to-router leg of the connection would be helpful more broadly beyond just bufferbloat. I often want to diagnose whether my connection issues or speed drops are happening due to an issue with my modem (and more generally the WAN connection) or if it's an issue with my wifi connection. <br/></div><div class=""><br/></div><div class="">I guess I don't quite understand this part though: "iperf2/3 or netperf on a router is not all that representative for its routing performance." What exactly do you mean here?<br/></div><div class=""><br/></div><blockquote class=""><div class="">Most recent discussion moved over to<span class=""> </span><a style="text-decoration-color:initial;text-decoration-style:initial;text-decoration-thickness:initial;text-decoration-line:none;color:rgb(84, 172, 220);" class="" href="https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379" rel="noopener noreferrer" target="_blank">https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379</a><br/></div></blockquote><div class=""><br/></div><div class="">Thanks! I have a lot of catching up to do on that thread, and some of it is definitely above my pay grade :).<br/></div><div class=""><br/></div><blockquote class=""><div class=""><span style="text-decoration-color:initial;text-decoration-style:initial;text-decoration-thickness:initial;font-weight:400;" class="">I think this ideally would be solved at the 3GPPP level</span><br/></div></blockquote><div class=""><div class=""><br/></div><div class="">Agreed. I wonder if there's anything we can do to encourage them to pay attention to this.<br/></div></div><div class=""><br/></div><div class="">Best regards,<br/></div><div class=""><br/></div><div class="">Sina.<br/></div></div></div><br/><div class="sh-quoted-content"><div class=""><div class="gmail_quote">On Tue, Oct 18, 2022 at 12:04 PM, Sebastian Moeller <span dir="ltr" class=""><<a href="mailto:moeller0@gmx.de" target="_blank" class="">moeller0@gmx.de</a>></span> wrote:<br/><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_extra"><div class="gmail_quote"><p class="">Hi Sina,
<br/></p><p class="">
On 18 October 2022 19:17:16 CEST, Sina Khanifar via Bloat <<a target="_blank" rel="noopener noreferrer" href="mailto:bloat@lists.bufferbloat.net" class="">bloat@<wbr/>lists.<wbr/>bufferbloat.<wbr/>net</a>> wrote:
<br/></p><blockquote class=""><blockquote class=""><p class="">
I can't help but wonder tho... are you collecting any statistics, over
time, as to how much better the problem is getting?
<br/></p></blockquote><p class="">
We are collecting anonymized data, but we haven't analyzed it yet. If we get a bit of time we'll look at that hopefully.
<br/></p></blockquote><p class="">
[SM] Just an observation, using Safari I see large maximal delays (like a small group of samples far out to the right of the bulk) for both down- and upload that essentially disappear when I switch to firefox.
Now I tend to have a ton of tabs open in Safari while I only open firefox for dedicated use-cases with a few tabs at most, so I do not intend to throw shade on Safari here; my point is more browsers can and do affect the reported latency numbers, of you want to be able to test this, maybe ask users to use the OS browser (safari, edge, konqueror ;) ) as well as firefox and chrome so you can directly compare across browsers?
<br/></p><blockquote class=""><blockquote class=""><p class="">
And any chance they could do something similar explaining wifi?
<br/></p></blockquote><p class="">
I'm actually not exactly sure what mitigations exist for WiFi at the moment - is there something I can read?
<br/></p><p class="">
On this note: when we were building our test one of the things we really wished existed was a standardized way to test latency and throughput to routers.
<br/></p></blockquote><p class="">
[SM] traceroute/mtr albeit not sure how well this approach works from inside the browser, can you e.g. control TTL and do you receive error messages via ICMP?
<br/></p><p class="">
It would be super helpful if there was a standard in consumer routers that allowed users to both ping and fetch 0kB fils from their routers, and also run download/upload tests.
<br/></p><p class="">
[SM] I think I see where you are coming from here. Over in the OpenWrt forum we often see that server performance with iperf2/3 or netperf on a router is not all that representative for its routing performance.
What do you expect to deduce from upload/download to the router? (I might misunderstand your point by a mile, if so please elaborate)
<br/></p><p class="">
Regards
<br/>
Sebastian
</p><blockquote class=""><blockquote class=""><p class="">
I think one more wispa conference will be a clean sweep of everyone in the
fixed wireless market to not only adopt these algorithms for plan
enforcement, but even more directly on the radios and more CPE.
<br/></p></blockquote><p class="">
T-Mobile has signed up 1m+ people to their new Home Internet over 5G, and all of them have really meaningful bufferbloat issues. I've been pointing folks who reach out to this thread ( <a target="_blank" rel="noopener noreferrer" href="https://forum.openwrt.org/t/cake-w-adaptive-bandwidth-historic/108848" class="">https:/<wbr/>/<wbr/>forum.<wbr/>openwrt.<wbr/>org/<wbr/>t/<wbr/>cake-w-adaptive-bandwidth-historic/<wbr/>108848</a> ) about cake-autorate and sqm-autorate, but ideally it would be fixed at a network level, just not sure how to apply pressure (I'm in contact with the T-Mobile Home Internet team, but I think this is above their heads).
<br/></p><p class="">
On Mon, Oct 17, 2022 at 8:15 PM, Dave Taht < <a target="_blank" rel="noopener noreferrer" href="mailto:dave.taht@gmail.com" class="">dave.<wbr/>taht@<wbr/>gmail.<wbr/>com</a> > wrote:
<br/></p><blockquote class=""><p class="">
<span class="sh-date" data-date-isostring="2022-10-17T07:00:00.000Z">On Mon, Oct 17</span>, 2022 at 7:51 PM Sina Khanifar < sina@ waveform. com (
<a target="_blank" rel="noopener noreferrer" href="mailto:sina@waveform.com" class="">sina@<wbr/>waveform.<wbr/>com</a> ) > wrote:
<br/></p><blockquote class=""><p class="">
Positive or negative, I can claim a bit of credit for this video :). We've
been working with LTT on a few projects and we pitched them on doing
something around bufferbloat. We've seen more traffic to our Waveforn test
than ever before, which has been fun!
<br/></p></blockquote><p class="">
Thank you. Great job with that video! And waveform has become the goto
site for many now.
<br/></p><p class="">
I can't help but wonder tho... are you collecting any statistics, over
time, as to how much better the problem is getting?
<br/></p><p class="">
And any chance they could do something similar explaining wifi?
<br/></p><p class="">
...
<br/></p><p class="">
I was just at WISPA conference week before last. Preseem's booth
<br/>
(fq_codel) was always packed. Vilo living had put cake in their wifi 6
product. A
<br/>
keynote speaker had deployed it and talked about it with waveform results
on the big screen (2k people there). A large wireless vendor demo'd
privately to me their flent results before/after cake on their next-gen
radios... and people dissed tarana without me prompting for their bad
bufferbloat... and the best thing of all that happened to me was...
besides getting a hug from a young lady (megan) who'd salvaged her
schooling in alaska using sqm - I walked up to the paraqum booth
<br/>
(another large QoE middlebox maker centered more in india) and asked.
</p><p class="">
"So... do y'all have fq_codel yet?"
<br/></p><p class="">
And they smiled and said: "No, we have something better... we've got
cake."
<br/></p><p class="">
"Cake? What's that?" - I said, innocently.
<br/></p><p class="">
They then stepped me through their 200Gbps (!!) product, which uses a
bunch of offloads, and can track rtt down to a ms with the intel ethernet
card they were using. They'd modifed cake to provide 16 (?) levels of
service, and were running under dpdk (I am not sure if cake was). It was a
great, convincing pitch...
<br/></p><p class="">
... then I told 'em who I was. There's a video of the in-both concert
after.
<br/></p><p class="">
...
<br/></p><p class="">
The downside to me (and the subject of my talk) was that in nearly every
person I talked to, fq_codel was viewed as a means to better subscriber
bandwidth plan enforcement (which is admittedly the market that preseem
pioneered) and it was not understood that I'd got involved in this whole
thing because I'd wanted an algorithm to deal with "rain fade", running
directly on the radios. People wanted to use the statistics on the radios
to drive the plan enforcement better
<br/>
(which is an ok approach, I guess), and for 10+ I'd been whinging about
the... physics.
</p><p class="">
So I ranted about rfc7567 a lot and begged people now putting routerOS
<br/>
7.2 and later out there (mikrotik is huge in this market), to kill their
fifos and sfqs at the native rates of the interfaces... and watch their
network improve that way also.
</p><p class="">
I think one more wispa conference will be a clean sweep of everyone in the
fixed wireless market to not only adopt these algorithms for plan
enforcement, but even more directly on the radios and more CPE.
<br/></p><p class="">
I also picked up enough consulting business to keep me busy the rest of
this year, and possibly more than I can handle (anybody looking?)
<br/></p><p class="">
I wonder what will happen at a fiber conference?
<br/></p><blockquote class=""><p class="">
<span class="sh-date" data-date-isostring="2022-10-17T07:00:00.000Z">On Mon, Oct 17</span>, 2022 at 7:45 PM Dave Taht via Bloat < bloat@ lists. bufferbloat.
net ( <a target="_blank" rel="noopener noreferrer" href="mailto:bloat@lists.bufferbloat.net" class="">bloat@<wbr/>lists.<wbr/>bufferbloat.<wbr/>net</a> ) > wrote:
<br/></p><blockquote class=""><p class="">
<span class="sh-date" data-date-isostring="2022-10-17T07:00:00.000Z">On Mon, Oct 17</span>, 2022 at 5:02 PM Stuart Cheshire < cheshire@ apple. com (
<a target="_blank" rel="noopener noreferrer" href="mailto:cheshire@apple.com" class="">cheshire@<wbr/>apple.<wbr/>com</a> ) > wrote:
<br/></p><blockquote class=""><p class="">
On <span class="sh-date" data-date-isostring="2022-10-09T07:00:00.000Z">9 Oct 2022</span>, at 06:14, Dave Taht via Make-wifi-fast < make-wifi-fast@ lists.
bufferbloat. net ( <a target="_blank" rel="noopener noreferrer" href="mailto:make-wifi-fast@lists.bufferbloat.net" class="">make-wifi-fast@<wbr/>lists.<wbr/>bufferbloat.<wbr/>net</a> ) > wrote:
<br/></p><blockquote class=""><p class="">
This was so massively well done, I cried. Does anyone know how to get in
touch with the ifxit folk?
<br/></p><p class="">
https:/ / www. youtube. com/ watch?v=UICh3ScfNWI (
<a target="_blank" rel="noopener noreferrer" href="https://www.youtube.com/watch?v=UICh3ScfNWI" class="">https:/<wbr/>/<wbr/>www.<wbr/>youtube.<wbr/>com/<wbr/>watch?v=UICh3ScfNWI</a> )
<br/></p></blockquote><p class="">
I’m surprised that you liked this video. It seems to me that it repeats
all the standard misinformation. The analogy they use is the standard
terrible example of waiting in a long line at a grocery store, and the
<br/>
“solution” is letting certain traffic “jump the line, angering everyone
behind them”.
</p></blockquote><p class="">
Accuracy be damned. The analogy to common experience resonates more.
<br/></p><blockquote class=""><p class="">
Some quotes from the video:
<br/></p><blockquote class=""><p class="">
it would be so much more efficient for them to let you skip the line and
just check out, especially since you’re in a hurry, but they’re rudely
refusing
<br/></p></blockquote></blockquote><p class="">
I think the person with the cheetos pulling out a gun and shooting
everyone in front of him (AQM) would not go down well.
<br/></p><blockquote class=""><blockquote class=""><p class="">
to go back to our grocery store analogy this would be like if a worker saw
you standing at the back ... and either let you skip to the front of the
line or opens up an express lane just for you
<br/></p></blockquote></blockquote><p class="">
Actually that analogy is fairly close to fair queuing. The multiple
checker analogy is one of the most common analogies in queue theory
itself.
<br/></p><blockquote class=""><p class="">
The video describes the problem of bufferbloat, and then describes the
same failed solution that hasn’t worked for the last three decades.
<br/></p></blockquote><p class="">
Hmm? It establishes the scenario, explains the problem *quickly*, disses
gamer routers for not getting it right.. *points to an accurate test*, and
then to the ideas and products that *actually work* with "smart queueing",
with a screenshot of the most common
<br/>
(eero's optimize for gaming and videoconferencing), and fq_codel and cake
<br/>
*by name*, and points folk at the best known solution available, openwrt.
</p><p class="">
Bing, baddabang, boom. Also the comments were revealing. A goodly
percentage already knew the problem, more than a few were inspired to take
the test,
<br/>
there was a whole bunch of "Aha!" success stories and 360k views, which is
more people than we've ever been able to reach in for example, a nanog
conference.
</p><p class="">
I loved that folk taking the test actually had quite a few A results,
without having had to do anything. At least some ISPs are getting it more
right now!
<br/></p><p class="">
At this point I think gamers in particular know what "brands" we've tried
to establish - "Smart queues", "SQM", "OpenWrt", fq_codel and now "cake"
are "good" things to have, and are stimulating demand by asking for them,
It's certainly working out better and better for evenroute, firewalla,
ubnt and others, and I saw an uptick in questions about this on various
user forums.
<br/></p><p class="">
I even like that there's a backlash now of people saying "fixing
bufferbloat doesn't solve everything" -
<br/></p><blockquote class=""><p class="">
Describing the obvious simple-minded (wrong) solution that any normal
person would think of based on their personal human experience waiting in
grocery stores and airports, is not describing the solution to
bufferbloat. The solution to bufferbloat is not that if you are privileged
then you get to “skip to the front of the line”. The solution to
bufferbloat is that there is no line!
<br/></p></blockquote><p class="">
I like the idea of a guru floating above a grocery cart with a better
string of explanations, explaining
<br/></p><p class="">
- "no, grasshopper, the solution to bufferbloat is no line... at all".
<br/></p><blockquote class=""><p class="">
With grocery stores and airports people’s arrivals are independent and not
controlled. There is no way for a grocery store or airport to generate
backpressure to tell people to wait at home when a queue begins to form.
The key to solving bufferbloat is generating timely backpressure to
prevent the queue forming in the first place, not accepting a huge queue
and then deciding who deserves special treatment to get better service
than all the other peons who still have to wait in a long queue, just like
before.
<br/></p></blockquote><p class="">
I am not huge on the word "backpressure" here. Needs to signal the other
side to slow down, is more accurate. So might say timely signalling rather
than timely backpressure?
<br/></p><p class="">
Other feedback I got was that the video was too smarmy (I agree),
different audiences than gamers need different forms of outreach...
<br/></p><p class="">
but to me, winning the gamers has always been one of the most important
things, as they make a lot of buying decisions, and they benefit the most
for
<br/>
fq and packet prioritization as we do <span class="sh-date" data-date-isostring="2022-10-18T22:00:00.000Z">today</span> in gamer routers and in cake +
qosify.
</p><p class="">
maybe that gets in the way of more serious markets. Certainly I would like
another video explaining what goes wrong with videoconferencing.
<br/></p><blockquote class=""><p class="">
Stuart Cheshire
<br/></p></blockquote><p class="">
--
<br/>
This song goes out to all the folk that thought Stadia would work: https:/
<br/>
/ www. linkedin. com/ posts/ dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
<br/>
(
<br/>
<a target="_blank" rel="noopener noreferrer" href="https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz" class="">https:/<wbr/>/<wbr/>www.<wbr/>linkedin.<wbr/>com/<wbr/>posts/<wbr/>dtaht_the-mushroom-song-activity-6981366665607352320-FXtz</a>
<br/>
) Dave Täht CEO, TekLibre, LLC
<br/>
_______________________________________________
<br/>
Bloat mailing list
<br/>
Bloat@ lists. bufferbloat. net ( <a target="_blank" rel="noopener noreferrer" href="mailto:Bloat@lists.bufferbloat.net">Bloat@lists.bufferbloat.net</a> )
<a target="_blank" rel="noopener noreferrer" href="https://lists.bufferbloat.net/listinfo/bloat">https://lists.bufferbloat.net/listinfo/bloat</a>
</p></blockquote></blockquote><p class="">
--
<br/>
This song goes out to all the folk that thought Stadia would work: https:/
<br/>
/ www. linkedin. com/ posts/ dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
<br/>
(
<br/>
<a target="_blank" rel="noopener noreferrer" href="https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz" class="">https:/<wbr/>/<wbr/>www.<wbr/>linkedin.<wbr/>com/<wbr/>posts/<wbr/>dtaht_the-mushroom-song-activity-6981366665607352320-FXtz</a>
<br/>
) Dave Täht CEO, TekLibre, LLC
</p></blockquote></blockquote><p class="">
--
<br/>
Sent from my Android device with K-9 Mail. Please excuse my brevity.</p></div></div></blockquote></div></div></div></div><div><br/></div></div><div><div style="display: none; border: 0px; width: 0px; height: 0px; overflow: hidden; visibility: hidden;"><img src="https://r.superhuman.com/7VwF79K8ZwJ-GwWmsXu7psMSwxaQu9S6UrxwBUTNUkyKbXt8IiommCoOiH4WVBKcVTltn_0lLmhN9jmdPE_GFwlUoycSXXO0Ay_38mOyGGpsZNka1Ufu8g_z7nSsVB_sfUkUlcBZuqeqEITw_nH2eAhSOEAPNg5CFLciDdVrgpCODLHZc-FlKERRZDw16oY.gif" alt=" " width="1" height="0" style="display: none; border: 0px; width: 0px; height: 0px; overflow: hidden; visibility: hidden;"/><!-- --></div></div></div></body></html>