From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-vs1-xe2b.google.com (mail-vs1-xe2b.google.com [IPv6:2607:f8b0:4864:20::e2b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 6C23C3B29D for ; Thu, 20 Oct 2022 01:15:51 -0400 (EDT) Received: by mail-vs1-xe2b.google.com with SMTP id a2so19634771vsc.13 for ; Wed, 19 Oct 2022 22:15:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=waveform.com; s=google; h=cc:to:in-reply-to:references:subject:message-id:date:from :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=pNfraTl1vkDEBgq2BOlGdh4eab1F32Lu1nb0RbA1VSE=; b=UzZNotv28H4wQqYLeBmYifzlTB6Vbhk0LFDlK3oFiMxJmpqk5HWbZiVoxQrQ8dBdgj z6oXQF6ljsnTeF11dFF2yb9R0eof3ZpcRAjtlhEYAZVljRUwH0DfvrLIVbl46jrAE9Tj DSnqhd0F1s9OoFLSGkyPxh2I6xbuDdCUEannkIShSMWmrBEX2xGkPkV1XYzsx8KVygwi b0AQBMyPoZfxvpLgqlBBLkwqWbkS4v98talw2dfbIpLNaSlWI8dMPGls4F/H4nArdU6U C58ozzqe6D6bxpQDarhRBpWrcdp4d3EUirCVbbP8CDF0QvDfmHBimOX+Ri7JI+MTlo/3 0ptg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:in-reply-to:references:subject:message-id:date:from :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=pNfraTl1vkDEBgq2BOlGdh4eab1F32Lu1nb0RbA1VSE=; b=VmRHZYCJPPxi8s+Ihi0NQ5wqOPUVAkNF/X6CYdibBvobl6+FOoGB743b6bNehBU9K+ 51YHQaknVzefCbdv9NYp4y4xiAiIMtzk6bcj3/oXrae+ZaeIAkoy3V5UKyQ5ve08Aay3 EPpBtNDFyeXtj3Uwrt13SoBivNkhDWW0er7doabLgFf4ZwrZ+8Mi/uPU/3ANW/jlsyHR Nkhf3l2uShi8PbR0/2p97ttBDuJRkIeraNutOLJVIaDuHQ/nxfO6683fNa+0g4+QE8ph gcA+IjI7sZ0qWTVUsU29+G153blP7iHtkjRG+6fz9jAtj46SDEkpxAFvVCizPI/0HTcm kbxg== X-Gm-Message-State: ACrzQf1xjT84B8860X36EYKEB3i9RpCjCylun5MdgOn4K29vWFb8vB/8 BQXd8mYsgXEmn+WkkETRZhtw5ezKmQjFzHL25L/tSS+8Od2/6GwE0MoGa6ZhUPSUD+OOvMKkqNJ 9oKQ0IKphZswHheta9GV7W3h3GRSird75BW53M2Tzd+Xww3fglY3NR9GP++/+WCwsbvV/BHQhCt cmCSoBTw== X-Google-Smtp-Source: AMsMyM4ErkQIFFL8OYx9rXWeShE2kaKrlCuQPd7wUWHBvSNdmUACo05SdBjB/j6IIeQ2y7HVAYQpsw== X-Received: by 2002:a67:b805:0:b0:3a7:a708:20a9 with SMTP id i5-20020a67b805000000b003a7a70820a9mr5203629vsf.64.1666242950155; Wed, 19 Oct 2022 22:15:50 -0700 (PDT) Received: from localhost (0.92.231.35.bc.googleusercontent.com. [35.231.92.0]) by smtp.gmail.com with UTF8SMTPSA id n11-20020ab01e4b000000b003b44dcec32asm2695355uak.16.2022.10.19.22.15.50 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 19 Oct 2022 22:15:50 -0700 (PDT) Mime-Version: 1.0 From: "Sina Khanifar" Message-ID: References: <938D9D45-DADA-4291-BD8A-84E4257CEE49@apple.com> <31A8072A-2482-4395-A521-64350035D8DD@gmx.de> X-Superhuman-ID: l9gm0mrd.182095c0-0888-4b47-bc62-e7af69382901 X-Superhuman-Draft-ID: draft0048ea2b9b9793bc In-Reply-To: <31A8072A-2482-4395-A521-64350035D8DD@gmx.de> To: "Sebastian Moeller" Cc: "Sina Khanifar" , "Dave Taht" , "Cake List" , "Make-Wifi-fast" , "Rpm" X-Mailer: Superhuman Web (2022-10-18T22:12:12Z) Content-Type: multipart/alternative; boundary=a00aad616cc2c3b0f180550026cfbbf30ab098aff9069214a780a4a411f0 X-Mailman-Approved-At: Mon, 29 Apr 2024 18:18:51 -0400 Subject: Re: [Cake] [Bloat] A quick report from the WISPA conference X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Date: Thu, 20 Oct 2022 05:15:51 -0000 X-Original-Date: Thu, 20 Oct 2022 05:15:47 +0000 X-List-Received-Date: Thu, 20 Oct 2022 05:15:51 -0000 --a00aad616cc2c3b0f180550026cfbbf30ab098aff9069214a780a4a411f0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Hi Sebastian, >=20 > [SM] Just an observation, using Safari I see large maximal delays (like a > small group of samples far out to the right of the bulk) for both down- > and upload that essentially disappear when I switch to firefox. Now I ten= d > to have a ton of tabs open in Safari while I only open firefox for > dedicated use-cases with a few tabs at most, so I do not intend to throw > shade on Safari here; my point is more browsers can and do affect the > reported latency numbers, of you want to be able to test this, maybe ask > users to use the OS browser (safari, edge, konqueror ;) ) as well as > firefox and chrome so you can directly compare across browsers? >=20 I believe this is because we use the WebTiming APIs to get a more accurate = latency numbers, but the API isn't fully supported on Safari. As such, late= ncy measurements in Safari are much less accurate than in Firefox and Chrom= e. >=20 > traceroute/mtr albeit not sure how well this approach works from inside > the browser, can you e.g. control TTL and do you receive error messages > via ICMP? >=20 Unfortunately traceroutes via the browser don't really work :(. And I don't= believe we can control TTL or see ICMP error messages=C2=A0either, though = I haven't dug into this very deeply. >=20 >=20 >=20 > Over in the OpenWrt forum we often see that server performance with > iperf2/3 or netperf on a router is not all that representative for its > routing performance. What do you expect to deduce from upload/download to > the router? (I might misunderstand your point by a mile, if so please > elaborate) >=20 >=20 >=20 The goal would be to test the "local" latency, throughput, and bufferbloat = between the user's device and the router, and then compare this with the la= tency, throughput, and bufferbloat when DL/ULing to a remote server. This would reveal whether the dominant source of increase in latency under = load is=C2=A0at the router's WAN interface or somewhere between the router = and the user (e.g. WiFi, ethernet, powerline, Moca devices, PtP connections= , etc). Being able to test the user-to-router leg of the connection would be helpfu= l more broadly beyond just bufferbloat.=C2=A0I often want to diagnose wheth= er my connection issues or speed drops are happening due to an issue with m= y modem (and more generally the WAN connection) or if it's an issue with my= wifi connection. I guess I don't quite understand this part though: "iperf2/3 or netperf on = a router is not all that representative for its routing performance." What = exactly do you mean here? >=20 > =E2=80=8BMost recent discussion moved over to https://forum.openwrt.org/t= /cake-w-adaptive-bandwidth/135379 >=20 >=20 >=20 Thanks! I have a lot of catching up to do on that thread, and some of it is= definitely above my pay grade :). >=20 > =E2=80=8B I think this ideally would be solved at the 3GPPP level >=20 >=20 Agreed. I wonder if there's anything we can do to encourage them to pay att= ention to this. Best regards, Sina. On Tue, Oct 18, 2022 at 12:04 PM, Sebastian Moeller < moeller0@gmx.de > wro= te: >=20 >=20 >=20 > Hi Sina, >=20 >=20 >=20 >=20 > On 18 October 2022 19:17:16 CEST, Sina Khanifar via Bloat < bloat@ lists.= bufferbloat. > net ( bloat@lists.bufferbloat.net ) > wrote: >=20 >=20 >=20 >>=20 >>>=20 >>>=20 >>> I can't help but wonder tho... are you collecting any statistics, over >>> time, as to how much better the problem is getting? >>>=20 >>>=20 >>>=20 >>=20 >>=20 >>=20 >> We are collecting anonymized=C2=A0data, but we haven't analyzed it yet. = If we >> get a bit of time we'll look at that hopefully. >>=20 >>=20 >>=20 >=20 >=20 >=20 > [SM] Just an observation, using Safari I see large maximal delays (like a > small group of samples far out to the right of the bulk) for both down- > and upload that essentially disappear when I switch to firefox. Now I ten= d > to have a ton of tabs open in Safari while I only open firefox for > dedicated use-cases with a few tabs at most, so I do not intend to throw > shade on Safari here; my point is more browsers can and do affect the > reported latency numbers, of you want to be able to test this, maybe ask > users to use the OS browser (safari, edge, konqueror ;) ) as well as > firefox and chrome so you can directly compare across browsers? >=20 >=20 >=20 >>=20 >>>=20 >>>=20 >>> And any chance they could do something similar explaining wifi? >>>=20 >>>=20 >>>=20 >>=20 >>=20 >>=20 >> I'm actually not exactly sure what mitigations exist for WiFi at the >> moment - is there something I can read? >>=20 >>=20 >>=20 >>=20 >> On this note: when we were building our test one of the things we really >> wished existed was a standardized way to test latency and throughput to >> routers. >>=20 >>=20 >>=20 >=20 >=20 >=20 > [SM] traceroute/mtr albeit not sure how well this approach works from > inside the browser, can you e.g. control TTL and do you receive error > messages via ICMP? >=20 >=20 >=20 >=20 > It would be super helpful if there was a standard in consumer routers=C2= =A0that > allowed users to both ping and fetch 0kB fils from their routers, and als= o > run download/upload tests. >=20 >=20 >=20 >=20 > [SM] I think I see where you are coming from here. Over in the OpenWrt > forum we often see that server performance with iperf2/3 or netperf on a > router is not all that representative for its routing performance. What d= o > you expect to deduce from upload/download to the router? (I might > misunderstand your point by a mile, if so please elaborate) >=20 >=20 >=20 >=20 > Regards > Sebastian >=20 >=20 >>=20 >>>=20 >>>=20 >>> I think one more wispa conference will be a clean sweep of everyone in = the >>> fixed wireless market to not only adopt these algorithms for plan >>> enforcement, but even more directly on the radios and more CPE. >>>=20 >>>=20 >>>=20 >>=20 >>=20 >>=20 >> T-Mobile has signed up 1m+ people to their new Home Internet over 5G, an= d >> all of them have really meaningful bufferbloat issues. I've been pointin= g >> folks who reach out to this thread ( https:/ / forum. openwrt. org/ t/ c= ake-w-adaptive-bandwidth-historic/ >> 108848 ( >> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth-historic/108848 ) = ) >> about cake-autorate and sqm-autorate, but ideally it would be fixed at a >> network level, just not sure how to apply pressure (I'm in contact with >> the T-Mobile Home Internet team, but I think this is above their heads). >>=20 >>=20 >>=20 >>=20 >> On Mon, Oct 17, 2022 at 8:15 PM, Dave Taht < dave. taht@ gmail. com ( >> dave.taht@gmail.com ) > wrote: >>=20 >>=20 >>=20 >>>=20 >>>=20 >>> On Mon, Oct 17 , 2022 at 7:51 PM Sina Khanifar < sina@ waveform. com ( = sina@ >>> waveform. com ( sina@waveform.com ) ) > wrote: >>>=20 >>>=20 >>>=20 >>>>=20 >>>>=20 >>>> Positive or negative, I can claim a bit of credit for this video :). W= e've >>>> been working with LTT on a few projects and we pitched them on doing >>>> something around bufferbloat. We've seen more traffic to our Waveforn = test >>>> than ever before, which has been fun! >>>>=20 >>>>=20 >>>>=20 >>>=20 >>>=20 >>>=20 >>> Thank you. Great job with that video! And waveform has become the goto >>> site for many now. >>>=20 >>>=20 >>>=20 >>>=20 >>> I can't help but wonder tho... are you collecting any statistics, over >>> time, as to how much better the problem is getting? >>>=20 >>>=20 >>>=20 >>>=20 >>> And any chance they could do something similar explaining wifi? >>>=20 >>>=20 >>>=20 >>>=20 >>> ... >>>=20 >>>=20 >>>=20 >>>=20 >>> I was just at WISPA conference week before last. Preseem's booth >>> (fq_codel) was always packed. Vilo living had put cake in their wifi 6 >>> product. A >>> keynote speaker had deployed it and talked about it with waveform resul= ts >>> on the big screen (2k people there). A large wireless vendor demo'd >>> privately to me their flent results before/after cake on their next-gen >>> radios... and people dissed tarana without me prompting for their bad >>> bufferbloat... and the best thing of all that happened to me was... >>> besides getting a hug from a young lady (megan) who'd salvaged her >>> schooling in alaska using sqm - I walked up to the paraqum booth >>> (another large QoE middlebox maker centered more in india) and asked. >>>=20 >>>=20 >>>=20 >>> "So... do y'all have fq_codel yet?" >>>=20 >>>=20 >>>=20 >>>=20 >>> And they smiled and said: "No, we have something better... we've got >>> cake." >>>=20 >>>=20 >>>=20 >>>=20 >>> "Cake? What's that?" - I said, innocently. >>>=20 >>>=20 >>>=20 >>>=20 >>> They then stepped me through their 200Gbps (!!) product, which uses a >>> bunch of offloads, and can track rtt down to a ms with the intel ethern= et >>> card they were using. They'd modifed cake to provide 16 (?) levels of >>> service, and were running under dpdk (I am not sure if cake was). It wa= s a >>> great, convincing pitch... >>>=20 >>>=20 >>>=20 >>>=20 >>> ... then I told 'em who I was. There's a video of the in-both concert >>> after. >>>=20 >>>=20 >>>=20 >>>=20 >>> ... >>>=20 >>>=20 >>>=20 >>>=20 >>> The downside to me (and the subject of my talk) was that in nearly ever= y >>> person I talked to, fq_codel was viewed as a means to better subscriber >>> bandwidth plan enforcement (which is admittedly the market that preseem >>> pioneered) and it was not understood that I'd got involved in this whol= e >>> thing because I'd wanted an algorithm to deal with "rain fade", running >>> directly on the radios. People wanted to use the statistics on the radi= os >>> to drive the plan enforcement better >>> (which is an ok approach, I guess), and for 10+ I'd been whinging about >>> the... physics. >>>=20 >>>=20 >>>=20 >>> So I ranted about rfc7567 a lot and begged people now putting routerOS >>> 7.2 and later out there (mikrotik is huge in this market), to kill thei= r >>> fifos and sfqs at the native rates of the interfaces... and watch their >>> network improve that way also. >>>=20 >>>=20 >>>=20 >>> I think one more wispa conference will be a clean sweep of everyone in = the >>> fixed wireless market to not only adopt these algorithms for plan >>> enforcement, but even more directly on the radios and more CPE. >>>=20 >>>=20 >>>=20 >>>=20 >>> I also picked up enough consulting business to keep me busy the rest of >>> this year, and possibly more than I can handle (anybody looking?) >>>=20 >>>=20 >>>=20 >>>=20 >>> I wonder what will happen at a fiber conference? >>>=20 >>>=20 >>>=20 >>>>=20 >>>>=20 >>>> On Mon, Oct 17 , 2022 at 7:45 PM Dave Taht via Bloat < bloat@ lists. >>>> bufferbloat. net ( bloat@ lists. bufferbloat. net ( >>>> bloat@lists.bufferbloat.net ) ) > wrote: >>>>=20 >>>>=20 >>>>=20 >>>>>=20 >>>>>=20 >>>>> On Mon, Oct 17 , 2022 at 5:02 PM Stuart Cheshire < cheshire@ apple. c= om ( cheshire@ >>>>> apple. com ( cheshire@apple.com ) ) > wrote: >>>>>=20 >>>>>=20 >>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> On 9 Oct 2022 , at 06:14, Dave Taht via Make-wifi-fast < make-wifi-f= ast@ >>>>>> lists. bufferbloat. net ( make-wifi-fast@ lists. bufferbloat. net ( >>>>>> make-wifi-fast@lists.bufferbloat.net ) ) > wrote: >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>> This was so massively well done, I cried. Does anyone know how to g= et in >>>>>>> touch with the ifxit folk? >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>> https:/ / www. youtube. com/ watch?v=3DUICh3ScfNWI ( https:/ / www.= youtube. >>>>>>> com/ watch?v=3DUICh3ScfNWI ( https://www.youtube.com/watch?v=3DUICh= 3ScfNWI ) ) >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> I=E2=80=99m surprised that you liked this video. It seems to me that= it repeats >>>>>> all the standard misinformation. The analogy they use is the standar= d >>>>>> terrible example of waiting in a long line at a grocery store, and t= he >>>>>> =E2=80=9Csolution=E2=80=9D is letting certain traffic =E2=80=9Cjump = the line, angering everyone >>>>>> behind them=E2=80=9D. >>>>>>=20 >>>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>> Accuracy be damned. The analogy to common experience resonates more. >>>>>=20 >>>>>=20 >>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> Some quotes from the video: >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>> it would be so much more efficient for them to let you skip the lin= e and >>>>>>> just check out, especially since you=E2=80=99re in a hurry, but the= y=E2=80=99re rudely >>>>>>> refusing >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>> I think the person with the cheetos pulling out a gun and shooting >>>>> everyone in front of him (AQM) would not go down well. >>>>>=20 >>>>>=20 >>>>>=20 >>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>> to go back to our grocery store analogy this would be like if a wor= ker saw >>>>>>> you standing at the back ... and either let you skip to the front o= f the >>>>>>> line or opens up an express lane just for you >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>> Actually that analogy is fairly close to fair queuing. The multiple >>>>> checker analogy is one of the most common analogies in queue theory >>>>> itself. >>>>>=20 >>>>>=20 >>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> The video describes the problem of bufferbloat, and then describes t= he >>>>>> same failed solution that hasn=E2=80=99t worked for the last three d= ecades. >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>> Hmm? It establishes the scenario, explains the problem *quickly*, dis= ses >>>>> gamer routers for not getting it right.. *points to an accurate test*= , and >>>>> then to the ideas and products that *actually work* with "smart queue= ing", >>>>> with a screenshot of the most common >>>>> (eero's optimize for gaming and videoconferencing), and fq_codel and = cake >>>>> *by name*, and points folk at the best known solution available, open= wrt. >>>>>=20 >>>>>=20 >>>>>=20 >>>>> Bing, baddabang, boom. Also the comments were revealing. A goodly >>>>> percentage already knew the problem, more than a few were inspired to= take >>>>> the test, >>>>> there was a whole bunch of "Aha!" success stories and 360k views, whi= ch is >>>>> more people than we've ever been able to reach in for example, a nano= g >>>>> conference. >>>>>=20 >>>>>=20 >>>>>=20 >>>>> I loved that folk taking the test actually had quite a few A results, >>>>> without having had to do anything. At least some ISPs are getting it = more >>>>> right now! >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>> At this point I think gamers in particular know what "brands" we've t= ried >>>>> to establish - "Smart queues", "SQM", "OpenWrt", fq_codel and now "ca= ke" >>>>> are "good" things to have, and are stimulating demand by asking for t= hem, >>>>> It's certainly working out better and better for evenroute, firewalla= , >>>>> ubnt and others, and I saw an uptick in questions about this on vario= us >>>>> user forums. >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>> I even like that there's a backlash now of people saying "fixing >>>>> bufferbloat doesn't solve everything" - >>>>>=20 >>>>>=20 >>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> Describing the obvious simple-minded (wrong) solution that any norma= l >>>>>> person would think of based on their personal human experience waiti= ng in >>>>>> grocery stores and airports, is not describing the solution to >>>>>> bufferbloat. The solution to bufferbloat is not that if you are priv= ileged >>>>>> then you get to =E2=80=9Cskip to the front of the line=E2=80=9D. The= solution to >>>>>> bufferbloat is that there is no line! >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>> I like the idea of a guru floating above a grocery cart with a better >>>>> string of explanations, explaining >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>> - "no, grasshopper, the solution to bufferbloat is no line... at all"= . >>>>>=20 >>>>>=20 >>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> With grocery stores and airports people=E2=80=99s arrivals are indep= endent and not >>>>>> controlled. There is no way for a grocery store or airport to genera= te >>>>>> backpressure to tell people to wait at home when a queue begins to f= orm. >>>>>> The key to solving bufferbloat is generating timely backpressure to >>>>>> prevent the queue forming in the first place, not accepting a huge q= ueue >>>>>> and then deciding who deserves special treatment to get better servi= ce >>>>>> than all the other peons who still have to wait in a long queue, jus= t like >>>>>> before. >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>> I am not huge on the word "backpressure" here. Needs to signal the ot= her >>>>> side to slow down, is more accurate. So might say timely signalling r= ather >>>>> than timely backpressure? >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>> Other feedback I got was that the video was too smarmy (I agree), >>>>> different audiences than gamers need different forms of outreach... >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>> but to me, winning the gamers has always been one of the most importa= nt >>>>> things, as they make a lot of buying decisions, and they benefit the = most >>>>> for >>>>> fq and packet prioritization as we do today in gamer routers and in c= ake + >>>>> qosify. >>>>>=20 >>>>>=20 >>>>>=20 >>>>> maybe that gets in the way of more serious markets. Certainly I would= like >>>>> another video explaining what goes wrong with videoconferencing. >>>>>=20 >>>>>=20 >>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> Stuart Cheshire >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>> -- >>>>> This song goes out to all the folk that thought Stadia would work: ht= tps:/ >>>>>=20 >>>>> / www. linkedin. com/ posts/ >>>>> dtaht_the-mushroom-song-activity-6981366665607352320-FXtz >>>>> ( >>>>> https:/ / www. linkedin. com/ posts/ dtaht_the-mushroom-song-activity= -6981366665607352320-FXtz >>>>> ( >>>>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-69813= 66665607352320-FXtz >>>>> ) >>>>> ) Dave T=C3=A4ht CEO, TekLibre, LLC >>>>> _______________________________________________ >>>>> Bloat mailing list >>>>> Bloat@ lists. bufferbloat. net ( Bloat@lists.bufferbloat.net ) https:= //lists.bufferbloat.net/listinfo/bloat >>>>>=20 >>>>>=20 >>>>>=20 >>>>=20 >>>>=20 >>>=20 >>>=20 >>>=20 >>> -- >>> This song goes out to all the folk that thought Stadia would work: http= s:/ >>>=20 >>> / www. linkedin. com/ posts/ >>> dtaht_the-mushroom-song-activity-6981366665607352320-FXtz >>> ( >>> https:/ / www. linkedin. com/ posts/ dtaht_the-mushroom-song-activity-6= 981366665607352320-FXtz >>> ( >>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366= 665607352320-FXtz >>> ) >>> ) Dave T=C3=A4ht CEO, TekLibre, LLC >>>=20 >>>=20 >>=20 >>=20 >=20 >=20 >=20 > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. >=20 >=20 > --a00aad616cc2c3b0f180550026cfbbf30ab098aff9069214a780a4a411f0 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=UTF-8
Hi Sebastian,

[SM] Just an observation, using Saf= ari I see large maximal delays (like a small group of samples far out to th= e right of the bulk) for both down- and upload that essentially disappear w= hen I switch to firefox. Now I tend to have a ton of tabs open in Safari while I only open firefox f= or dedicated use-cases with a few tabs at most, so I do not intend to throw= shade on Safari here; my point is more browsers can and do affect the repo= rted latency numbers, of you want to be able to test this, maybe ask users = to use the OS browser (safari, edge, konqueror ;) ) as well as firefox and = chrome so you can directly compare across browsers?

I believe this is because we use= the WebTiming APIs to get a more accurate latency numbers, but the API isn= 't fully supported on Safari. As such, latency measurements in Safari a= re much less accurate than in Firefox and Chrome.=C2=A0

traceroute/mtr alb= eit not sure how well this approach works from inside the browser, can you = e.g. control TTL and do you receive error messages via ICMP?

Unfortunately tracerout= es via the browser don't really work :(. And I don't believe we can= control TTL or see ICMP error messages=C2=A0either, though I haven't d= ug into this very deeply.

Over in the OpenWrt forum we oft= en see that server performance with iperf2/3 or netperf on a router is not = all that representative for its routing performance. What do you expect to deduce from upload/download to the router? (I might m= isunderstand your point by a mile, if so please elaborate)


The goal would be to= test the "local" latency, throughput, and bufferbloat between the = user's device and the router, and then compare this with the latency, t= hroughput, and bufferbloat when DL/ULing to a remote server.

This would reveal whether=C2=A0the dominant source of increase in late= ncy under load is=C2=A0at the router's WAN interface or somewhere betwe= en the router and the user (e.g. WiFi, ethernet, powerline, Moca devices, P= tP connections, etc).

Being able to test the user-to-router leg of the connection would be= helpful more broadly beyond just bufferbloat.=C2=A0I often want to diagnos= e whether my connection issues or speed drops are happening due to an issue= with my modem (and more generally the WAN connection) or if it's an is= sue with my wifi connection.=C2=A0

I guess I don't quite understand this part though: "ip= erf2/3 or netperf on a router is not all that representative for its routin= g performance." What exactly do you mean here?

=E2=80=8BMost recent di= scussion moved over to=C2=A0https://forum.openwrt.org/t/cake-w-ada= ptive-bandwidth/135379

Thanks! I have a lot of catching up to do on that thread,= and some of it is definitely above my pay grade :).

=E2=80=8BI think this ideally w= ould be solved at the 3GPPP level

Agreed. I wonder if there&= #39;s anything we can do to encourage them to pay attention to this.

Best regards,

Sina.
On Tue, Oct 18, 2022 at 12:04 PM, Sebastian Moeller <moeller0@gmx.de> wrote:

Hi Sin= a,

On 18 October 2022 19:17:16 CEST, Sina Khanifar via Bloat <bloat@lists.bufferbloat.net> wrote= :

I can't help but wonder tho... are you collecting any statistics, over time, as to how much better the problem is getting?

We are collecting anonymized=C2=A0data, but we haven't analyzed it yet.= If we get a bit of time we'll look at that hopefully.

[SM] Just an observation, using Safari I see large maximal delays (like a s= mall group of samples far out to the right of the bulk) for both down- and = upload that essentially disappear when I switch to firefox. Now I tend to have a ton of tabs open in Safari while I only open firefox f= or dedicated use-cases with a few tabs at most, so I do not intend to throw= shade on Safari here; my point is more browsers can and do affect the repo= rted latency numbers, of you want to be able to test this, maybe ask users = to use the OS browser (safari, edge, konqueror ;) ) as well as firefox and = chrome so you can directly compare across browsers?

And any chance they could do something similar explaining wifi?

I'm actually not exactly sure what mitigations exist for WiFi at the mo= ment - is there something I can read?

On this note: when we were building our test one of the things we really wi= shed existed was a standardized way to test latency and throughput to route= rs.=20

[SM] traceroute/mtr albeit not sure how well this approach works from insid= e the browser, can you e.g. control TTL and do you receive error messages v= ia ICMP?

It would be super helpful if there was a standard in consumer routers=C2=A0= that allowed users to both ping and fetch 0kB fils from their routers, and = also run download/upload tests.

[SM] I think I see where you are coming from here. Over in the OpenWrt foru= m we often see that server performance with iperf2/3 or netperf on a router= is not all that representative for its routing performance. What do you expect to deduce from upload/download to the router? (I might m= isunderstand your point by a mile, if so please elaborate)

Regards
Sebastian

I think one more wispa conference will be a clean sweep of everyone in the fixed wireless market to not only adopt these algorithms for plan enforcement, but even more directly on the radios and more CPE.

T-Mobile has signed up 1m+ people to their new Home Internet over 5G, and a= ll of them have really meaningful bufferbloat issues. I've been pointin= g folks who reach out to this thread ( https://forum.openwrt.o= rg/t/cake-w-adaptive-bandwidth-historic/108848 ) abou= t cake-autorate and sqm-autorate, but ideally it would be fixed at a networ= k level, just not sure how to apply pressure (I'm in contact with the T= -Mobile Home Internet team, but I think this is above their heads).

On Mon, Oct 17, 2022 at 8:15 PM, Dave Taht < dave.<= wbr/>taht@gmail.com > wrote:

On= Mon, Oct 17, 2022 at 7:51 PM Sina Khanifar < sina@ waveform. com= ( sina@waveform.com ) > wrote:

Positive or negative, I can claim a bit of credit for this video :). We'= ;ve been working with LTT on a few projects and we pitched them on doing something around bufferbloat. We've seen more traffic to our Waveforn t= est than ever before, which has been fun!

Thank you. Great job with that video! And waveform has become the goto site for many now.

I can't help but wonder tho... are you collecting any statistics, over time, as to how much better the problem is getting?

And any chance they could do something similar explaining wifi?

...

I was just at WISPA conference week before last. Preseem's booth
(fq_codel) was always packed. Vilo living had put cake in their wifi 6 product. A
keynote speaker had deployed it and talked about it with waveform results on the big screen (2k people there). A large wireless vendor demo'd privately to me their flent results before/after cake on their next-gen radios... and people dissed tarana without me prompting for their bad bufferbloat... and the best thing of all that happened to me was... besides getting a hug from a young lady (megan) who'd salvaged her schooling in alaska using sqm - I walked up to the paraqum booth
(another large QoE middlebox maker centered more in india) and asked.

"So... do y'all have fq_codel yet?"

And they smiled and said: "No, we have something better... we've go= t cake."

"Cake? What's that?" - I said, innocently.

They then stepped me through their 200Gbps (!!) product, which uses a bunch of offloads, and can track rtt down to a ms with the intel ethernet card they were using. They'd modifed cake to provide 16 (?) levels of service, and were running under dpdk (I am not sure if cake was). It was a great, convincing pitch...

... then I told 'em who I was. There's a video of the in-both conce= rt after.

...

The downside to me (and the subject of my talk) was that in nearly every person I talked to, fq_codel was viewed as a means to better subscriber bandwidth plan enforcement (which is admittedly the market that preseem pioneered) and it was not understood that I'd got involved in this whol= e thing because I'd wanted an algorithm to deal with "rain fade",= running directly on the radios. People wanted to use the statistics on the radios to drive the plan enforcement better
(which is an ok approach, I guess), and for 10+ I'd been whinging about the... physics.

So I ranted about rfc7567 a lot and begged people now putting routerOS
7.2 and later out there (mikrotik is huge in this market), to kill their fifos and sfqs at the native rates of the interfaces... and watch their network improve that way also.

I think one more wispa conference will be a clean sweep of everyone in the fixed wireless market to not only adopt these algorithms for plan enforcement, but even more directly on the radios and more CPE.

I also picked up enough consulting business to keep me busy the rest of this year, and possibly more than I can handle (anybody looking?)

I wonder what will happen at a fiber conference?

On= Mon, Oct 17, 2022 at 7:45 PM Dave Taht via Bloat < bloat@ lists.= bufferbloat. net ( bloat@lists.bufferbloat.net ) > wrote:

On= Mon, Oct 17, 2022 at 5:02 PM Stuart Cheshire < cheshire@ apple. = com ( cheshire@apple.com ) > wrote:

On 9 Oct 2022, at 06:14, Dave Taht via Make-wifi-fast < make-wifi-f= ast@ lists. bufferbloat. net ( make-wifi-fast@lists.bufferbloat.net ) > wrote:

This was so massively well done, I cried. Does anyone know how to get in touch with the ifxit folk?

https:/ / www. youtube. com/ watch?v=3DUICh3ScfNWI ( https://www.youtu= be.com/watch?v=3DUICh3ScfNWI )

I=E2=80=99m surprised that you liked this video. It seems to me that it rep= eats all the standard misinformation. The analogy they use is the standard terrible example of waiting in a long line at a grocery store, and the
=E2=80=9Csolution=E2=80=9D is letting certain traffic =E2=80=9Cjump the lin= e, angering everyone behind them=E2=80=9D.

Accuracy be damned. The analogy to common experience resonates more.

Some quotes from the video:

it would be so much more efficient for them to let you skip the line and just check out, especially since you=E2=80=99re in a hurry, but they=E2=80= =99re rudely refusing

I think the person with the cheetos pulling out a gun and shooting everyone in front of him (AQM) would not go down well.

to go back to our grocery store analogy this would be like if a worker saw you standing at the back ... and either let you skip to the front of the line or opens up an express lane just for you

Actually that analogy is fairly close to fair queuing. The multiple checker analogy is one of the most common analogies in queue theory itself.

The video describes the problem of bufferbloat, and then describes the same failed solution that hasn=E2=80=99t worked for the last three decades.

Hmm? It establishes the scenario, explains the problem *quickly*, disses gamer routers for not getting it right.. *points to an accurate test*, and then to the ideas and products that *actually work* with "smart queuein= g", with a screenshot of the most common
(eero's optimize for gaming and videoconferencing), and fq_codel and ca= ke
*by name*, and points folk at the best known solution available, openwrt.

Bing, baddabang, boom. Also the comments were revealing. A goodly percentage already knew the problem, more than a few were inspired to take the test,
there was a whole bunch of "Aha!" success stories and 360k views, w= hich is more people than we've ever been able to reach in for example, a nanog conference.

I loved that folk taking the test actually had quite a few A results, without having had to do anything. At least some ISPs are getting it more right now!

At this point I think gamers in particular know what "brands" we= 9;ve tried to establish - "Smart queues", "SQM", "OpenWrt", fq= _codel and now "cake" are "good" things to have, and are stimulating demand by asking for= them, It's certainly working out better and better for evenroute, firewalla, ubnt and others, and I saw an uptick in questions about this on various user forums.

I even like that there's a backlash now of people saying "fixing bufferbloat doesn't solve everything" -

Describing the obvious simple-minded (wrong) solution that any normal person would think of based on their personal human experience waiting in grocery stores and airports, is not describing the solution to bufferbloat. The solution to bufferbloat is not that if you are privileged then you get to =E2=80=9Cskip to the front of the line=E2=80=9D. The soluti= on to bufferbloat is that there is no line!

I like the idea of a guru floating above a grocery cart with a better string of explanations, explaining

- "no, grasshopper, the solution to bufferbloat is no line... at all= 4;.

With grocery stores and airports people=E2=80=99s arrivals are independent = and not controlled. There is no way for a grocery store or airport to generate backpressure to tell people to wait at home when a queue begins to form. The key to solving bufferbloat is generating timely backpressure to prevent the queue forming in the first place, not accepting a huge queue and then deciding who deserves special treatment to get better service than all the other peons who still have to wait in a long queue, just like before.

I am not huge on the word "backpressure" here. Needs to signal the = other side to slow down, is more accurate. So might say timely signalling rather than timely backpressure?

Other feedback I got was that the video was too smarmy (I agree), different audiences than gamers need different forms of outreach...

but to me, winning the gamers has always been one of the most important things, as they make a lot of buying decisions, and they benefit the most for
fq and packet prioritization as we do today in gamer routers and in ca= ke + qosify.

maybe that gets in the way of more serious markets. Certainly I would like another video explaining what goes wrong with videoconferencing.

Stuart Cheshire

--
This song goes out to all the folk that thought Stadia would work: https:/
/ www. linkedin. com/ posts/ dtaht_the-mushroom-song-activity-6981366665607= 352320-FXtz
(
https://www.linkedin.com/posts/= dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
) Dave T=C3=A4ht CEO, TekLibre, LLC
_______________________________________________
Bloat mailing list
Bloat@ lists. bufferbloat. net ( Bloat@lists.bufferbloat.= net ) https://lists.bufferbloat.net/listinfo/bloat

--
This song goes out to all the folk that thought Stadia would work: https:/
/ www. linkedin. com/ posts/ dtaht_the-mushroom-song-activity-6981366665607= 352320-FXtz
(
https://www.linkedin.com/posts/= dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
) Dave T=C3=A4ht CEO, TekLibre, LLC

--=20
Sent from my Android device with K-9 Mail. Please excuse my brevity.


--a00aad616cc2c3b0f180550026cfbbf30ab098aff9069214a780a4a411f0--