From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 14BDD3B29E for ; Mon, 24 Oct 2022 14:14:15 -0400 (EDT) Received: by mail-wm1-x32a.google.com with SMTP id bg9-20020a05600c3c8900b003bf249616b0so7133529wmb.3 for ; Mon, 24 Oct 2022 11:14:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=1kIXqX4ApdY8OcyTv3bqCHWplsPc7w1CiUAoS5JuU24=; b=F8wqpX/7AcrpVnGopaR1WDKdYM+dGJVFk5zCB453x8zH4pNyuHR+Hp2pRUrwEqzqBK zKPc6jaCRT+1u+sSdD4kPZkf0chybsj3fafAj8rf7WBx/SMk8Y6PSr2uJdlwA50xGz+G v3lM0D8tdvS8jCAs9/9WMOwKkBRxfxF+GLhBfHOXxJwteXdzB7oBEtiStEeqpJPEGuxZ 29UYCcojccsRT7nxR7FEUogwN9tqfm17ri5h0TFkSnP2M4ul9WF7wgtRpGb1s1OuPsrY A1O7ZR9eKKPwB6mreSrgvEG21AmuRt7irmdsvg4+yZZFj5ni6JA0RH+kZwjLnI9AlkF3 1ZXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1kIXqX4ApdY8OcyTv3bqCHWplsPc7w1CiUAoS5JuU24=; b=gFkSjx3ebF5HNCdRuurTLOIILZg4zTbFHxbsfSepF7EhQD5/v3IPHqnC5mGzFJnF9Q oRtqHGK1f83ficHkXesNAwweLLxwrwraQl/rFT7C4XZu28Pv4kf0KWe8X4pb/6XBCitP x/X5CiA5r/3EwkPfROfS0HDz1aaCUidpTMSntYQWKMKC4m19UbDlVp1Hs5z+Fclv6bz+ LLwBcUGw0LtqlhCmPjN4wN44P1EaMJZH3cAfxZzMqJcodDKNm7DKGHMROmXz5VM9/sFj 4fLCx1hVSxVo449qw1UVZzIFEL9zorA2w7pFm9fLLebVyKStalaGqTKHx2zqwX68z9xt lUZQ== X-Gm-Message-State: ACrzQf2mw24SNzVxwKLDilEiGD3W++wN2/PtQ7gsvny+hgcLFozTLf3X PuDJoCcTwdcxm8QDm/MffXq/U5is5GukTKF+jzo= X-Google-Smtp-Source: AMsMyM5oHHPOtbiclWr4z1rB7bV4mZRnuqSZKRAhnN235YlNeqFe1wEVSeI0ub6kgiD1eB5rDBJUs/0ffBtK+b2jOSo= X-Received: by 2002:a05:600c:2104:b0:3c6:b7f0:cfd0 with SMTP id u4-20020a05600c210400b003c6b7f0cfd0mr44127673wml.128.1666635253886; Mon, 24 Oct 2022 11:14:13 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Dave Taht Date: Mon, 24 Oct 2022 11:14:00 -0700 Message-ID: To: dan Cc: =?UTF-8?Q?Robert_Chac=C3=B3n?= , libreqos@lists.bufferbloat.net Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [LibreQoS] ookla speedtest results? X-BeenThere: libreqos@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Many ISPs need the kinds of quality shaping cake can do List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2022 18:14:15 -0000 On Mon, Oct 24, 2022 at 11:03 AM dan wrote: > > We're pretty far past that hardware. Minimum plans at most sites are now= 100x20 and we're moving rapidly to 200x200-1Gx1G plans. The implication b= eing that we need 4x4 AX wifi radios and that's real tough in an openwrt co= mpatible hardware right now, not a ton of options. At a gbit, most router designs fall over in the first place, so I have leaned away from the all-in-one approach to date, in favor of a base unit + wifi satellites. Now things are getting better since this had been written: https://forum.openwrt.org/t/so-you-have-500mbps-1gbps-fiber-and-need-a-rout= er-read-this-first/90305 But it's still quite hard to push a gbit, do firewalling, etc, etc. > > Flat out though, wisps in general simply do not have the option of contro= lling every client side router or even sending out new/compatible routers f= or each customer to pull this off. It's a non-starter for us. It may be= better but it's just not an option that can be pursued. ingress and egres= s edge shaping is what we need. I agree that it is impossible to control every client side router, and that will forevermore retain the demand for middlebox shaping. I was trying to say, it would be better if as many routers as possible - especially the wifi ones - also managed bufferbloat better, especially at range. Most of the wifi routers I've tested hit a performance cliff at ever decreasing ranges, it seems, mesh wifi (even eeros) have really terrible responsiveness, and so on. Stuff in the 6 band is really flaky still. One of the opportunities in herbert's new cpumap-qoe code is to actually try to manage rtts better on the prem wifi, from the center. > On Mon, Oct 24, 2022 at 11:57 AM Dave Taht wrote: >> >> On Mon, Oct 24, 2022 at 10:46 AM dan wrote: >> > >> > Some sort of % drop is better than count from an operator's perspectiv= e. I'm actually happy with 'scaling issues' of % on small vs large plan be= cause those small plans are the one's we probably want to push for upgrades= if they're seeing a lot of drops. >> > >> > That said, a 'score' method works really well from a high level. And = that can be proportionally adjusted so a slightly higher % in a small plan = gives the same score as a larger plan would with fewer drops. >> > >> > >> > Also, on the SNMP statement above, we had the same issue with preseem = being hyper aggressive to pull stats. Some mikrotik versions have some snm= p flaws and preseem can peg the CPU. Hasn't happened in a while but that w= as an issue. >> >> I gave up on mikrotik long before they started working on routeros7, >> and just reflashed everything to openwrt. >> >> I have a nice little custom build for it assembled for the ubnt agws, >> and the mikrotik hap ac lite. Anyone using those? >> >> I would hope they don't have the snmp problem you describe, and if >> they did I could get it fixed in days. >> >> Vastly improved the wifi in particular, see before/after here: >> https://forum.openwrt.org/t/aql-and-the-ath10k-is-lovely/59002/ >> >> (After about 40Mbits, the bloat starts moving to the wifi) >> >> Most of my work in the past 9 months has been on improving the >> mediatek mt76 derived products. I left off here: >> https://forum.openwrt.org/t/aql-and-the-ath10k-is-lovely/59002/908 >> >> A lot of this stuff is shipping commercially in the openwifi project, >> if you haven't been paying attention to that. >> >> > On Mon, Oct 24, 2022 at 11:42 AM Dave Taht wrote= : >> >> >> >> On Mon, Oct 24, 2022 at 10:35 AM Robert Chac=C3=B3n >> >> wrote: >> >> > >> >> > > Presently (in v1.3.), robert (in grafina?) is summing drops + mar= ks. >> >> > which is a good idea, however perhaps drops, marks and drops+marks >> >> > would be better. >> >> > >> >> > Yup, in InfluxDB for the stats by Tin (diffserv4) we display true d= rops as (trueDrops =3D ecn_mark + drops - ack_drops). But we don't do it by= sub quite yet. >> >> > And ok, drops, marks and drops+marks can be shown instead - by subs= criber or by Node (AP/Site). >> >> > Dave - is it ok if we graph this data as a percentage of packets se= nt? So for drops, drop / sent_packets ? I just imagine that would make com= paring different subscribers' connections easier. >> >> >> >> Drops and marks are proportional to the plan. You get a heck of a lot >> >> more drops at 5mbits than 50, and the relationship >> >> is not linear, closer to quadratic. >> >> >> >> So comparing customers at their given tiers makes more sense than >> >> across the board. >> >> >> >> As for a percentage, you should see a clear difference in percentages >> >> across tiers. >> >> >> >> I know I'm not quite answering your question well, I'm a big believer >> >> in just getting lots of data, graphing it every which way, >> >> and seeing what's useful. >> >> >> >> > Question for all - is having drops / marks/ drops+marks by subscrib= er/circuit worth the extra modest InfluxDB CPU use? Or should we just show = those stats by AP/Node? >> >> > If preseem currently offers such stats at a subscriber level we can= do the same, just wanna be sure I'm not overtaxing CPU. >> >> > >> >> > On Mon, Oct 24, 2022 at 11:14 AM Dave Taht via LibreQoS wrote: >> >> >> >> >> >> Some of the context of my request for libreqos on vs off is from h= ere, >> >> >> presentation in the wee hours, wed morning. >> >> >> >> >> >> https://www.linkedin.com/feed/update/urn:li:activity:6989281945868= 283904/ >> >> >> >> >> >> On Mon, Oct 24, 2022 at 9:48 AM dan via LibreQoS >> >> >> wrote: >> >> >> > >> >> >> > we also have a bunch of 25/5. It would be nice to get some more= detailed numbers from ookla tests. >> >> >> >> >> >> +1! We've argued methods with many a provider, and ookla and samkn= ows >> >> >> are new to this game. >> >> >> >> >> >> As one example I don't know if they use decimal or binary mbits! >> >> >> >> >> >> ... >> >> >> >> >> >> ack-filtering in the libreqos case can and will drop acks in both >> >> >> directions, due to the structure of how htb >> >> >> works in this scenario, at any bandwidth ratio. I *think* it does = very >> >> >> little harm but I would use the ack-filter >> >> >> not the aggressive ack filter out of caution. >> >> >> >> >> >> Secondly if you are gathering drop statistics from the higher leve= l >> >> >> htb categories, total drops will go WAY up, >> >> >> and it is best to use the cake drop, ack_drop outputs statistics d= irectly. >> >> >> >> >> >> ack_drops have nothing to do with congestion control and should be >> >> >> ignored in most cases. >> >> >> >> >> >> Also I keep hoping more will pull out ecn marks separately. Public= use >> >> >> is way up by e2e transports, and I worry that a high ecn ratio is = a >> >> >> e2e network error, which might show up as a big backlog. >> >> >> >> >> >> Presently (in v1.3.), robert (in grafina?) is summing drops + mark= s. >> >> >> which is a good idea, however perhaps drops, marks and drops+marks >> >> >> would be better. >> >> >> >> >> >> An inverse log scale is needed to plot marks, drops, and ack_drops= . >> >> >> >> >> >> I'll always be of the opinion that it's best to run cake on egress= at >> >> >> the cpe, first, doing all this work before it hits the wire. >> >> >> Are any of you doing that? Openwrt/dd-wrt/firewalla/mikrotik 7.2? >> >> >> >> >> >> >We also have a lot of Eero units out there, so watching for the e= ero speed test would also be great. >> >> >> >> >> >> eero 5s have cake (and fq_codel on the wifi), eero 6s have a somew= hat >> >> >> buggy fq_codel offload: >> >> >> >> >> >> https://www.reddit.com/r/eero/comments/u7xm83/gen_2_sqm_vs_gen_3_s= qm_stick_with_gen_2_if_you/ >> >> >> > On Mon, Oct 24, 2022 at 10:31 AM Herbert Wolverson via LibreQoS = wrote: >> >> >> >> >> >> >> >> > sniff DNS for those speed test lookups and trigger a packet c= apture for x duration? >> >> >> >> >> >> >> >> That's a good idea (I think there's a service somewhere for fin= ding speedtest >> >> >> >> servers?). We're discussing testing the ack-filter feature righ= t now, and something >> >> >> >> similar would work well for that. (We have a bunch of 25/5 and = similarly asymmetric >> >> >> >> connections, in theory it'll help...) >> >> >> >> >> >> >> >> On Mon, Oct 24, 2022 at 10:31 AM dan via LibreQoS wrote: >> >> >> >>> >> >> >> >>> sniff DNS for those speed test lookups and trigger a packet ca= pture for x duration? >> >> >> >>> >> >> >> >>> On Mon, Oct 24, 2022 at 9:27 AM Dave Taht wrote: >> >> >> >>>> >> >> >> >>>> On Mon, Oct 24, 2022 at 8:20 AM dan wro= te: >> >> >> >>>> > >> >> >> >>>> > Dave, I think that 'bouncing' is more CPU time in the brows= er. I don't think it's indicative over what's actually happening. >> >> >> >>>> >> >> >> >>>> A simultaneous packet capture and wireshark plot of the RTTs = would >> >> >> >>>> also be helpful. >> >> >> >>>> >> >> >> >>>> > >> >> >> >>>> > On Mon, Oct 24, 2022 at 9:09 AM Dave Taht via LibreQoS wrote: >> >> >> >>>> >> >> >> >> >>>> >> On Mon, Oct 24, 2022 at 8:02 AM Robert Chac=C3=B3n via Lib= reQoS >> >> >> >>>> >> wrote: >> >> >> >>>> >> > >> >> >> >>>> >> > I can run an Ookla test in the middle of the night with = LibreQoS off for a bit. >> >> >> >>>> >> >> >> >> >>>> >> Groovy. >> >> >> >>>> >> >> >> >> >>>> >> > From what I recall - without Libre we see download bloat= of 300ms or so. With Libre, it's gone. On waveform we see 0ms added bloat = each direction for most clients. >> >> >> >>>> >> >> >> >> >>>> >> The numbers report by speedtest bounce around a lot, and t= hey tend to >> >> >> >>>> >> pick a lower number as a final result than I'd >> >> >> >>>> >> like. If there's a way to capture a movie of it... >> >> >> >>>> >> > >> >> >> >>>> >> > And I like the "monitor only" mode idea. >> >> >> >>>> >> > Perhaps we could create the HTB tree, but just not attac= h the CAKE qdisc? >> >> >> >>>> >> > Technically even HTB could reduce latency by itself on o= verloaded APs , so it wouldn't be true passive monitoring, but it would all= ow us to compare the before/after of CAKE while still having qdiscs we can = point cpumap-pping to. >> >> >> >>>> >> > >> >> >> >>>> >> > Alternatively, maybe we could use eBPF PPing just for th= is passive monitoring period? >> >> >> >>>> >> > >> >> >> >>>> >> > >> >> >> >>>> >> > On Mon, Oct 24, 2022 at 8:02 AM Herbert Wolverson via Li= breQoS wrote: >> >> >> >>>> >> >> >> >> >> >>>> >> >> Are you looking for a shaped result? That'll show "what= ever bandwidth >> >> >> >>>> >> >> is left on the sector" without Libre, and around the cu= stomer's paid-for >> >> >> >>>> >> >> target with Libre (hopefully!). >> >> >> >>>> >> >> >> >> >> >>>> >> >> One thing Preseem does well is that their onboarding pr= ocess includes >> >> >> >>>> >> >> "leave it in monitoring-only mode" for a week, gatherin= g data before you >> >> >> >>>> >> >> pull the switch (although most ISPs I've talked to just= pull the switch...) >> >> >> >>>> >> >> It's quite enlightening to see a before/after, even wit= h only FQ_CODEL >> >> >> >>>> >> >> as the shaper. >> >> >> >>>> >> >> >> >> >> >>>> >> >> We could probably do something similar with a "monitor = only" mode that >> >> >> >>>> >> >> doesn't enable any queues (is there a TC equivalent to = "dummy" on >> >> >> >>>> >> >> FreeBSD?). >> >> >> >>>> >> >> >> >> >> >>>> >> >> On Mon, Oct 24, 2022 at 8:52 AM Dave Taht via LibreQoS = wrote: >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> I am curious if anyone here has speedtest results with= libreqos on and >> >> >> >>>> >> >>> off, and can send a screen shot? >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> -- >> >> >> >>>> >> >>> This song goes out to all the folk that thought Stadia= would work: >> >> >> >>>> >> >>> https://www.linkedin.com/posts/dtaht_the-mushroom-song= -activity-6981366665607352320-FXtz >> >> >> >>>> >> >>> Dave T=C3=A4ht CEO, TekLibre, LLC >> >> >> >>>> >> >>> _______________________________________________ >> >> >> >>>> >> >>> LibreQoS mailing list >> >> >> >>>> >> >>> LibreQoS@lists.bufferbloat.net >> >> >> >>>> >> >>> https://lists.bufferbloat.net/listinfo/libreqos >> >> >> >>>> >> >> >> >> >> >>>> >> >> _______________________________________________ >> >> >> >>>> >> >> LibreQoS mailing list >> >> >> >>>> >> >> LibreQoS@lists.bufferbloat.net >> >> >> >>>> >> >> https://lists.bufferbloat.net/listinfo/libreqos >> >> >> >>>> >> > >> >> >> >>>> >> > >> >> >> >>>> >> > >> >> >> >>>> >> > -- >> >> >> >>>> >> > Robert Chac=C3=B3n >> >> >> >>>> >> > CEO | JackRabbit Wireless LLC >> >> >> >>>> >> > _______________________________________________ >> >> >> >>>> >> > LibreQoS mailing list >> >> >> >>>> >> > LibreQoS@lists.bufferbloat.net >> >> >> >>>> >> > https://lists.bufferbloat.net/listinfo/libreqos >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> -- >> >> >> >>>> >> This song goes out to all the folk that thought Stadia wou= ld work: >> >> >> >>>> >> https://www.linkedin.com/posts/dtaht_the-mushroom-song-act= ivity-6981366665607352320-FXtz >> >> >> >>>> >> Dave T=C3=A4ht CEO, TekLibre, LLC >> >> >> >>>> >> _______________________________________________ >> >> >> >>>> >> LibreQoS mailing list >> >> >> >>>> >> LibreQoS@lists.bufferbloat.net >> >> >> >>>> >> https://lists.bufferbloat.net/listinfo/libreqos >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> -- >> >> >> >>>> This song goes out to all the folk that thought Stadia would = work: >> >> >> >>>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activi= ty-6981366665607352320-FXtz >> >> >> >>>> Dave T=C3=A4ht CEO, TekLibre, LLC >> >> >> >>> >> >> >> >>> _______________________________________________ >> >> >> >>> LibreQoS mailing list >> >> >> >>> LibreQoS@lists.bufferbloat.net >> >> >> >>> https://lists.bufferbloat.net/listinfo/libreqos >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> >> LibreQoS mailing list >> >> >> >> LibreQoS@lists.bufferbloat.net >> >> >> >> https://lists.bufferbloat.net/listinfo/libreqos >> >> >> > >> >> >> > _______________________________________________ >> >> >> > LibreQoS mailing list >> >> >> > LibreQoS@lists.bufferbloat.net >> >> >> > https://lists.bufferbloat.net/listinfo/libreqos >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> This song goes out to all the folk that thought Stadia would work: >> >> >> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-69= 81366665607352320-FXtz >> >> >> Dave T=C3=A4ht CEO, TekLibre, LLC >> >> >> _______________________________________________ >> >> >> LibreQoS mailing list >> >> >> LibreQoS@lists.bufferbloat.net >> >> >> https://lists.bufferbloat.net/listinfo/libreqos >> >> > >> >> > >> >> > >> >> > -- >> >> > Robert Chac=C3=B3n >> >> > CEO | JackRabbit Wireless LLC >> >> >> >> >> >> >> >> -- >> >> This song goes out to all the folk that thought Stadia would work: >> >> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-69813= 66665607352320-FXtz >> >> Dave T=C3=A4ht CEO, TekLibre, LLC >> >> >> >> -- >> This song goes out to all the folk that thought Stadia would work: >> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-69813666= 65607352320-FXtz >> Dave T=C3=A4ht CEO, TekLibre, LLC --=20 This song goes out to all the folk that thought Stadia would work: https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-69813666656= 07352320-FXtz Dave T=C3=A4ht CEO, TekLibre, LLC