From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 3E16A3B29E for ; Mon, 24 Oct 2022 14:23:00 -0400 (EDT) Received: by mail-wr1-x433.google.com with SMTP id y16so7046493wrt.12 for ; Mon, 24 Oct 2022 11:23:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=LWxSJNh0X0Y9saOOxjH2YpcONh776ymbgBKHXmQf6B8=; b=HA3KAjfGLwGEJoJmHO8Y5xlhhz4DMVauWtPRrdM6+RBKpB/HPr0fOIHQrNbaT4XtVU 4X2hU+FlJM/Ow1muGYAHwMWOerhjFpaOY9mhkkYwfwaVP0fURU/BkuoyKJa6G5vNOUDL 5hx75eslP3zmYin4k4QdfCZ5sywHeD8BnZPBsE5+lHxP418y5g49ky4XWtrvYvtCpHh4 QB/sGkAyMN5qWFRMVeL70/CIEHeGRqMcroBcGwSV/o0U6OSSWcz6EPMCjoieC5acHVTw K7yx5eocgWb9MWnwr8kpRMniPLDxLgLoLhKJLvgedWYVrTYBFehKD9bhpOrFiEoIiTOQ BElg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LWxSJNh0X0Y9saOOxjH2YpcONh776ymbgBKHXmQf6B8=; b=5+xdHKycOJKrgkuBJLhhiAsK+WpmbnsJyfw2MLPhtFXCIt7WOhHI/1//aTarQeklrh iwm6Y9zXyGopGwKDrXA0cKrrUYm2kp2RxHQX+SfaqTKqP0P1CsqHolTFi7mLtt1Jr+ca 6I85pmNTRVScpXhRFArxOdiM6jBvAhEOS3p9ytkXj7bMYGtyDQ0pLNH6xClmiOUoKrsp nfzi7pUbT8qQqih02Y/wXdMm9gNk1T210+qw9Z3fuLXJhYcKScIiqHNVAsHMctCw9gi0 CMuZ4WsleHCRLegURDLSnPm9ACDtNuzI0Yvn4Y/JVot5JXLtVkBGowZe8TsUqBHMqxdm FXxA== X-Gm-Message-State: ACrzQf3y1LjIjg+1z3gc1pY6H+swyiVt6CoQ4HUMbU+8M0WVxtB7thpY 0ANBMwtXfyPOxjAvWzXvI7PzqbaE0sbYLU3T93k= X-Google-Smtp-Source: AMsMyM4MJG/WrAy+8r5ONY0k9cuoBsFWPF0v707wJVT/DKenHlApjCcR0ztwVv4VAZHBklKyhHgHxfa9L5oNonDdpcs= X-Received: by 2002:a05:6000:3cf:b0:231:6ed6:e978 with SMTP id b15-20020a05600003cf00b002316ed6e978mr21994315wrg.500.1666635778871; Mon, 24 Oct 2022 11:22:58 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Dave Taht Date: Mon, 24 Oct 2022 11:22:46 -0700 Message-ID: To: Herbert Wolverson Cc: libreqos@lists.bufferbloat.net Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [LibreQoS] ookla speedtest results? X-BeenThere: libreqos@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Many ISPs need the kinds of quality shaping cake can do List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2022 18:23:00 -0000 On Mon, Oct 24, 2022 at 11:06 AM Herbert Wolverson via LibreQoS wrote: > > I think we have a handful of HAP lite's left? They aren't great, but they= are cheap - so > we have a small apartment complex using them as client-side devices. The > complex is a half-way house for paroled offenders, and they destroy a *lo= t* > of hardware. :-| > > The biggest issue we've had (other than they aren't very resilient to som= eone > who needs anger-management classes) is the antenna. There's just not a lo= t > of gain there. I agree on the antenna front. There are plenty of other devices with better antennas. Still, I'm frugal, I keep hoping the stuff we are turning into green waste ends up instead, reflashed, operating at the bandwidths available in the third world, ipv6 capable, etc. I regularly ran routers down to SJDS, nicaragua precovid, and gave them to coffee shops and airbnbs. There are still a lot of people stuck (and unable to upgrade) at 25/3, too, where a 20 dollar router with cake and better wifi on it, would make a difference. I am ever more aware that my hippie-dippy approach to life is not shared by most! Here's a 3+ year old proposal that got no traction. https://docs.google.com/document/d/1T21on7g1MqQZoK91epUdxLYFGdtyLRgBat0VXoC= 9e3I/edit I was thinking with a tax subsidy, perhaps we could find uses for these older routers, as meshy stuff, at least. To this day, tho, I have dozens of routers, running modern openwrt, still in use, doing an adaquate job. My favorite is a picostation above a pool I like, still serving up 50+Mbit second, which was one of the very first routers I reflashed to fq_codel back in 2012. I have every reason to believe it will still be working 10 years from now. > > On Mon, Oct 24, 2022 at 1:03 PM dan via LibreQoS wrote: >> >> We're pretty far past that hardware. Minimum plans at most sites are no= w 100x20 and we're moving rapidly to 200x200-1Gx1G plans. The implication = being that we need 4x4 AX wifi radios and that's real tough in an openwrt c= ompatible hardware right now, not a ton of options. >> >> Flat out though, wisps in general simply do not have the option of contr= olling every client side router or even sending out new/compatible routers = for each customer to pull this off. It's a non-starter for us. It may b= e better but it's just not an option that can be pursued. ingress and egre= ss edge shaping is what we need. >> >> On Mon, Oct 24, 2022 at 11:57 AM Dave Taht wrote: >>> >>> On Mon, Oct 24, 2022 at 10:46 AM dan wrote: >>> > >>> > Some sort of % drop is better than count from an operator's perspecti= ve. I'm actually happy with 'scaling issues' of % on small vs large plan b= ecause those small plans are the one's we probably want to push for upgrade= s if they're seeing a lot of drops. >>> > >>> > That said, a 'score' method works really well from a high level. And= that can be proportionally adjusted so a slightly higher % in a small plan= gives the same score as a larger plan would with fewer drops. >>> > >>> > >>> > Also, on the SNMP statement above, we had the same issue with preseem= being hyper aggressive to pull stats. Some mikrotik versions have some sn= mp flaws and preseem can peg the CPU. Hasn't happened in a while but that = was an issue. >>> >>> I gave up on mikrotik long before they started working on routeros7, >>> and just reflashed everything to openwrt. >>> >>> I have a nice little custom build for it assembled for the ubnt agws, >>> and the mikrotik hap ac lite. Anyone using those? >>> >>> I would hope they don't have the snmp problem you describe, and if >>> they did I could get it fixed in days. >>> >>> Vastly improved the wifi in particular, see before/after here: >>> https://forum.openwrt.org/t/aql-and-the-ath10k-is-lovely/59002/ >>> >>> (After about 40Mbits, the bloat starts moving to the wifi) >>> >>> Most of my work in the past 9 months has been on improving the >>> mediatek mt76 derived products. I left off here: >>> https://forum.openwrt.org/t/aql-and-the-ath10k-is-lovely/59002/908 >>> >>> A lot of this stuff is shipping commercially in the openwifi project, >>> if you haven't been paying attention to that. >>> >>> > On Mon, Oct 24, 2022 at 11:42 AM Dave Taht wrot= e: >>> >> >>> >> On Mon, Oct 24, 2022 at 10:35 AM Robert Chac=C3=B3n >>> >> wrote: >>> >> > >>> >> > > Presently (in v1.3.), robert (in grafina?) is summing drops + ma= rks. >>> >> > which is a good idea, however perhaps drops, marks and drops+marks >>> >> > would be better. >>> >> > >>> >> > Yup, in InfluxDB for the stats by Tin (diffserv4) we display true = drops as (trueDrops =3D ecn_mark + drops - ack_drops). But we don't do it b= y sub quite yet. >>> >> > And ok, drops, marks and drops+marks can be shown instead - by sub= scriber or by Node (AP/Site). >>> >> > Dave - is it ok if we graph this data as a percentage of packets s= ent? So for drops, drop / sent_packets ? I just imagine that would make co= mparing different subscribers' connections easier. >>> >> >>> >> Drops and marks are proportional to the plan. You get a heck of a lo= t >>> >> more drops at 5mbits than 50, and the relationship >>> >> is not linear, closer to quadratic. >>> >> >>> >> So comparing customers at their given tiers makes more sense than >>> >> across the board. >>> >> >>> >> As for a percentage, you should see a clear difference in percentage= s >>> >> across tiers. >>> >> >>> >> I know I'm not quite answering your question well, I'm a big believe= r >>> >> in just getting lots of data, graphing it every which way, >>> >> and seeing what's useful. >>> >> >>> >> > Question for all - is having drops / marks/ drops+marks by subscri= ber/circuit worth the extra modest InfluxDB CPU use? Or should we just show= those stats by AP/Node? >>> >> > If preseem currently offers such stats at a subscriber level we ca= n do the same, just wanna be sure I'm not overtaxing CPU. >>> >> > >>> >> > On Mon, Oct 24, 2022 at 11:14 AM Dave Taht via LibreQoS wrote: >>> >> >> >>> >> >> Some of the context of my request for libreqos on vs off is from = here, >>> >> >> presentation in the wee hours, wed morning. >>> >> >> >>> >> >> https://www.linkedin.com/feed/update/urn:li:activity:698928194586= 8283904/ >>> >> >> >>> >> >> On Mon, Oct 24, 2022 at 9:48 AM dan via LibreQoS >>> >> >> wrote: >>> >> >> > >>> >> >> > we also have a bunch of 25/5. It would be nice to get some mor= e detailed numbers from ookla tests. >>> >> >> >>> >> >> +1! We've argued methods with many a provider, and ookla and samk= nows >>> >> >> are new to this game. >>> >> >> >>> >> >> As one example I don't know if they use decimal or binary mbits! >>> >> >> >>> >> >> ... >>> >> >> >>> >> >> ack-filtering in the libreqos case can and will drop acks in both >>> >> >> directions, due to the structure of how htb >>> >> >> works in this scenario, at any bandwidth ratio. I *think* it does= very >>> >> >> little harm but I would use the ack-filter >>> >> >> not the aggressive ack filter out of caution. >>> >> >> >>> >> >> Secondly if you are gathering drop statistics from the higher lev= el >>> >> >> htb categories, total drops will go WAY up, >>> >> >> and it is best to use the cake drop, ack_drop outputs statistics = directly. >>> >> >> >>> >> >> ack_drops have nothing to do with congestion control and should b= e >>> >> >> ignored in most cases. >>> >> >> >>> >> >> Also I keep hoping more will pull out ecn marks separately. Publi= c use >>> >> >> is way up by e2e transports, and I worry that a high ecn ratio is= a >>> >> >> e2e network error, which might show up as a big backlog. >>> >> >> >>> >> >> Presently (in v1.3.), robert (in grafina?) is summing drops + mar= ks. >>> >> >> which is a good idea, however perhaps drops, marks and drops+mark= s >>> >> >> would be better. >>> >> >> >>> >> >> An inverse log scale is needed to plot marks, drops, and ack_drop= s. >>> >> >> >>> >> >> I'll always be of the opinion that it's best to run cake on egres= s at >>> >> >> the cpe, first, doing all this work before it hits the wire. >>> >> >> Are any of you doing that? Openwrt/dd-wrt/firewalla/mikrotik 7.2? >>> >> >> >>> >> >> >We also have a lot of Eero units out there, so watching for the = eero speed test would also be great. >>> >> >> >>> >> >> eero 5s have cake (and fq_codel on the wifi), eero 6s have a some= what >>> >> >> buggy fq_codel offload: >>> >> >> >>> >> >> https://www.reddit.com/r/eero/comments/u7xm83/gen_2_sqm_vs_gen_3_= sqm_stick_with_gen_2_if_you/ >>> >> >> > On Mon, Oct 24, 2022 at 10:31 AM Herbert Wolverson via LibreQoS= wrote: >>> >> >> >> >>> >> >> >> > sniff DNS for those speed test lookups and trigger a packet = capture for x duration? >>> >> >> >> >>> >> >> >> That's a good idea (I think there's a service somewhere for fi= nding speedtest >>> >> >> >> servers?). We're discussing testing the ack-filter feature rig= ht now, and something >>> >> >> >> similar would work well for that. (We have a bunch of 25/5 and= similarly asymmetric >>> >> >> >> connections, in theory it'll help...) >>> >> >> >> >>> >> >> >> On Mon, Oct 24, 2022 at 10:31 AM dan via LibreQoS wrote: >>> >> >> >>> >>> >> >> >>> sniff DNS for those speed test lookups and trigger a packet c= apture for x duration? >>> >> >> >>> >>> >> >> >>> On Mon, Oct 24, 2022 at 9:27 AM Dave Taht wrote: >>> >> >> >>>> >>> >> >> >>>> On Mon, Oct 24, 2022 at 8:20 AM dan wr= ote: >>> >> >> >>>> > >>> >> >> >>>> > Dave, I think that 'bouncing' is more CPU time in the brow= ser. I don't think it's indicative over what's actually happening. >>> >> >> >>>> >>> >> >> >>>> A simultaneous packet capture and wireshark plot of the RTTs= would >>> >> >> >>>> also be helpful. >>> >> >> >>>> >>> >> >> >>>> > >>> >> >> >>>> > On Mon, Oct 24, 2022 at 9:09 AM Dave Taht via LibreQoS wrote: >>> >> >> >>>> >> >>> >> >> >>>> >> On Mon, Oct 24, 2022 at 8:02 AM Robert Chac=C3=B3n via Li= breQoS >>> >> >> >>>> >> wrote: >>> >> >> >>>> >> > >>> >> >> >>>> >> > I can run an Ookla test in the middle of the night with= LibreQoS off for a bit. >>> >> >> >>>> >> >>> >> >> >>>> >> Groovy. >>> >> >> >>>> >> >>> >> >> >>>> >> > From what I recall - without Libre we see download bloa= t of 300ms or so. With Libre, it's gone. On waveform we see 0ms added bloat= each direction for most clients. >>> >> >> >>>> >> >>> >> >> >>>> >> The numbers report by speedtest bounce around a lot, and = they tend to >>> >> >> >>>> >> pick a lower number as a final result than I'd >>> >> >> >>>> >> like. If there's a way to capture a movie of it... >>> >> >> >>>> >> > >>> >> >> >>>> >> > And I like the "monitor only" mode idea. >>> >> >> >>>> >> > Perhaps we could create the HTB tree, but just not atta= ch the CAKE qdisc? >>> >> >> >>>> >> > Technically even HTB could reduce latency by itself on = overloaded APs , so it wouldn't be true passive monitoring, but it would al= low us to compare the before/after of CAKE while still having qdiscs we can= point cpumap-pping to. >>> >> >> >>>> >> > >>> >> >> >>>> >> > Alternatively, maybe we could use eBPF PPing just for t= his passive monitoring period? >>> >> >> >>>> >> > >>> >> >> >>>> >> > >>> >> >> >>>> >> > On Mon, Oct 24, 2022 at 8:02 AM Herbert Wolverson via L= ibreQoS wrote: >>> >> >> >>>> >> >> >>> >> >> >>>> >> >> Are you looking for a shaped result? That'll show "wha= tever bandwidth >>> >> >> >>>> >> >> is left on the sector" without Libre, and around the c= ustomer's paid-for >>> >> >> >>>> >> >> target with Libre (hopefully!). >>> >> >> >>>> >> >> >>> >> >> >>>> >> >> One thing Preseem does well is that their onboarding p= rocess includes >>> >> >> >>>> >> >> "leave it in monitoring-only mode" for a week, gatheri= ng data before you >>> >> >> >>>> >> >> pull the switch (although most ISPs I've talked to jus= t pull the switch...) >>> >> >> >>>> >> >> It's quite enlightening to see a before/after, even wi= th only FQ_CODEL >>> >> >> >>>> >> >> as the shaper. >>> >> >> >>>> >> >> >>> >> >> >>>> >> >> We could probably do something similar with a "monitor= only" mode that >>> >> >> >>>> >> >> doesn't enable any queues (is there a TC equivalent to= "dummy" on >>> >> >> >>>> >> >> FreeBSD?). >>> >> >> >>>> >> >> >>> >> >> >>>> >> >> On Mon, Oct 24, 2022 at 8:52 AM Dave Taht via LibreQoS= wrote: >>> >> >> >>>> >> >>> >>> >> >> >>>> >> >>> I am curious if anyone here has speedtest results wit= h libreqos on and >>> >> >> >>>> >> >>> off, and can send a screen shot? >>> >> >> >>>> >> >>> >>> >> >> >>>> >> >>> -- >>> >> >> >>>> >> >>> This song goes out to all the folk that thought Stadi= a would work: >>> >> >> >>>> >> >>> https://www.linkedin.com/posts/dtaht_the-mushroom-son= g-activity-6981366665607352320-FXtz >>> >> >> >>>> >> >>> Dave T=C3=A4ht CEO, TekLibre, LLC >>> >> >> >>>> >> >>> _______________________________________________ >>> >> >> >>>> >> >>> LibreQoS mailing list >>> >> >> >>>> >> >>> LibreQoS@lists.bufferbloat.net >>> >> >> >>>> >> >>> https://lists.bufferbloat.net/listinfo/libreqos >>> >> >> >>>> >> >> >>> >> >> >>>> >> >> _______________________________________________ >>> >> >> >>>> >> >> LibreQoS mailing list >>> >> >> >>>> >> >> LibreQoS@lists.bufferbloat.net >>> >> >> >>>> >> >> https://lists.bufferbloat.net/listinfo/libreqos >>> >> >> >>>> >> > >>> >> >> >>>> >> > >>> >> >> >>>> >> > >>> >> >> >>>> >> > -- >>> >> >> >>>> >> > Robert Chac=C3=B3n >>> >> >> >>>> >> > CEO | JackRabbit Wireless LLC >>> >> >> >>>> >> > _______________________________________________ >>> >> >> >>>> >> > LibreQoS mailing list >>> >> >> >>>> >> > LibreQoS@lists.bufferbloat.net >>> >> >> >>>> >> > https://lists.bufferbloat.net/listinfo/libreqos >>> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> -- >>> >> >> >>>> >> This song goes out to all the folk that thought Stadia wo= uld work: >>> >> >> >>>> >> https://www.linkedin.com/posts/dtaht_the-mushroom-song-ac= tivity-6981366665607352320-FXtz >>> >> >> >>>> >> Dave T=C3=A4ht CEO, TekLibre, LLC >>> >> >> >>>> >> _______________________________________________ >>> >> >> >>>> >> LibreQoS mailing list >>> >> >> >>>> >> LibreQoS@lists.bufferbloat.net >>> >> >> >>>> >> https://lists.bufferbloat.net/listinfo/libreqos >>> >> >> >>>> >>> >> >> >>>> >>> >> >> >>>> >>> >> >> >>>> -- >>> >> >> >>>> This song goes out to all the folk that thought Stadia would= work: >>> >> >> >>>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activ= ity-6981366665607352320-FXtz >>> >> >> >>>> Dave T=C3=A4ht CEO, TekLibre, LLC >>> >> >> >>> >>> >> >> >>> _______________________________________________ >>> >> >> >>> LibreQoS mailing list >>> >> >> >>> LibreQoS@lists.bufferbloat.net >>> >> >> >>> https://lists.bufferbloat.net/listinfo/libreqos >>> >> >> >> >>> >> >> >> _______________________________________________ >>> >> >> >> LibreQoS mailing list >>> >> >> >> LibreQoS@lists.bufferbloat.net >>> >> >> >> https://lists.bufferbloat.net/listinfo/libreqos >>> >> >> > >>> >> >> > _______________________________________________ >>> >> >> > LibreQoS mailing list >>> >> >> > LibreQoS@lists.bufferbloat.net >>> >> >> > https://lists.bufferbloat.net/listinfo/libreqos >>> >> >> >>> >> >> >>> >> >> >>> >> >> -- >>> >> >> This song goes out to all the folk that thought Stadia would work= : >>> >> >> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6= 981366665607352320-FXtz >>> >> >> Dave T=C3=A4ht CEO, TekLibre, LLC >>> >> >> _______________________________________________ >>> >> >> LibreQoS mailing list >>> >> >> LibreQoS@lists.bufferbloat.net >>> >> >> https://lists.bufferbloat.net/listinfo/libreqos >>> >> > >>> >> > >>> >> > >>> >> > -- >>> >> > Robert Chac=C3=B3n >>> >> > CEO | JackRabbit Wireless LLC >>> >> >>> >> >>> >> >>> >> -- >>> >> This song goes out to all the folk that thought Stadia would work: >>> >> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981= 366665607352320-FXtz >>> >> Dave T=C3=A4ht CEO, TekLibre, LLC >>> >>> >>> >>> -- >>> This song goes out to all the folk that thought Stadia would work: >>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366= 665607352320-FXtz >>> Dave T=C3=A4ht CEO, TekLibre, LLC >> >> _______________________________________________ >> LibreQoS mailing list >> LibreQoS@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/libreqos > > _______________________________________________ > LibreQoS mailing list > LibreQoS@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/libreqos --=20 This song goes out to all the folk that thought Stadia would work: https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-69813666656= 07352320-FXtz Dave T=C3=A4ht CEO, TekLibre, LLC