From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pg1-x52d.google.com (mail-pg1-x52d.google.com [IPv6:2607:f8b0:4864:20::52d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id C897F3B29E for ; Mon, 24 Oct 2022 13:41:30 -0400 (EDT) Received: by mail-pg1-x52d.google.com with SMTP id f193so9306462pgc.0 for ; Mon, 24 Oct 2022 10:41:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=y2s6FdlTn27lsgBnLc7G7QzXHxlMUazXlF+f4KmI7IY=; b=ZlP0gYYby24MBQxOTRQVBitrBcXFaEwBlZtvwEeEIReBhQ3Tm8ga7Utl+G6Vi9bQOr wxKXKHye6nml934e9TCCWw17fQfeVNuxFupBVt/n5bTend/N+NJbi9w0u7RrUYLGJh4J 1Zyz6rv6SeYVilydbJNB/meOF1vyP+ej+1xH6ewkMiIdKWQvvKW97tY9hXgAnA7DrPTk UE3HEEKZpe/oRRtUpOJgfLtejgxIq6tfcbk5eyLSwx8SmXAFO7cWZHmR4IDQD89abwiG quMPom60UygCtvmy5zu0p0hO2qrFKubfirC9mF0xvbnPuPyK79Tx/MvyUcsSUA/CQpuX RRsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=y2s6FdlTn27lsgBnLc7G7QzXHxlMUazXlF+f4KmI7IY=; b=sPJ67aaqjeBMuc941HkcNzt7og49r5G/nvpHT/0ZLvrQpbysInBXu0nhljk0bW6Zzw ns9lS0CXOoLRWTdU3/IreJS8drGMt8l3o8UWOPH6bEU3tOyc/06DqNUzM6/RcHAbUABc FuE1H0YmVZGbIKB5NLp8Xgw3WKsxGPVMkxdUwRu4HJAsm7qMV2y0l8O+Iadgb6myAg0Z ENBunMIZroz7V9GkV47PEMz/RUhUh/5Hg9tOoR4BXVVYt1dMpfa8he+XR4OxY15W0Rf3 dY2gqjY7OF7YQvQ+eqz4vMLWYB8FdcmW/TqWJlro1w8/Fx7mRKxuc6ssfm3vN2EjbV2V 55rQ== X-Gm-Message-State: ACrzQf0e8UvK9GCZdKE/RwCto2JfY9h0WwclmkywHAmXOANI10O/wGbq etyXmOxxhYrNbXtkIUR97Qw5o3bTrJnerPY3cKYw9qDA X-Google-Smtp-Source: AMsMyM7OreEHXNQqBQ6yFq/UT+EsmLeZMlOBRHDVEmpp/AYijtqiC0tXgw5wo4gWFum+gJVRvJjEhrazAeEpq/MQgTA= X-Received: by 2002:a17:903:124e:b0:179:da2f:244e with SMTP id u14-20020a170903124e00b00179da2f244emr34693374plh.169.1666633278380; Mon, 24 Oct 2022 10:41:18 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Herbert Wolverson Date: Mon, 24 Oct 2022 12:41:07 -0500 Message-ID: Cc: libreqos@lists.bufferbloat.net Content-Type: multipart/alternative; boundary="00000000000087600e05ebcb4c00" Subject: Re: [LibreQoS] ookla speedtest results? X-BeenThere: libreqos@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Many ISPs need the kinds of quality shaping cake can do List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2022 17:41:31 -0000 --00000000000087600e05ebcb4c00 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Preseem is a little funky in actually showing any queue statistics, because it also pounds your devices with SNMP requests and gathers things like TCP drops/retransmits from there (and uses it to try and figure out their hierarchy). It was actually one of the major problems we had with it - we have a few EdgeRouters around, and they have a bug that they utilize ram disk space on a successful SNMPv2 query. They don't leak a lot of RAM, just a few bytes per request - but Preseem pounds the living daylights of out them, and we had to implement a "reboot all the EdgeRouters within 30 days" policy to avoid the things silently locking up. Ugh. I'd love to be able to provide cake at the end-point, but it's not very practical. Our customers use a wide range of devices (and CPEs, even a few old M2 devices still up!), and a "use the router we provide" policy isn't going to fly for quite a few of them - they just really don't like that. I've had UI issues with displaying "drops". Calling it that terrifies our support people. I'd love to have another name that conveys the meaning! On Mon, Oct 24, 2022 at 12:35 PM Robert Chac=C3=B3n via LibreQoS < libreqos@lists.bufferbloat.net> wrote: > > Presently (in v1.3.), robert (in grafina?) is summing drops + marks. > which is a good idea, however perhaps drops, marks and drops+marks > would be better. > > Yup, in InfluxDB for the stats by Tin (diffserv4) we display true drops a= s > (trueDrops =3D ecn_mark + drops - ack_drops). But we don't do it by sub q= uite > yet. > And ok, drops, marks and drops+marks can be shown instead - by subscriber > or by Node (AP/Site). > Dave - is it ok if we graph this data as a percentage of packets sent? So > for drops, drop / sent_packets ? I just imagine that would make comparin= g > different subscribers' connections easier. > > Question for all - is having drops / marks/ drops+marks by > subscriber/circuit worth the extra modest InfluxDB CPU use? Or should we > just show those stats by AP/Node? > If preseem currently offers such stats at a subscriber level we can do th= e > same, just wanna be sure I'm not overtaxing CPU. > > On Mon, Oct 24, 2022 at 11:14 AM Dave Taht via LibreQoS < > libreqos@lists.bufferbloat.net> wrote: > >> Some of the context of my request for libreqos on vs off is from here, >> presentation in the wee hours, wed morning. >> >> https://www.linkedin.com/feed/update/urn:li:activity:6989281945868283904= / >> >> On Mon, Oct 24, 2022 at 9:48 AM dan via LibreQoS >> wrote: >> > >> > we also have a bunch of 25/5. It would be nice to get some more >> detailed numbers from ookla tests. >> >> +1! We've argued methods with many a provider, and ookla and samknows >> are new to this game. >> >> As one example I don't know if they use decimal or binary mbits! >> >> ... >> >> ack-filtering in the libreqos case can and will drop acks in both >> directions, due to the structure of how htb >> works in this scenario, at any bandwidth ratio. I *think* it does very >> little harm but I would use the ack-filter >> not the aggressive ack filter out of caution. >> >> Secondly if you are gathering drop statistics from the higher level >> htb categories, total drops will go WAY up, >> and it is best to use the cake drop, ack_drop outputs statistics directl= y. >> >> ack_drops have nothing to do with congestion control and should be >> ignored in most cases. >> >> Also I keep hoping more will pull out ecn marks separately. Public use >> is way up by e2e transports, and I worry that a high ecn ratio is a >> e2e network error, which might show up as a big backlog. >> >> Presently (in v1.3.), robert (in grafina?) is summing drops + marks. >> which is a good idea, however perhaps drops, marks and drops+marks >> would be better. >> >> An inverse log scale is needed to plot marks, drops, and ack_drops. >> >> I'll always be of the opinion that it's best to run cake on egress at >> the cpe, first, doing all this work before it hits the wire. >> Are any of you doing that? Openwrt/dd-wrt/firewalla/mikrotik 7.2? >> >> >We also have a lot of Eero units out there, so watching for the eero >> speed test would also be great. >> >> eero 5s have cake (and fq_codel on the wifi), eero 6s have a somewhat >> buggy fq_codel offload: >> >> >> https://www.reddit.com/r/eero/comments/u7xm83/gen_2_sqm_vs_gen_3_sqm_sti= ck_with_gen_2_if_you/ >> > On Mon, Oct 24, 2022 at 10:31 AM Herbert Wolverson via LibreQoS < >> libreqos@lists.bufferbloat.net> wrote: >> >> >> >> > sniff DNS for those speed test lookups and trigger a packet capture >> for x duration? >> >> >> >> That's a good idea (I think there's a service somewhere for finding >> speedtest >> >> servers?). We're discussing testing the ack-filter feature right now, >> and something >> >> similar would work well for that. (We have a bunch of 25/5 and >> similarly asymmetric >> >> connections, in theory it'll help...) >> >> >> >> On Mon, Oct 24, 2022 at 10:31 AM dan via LibreQoS < >> libreqos@lists.bufferbloat.net> wrote: >> >>> >> >>> sniff DNS for those speed test lookups and trigger a packet capture >> for x duration? >> >>> >> >>> On Mon, Oct 24, 2022 at 9:27 AM Dave Taht >> wrote: >> >>>> >> >>>> On Mon, Oct 24, 2022 at 8:20 AM dan wrote: >> >>>> > >> >>>> > Dave, I think that 'bouncing' is more CPU time in the browser. I >> don't think it's indicative over what's actually happening. >> >>>> >> >>>> A simultaneous packet capture and wireshark plot of the RTTs would >> >>>> also be helpful. >> >>>> >> >>>> > >> >>>> > On Mon, Oct 24, 2022 at 9:09 AM Dave Taht via LibreQoS < >> libreqos@lists.bufferbloat.net> wrote: >> >>>> >> >> >>>> >> On Mon, Oct 24, 2022 at 8:02 AM Robert Chac=C3=B3n via LibreQoS >> >>>> >> wrote: >> >>>> >> > >> >>>> >> > I can run an Ookla test in the middle of the night with >> LibreQoS off for a bit. >> >>>> >> >> >>>> >> Groovy. >> >>>> >> >> >>>> >> > From what I recall - without Libre we see download bloat of >> 300ms or so. With Libre, it's gone. On waveform we see 0ms added bloat e= ach >> direction for most clients. >> >>>> >> >> >>>> >> The numbers report by speedtest bounce around a lot, and they >> tend to >> >>>> >> pick a lower number as a final result than I'd >> >>>> >> like. If there's a way to capture a movie of it... >> >>>> >> > >> >>>> >> > And I like the "monitor only" mode idea. >> >>>> >> > Perhaps we could create the HTB tree, but just not attach the >> CAKE qdisc? >> >>>> >> > Technically even HTB could reduce latency by itself on >> overloaded APs , so it wouldn't be true passive monitoring, but it would >> allow us to compare the before/after of CAKE while still having qdiscs w= e >> can point cpumap-pping to. >> >>>> >> > >> >>>> >> > Alternatively, maybe we could use eBPF PPing just for this >> passive monitoring period? >> >>>> >> > >> >>>> >> > >> >>>> >> > On Mon, Oct 24, 2022 at 8:02 AM Herbert Wolverson via LibreQoS= < >> libreqos@lists.bufferbloat.net> wrote: >> >>>> >> >> >> >>>> >> >> Are you looking for a shaped result? That'll show "whatever >> bandwidth >> >>>> >> >> is left on the sector" without Libre, and around the >> customer's paid-for >> >>>> >> >> target with Libre (hopefully!). >> >>>> >> >> >> >>>> >> >> One thing Preseem does well is that their onboarding process >> includes >> >>>> >> >> "leave it in monitoring-only mode" for a week, gathering data >> before you >> >>>> >> >> pull the switch (although most ISPs I've talked to just pull >> the switch...) >> >>>> >> >> It's quite enlightening to see a before/after, even with only >> FQ_CODEL >> >>>> >> >> as the shaper. >> >>>> >> >> >> >>>> >> >> We could probably do something similar with a "monitor only" >> mode that >> >>>> >> >> doesn't enable any queues (is there a TC equivalent to "dummy= " >> on >> >>>> >> >> FreeBSD?). >> >>>> >> >> >> >>>> >> >> On Mon, Oct 24, 2022 at 8:52 AM Dave Taht via LibreQoS < >> libreqos@lists.bufferbloat.net> wrote: >> >>>> >> >>> >> >>>> >> >>> I am curious if anyone here has speedtest results with >> libreqos on and >> >>>> >> >>> off, and can send a screen shot? >> >>>> >> >>> >> >>>> >> >>> -- >> >>>> >> >>> This song goes out to all the folk that thought Stadia would >> work: >> >>>> >> >>> >> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-69813666= 65607352320-FXtz >> >>>> >> >>> Dave T=C3=A4ht CEO, TekLibre, LLC >> >>>> >> >>> _______________________________________________ >> >>>> >> >>> LibreQoS mailing list >> >>>> >> >>> LibreQoS@lists.bufferbloat.net >> >>>> >> >>> https://lists.bufferbloat.net/listinfo/libreqos >> >>>> >> >> >> >>>> >> >> _______________________________________________ >> >>>> >> >> LibreQoS mailing list >> >>>> >> >> LibreQoS@lists.bufferbloat.net >> >>>> >> >> https://lists.bufferbloat.net/listinfo/libreqos >> >>>> >> > >> >>>> >> > >> >>>> >> > >> >>>> >> > -- >> >>>> >> > Robert Chac=C3=B3n >> >>>> >> > CEO | JackRabbit Wireless LLC >> >>>> >> > _______________________________________________ >> >>>> >> > LibreQoS mailing list >> >>>> >> > LibreQoS@lists.bufferbloat.net >> >>>> >> > https://lists.bufferbloat.net/listinfo/libreqos >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> -- >> >>>> >> This song goes out to all the folk that thought Stadia would wor= k: >> >>>> >> >> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-69813666= 65607352320-FXtz >> >>>> >> Dave T=C3=A4ht CEO, TekLibre, LLC >> >>>> >> _______________________________________________ >> >>>> >> LibreQoS mailing list >> >>>> >> LibreQoS@lists.bufferbloat.net >> >>>> >> https://lists.bufferbloat.net/listinfo/libreqos >> >>>> >> >>>> >> >>>> >> >>>> -- >> >>>> This song goes out to all the folk that thought Stadia would work: >> >>>> >> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-69813666= 65607352320-FXtz >> >>>> Dave T=C3=A4ht CEO, TekLibre, LLC >> >>> >> >>> _______________________________________________ >> >>> LibreQoS mailing list >> >>> LibreQoS@lists.bufferbloat.net >> >>> https://lists.bufferbloat.net/listinfo/libreqos >> >> >> >> _______________________________________________ >> >> LibreQoS mailing list >> >> LibreQoS@lists.bufferbloat.net >> >> https://lists.bufferbloat.net/listinfo/libreqos >> > >> > _______________________________________________ >> > LibreQoS mailing list >> > LibreQoS@lists.bufferbloat.net >> > https://lists.bufferbloat.net/listinfo/libreqos >> >> >> >> -- >> This song goes out to all the folk that thought Stadia would work: >> >> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-69813666= 65607352320-FXtz >> Dave T=C3=A4ht CEO, TekLibre, LLC >> _______________________________________________ >> LibreQoS mailing list >> LibreQoS@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/libreqos >> > > > -- > Robert Chac=C3=B3n > CEO | JackRabbit Wireless LLC > _______________________________________________ > LibreQoS mailing list > LibreQoS@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/libreqos > --00000000000087600e05ebcb4c00 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Preseem is a little funky in actually showing any que= ue statistics, because it also pounds
your devices with SNMP requ= ests and gathers things like TCP drops/retransmits from
there (an= d uses it to try and figure out their hierarchy).

=
It was actually one of the major problems we had with it - we have a f= ew EdgeRouters
around, and they have a bug that they utilize= ram disk space on a successful SNMPv2 query.
They don't leak= a lot of RAM, just a few bytes per request - but Preseem pounds the
<= div>living daylights of out them, and we had to implement a "reboot al= l the EdgeRouters within
30 days" policy to avoid the things= silently locking up. Ugh.

I'd love to be able= to provide cake at the end-point, but it's not very practical. Our
customers use a wide range of devices (and CPEs, even a few old M2 d= evices
still up!), and a "use the router we provide" po= licy isn't going to fly for quite
a few of them - they just r= eally don't like that.

I've had UI issues = with displaying "drops". Calling it that terrifies our support
people. I'd love to have another name that conveys the meaning!=

On Mon, Oct 24, 2022 at 12:35 PM Robert Chac=C3=B3n via LibreQoS = <libreqos@lists.buffer= bloat.net> wrote:
> Presently (in v1.3.), rob= ert (in grafina?) is summing drops + marks.
which is a good idea, however perhaps drops, marks and drops+marks
would be better.

Yup, in InfluxDB for the stats by= Tin (diffserv4) we display true drops as (trueDrops =3D ecn_mark + drops -= ack_drops). But we don't do it by sub quite yet.
And ok,= drops, marks and drops+marks can be shown instead - by subscriber or by No= de (AP/Site).
Dave - is it ok if we graph this data as a percenta= ge of packets sent? So for drops, drop / sent_packets ? I just imagine that= =C2=A0 would make comparing different subscribers' connections easier.<= br>

Question for all - is having drops / marks/ dr= ops+marks by subscriber/circuit worth the extra modest InfluxDB CPU use? Or= should we just show those stats by AP/Node?
If preseem curre= ntly offers such stats at a subscriber level we can do the same, just wanna= be sure I'm not overtaxing CPU.

On Mon, Oct 24, 2022 at= 11:14 AM Dave Taht via LibreQoS <libreqos@lists.bufferbloat.net> wrote:=
Some of the con= text of my request for libreqos on vs off is from here,
presentation in the wee hours, wed morning.

https://www.linkedin.com/fe= ed/update/urn:li:activity:6989281945868283904/

On Mon, Oct 24, 2022 at 9:48 AM dan via LibreQoS
<lib= reqos@lists.bufferbloat.net> wrote:
>
> we also have a bunch of 25/5.=C2=A0 It would be nice to get some more = detailed numbers from ookla tests.

+1! We've argued methods with many a provider, and ookla and samknows are new to this game.

As one example I don't know if they use decimal or binary mbits!

...

ack-filtering in the libreqos case can and will drop acks in both
directions, due to the structure of how htb
works in this scenario, at any bandwidth ratio. I *think* it does very
little harm but I would use the ack-filter
not the aggressive ack filter out of caution.

Secondly if you are gathering drop statistics from the higher level
htb categories, total drops will go WAY up,
and it is best to use the cake drop, ack_drop outputs statistics directly.<= br>
ack_drops have nothing to do with congestion control and should be
ignored in most cases.

Also I keep hoping more will pull out ecn marks separately. Public use
is way up by e2e transports, and I worry that a high ecn ratio is a
e2e network error, which might show up as a big backlog.

Presently (in v1.3.), robert (in grafina?) is summing drops + marks.
which is a good idea, however perhaps drops, marks and drops+marks
would be better.

An inverse log scale is needed to plot marks, drops, and ack_drops.

I'll always be of the opinion that it's best to run cake on egress = at
the cpe, first, doing all this work before it hits the wire.
Are any of you doing that? Openwrt/dd-wrt/firewalla/mikrotik 7.2?

>We also have a lot of Eero units out there, so watching for the eero sp= eed test would also be great.

eero 5s have cake (and fq_codel on the wifi), eero 6s have a somewhat
buggy fq_codel offload:

https:/= /www.reddit.com/r/eero/comments/u7xm83/gen_2_sqm_vs_gen_3_sqm_stick_with_ge= n_2_if_you/
> On Mon, Oct 24, 2022 at 10:31 AM Herbert Wolverson via LibreQoS <libreqos@= lists.bufferbloat.net> wrote:
>>
>> > sniff DNS for those speed test lookups and trigger a packet c= apture for x duration?
>>
>> That's a good idea (I think there's a service somewhere fo= r finding speedtest
>> servers?). We're discussing testing the ack-filter feature rig= ht now, and something
>> similar would work well for that. (We have a bunch of 25/5 and sim= ilarly asymmetric
>> connections, in theory it'll help...)
>>
>> On Mon, Oct 24, 2022 at 10:31 AM dan via LibreQoS <libreqos@lists.buff= erbloat.net> wrote:
>>>
>>> sniff DNS for those speed test lookups and trigger a packet ca= pture for x duration?
>>>
>>> On Mon, Oct 24, 2022 at 9:27 AM Dave Taht <dave.taht@gmail.com> wrote:=
>>>>
>>>> On Mon, Oct 24, 2022 at 8:20 AM dan <dandenson@gmail.com> wrote: >>>> >
>>>> > Dave, I think that 'bouncing' is more CPU tim= e in the browser.=C2=A0 I don't think it's indicative over what'= ;s actually happening.
>>>>
>>>> A simultaneous packet capture and wireshark plot of the RT= Ts would
>>>> also be helpful.
>>>>
>>>> >
>>>> > On Mon, Oct 24, 2022 at 9:09 AM Dave Taht via LibreQo= S <l= ibreqos@lists.bufferbloat.net> wrote:
>>>> >>
>>>> >> On Mon, Oct 24, 2022 at 8:02 AM Robert Chac=C3=B3= n via LibreQoS
>>>> >> <libreqos@lists.bufferbloat.net> wrote:
>>>> >> >
>>>> >> > I can run an Ookla test in the middle of the= night with LibreQoS off for a bit.
>>>> >>
>>>> >> Groovy.
>>>> >>
>>>> >> > From what I recall - without Libre we see do= wnload bloat of 300ms or so. With Libre, it's gone. On waveform we see = 0ms added bloat each direction for most clients.
>>>> >>
>>>> >> The numbers report by speedtest bounce around a l= ot, and they tend to
>>>> >> pick a lower number as a final result than I'= d
>>>> >> like. If there's a way to capture a movie of = it...
>>>> >> >
>>>> >> > And I like the "monitor only" mode= idea.
>>>> >> > Perhaps we could create the HTB tree, but ju= st not attach the CAKE qdisc?
>>>> >> > Technically even HTB could reduce latency by= itself on overloaded APs , so it wouldn't be true passive monitoring, = but it would allow us to compare the before/after of CAKE while still havin= g qdiscs we can point cpumap-pping to.
>>>> >> >
>>>> >> > Alternatively, maybe we could use eBPF PPing= just for this passive monitoring period?
>>>> >> >
>>>> >> >
>>>> >> > On Mon, Oct 24, 2022 at 8:02 AM Herbert Wolv= erson via LibreQoS <libreqos@lists.bufferbloat.net> wrote:
>>>> >> >>
>>>> >> >> Are you looking for a shaped result? Tha= t'll show "whatever bandwidth
>>>> >> >> is left on the sector" without Libr= e, and around the customer's paid-for
>>>> >> >> target with Libre (hopefully!).
>>>> >> >>
>>>> >> >> One thing Preseem does well is that thei= r onboarding process includes
>>>> >> >> "leave it in monitoring-only mode&q= uot; for a week, gathering data before you
>>>> >> >> pull the switch (although most ISPs I= 9;ve talked to just pull the switch...)
>>>> >> >> It's quite enlightening to see a bef= ore/after, even with only FQ_CODEL
>>>> >> >> as the shaper.
>>>> >> >>
>>>> >> >> We could probably do something similar w= ith a "monitor only" mode that
>>>> >> >> doesn't enable any queues (is there = a TC equivalent to "dummy" on
>>>> >> >> FreeBSD?).
>>>> >> >>
>>>> >> >> On Mon, Oct 24, 2022 at 8:52 AM Dave Tah= t via LibreQoS <libreqos@lists.bufferbloat.net> wrote:
>>>> >> >>>
>>>> >> >>> I am curious if anyone here has spee= dtest results with libreqos on and
>>>> >> >>> off, and can send a screen shot?
>>>> >> >>>
>>>> >> >>> --
>>>> >> >>> This song goes out to all the folk t= hat thought Stadia would work:
>>>> >> >>> https://www.linkedin.com/posts/dtaht_the-mushroo= m-song-activity-6981366665607352320-FXtz
>>>> >> >>> Dave T=C3=A4ht CEO, TekLibre, LLC >>>> >> >>> ____________________________________= ___________
>>>> >> >>> LibreQoS mailing list
>>>> >> >>> LibreQoS@lists.bufferbloat.net
>>>> >> >>> https://lists.= bufferbloat.net/listinfo/libreqos
>>>> >> >>
>>>> >> >> ________________________________________= _______
>>>> >> >> LibreQoS mailing list
>>>> >> >> LibreQoS@lists.bufferbloat.net
>>>> >> >> https://lists.buff= erbloat.net/listinfo/libreqos
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > --
>>>> >> > Robert Chac=C3=B3n
>>>> >> > CEO | JackRabbit Wireless LLC
>>>> >> > ____________________________________________= ___
>>>> >> > LibreQoS mailing list
>>>> >> > LibreQoS@lists.bufferbloat.net
>>>> >> > https://lists.bufferbl= oat.net/listinfo/libreqos
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> This song goes out to all the folk that thought S= tadia would work:
>>>> >> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activi= ty-6981366665607352320-FXtz
>>>> >> Dave T=C3=A4ht CEO, TekLibre, LLC
>>>> >> _______________________________________________ >>>> >> LibreQoS mailing list
>>>> >> LibreQoS@lists.bufferbloat.net
>>>> >> https://lists.bufferbloat.n= et/listinfo/libreqos
>>>>
>>>>
>>>>
>>>> --
>>>> This song goes out to all the folk that thought Stadia wou= ld work:
>>>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-698136= 6665607352320-FXtz
>>>> Dave T=C3=A4ht CEO, TekLibre, LLC
>>>
>>> _______________________________________________
>>> LibreQoS mailing list
>>> LibreQoS@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/l= ibreqos
>>
>> _______________________________________________
>> LibreQoS mailing list
>> LibreQoS@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/libre= qos
>
> _______________________________________________
> LibreQoS mailing list
> Li= breQoS@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/libreqos<= /a>



--
This song goes out to all the folk that thought Stadia would work:
https://www.= linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXt= z
Dave T=C3=A4ht CEO, TekLibre, LLC
_______________________________________________
LibreQoS mailing list
LibreQo= S@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/libreqos


--
Robert Chac=C3=B3n
CEO | JackRabbit Wireless LLC
_______________________________________________
LibreQoS mailing list
LibreQo= S@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/libreqos
--00000000000087600e05ebcb4c00--