From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oi1-x233.google.com (mail-oi1-x233.google.com [IPv6:2607:f8b0:4864:20::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id C95543CB37 for ; Wed, 1 May 2024 17:12:30 -0400 (EDT) Received: by mail-oi1-x233.google.com with SMTP id 5614622812f47-3c74b643aebso4264849b6e.0 for ; Wed, 01 May 2024 14:12:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1714597950; x=1715202750; darn=lists.bufferbloat.net; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=hcKCIWvhfCd3a7AvYD3bm+s4ngrfS3XAm8pR5v0paBw=; b=erUKTHPCNXSBKx/NRGVmOy045t48yt5dQnoD+PNwD9LHTgBCczp2POML7o5JmpdsK3 26t2jJZBNrMxpPREWeU4VrWvR1gLd7oc36GxGmvOByE7Y8kIDOnhVtMYDDRcAGB8nRlR IimsDOLlSqbJrdMqFvTz5UXA+xHWdrH2sGWbgGGE4l9Ds6UC8JYOPnsRtRbEmaC8LEcB FMbENFyt6VcBM+pzmtCIPfzdz+kTJ4gPHzn/epKrfE5iVqDQWI6XtHrmA6kPpvj3Jp0S EaKdkWEoSWEnNeHrT2waEgPDsw2eOv5EGknx8jhzcTPeuTNQRuY0G+ktvXXAqfGW5Pz/ /QvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714597950; x=1715202750; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=hcKCIWvhfCd3a7AvYD3bm+s4ngrfS3XAm8pR5v0paBw=; b=SsB0/xHWIMCDyD9Sknhzx9DMhRa8jItrYWPSAXGRpFhwsH7FFI9wW030Im0WeZN2x5 iQr83x1Tu3ag9aDQVDCA+KjecAZ0GvYHacWWkmKQSFhZ2SbWgERFpiY1YRWFbTnuU/IB iL0FfXDbgbfwknJvTtVejWwP35z6itqdR015PbvbXSj4ReHNOIE42J0kNBD9vP54+L+3 ROcB4VdZELiubiAgs650SvDc0rCKmAkSzpHucpHzlF2rnb4P0jlgyPnk2v3dF94xtAim I63XC1/y9vMOYsPD2HnEOr7uW2QXfviqowMoTB5MRVlNKi04wSmfkKyi2ic5BF9G8vuJ OUbw== X-Forwarded-Encrypted: i=1; AJvYcCU2D7yt6p7QgM4mc+oqY76ikomaEbasNOz0yXk061Ezgc9rzwIbUm6RWKrI4giT2mzh0JCTAScLYS3Vx2DUU7BwM+0ygZqBmLcdLfD/QhM= X-Gm-Message-State: AOJu0Yw9yMsFEjuJE6Xp/gaft1M10vptoxKeYu7oenZBzbaNng8PXI21 QCIp0QWhZf+17OxlQUqR9JYCrMIgAaKgLsPkT7bLmjaEBj5KUk+Wjw01Ko1plbFE+tQp0jat1/Y aioDMfed+qJNEPytjzX1cfxcIVV8= X-Google-Smtp-Source: AGHT+IG04GskNCZdZrQ1RB4Y13SgCuIm4c1WGJqSenOzwhZfgsQ2tYy777RJpP9vRpmrajI6AbvK/OF6c4dnqVv1KoQ= X-Received: by 2002:a05:6808:1308:b0:3c5:eed2:e244 with SMTP id y8-20020a056808130800b003c5eed2e244mr190877oiv.30.1714597949765; Wed, 01 May 2024 14:12:29 -0700 (PDT) MIME-Version: 1.0 References: <438B1BC4-D465-497A-B6BA-700E1D411036@ieee.org> <79C02ABB-B2A6-4B4D-98F4-6540D3F96EBB@ieee.org> <7E918B58-382A-4793-A144-13A7075CA56C@connectivitycap.com> <13rq2389-9012-p95n-s494-q3pp070s497n@ynat.uz> <6qop2p3o-351p-788q-q1q2-86sosnq3rn21@ynat.uz> In-Reply-To: <6qop2p3o-351p-788q-q1q2-86sosnq3rn21@ynat.uz> From: Frantisek Borsik Date: Wed, 1 May 2024 23:11:53 +0200 Message-ID: To: David Lang Cc: Eugene Y Chang , Colin_Higbie , Dave Taht via Starlink Content-Type: multipart/alternative; boundary="000000000000ba837f06176af1a6" Subject: Re: [Starlink] =?utf-8?q?It=CA=BCs_the_Latency=2C_FCC?= X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 May 2024 21:12:30 -0000 --000000000000ba837f06176af1a6 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Eugene, this is one of the ISP examples of using OpenWrt, CAKE & FQ-CoDel to fix not only his network, but also to refurbish an old device - when the vendor didn't give a flying F: https://blog.nafiux.com/posts/cnpilot_r190w_openwrt_bufferbloat_fqcodel_cak= e/ Here is also the list of OpenWrt supported HW: https://openwrt.org/supported_devices If you/ISP want to go mainstream, MikroTik will be a good option. This is a great place to start (not only for your ISP): https://www.bufferbloat.net/projects/bloat/wiki/What_can_I_do_about_Bufferb= loat/ All the best, Frank Frantisek (Frank) Borsik https://www.linkedin.com/in/frantisekborsik Signal, Telegram, WhatsApp: +421919416714 iMessage, mobile: +420775230885 Skype: casioa5302ca frantisek.borsik@gmail.com On Wed, May 1, 2024 at 9:18=E2=80=AFPM David Lang via Starlink < starlink@lists.bufferbloat.net> wrote: > On Wed, 1 May 2024, Eugene Y Chang wrote: > > > Thanks David, > > > > > >> On Apr 30, 2024, at 6:12 PM, David Lang wrote: > >> > >> On Tue, 30 Apr 2024, Eugene Y Chang wrote: > >> > >>> I=E2=80=99m not completely up to speed on the gory details. Please hu= mor me. I > am pretty good on the technical marketing magic. > >>> > >>> What is the minimum configuration of an ISP infrastructure where we > can show an A/B (before and after) test? > >>> It can be a simplified scenario. The simpler, the better. We can talk > through the issues of how minimal is adequate. Of course and ISP engineer > will argue against simplicity. > >> > >> I did not see a very big improvement on a 4/.5 dsl link, but there was > improvement. > > > > Would a user feel the improvement with a 10 minute session: > > shopping on Amazon? > > using Salesforce? > > working with a shared Google doc? > > When it's only a single user, they are unlikely to notice any difference. > > But if you have one person on zoom, a second downloading something, and a > third > on Amazon, it doesn't take much to notice a difference. > > >> if you put openwrt on the customer router and configure cake with the > targeted bandwith at ~80% of line speed, you will usually see a drastic > improvement for just about any connection. > > > > Are you saying some of the benefits can be realized with just upgrading > the > > subscriber=E2=80=99s router? This makes adoption harder because the sub= scriber > will > > lose the ISP=E2=80=99s support for any connectivity issues. If a demo i= mpresses > the > > subscribers, the ISP still needs to embrace this change; otherwise the > ISP > > will wash their hands of any subscriber problems. > > Yes, just upgrading the subscriber's device with cake and configuring it > appropriately largely solves the problem (at the cost of sacraficing > bandwith > because cake isn't working directly on the data flowing from the ISP to > the > client, and so it has to work indirectly to get the Internet server to > slow down > instead and that's a laggy, imperect work-around. If the ISPs router does > active > queue management with fq_codel, then you don't have to do this. > > This is how we know this works, many of use have been doing this for year= s > (see > the bufferbloat mailing list and it's archives_ > > >> If you can put fq_codel on both ends of the link, you can usually skip > capping the bandwidth. > > > > This is good if this means the benefits can be achieved with just the > CPE. This also limits the changes to subscribers that care. > > fq_codel on the ISPs router for downlink, and on the subscribers router > for > uplink. > > putting cake on the router on the subscriber's end and tuning it > appropriately > can achieve most of the benefit, but is more work to configure. > > >> > >> unfortunantly, it's not possible to just add this to the ISPs existing > hardware without having the source for the firmware there (and if they ha= ve > their queues in ASICs it's impossible to change them. > > > > Is this just an alternative to having the change at the CPE? > > Yes this is harder for routers in the network. > > simple fq_codel on both ends of the bottleneck connection works quite wel= l > without any configuration. Cake adds some additional fairness capabilitie= s > and > has a mode to work around the router on the other end of the bottleneck > not > doing active queue management > > >> If you can point at the dramatic decrease in latency, with no bandwidt= h > losses, that Starlink has achieved on existing hardware, that may help. > > > > This is good to know for the engineers. This adds confusion with the > subscribers. > > > >> > >> There are a number of ISPs around the world that have implemented > active queue management and report very good results from doing so. > > > > Can we get these ISPs to publically report how they have achieved great > latency reduction? > > We can help them get credit for caring about their subscribers. It > would/could be a (short term) competitive advantage. > > Of course their competitors will (might) adopt these changes and > eliminate the advantage, BUT the subscribers will retain glow of the > initial marketing for a much longer time. > > several of them have done so, I think someone else posted a report from > one in > this thread. > > >> But showing that their existing hardware can do it when their upstream > vendor doesn't support it is going to be hard. > > > > Is the upstream vendor a network provider or a computing center? > > Getting good latency from the subscriber, through the access network to > the edge computing center and CDNs would be great. The CDNs would harvest > the benefits. The other computing configurations would have make the chan= ge > to be competitive. > > I'm talking about the manufacturer of the routers that the ISPs deploy at > the > last hop before getting to the subscriber, and the router on the > subscriber end > of the link (although most of those are running some variation of openWRT= , > so > turning it on would not be significant work for the manufacturer) > > > We wouild have done our part at pushing the next round of adoption. > > Many of us have been pushing this for well over a decade. Getting > Starlink's > attention to address their bufferbloat issues is a major success. > > David Lang > > > Gene > > > >> > >> David Lang > >> > >>> > >>> We will want to show the human visible impact and not debate good or > not so good measurements. If we get the business and community subscriber= s > on our side, we win. > >>> > >>> Note: > >>> Stage 1 is to show we have a pure software fix (that can work on thei= r > hardware). The fix is =E2=80=9Cso dramatic=E2=80=9D that subscribers can = experience it > without debating measurements. > >>> Stage 2 discusses why the ISP should demand that their equipment > vendors add this software. (The software could already be available, but > the ISP doesn=E2=80=99t think it is worth the trouble to enable it.) Noth= ing will > happen unless we stay engaged. We need to keep the subscribers engaged, t= oo. > >>> > >>> Should we have a conference call to discuss this? > >>> > >>> > >>> Gene > >>> ---------------------------------------------- > >>> Eugene Chang > >>> IEEE Life Senior Member > >>> > >>> > >>> > >>>> On Apr 30, 2024, at 3:52 PM, Jim Forster > wrote: > >>>> > >>>> Gene, David, > >>>> =E2=80=98m > >>>> Agreed that the technical problem is largely solved with cake & code= l. > >>>> > >>>> Also that demos are good. How to do one for this problem> > >>>> > >>>> =E2=80=94 Jim > >>>> > >>>>> The bandwidth mantra has been used for so long that a technical > discussion cannot unseat the mantra. > >>>>> Some technical parties use the mantra to sell more, faster, > ineffective service. Gullible customers accept that they would be happy i= f > they could afford even more speed. > >>>>> > >>>>> Shouldn=E2=80=99t we create a demo to show the solution? > >>>>> To show is more effective than to debate. It is impossible to > explain to some people. > >>>>> Has anyone tried to create a demo (to unseat the bandwidth mantra)? > >>>>> Is an effective demo too complicated to create? > >>>>> I=E2=80=99d be glad to participate in defining a demo and publicity= campaign. > >>>>> > >>>>> Gene > >>>>> > >>>>> > >>>>>> On Apr 30, 2024, at 2:36 PM, David Lang david@lang.hm> >> wrote: > >>>>>> > >>>>>> On Tue, 30 Apr 2024, Eugene Y Chang via Starlink wrote: > >>>>>> > >>>>>>> I am always surprised how complicated these discussions become. > (Surprised mostly because I forgot the kind of issues this community care > about.) The discussion doesn=E2=80=99t shed light on the following scenar= ios. > >>>>>>> > >>>>>>> While watching stream content, activating controls needed to > switch content sometimes (often?) have long pauses. I attribute that to > buffer bloat and high latency. > >>>>>>> > >>>>>>> With a happy household user watching streaming media, a second > user could have terrible shopping experience with Amazon. The interactive > response could be (is often) horrible. (Personally, I would be doing emai= l > and working on a shared doc. The Amazon analogy probably applies to more > people.) > >>>>>>> > >>>>>>> How can we deliver graceful performance to both persons in a > household? > >>>>>>> Is seeking graceful performance too complicated to improve? > >>>>>>> (I said =E2=80=9Cgraceful=E2=80=9D to allow technical flexibility= .) > >>>>>> > >>>>>> it's largely a solved problem from a technical point of view. > fq_codel and cake solve this. > >>>>>> > >>>>>> The solution is just not deployed widely, instead people argue tha= t > more bandwidth is needed instead. > > > >_______________________________________________ > Starlink mailing list > Starlink@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/starlink > --000000000000ba837f06176af1a6 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Eugene, this is one of the ISP examples of using OpenWrt, = CAKE & FQ-CoDel to fix not only his network, but also to refurbish an o= ld device - when the vendor didn't give a flying F:
= https://blog.nafiux.com/posts/cnpilot_r190w_openwrt_bufferbloat_fqcodel_cak= e/

Here is also the list of OpenWrt supported HW= :=C2=A0https://openwrt.or= g/supported_devices=C2=A0
If you/ISP want to go mainstream, M= ikroTik will be a good option.

This is a great pla= ce to start (not only for your ISP):=C2=A0https://www.buf= ferbloat.net/projects/bloat/wiki/What_can_I_do_about_Bufferbloat/
=

=
All the best,

Frank

Frantisek (Frank) Borsik=

<= /u>=C2=A0

https://www.linkedin.com/in/frantisekborsik=

Sign= al, Telegram, WhatsApp: +421919416714=C2=A0

iMessage, mobile: +420775230885

Skype: ca= sioa5302ca

frantisek.borsik@gmail.com



On Wed, May 1, 2024 at 9:18=E2=80=AFPM David Lang via Starlink &l= t;starlink@lists.bufferbl= oat.net> wrote:
On Wed, 1 May 2024, Eugene= Y Chang wrote:

> Thanks David,
>
>
>> On Apr 30, 2024, at 6:12 PM, David Lang <david@lang.hm> wrote:
>>
>> On Tue, 30 Apr 2024, Eugene Y Chang wrote:
>>
>>> I=E2=80=99m not completely up to speed on the gory details. Pl= ease humor me. I am pretty good on the technical marketing magic.
>>>
>>> What is the minimum configuration of an ISP infrastructure whe= re we can show an A/B (before and after) test?
>>> It can be a simplified scenario. The simpler, the better. We c= an talk through the issues of how minimal is adequate. Of course and ISP en= gineer will argue against simplicity.
>>
>> I did not see a very big improvement on a 4/.5 dsl link, but there= was improvement.
>
> Would a user feel the improvement with a 10 minute session:
> shopping on Amazon?
> using Salesforce?
> working with a shared Google doc?

When it's only a single user, they are unlikely to notice any differenc= e.

But if you have one person on zoom, a second downloading something, and a t= hird
on Amazon, it doesn't take much to notice a difference.

>> if you put openwrt on the customer router and configure cake with = the targeted bandwith at ~80% of line speed, you will usually see a drastic= improvement for just about any connection.
>
> Are you saying some of the benefits can be realized with just upgradin= g the
> subscriber=E2=80=99s router? This makes adoption harder because the su= bscriber will
> lose the ISP=E2=80=99s support for any connectivity issues. If a demo = impresses the
> subscribers, the ISP still needs to embrace this change; otherwise the= ISP
> will wash their hands of any subscriber problems.

Yes, just upgrading the subscriber's device with cake and configuring i= t
appropriately largely solves the problem (at the cost of sacraficing bandwi= th
because cake isn't working directly on the data flowing from the ISP to= the
client, and so it has to work indirectly to get the Internet server to slow= down
instead and that's a laggy, imperect work-around. If the ISPs router do= es active
queue management with fq_codel, then you don't have to do this.

This is how we know this works, many of use have been doing this for years = (see
the bufferbloat mailing list and it's archives_

>> If you can put fq_codel on both ends of the link, you can usually = skip capping the bandwidth.
>
> This is good if this means the benefits can be achieved with just the = CPE. This also limits the changes to subscribers that care.

fq_codel on the ISPs router for downlink, and on the subscribers router for=
uplink.

putting cake on the router on the subscriber's end and tuning it approp= riately
can achieve most of the benefit, but is more work to configure.

>>
>> unfortunantly, it's not possible to just add this to the ISPs = existing hardware without having the source for the firmware there (and if = they have their queues in ASICs it's impossible to change them.
>
> Is this just an alternative to having the change at the CPE?
> Yes this is harder for routers in the network.

simple fq_codel on both ends of the bottleneck connection works quite well =
without any configuration. Cake adds some additional fairness capabilities = and
has a mode to work around the router on the other end of the bottleneck not=
doing active queue management

>> If you can point at the dramatic decrease in latency, with no band= width losses, that Starlink has achieved on existing hardware, that may hel= p.
>
> This is good to know for the engineers. This adds confusion with the s= ubscribers.
>
>>
>> There are a number of ISPs around the world that have implemented = active queue management and report very good results from doing so.
>
> Can we get these ISPs to publically report how they have achieved grea= t latency reduction?
> We can help them get credit for caring about their subscribers. It wou= ld/could be a (short term) competitive advantage.
> Of course their competitors will (might) adopt these changes and elimi= nate the advantage, BUT the subscribers will retain glow of the initial mar= keting for a much longer time.

several of them have done so, I think someone else posted a report from one= in
this thread.

>> But showing that their existing hardware can do it when their upst= ream vendor doesn't support it is going to be hard.
>
> Is the upstream vendor a network provider or a computing center?
> Getting good latency from the subscriber, through the access network t= o the edge computing center and CDNs would be great. The CDNs would harvest= the benefits. The other computing configurations would have make the chang= e to be competitive.

I'm talking about the manufacturer of the routers that the ISPs deploy = at the
last hop before getting to the subscriber, and the router on the subscriber= end
of the link (although most of those are running some variation of openWRT, = so
turning it on would not be significant work for the manufacturer)

> We wouild have done our part at pushing the next round of adoption.
Many of us have been pushing this for well over a decade. Getting Starlink&= #39;s
attention to address their bufferbloat issues is a major success.

David Lang

> Gene
>
>>
>> David Lang
>>
>>>
>>> We will want to show the human visible impact and not debate g= ood or not so good measurements. If we get the business and community subsc= ribers on our side, we win.
>>>
>>> Note:
>>> Stage 1 is to show we have a pure software fix (that can work = on their hardware). The fix is =E2=80=9Cso dramatic=E2=80=9D that subscribe= rs can experience it without debating measurements.
>>> Stage 2 discusses why the ISP should demand that their equipme= nt vendors add this software. (The software could already be available, but= the ISP doesn=E2=80=99t think it is worth the trouble to enable it.) Nothi= ng will happen unless we stay engaged. We need to keep the subscribers enga= ged, too.
>>>
>>> Should we have a conference call to discuss this?
>>>
>>>
>>> Gene
>>> ----------------------------------------------
>>> Eugene Chang
>>> IEEE Life Senior Member
>>>
>>>
>>>
>>>> On Apr 30, 2024, at 3:52 PM, Jim Forster <jim@connectivitycap.com= > wrote:
>>>>
>>>> Gene, David,
>>>> =E2=80=98m
>>>> Agreed that the technical problem is largely solved with c= ake & codel.
>>>>
>>>> Also that demos are good. How to do one for this problem&g= t;
>>>>
>>>>=C2=A0 =E2=80=94 Jim
>>>>
>>>>> The bandwidth mantra has been used for so long that a = technical discussion cannot unseat the mantra.
>>>>> Some technical parties use the mantra to sell more, fa= ster, ineffective service. Gullible customers accept that they would be hap= py if they could afford even more speed.
>>>>>
>>>>> Shouldn=E2=80=99t we create a demo to show the solutio= n?
>>>>> To show is more effective than to debate. It is imposs= ible to explain to some people.
>>>>> Has anyone tried to create a demo (to unseat the bandw= idth mantra)?
>>>>> Is an effective demo too complicated to create?
>>>>> I=E2=80=99d be glad to participate in defining a demo = and publicity campaign.
>>>>>
>>>>> Gene
>>>>>
>>>>>
>>>>>> On Apr 30, 2024, at 2:36 PM, David Lang <david@lang.hm <mailto:<= a href=3D"mailto:david@lang.hm" target=3D"_blank">david@lang.hm> <= ;mailto:david@lang.hm <mailto:david@lang.= hm>>> wrote:
>>>>>>
>>>>>> On Tue, 30 Apr 2024, Eugene Y Chang via Starlink w= rote:
>>>>>>
>>>>>>> I am always surprised how complicated these di= scussions become. (Surprised mostly because I forgot the kind of issues thi= s community care about.) The discussion doesn=E2=80=99t shed light on the f= ollowing scenarios.
>>>>>>>
>>>>>>> While watching stream content, activating cont= rols needed to switch content sometimes (often?) have long pauses. I attrib= ute that to buffer bloat and high latency.
>>>>>>>
>>>>>>> With a happy household user watching streaming= media, a second user could have terrible shopping experience with Amazon. = The interactive response could be (is often) horrible. (Personally, I would= be doing email and working on a shared doc. The Amazon analogy probably ap= plies to more people.)
>>>>>>>
>>>>>>> How can we deliver graceful performance to bot= h persons in a household?
>>>>>>> Is seeking graceful performance too complicate= d to improve?
>>>>>>> (I said =E2=80=9Cgraceful=E2=80=9D to allow te= chnical flexibility.)
>>>>>>
>>>>>> it's largely a solved problem from a technical= point of view. fq_codel and cake solve this.
>>>>>>
>>>>>> The solution is just not deployed widely, instead = people argue that more bandwidth is needed instead.
>
>_______________________________________________
Starlink mailing list
Starlin= k@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink
--000000000000ba837f06176af1a6--