From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from vsmx001.dclux.xion.oxcs.net (vsmx001.dclux.xion.oxcs.net [185.74.65.81]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 3B5243B29E for ; Mon, 13 Mar 2023 12:35:38 -0400 (EDT) Received: from proxy-2.proxy.oxio.ns.xion.oxcs.net (proxy-2.proxy.oxio.ns.xion.oxcs.net [83.61.18.4]) by mx-out.dclux.xion.oxcs.net (Postfix) with ESMTPA id EF3678C03A8 for ; Mon, 13 Mar 2023 16:35:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=dclux.xion.oxcs.net; s=mail1; t=1678725337; bh=eCt2oI+Mkkb5OkYodwu5R0sJYpgulpeZeWanVxx8moM=; h=Date:From:To:In-Reply-To:References:Subject:From; b=ZrXVBvXLDwRfRZK65Li/oISVL4HDIB8nokASm5c5XJUoxcFetHEex7btm9L70qnWD RpzX6xh0Nd/t9Lb6FAAYhZyRvhenAKjSo+rVQLqdiD706Is4MaBWLf/gRC2wPCYwiS 56VrLhNUsc2yGdoHUJIlaiNHz5MEzBV166r6D8/v6T4Ob2oGC7SiCZtBcsBFyRzorE p3vWQbTnC5Qjki+r1cKIcMytQGJuxy/4CbrKxZpgiySozAIr8PV7pbYeZPbOKuSEs/ /4t6F8PA7a3TN4z+0glGIkgJ+gCrUftO6iV1MqDAZ1yPuWh23v/Jrxdf8WHX+ghfW1 G4GRLusMikIpA== Date: Mon, 13 Mar 2023 17:35:29 +0100 From: Mike Puchol To: starlink@lists.bufferbloat.net Message-ID: <7dd2c461-3312-4af5-b581-b6878313fef5@Spark> In-Reply-To: <45AA2C67-CC59-4222-991C-F43D08699F90@gmx.de> References: <1672786712.106922180@apps.rackspace.com> <77CCAD19-07E0-4F9E-88C1-D207CF7BF376@cable.comcast.com> <83ffc0dad19e3343e49271889369cefc@rjmcmahon.com> <45AA2C67-CC59-4222-991C-F43D08699F90@gmx.de> X-Readdle-Message-ID: 7dd2c461-3312-4af5-b581-b6878313fef5@Spark MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="640f50d6_49c0e823_3617" X-VadeSecure-Status: LEGIT X-VADE-STATUS: LEGIT Subject: Re: [Starlink] UnderBloat on fiber and wisps X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Mar 2023 16:35:38 -0000 --640f50d6_49c0e823_3617 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline > =22How can I accurately distinguish a constraint in supply from a reduc= tion in demand=3F=22 Serious answer: you look at the increase or decrease of customer tickets = and complaints. Practical example: a few years ago, we had two fixed wireless access netw= orks, about 3km apart, each fed by an individual fiber local loop to our = DC. I had the (in hindsight terrible) idea to back each network's fiber w= ith the other, using two wireless point-to-point hops with a tower in the= middle, which also had customers served by it. When one fiber went down, the traffic from the tower in the middle had to= still travel to its own fiber POP (as the router doing the routing was t= here) then trombone back over the same wireless link, across the middle t= ower again, and onwards to the functioning fiber. The net effect was a se= rious constraint on supply. The result was a deluge of complaints which caused us to discontinue this= approach, and which that taught me two lessons: your customers vote with= their wallet, will tell you when you are doing a bad job, and you better= listen. The second lesson is that customers prefer no service at all, to a severe= ly degraded service. It's better to know =22I can read a book for two hou= rs while they fix the fiber=22 than to have a randomized poor quality of = service. =46ootnote: we don't experiment with customers, so we never managed to ru= n experiments to test the tolerance to degraded service, and where the li= ne between =22OK(ish)=22 and =22unsuable=22 lies, in terms of demand vs s= upply ratio. Effectively, if we contend 4:1, and we suddenly failed over = to a link that's contended 40:1, is that acceptable=3F Is 100:1=3F It'd b= e great to hear other's experiences and ideas. Best, Mike On Mar 13, 2023 at 17:09 +0100, Sebastian Moeller via Starlink , wrote: > > > > On Mar 13, 2023, at 17:04, Dave Taht wrote: > > > > On Mon, Mar 13, 2023 at 8:08=E2=80=AFAM Jeremy Austin via Rpm > > wrote: > > > > > > > > > > > > On Mon, Mar 13, 2023 at 3:02=E2=80=AFAM Sebastian Moeller via Starl= ink wrote: > > > > > > > > Hi Dan, > > > > > > > > > > > > > On Jan 9, 2023, at 20:56, dan via Rpm wrote: > > > > > > > > > > You don't need to generate the traffic on a link to measure how= > > > > > much traffic a link can handle. > > > > > > > > =5BSM=5D OK, I will bite, how do you measure achievable throughpu= t without actually generating it=3F Packet-pair techniques are notoriousl= y imprecise and have funny failure modes. > > > > > > > > > I am also looking forward to the full answer to this question. Whil= e one can infer when a link is saturated by mapping network topology onto= latency sampling, it can have on the order of 30% error, given that ther= e are multiple causes of increased latency beyond proximal congestion. > > > > > > A question I commonly ask network engineers or academics is =22How = can I accurately distinguish a constraint in supply from a reduction in d= emand=3F=22 > > > > This is an insanely good point. In looking over the wisp > > configurations I have to date, many are using S=46Q which has a defau= lt > > packet limit of 128 packets. Many are using S=46Q with a *even shorte= r* > > packet limit, which looks good on speedtests which open many flows > > (keown's sqrt(flows) for bdp), but is *lousy* for allowing a single > > flow to achieve full rate (the more common case for end-user QoE). > > > > I have in general tried to get mikrotik folk at least, to switch away= > > from fifos, red, and sfq to wards fq=5Fcodel or cake at the defaults > > (good to 10Gbit) in part, due to this. > > > > I think S=46Q 128 really starts tapping out on most networks at aroun= d > > the 200Mbit level, and above 400, really, really does not have enough= > > queue, so the net result is that wisps attempting to provide higher > > levels of service are not actually providing it in the real world, an= > > accidental constraint in supply. > > > > I have a blog piece, long in draft, called =22underbloat=22, talking = to > > this. Also I have no seen multiple fiber installs that had had a > > reasonable 50ms =46I=46O buffer for 100Mbit, but when upgraded to 1gb= it, > > left it at 5ms, which has bad sideffects for all traffic. > > > > To me it looks also that at least some ubnt radios are =46Qd and unde= rbuffered. > > This is why I tend to describe bufferbloat as a problem of over-sized a= nd under-managed buffers hoping to imply that reducing the buffersize is = not the only or even best remedy here. Once proberly managed large buffer= s do no harm (except wasting memory for most of the time, but since that = buys some resilience that is not that bad). > > Regards > Sebastian > > P.S.: This is a bit of a pendulum thing where one simplistic =22solutio= n=22 too-large-buffers gets replaced with another simplistic solution too= -small-buffers ;) > > > > > > > > -- > > > -- > > > Jeremy Austin > > > Sr. Product Manager > > > Preseem =7C Aterlo Networks > > > preseem.com > > > > > > Book a Call: https://app.hubspot.com/meetings/jeremy548 > > > Phone: 1-833-733-7336 x718 > > > Email: jeremy=40preseem.com > > > > > > Stay Connected with Newsletters & More: https://preseem.com/stay-co= nnected/ > > > =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F > > > Rpm mailing list > > > Rpm=40lists.bufferbloat.net > > > https://lists.bufferbloat.net/listinfo/rpm > > > > > > > > -- > > Come Heckle Mar 6-9 at: https://www.understandinglatency.com/ > > Dave T=C3=A4ht CEO, TekLibre, LLC > > =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F > Starlink mailing list > Starlink=40lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/starlink --640f50d6_49c0e823_3617 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
=22How can I accurately distinguish a constraint in supply from a reduct= ion in demand=3F=22

Serious answer: you look at the increase or decrease of customer tickets = and complaints.

Practical example: a few years ago, we had two fixed wireless access netw= orks, about 3km apart, each fed by an individual fiber local loop to our = DC. I had the (in hindsight terrible) idea to back each network's fiber w= ith the other, using two wireless point-to-point hops with a tower in the= middle, which also had customers served by it.

When one fiber went down, the traffic from the tower in the middle had to= still travel to its own fiber POP (as the router doing the routing was t= here) then trombone back over the same wireless link, across the middle t= ower again, and onwards to the functioning fiber. The net effect was a se= rious constraint on supply.

The result was a deluge of complaints which caused us to discontinue this= approach, and which that taught me two lessons: your customers vote with= their wallet, will tell you when you are doing a bad job, and you better= listen.&=23160;

The second lesson is that customers prefer no service at all, to a severe= ly degraded service. It's better to know =22I can read a book for two hou= rs while they fix the fiber=22 than to have a randomized poor quality of = service.

=46ootnote: we don't experiment with customers, so we never managed to ru= n experiments to test the tolerance to degraded service, and where the li= ne between =22OK(ish)=22 and =22unsuable=22 lies, in terms of demand vs s= upply ratio. Effectively, if we contend 4:1, and we suddenly failed over = to a link that's contended 40:1, is that acceptable=3F Is 100:1=3F It'd b= e great to hear other's experiences and ideas.

Best,

Mike
On Mar 13, 2023 at 17:09 +0100, Seb= astian Moeller via Starlink <starlink=40lists.bufferbloat.net>, wro= te:


On Mar 13, 2023, at 17:04, Dave Taht <da= ve.taht=40gmail.com> wrote:

On Mon, Mar 13, 2023 at 8:08=E2=80=AFAM Jeremy Austin via Rpm
<rpm=40lists.bufferbloat.net> wrote:



On Mon, Mar 13, 2023 at 3:02=E2=80=AFAM Sebastian Moeller via Starlink &l= t;starlink=40lists.bufferbloat.net> wrote:

Hi Dan,


On Jan 9, 2023, at 20:56, dan via Rpm <r= pm=40lists.bufferbloat.net> wrote:

You don't need to generate the traffic on a link to measure how
much traffic a link can handle.

=5BSM=5D OK, I will bite, how do you measure achievable throughput withou= t actually generating it=3F Packet-pair techniques are notoriously imprec= ise and have funny failure modes.


I am also looking forward to the full answer to this question. While one = can infer when a link is saturated by mapping network topology onto laten= cy sampling, it can have on the order of 30% error, given that there are = multiple causes of increased latency beyond proximal congestion.

A question I commonly ask network engineers or academics is =22How can I = accurately distinguish a constraint in supply from a reduction in demand=3F= =22

This is an insanely good point. In looking over the wisp
configurations I have to date, many are using S=46Q which has a default packet limit of 128 packets. Many are using S=46Q with a *even shorter* packet limit, which looks good on speedtests which open many flows
(keown's sqrt(flows) for bdp), but is *lousy* for allowing a single
= flow to achieve full rate (the more common case for end-user QoE).

I have in general tried to get mikrotik folk at least, to switch away
from fifos, red, and sfq to wards fq=5Fcodel or cake at the defaults
(good to 10Gbit) in part, due to this.

I think S=46Q 128 really starts tapping out on most networks at around the 200Mbit level, and above 400, really, really does not have enough
queue, so the net result is that wisps attempting to provide higher
= levels of service are not actually providing it in the real world, an
accidental constraint in supply.

I have a blog piece, long in draft, called =22underbloat=22, talking to this. Also I have no seen multiple fiber installs that had had a
reasonable 50ms =46I=46O buffer for 100Mbit, but when upgraded to 1gbit,<= br /> left it at 5ms, which has bad sideffects for all traffic.

To me it looks also that at least some ubnt radios are =46Qd and underbuf= fered.

This is why I tend to describe bufferbloat as a problem of over-sized and= under-managed buffers hoping to imply that reducing the buffersize is no= t the only or even best remedy here. Once proberly managed large buffers = do no harm (except wasting memory for most of the time, but since that bu= ys some resilience that is not that bad).

Regards
Sebastian

P.S.: This is a bit of a pendulum thing where one simplistic =22solution=22= too-large-buffers gets replaced with another simplistic solution too-sma= ll-buffers ;)




--
--
Jeremy Austin
Sr. Product Manager
Preseem =7C Aterlo Networks
preseem.com

Book a Call: https://app.hubspot.com/meetings/jeremy548
Phone: 1-833-733-7336 x718
Email: jeremy=40preseem.com

Stay Connected with Newsletters & More: https://preseem.com/stay-conn= ected/
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
Rpm mailing list
Rpm=40lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/rpm



--
Come Heckle Mar 6-9 at: https://www.understandinglatency.com/
Dave T=C3=A4ht CEO, TekLibre, LLC

=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
Starlink mailing list
Starlink=40lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink
--640f50d6_49c0e823_3617--