From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io1-xd2c.google.com (mail-io1-xd2c.google.com [IPv6:2607:f8b0:4864:20::d2c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 6181A3B29E for ; Fri, 16 Jul 2021 13:13:54 -0400 (EDT) Received: by mail-io1-xd2c.google.com with SMTP id p186so11399143iod.13 for ; Fri, 16 Jul 2021 10:13:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nathan.io; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=3ZdaULf6XUATFtaLUxM3TMpmH97LIQhqzXzm+Vzg2OM=; b=PPK9Rqx1wwq8k9OCRsUd9VLYK6DAuLMOfrRYP2N8Z3qnZy/x5YaL33wjAdKggaG3ML VvHpwhO8X2ZXjJHChJujh9d4cbpBUDjJ0DS4jeIYC/JUCtwHEeJxoAlsqDXwezJFXVbD 2EcfQ6r7C3v7M/ep9uwI8VynfZcNVdrvJppiffNY2TCrpLnEwnu4/w+tCdIPaUeUFHc+ 4gZxCZzEkm3XfNivfE+UPUrGFe0HttET6ZmUNIbPAgReHxhN7wWIAfVLGCuVOdVmnd4b pTru6BzUyOFbDdqJbeqIfeTVD/M1q+uGBUp4ecjgXx90n63flnNTBzsL06ZnhX08DdMT GQ8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=3ZdaULf6XUATFtaLUxM3TMpmH97LIQhqzXzm+Vzg2OM=; b=HTInBf/Iw76Lufnyn4nTDxZqrOmqjgk3vmpPy0OVcqlJQ1auh/CBLbeO44/W2aIydj 0SzIIketSGGqoMxJfcr3OEChyiNcd7IoaQIa+WVFAoxzLA5nmOkiZ67K3ZRLTTb/27Tu NxtlTB53BHcFXnMcOiJPIHZmeTr1tsdZ7KjmePcOYpGKryw3WNijDFPg6QukQLXqvo1z 1WccVIWrsixD9ZUo9t/EMfCjlI2IneJPX1ovva+Oc/TyaZeesayIyHvwgSnhhX7w1w1B luK04NGeOvgqJII19BBwdHO2Hvi25NNIzmMHJTBC7Sni0tqtmFVf9wcuCVHEgbxT87Gu AOLg== X-Gm-Message-State: AOAM533Baaup+yAUn4zTYtLvmSb8TlluWHnY/zqe86zbcIiqKWdhyi83 HgXci2Nw80y3/pJMOyloU4FbCl77Y1WuNJ2caDvSKw== X-Google-Smtp-Source: ABdhPJxu6hmus+fbVuSxGuvjs4zdIB3raIAl5n5xZfkYlEaXZ8HDfcDMcZ9/Afj8mggdTIXK8GFKYqsWc9iAaxt0WHQ= X-Received: by 2002:a6b:6412:: with SMTP id t18mr7913660iog.64.1626455633585; Fri, 16 Jul 2021 10:13:53 -0700 (PDT) MIME-Version: 1.0 References: <1625856001.74681750@apps.rackspace.com> In-Reply-To: From: Nathan Owens Date: Fri, 16 Jul 2021 10:13:42 -0700 Message-ID: To: David Lang Cc: "Wheelock, Ian" , "starlink@lists.bufferbloat.net" , "David P. Reed" Content-Type: multipart/alternative; boundary="000000000000487aad05c740b6f6" Subject: Re: [Starlink] Starlink and bufferbloat status? X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jul 2021 17:13:54 -0000 --000000000000487aad05c740b6f6 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Elon said "foolish packet routing" for things over 20ms! Which seems crazy if you do some basic math: - Sat to User Terminal distance: 550-950km air/vacuum: 1.9 - 3.3ms - Sat to GW distance: 550-950km air/vacuum: 1.9 - 3.3ms - GW to PoP Distance: 50-800km fiber: 0.25 - 4ms - PoP to Internet Distance: 50km fiber: 0.25 - 0.5ms - Total one-way delay: 4.3 - 11.1ms - Theoretical minimum RTT: 8.6ms - 22.2ms, call it 15.4ms This includes no transmission delay, queuing delay, processing/fragmentation/reassembly/etc, and no time-division multiplexing. On Fri, Jul 16, 2021 at 10:09 AM David Lang wrote: > I think it depends on if you are looking at datacenter-to-datacenter > latency of > home to remote datacenter latency :-) > > my rule of thumb for cross US ping time has been 80-100ms latency (but > it's been > a few years since I tested it). > > I note that an article I saw today said that Elon is saying that latency > will > improve significantly in the near future, that up/down latency is ~20ms > and the > additional delays pushing it to the 80ms range are 'stupid packet routing= ' > problems that they are working on. > > If they are still in that level of optimization, it doesn't surprise me > that > they haven't really focused on the bufferbloat issue, they have more > obvious > stuff to fix first. > > David Lang > > > On Fri, 16 Jul 2021, Wheelock, Ian wrote: > > > Date: Fri, 16 Jul 2021 10:21:52 +0000 > > From: "Wheelock, Ian" > > To: David Lang , David P. Reed > > Cc: "starlink@lists.bufferbloat.net" > > Subject: Re: [Starlink] Starlink and bufferbloat status? > > > > Hi David > > In terms of the Latency that David (Reed) mentioned for California to > Massachusetts of about 17ms over the public internet, it seems a bit fast= er > than what I would expect. My own traceroute via my VDSL link shows 14ms > just to get out of the operator network. > > > > https://www.wondernetwork.com is a handy tool for checking geographic > ping perf between cities, and it shows a min of about 66ms for pings > between Boston and San Diego > https://wondernetwork.com/pings/boston/San%20Diego (so about 33ms for > 1-way transfer). > > > > Distance wise this is about 4,100 KM (2,500 M), and @2/3 speed of light > (through a pure fibre link of that distance) the propagation time is just > over 20ms. If the network equipment between the Boston and San Diego is > factored in, with some buffering along the way, 33ms does seem quite > reasonable over the 20ms for speed of light in fibre for that 1-way trans= fer > > > > -Ian Wheelock > > > > From: Starlink on behalf of > David Lang > > Date: Friday 9 July 2021 at 23:59 > > To: "David P. Reed" > > Cc: "starlink@lists.bufferbloat.net" > > Subject: Re: [Starlink] Starlink and bufferbloat status? > > > > IIRC, the definition of 'low latency' for the FCC was something like > 100ms, and Musk was predicting <40ms. roughly competitive with landlines, > and worlds better than geostationary satellite (and many > > External (mailto:david@lang.hm) > > > https://shared.outlook.inky.com/report?id=3DY29tbXNjb3BlL2lhbi53aGVlbG9ja= 0Bjb21tc2NvcGUuY29tL2I1MzFjZDA4OTZmMWI0Yzc5NzdiOTIzNmY3MTAzM2MxLzE2MjU4NzE1= NDkuNjU=3D#key=3D19e8545676e28e577c813de83a4cf1dc > https://www.inky.com/banner-faq/ https://www.inky.com > > > > IIRC, the definition of 'low latency' for the FCC was something like > 100ms, and > > Musk was predicting <40ms. > > > > roughly competitive with landlines, and worlds better than geostationar= y > > satellite (and many wireless ISPs) > > > > but when doing any serious testing of latency, you need to be wired to > the > > router, wifi introduces so much variability that it swamps the signal. > > > > David Lang > > > > On Fri, 9 Jul 2021, David P. Reed wrote: > > > >> Date: Fri, 9 Jul 2021 14:40:01 -0400 (EDT) > >> From: David P. Reed > >> To: starlink@lists.bufferbloat.net > >> Subject: [Starlink] Starlink and bufferbloat status? > >> > >> > >> Early measurements of performance of Starlink have shown significant > bufferbloat, as Dave Taht has shown. > >> > >> But... Starlink is a moving target. The bufferbloat isn't a hardware > issue, it should be completely manageable, starting by simple firmware > changes inside the Starlink system itself. For example, implementing > fq_codel so that bottleneck links just drop packets according to the Best > Practices RFC, > >> > >> So I'm hoping this has improved since Dave's measurements. How much ha= s > it improved? What's the current maximum packet latency under full > load, Ive heard anecdotally that a friend of a friend gets 84 msec. *pin= g > times under full load*, but he wasn't using flent or some other measureme= nt > tool of good quality that gives a true number. > >> > >> 84 msec is not great - it's marginal for Zoom quality experience (you > want latencies significantly less than 100 msec. as a rule of thumb for > teleconferencing quality). But it is better than Dave's measurements show= ed. > >> > >> Now Musk bragged that his network was "low latency" unlike other high > speed services, which means low end-to-end latency. That got him > permission from the FCC to operate Starlink at all. His number was, I > think, < 5 msec. 84 is a lot more than 5. (I didn't believe 5, because he > probably meant just the time from the ground station to the terminal > through the satellite. But I regularly get 17 msec. between California an= d > Massachusetts over the public Internet) > >> > >> So 84 might be the current status. That would mean that someone at > Srarlink might be paying some attention, but it is a long way from what > Musk implied. > >> > >> > >> PS: I forget the number of the RFC, but the number of packets queued o= n > an egress link should be chosen by taking the hardware bottleneck > throughput of any path, combined with an end-to-end Internet underlying > delay of about 10 msec. to account for hops between source and destinatio= n. > Lets say Starlink allocates 50 Mb/sec to each customer, packets are limit= ed > to 10,000 bits (1500 * 8), so the outbound queues should be limited to > about 0.01 * 50,000,000 / 10,000, which comes out to about 250 packets fr= om > each terminal of buffering, total, in the path from terminal to public > Internet, assuming the connection to the public Internet is not a problem= . > > _______________________________________________ > > Starlink mailing list > > Starlink@lists.bufferbloat.net > > > https://secure-web.cisco.com/1sNc_-1HhGCW7xdirt_lAoAy5Nn5T6UA85Scjn5BR7QH= Xtumhrf6RKn78SuRJG7DUKI3duggU9g6hJKW-Ze07HTczYqB9mBpIeALqk5drQ7nMvM8K7JbWfU= bPR7JSNrI75UjiNXQk0wslBfoOTvkMlRj5eMOZhps7DMGBRQTVAeTd5vwXoQtDgS6zLCcJkrcO2= S9MRSCC4f1I17SzgQJIwqo3LEwuN6lD-pkX0MFLqGr2zzsHw5eapd-VBlHu5reC4-OEn2zHkb7H= NzS1pcueF6tsUE1vFRsWs2SIOwU5MvbKe3J3Q6NRQ40cHI1AGd-i/https://lists.bufferbl= oat.net/listinfo/starlink > > > >_______________________________________________ > Starlink mailing list > Starlink@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/starlink > --000000000000487aad05c740b6f6 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Elon said "foolish packet routing" for = things over 20ms! Which seems crazy if you do some basic math:
  • S= at to User Terminal distance: 550-950km air/vacuum: 1.9 - 3.3ms
  • Sat= to GW distance: 550-950km air/vacuum: 1.9 - 3.3ms
  • GW to PoP Distan= ce: 50-800km fiber: 0.25 - 4ms
  • PoP to Internet Distance: 50km fiber= : 0.25 - 0.5ms
  • Total one-way delay: 4.3 - 11.1ms
  • Theoretica= l minimum RTT: 8.6ms - 22.2ms, call it 15.4ms
This includes = no transmission delay, queuing delay, processing/fragmentation/reassembly/e= tc, and no time-division multiplexing.

On Fri, Jul 16, 2021 at 1= 0:09 AM David Lang <david@lang.hm&g= t; wrote:
I thin= k it depends on if you are looking at datacenter-to-datacenter latency of <= br> home to remote datacenter latency :-)

my rule of thumb for cross US ping time has been 80-100ms latency (but it&#= 39;s been
a few years since I tested it).

I note that an article I saw today said that Elon is saying that latency wi= ll
improve significantly in the near future, that up/down latency is ~20ms and= the
additional delays pushing it to the 80ms range are 'stupid packet routi= ng'
problems that they are working on.

If they are still in that level of optimization, it doesn't surprise me= that
they haven't really focused on the bufferbloat issue, they have more ob= vious
stuff to fix first.

David Lang


=C2=A0 On Fri, 16 Jul 2021, Wheelock, Ian wrote:

> Date: Fri, 16 Jul 2021 10:21:52 +0000
> From: "Wheelock, Ian" <ian.wheelock@commscope.com>
> To: David Lang <= david@lang.hm>, David P. Reed <dpreed@deepplum.com>
> Cc: "starlink@lists.bufferbloat.net" <starlink@lists.bufferbloat.ne= t>
> Subject: Re: [Starlink] Starlink and bufferbloat status?
>
> Hi David
> In terms of the Latency that David (Reed) mentioned for California to = Massachusetts of about 17ms over the public internet, it seems a bit faster= than what I would expect. My own traceroute via my VDSL link shows 14ms ju= st to get out of the operator network.
>
> https://www.wondernetwork.com=C2=A0 is a handy tool for checki= ng geographic ping perf between cities, and it shows a min of about 66ms fo= r pings between Boston and San Diego https://wonde= rnetwork.com/pings/boston/San%20Diego (so about 33ms for 1-way transfer= ).
>
> Distance wise this is about 4,100 KM (2,500 M), and @2/3 speed of ligh= t (through a pure fibre link of that distance) the propagation time is just= over 20ms. If the network equipment between the Boston and San Diego is fa= ctored in, with some buffering along the way, 33ms does seem quite reasonab= le over the 20ms for speed of light in fibre for that 1-way transfer
>
> -Ian Wheelock
>
> From: Starlink <starlink-bounces@lists.bufferbloat.net> on = behalf of David Lang <david@lang.hm>
> Date: Friday 9 July 2021 at 23:59
> To: "David P. Reed" <dpreed@deepplum.com>
> Cc: "starlink@lists.bufferbloat.net" <starlink@lists.bufferbloat.ne= t>
> Subject: Re: [Starlink] Starlink and bufferbloat status?
>
> IIRC, the definition of 'low latency' for the FCC was somethin= g like 100ms, and Musk was predicting <40ms. roughly competitive with la= ndlines, and worlds better than geostationary satellite (and many
> External (mailto:da= vid@lang.hm)
> =C2=A0 https://shared.outlook.inky.com/report?id= =3DY29tbXNjb3BlL2lhbi53aGVlbG9ja0Bjb21tc2NvcGUuY29tL2I1MzFjZDA4OTZmMWI0Yzc5= NzdiOTIzNmY3MTAzM2MxLzE2MjU4NzE1NDkuNjU=3D#key=3D19e8545676e28e577c813de83a= 4cf1dc =C2=A0https://www.inky.com/banner-faq/ =C2=A0https://www.= inky.com
>
> IIRC, the definition of 'low latency' for the FCC was somethin= g like 100ms, and
> Musk was predicting <40ms.
> =C2=A0
> roughly competitive with landlines, and worlds better than geostationa= ry
> satellite (and many wireless ISPs)
> =C2=A0
> but when doing any serious testing of latency, you need to be wired to= the
> router, wifi introduces so much variability that it swamps the signal.=
> =C2=A0
> David Lang
> =C2=A0
> On Fri, 9 Jul 2021, David P. Reed wrote:
> =C2=A0
>> Date: Fri, 9 Jul 2021 14:40:01 -0400 (EDT)
>> From: David P. Reed <dpreed@deepplum.com>
>> To: starlink@lists.bufferbloat.net
>> Subject: [Starlink] Starlink and bufferbloat status?
>>
>>
>> Early measurements of performance of Starlink have shown significa= nt bufferbloat, as Dave Taht has shown.
>>
>> But...=C2=A0=C2=A0Starlink is a moving target. The bufferbloat isn= 't a hardware issue, it should be completely manageable, starting by si= mple firmware changes inside the Starlink system itself. For example, imple= menting fq_codel so that bottleneck links just drop packets according to th= e Best Practices RFC,
>>
>> So I'm hoping this has improved since Dave's measurements.= How much has it improved? What's the current maximum packet latency un= der full load,=C2=A0=C2=A0Ive heard anecdotally that a friend of a friend g= ets 84 msec. *ping times under full load*, but he wasn't using flent or= some other measurement tool of good quality that gives a true number.
>>
>> 84 msec is not great - it's marginal for Zoom quality experien= ce (you want latencies significantly less than 100 msec. as a rule of thumb= for teleconferencing quality). But it is better than Dave's measuremen= ts showed.
>>
>> Now Musk bragged that his network was "low latency" unli= ke other high speed services, which means low end-to-end latency.=C2=A0=C2= =A0That got him permission from the FCC to operate Starlink at all. His num= ber was, I think, < 5 msec. 84 is a lot more than 5. (I didn't belie= ve 5, because he probably meant just the time from the ground station to th= e terminal through the satellite. But I regularly get 17 msec. between Cali= fornia and Massachusetts over the public Internet)
>>
>> So 84 might be the current status. That would mean that someone at= Srarlink might be paying some attention, but it is a long way from what Mu= sk implied.
>>
>>
>> PS: I forget the number of the RFC, but the number of packets queu= ed on an egress link should be chosen by taking the hardware bottleneck thr= oughput of any path, combined with an end-to-end Internet underlying delay = of about 10 msec. to account for hops between source and destination. Lets = say Starlink allocates 50 Mb/sec to each customer, packets are limited to 1= 0,000 bits (1500 * 8), so the outbound queues should be limited to about 0.= 01 * 50,000,000 / 10,000, which comes out to about 250 packets from each te= rminal of buffering, total, in the path from terminal to public Internet, a= ssuming the connection to the public Internet is not a problem.
> _______________________________________________
> Starlink mailing list
> St= arlink@lists.bufferbloat.net
> https://secure-web.cisco.com/1sNc_-1HhGCW7xdirt_lAoAy5Nn5T6UA85Scjn5BR7Q= HXtumhrf6RKn78SuRJG7DUKI3duggU9g6hJKW-Ze07HTczYqB9mBpIeALqk5drQ7nMvM8K7JbWf= UbPR7JSNrI75UjiNXQk0wslBfoOTvkMlRj5eMOZhps7DMGBRQTVAeTd5vwXoQtDgS6zLCcJkrcO= 2S9MRSCC4f1I17SzgQJIwqo3LEwuN6lD-pkX0MFLqGr2zzsHw5eapd-VBlHu5reC4-OEn2zHkb7= HNzS1pcueF6tsUE1vFRsWs2SIOwU5MvbKe3J3Q6NRQ40cHI1AGd-i/https://lists.bufferb= loat.net/listinfo/starlink
>
>_______________________________________________
Starlink mailing list
Starlin= k@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink
--000000000000487aad05c740b6f6--