From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from vsmx001.dclux.xion.oxcs.net (vsmx001.dclux.xion.oxcs.net [185.74.65.81]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id BC6233B2A4 for ; Fri, 11 Jun 2021 20:02:47 -0400 (EDT) Received: from proxy-2.proxy.oxio.ns.xion.oxcs.net (proxy-2.proxy.oxio.ns.xion.oxcs.net [31.4.191.24]) by mx-out.dclux.xion.oxcs.net (Postfix) with SMTP id 3AA548C0DDF; Sat, 12 Jun 2021 00:02:44 +0000 (UTC) Date: Sat, 12 Jun 2021 01:09:00 +0200 From: Mike Puchol To: Dave Taht Cc: starlink@lists.bufferbloat.net Message-ID: In-Reply-To: References: <950B8EAF-90B9-41A6-951D-91821F591D41@teklibre.net> <01a7bed2-6f49-3d7d-eb5a-209031ee8070@gmail.com> <391A8897-5A1F-424A-9DD0-01B66824887B@teklibre.net> X-Readdle-Message-ID: be5b1d93-020c-465b-8c87-2e7ed28b62a8@Spark MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="60c3f9a2_7c83e458_3067" X-VadeSecure-Status: LEGIT X-VADE-STATUS: LEGIT Subject: Re: [Starlink] Microstate Accounting and the Nyquist problem X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Jun 2021 00:02:48 -0000 --60c3f9a2_7c83e458_3067 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline So the GPS is an STA8090 automotive-grade, makes sense as Dishy gets prop= er hot=21 Also supports sensor fusion for eg pointing in the right azimut= h.=C2=A0https://www.st.com/en/automotive-infotainment-and-telematics/gnss= -ics.html However, the beauty of GPS sync is we can choose our own time base extern= al to Dishy :-) Best, Mike On Jun 12, 2021, 01:00 +0200, Dave Taht , wrote: > > > > On Jun 11, 2021, at 3:39 PM, Dave Taht wrote: > > > > > > > > > On Jun 11, 2021, at 3:34 PM, Mike Puchol wrote= : > > > > > > We know that Starlink recalculates topology every 15 seconds (this = guy, who obviously has way too much spare time, came up with an indirect = observation of this interval:=C2=A0https://blog.beerriot.com/2021/02/14/s= tarlink-raster-scan/=C2=A0) > > > > > > If we could align with this, we could at least know when potential = changes in path delays happen, and try to observe other changes that happ= en at a similar cadence. > > > > > > Other thoughts, try to plug more details out of the gRPC data, setu= p GPS-synced probes with a device at the exit PoP, measure differences be= tween time-sync probes to an array of endpoints. > > > > > > > It=E2=80=99s ironic that the device has to have gps in it, and thus s= hould be =C2=A0able to provide perfect time to clients directly behind it= , isn=E2=80=99t. > > > > I haven=E2=80=99t captured a dhcp or dhcpv6 transaction yet myself, > > do they have a ntp option=3F > > > > What gps software or driver might they have used=3F (esr=E2=80=99s gp= sd is quite popular, but there are others) > > > > What=E2=80=99s the gps chip=3F > > > > It would be good to have solid time everywhere, as I am seeing clocks n= ot synced even close to 40ms accuracy of late. > > BTW, Eric Raymond (esr) is also one of the driving forces behind ntpsec= , along with gary and a few other people now on our list. > > =46or more details, see: > https://www.ntpsec.org/ > > Once upon a time, I sat in esr's basement hearing him rip much crud out= of the old ntpd codebase over the course of a very few days. The shouts = =E2=80=9CWhat=3F WHAAAT=3F=E2=80=9D and most of the other pithy comments = he made never made the git log. > > Over the years following the codebase got better and better, but adopti= on has been slow. > > An intro to that woefully underfunded project: > > NTPsec project - a secure, hardened, and improved implementation of Net= work Time Protocol derived from NTP Classic, Dave Mills=E2=80=99s origina= l. > NTPsec, as its name implies, is a more secure NTP. Our goal is to deliv= er code that can be used with confidence in deployments with the most str= ingent security, availability, and assurance requirements. > Towards that end we apply best practices and state-of-the art technolog= y in code auditing, verification, and testing. We begin with the most imp= ortant best practice: true open-source code review. The NTPsec code is av= ailable in a public git repository. One of our goals is to support broade= r community participation. > > > > > > Has nobody attacked the JTAG connector on a Dishy yet=3F > > I reached out to one of the teardown folk (mike (mikeonsoftware=3F)) mo= nths ago to get the debris but the rightest answer was to drill down into= it on a still-alive ones. > > > > > > > Best, > > > > > > Mike > > > On Jun 12, 2021, 00:14 +0200, David Collier-Brown , wrote: > > > > OK,=C2=A0Oh Smarter Colleagues, the challenge to you is to say if= there is a =22natural=22 place to capture state changes to get the data = we want, and if so, is it common or similar enough between drivers to be = worthy of attention=3F > > > > --dave > > > > On 2021-06-09 9:15 a.m., Dave Taht wrote: > > > > > > > > > > > > > > > > Begin forwarded message: > > > > > > > > > > > > =46rom:=C2=A0David Collier-Brown > > > > > > Subject:=C2=A0Microstate Accounting and the Nyquist problem > > > > > > Date:=C2=A0June 9, 2021 at 4:44:14 AM PDT > > > > > > To:=C2=A0Dave Taht > > > > > > Cc:=C2=A0Dave Collier-Brown > > > > > > Reply-To:=C2=A0davecb=40spamcop.net > > > > > > > > > > > > A million years ago (roughly around Solaris 9), Sun was suffe= ring from the same problems in measuring their dispatcher as you are with= =22sloshing=22. > > > > > > A CPU would be 100% busy in one microsecond, 10% busy in the = next gazillion, and the average CPU utilization for our sample period wou= ld be=C2=A0maybe=C2=A010.1, if the sampler happened to sample right when = the spike was happening. > > > > > > This was utterly useless for things like the fair-share sched= uler, so it got fixed in Solaris 10, by having the dispatcher record the = time a process (well, kernel thread) had spent in a state when the state = changed. > > > > > > Initially =22microstate accounting=22 could be toggled on and= off, but the branch-around cost more time than always doing the calculat= ion (as discovered by my mad friend =46red) and the kernel folks left it = on. It's on to this day. > > > > > > In Simon Sundberg's talk, the opportunity to measure occurs e= very 1,000 packets, when a suitable timestamp is provided. While the eBP=46= program can look at every packet and do after-the-fact book-keeping in a= map, that's only good if the phenomenon you're measuring is persistent e= nough that it's around for =7E2,000 packets. > > > > > > I'm going to suggest that the right place to record the infor= mation you want is right where the event happens.=C2=A0 Preferably in c c= ode, as performance is easy to mess up, but perhaps with an eBP=46 mechan= ism to export it. > > > > > > In previous Solaris work, I reliably found that exporting kst= ats was a darn sight harder than collecting them, and in Eric's blog post= =5B1=5D he notes that converting time is expensive and best done long aft= er collecting, when someone wanted to read the data. > > > > > > There was an effort to do kstats in Linux=5B2=5D, but it had = supposedly poor performance, and actual trouble when the clock frequency = changed. > > > > > > Is there, in your opinion, a =22natural=22 place to capture s= tate changes to get the data you want, and if so, is it common or similar= enough between drivers to be worthy of attention=3F > > > > > > --dave > > > > > > > > > > > > References: > > > > > > > > > > > > 1. Solaris:=C2=A0http://dtrace.org/blogs/eschrock/2004/10/13/= microstate-accounting-in-solaris-10/ > > > > > > 2. A failing Linux effort:=C2=A0https://lwn.net/Articles/1272= 96/,=C2=A0https://sourceforge.net/projects/microstate/ > > > > > > > > > > > > -- > > > > > > David Collier-Brown, =7C Always do right. This will g= ratify > > > > > > System Programmer and Author =7C some people and astonish the= rest > > > > > > davecb=40spamcop.net =7C -- Ma= rk Twain > > > > > > > > > =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= > > > > Starlink mailing list > > > > Starlink=40lists.bufferbloat.net > > > > https://lists.bufferbloat.net/listinfo/starlink > > > =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F > > > Starlink mailing list > > > Starlink=40lists.bufferbloat.net > > > https://lists.bufferbloat.net/listinfo/starlink > > > > =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F > > Starlink mailing list > > Starlink=40lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/starlink > --60c3f9a2_7c83e458_3067 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
So the GPS is an STA8090 automotive-grade, makes se= nse as Dishy gets proper hot=21 Also supports sensor fusion for eg pointi= ng in the right azimuth.&=23160;https://www.st.com/en/automotive-infotainment-and-telematics/gnss-ics.ht= ml

However, the beauty of GPS sync is we can choose our own time base extern= al to Dishy :-)

Best,

Mike
On Jun 12, 2021, 01:00 +0200, Dave = Taht <davet=40teklibre.net>, wrote:


On Jun 11, 2021, at 3:39 PM, Dave Taht <davet=40teklibre.net= > wrote:



On Jun 11, 2021, at 3:34 PM, Mike Puchol <mike=40starlink.sx>= wrote:

We know that Starlink recalculates t= opology every 15 seconds (this guy, who obviously has way too much spare = time, came up with an indirect observation of this interval:&=23160;https://blog.beerriot.com/2021/02/14/= starlink-raster-scan/&=23160;)

If we could align with this, we could at least know when potential change= s in path delays happen, and try to observe other changes that happen at = a similar cadence.

Other thoughts, try to plug more details out of the gRPC data, setup GPS-= synced probes with a device at the exit PoP, measure differences between = time-sync probes to an array of endpoints.


It=E2=80=99s ironic that the device has to have gps in it, and thus shoul= d be &=23160;able to provide perfect time to clients directly behind it, = isn=E2=80=99t.

I haven=E2=80=99t= captured a dhcp or dhcpv6 transaction yet myself,
do they have a nt= p option=3F

What gps software= or driver might they have used=3F (esr=E2=80=99s gpsd is quite popular, = but there are others)&=23160;

What=E2=80=99s th= e gps chip=3F


It would be good to have solid time everywhere, as I am seeing clock= s not synced even close to 40ms accuracy of late.

BTW, Eric Raymond (esr) is also one of the driving forces behind ntp= sec, along with gary and a few other people now on our list.

=46or more details, see:

Once upon a time, I sat in esr's basement hearing him rip much crud = out of the old ntpd codebase over the course of a very few days. The shou= ts =E2=80=9CWhat=3F WHAAAT=3F=E2=80=9D and most of the other pithy commen= ts he made never made the git log.&=23160;

Over the years following the codebase got better and better, but ado= ption has been slow.

An intro to that woefully underfunded project:

NTPsec project - a secure, hardened, and improved implementation of Netw= ork Time Protocol derived from NTP Classic, Dave Mills=E2=80=99s original= .
NTPsec, as its name implies, is a more secure NTP. Our goal is to delive= r code that can be used with confidence in deployments with the most stri= ngent security, availability, and assurance requirements.
Towards that end we apply best practices and state-of-the art technology= in code auditing, verification, and testing. We begin with the most impo= rtant best practice: true open-source code review. The NTPsec code is ava= ilable in a public git repository. One of our goals is to support broader= community participation.


Has nobody attacked the JTAG connect= or on a Dishy yet=3F

I reached out to one of the teardown folk (mike (mikeonsoftware=3F)) mont= hs ago to get the debris but the rightest answer was to drill down into i= t on a still-alive ones.


Best,

Mike
On Jun 12, 2021, 00:= 14 +0200, David Collier-Brown <davecb.42=40gmail.com>, wrote:

OK,&=23160;Oh Smarter Colleagues, the challenge to you is= to say if there is a =22natural=22 place to capture state changes to get= the data we want, and if so, is it common or similar enough between driv= ers to be worthy of attention=3F

--dave

On 2021-06-09 9:15 a.m., Dave Taht wro= te:


Begin forwarded message:

=46rom:&=23160;David Collier-Brown <davecb.42=40gmail.c= om>
Subject:&=23160;Microstate Accounting and the Nyquist pro= blem
Date:&=23160;June 9, 2021 at 4:44:14 AM PDT
To:&=23160;Dave Taht <davet=40teklibre.net>
Cc:&=23160;Dave Collier-Brown <dave.col= lier-brown=40indexexchange.com>
Reply-To:&=23160;davecb=40spamcop.net

A million years ago (roughly around Solaris 9), Sun was= suffering from the same problems in measuring their dispatcher as you ar= e with =22sloshing=22.

A CPU would be 100% busy in one microsecond, 10% busy i= n the next gazillion, and the average CPU utilization for our sample peri= od would be&=23160;maybe&=23160;<= /span>10.1, if the sampler happened to sample right when the spike was ha= ppening.

This was utterly useless for things like the fair-share= scheduler, so it got fixed in Solaris 10, by having the dispatcher recor= d the time a process (well, kernel thread) had spent in a state when the = state changed.

Initially =22microstate accounting=22 could be toggled = on and off, but the branch-around cost more time than always doing the ca= lculation (as discovered by my mad friend =46red) and the kernel folks le= ft it on. It's on to this day.

In Simon Sundberg's talk, the opportunity to measure oc= curs every 1,000 packets, when a suitable timestamp is provided. While th= e eBP=46 program can look at every packet and do after-the-fact book-keep= ing in a map, that's only good if the phenomenon you're measuring is pers= istent enough that it's around for =7E2,000 packets.

I'm going to suggest that the right place to record the= information you want is right where the event happens.&=23160; Preferabl= y in c code, as performance is easy to mess up, but perhaps with an eBP=46= mechanism to export it.

In previous Solaris work, I reliably found that exporti= ng kstats was a darn sight harder than collecting them, and in Eric's blo= g post=5B1=5D he notes that converting time is expensive and best done lo= ng after collecting, when someone wanted to read the data.

There was an effort to do kstats in Linux=5B2=5D, but i= t had supposedly poor performance, and actual trouble when the clock freq= uency changed.

Is there, in your opinion, a =22natural=22 place to cap= ture state changes to get the data you want, and if so, is it common or s= imilar enough between drivers to be worthy of attention=3F

--dave


References:

  1. Solaris:&=23= 160;http://dtr= ace.org/blogs/eschrock/2004/10/13/microstate-accounting-in-solaris-10/
  2. A failing Linux effort:&=23160;https://lwn.net/Articles/127296/,= &=23160;https://sourceforge.net/projects/microstate/
--  =20
David Collier-Brown,         =7C Always do right. This will gratify
System Programmer and Author =7C some people and astonish the rest
davecb=40spamcop.net          =
 =7C                      -- Mark Twain

=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
Starlink mailing list
St= arlink=40lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
Starlink mailing list
St= arlink=40lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink

=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F
Starlink mailing list
Starlink=40lists.bufferbloat= .net
https://lists.bufferbl= oat.net/listinfo/starlink

--60c3f9a2_7c83e458_3067--