[Starlink] Microstate Accounting and the Nyquist problem

Fri Jun 11 19:00:52 EDT 2021

> On Jun 11, 2021, at 3:39 PM, Dave Taht <davet at teklibre.net> wrote:
> 
> 
> 
>> On Jun 11, 2021, at 3:34 PM, Mike Puchol <mike at starlink.sx <mailto:mike at starlink.sx>> wrote:
>> 
>> We know that Starlink recalculates topology every 15 seconds (this guy, who obviously has way too much spare time, came up with an indirect observation of this interval: https://blog.beerriot.com/2021/02/14/starlink-raster-scan/ <https://blog.beerriot.com/2021/02/14/starlink-raster-scan/> )
>> 
>> If we could align with this, we could at least know when potential changes in path delays happen, and try to observe other changes that happen at a similar cadence.
>> 
>> Other thoughts, try to plug more details out of the gRPC data, setup GPS-synced probes with a device at the exit PoP, measure differences between time-sync probes to an array of endpoints.
>> 
> 
> It’s ironic that the device has to have gps in it, and thus should be  able to provide perfect time to clients directly behind it, isn’t.
> 
> I haven’t captured a dhcp or dhcpv6 transaction yet myself,
> do they have a ntp option?
> 
> What gps software or driver might they have used? (esr’s gpsd is quite popular, but there are others) 
> 
> What’s the gps chip?
> 

It would be good to have solid time everywhere, as I am seeing clocks not synced even close to 40ms accuracy of late.

BTW, Eric Raymond (esr) is also one of the driving forces behind ntpsec, along with gary and a few other people now on our list.

For more details, see:
https://www.ntpsec.org/

Once upon a time, I sat in esr's basement hearing him rip much crud out of the old ntpd codebase over the course of a very few days. The shouts “What? WHAAAT?” and most of the other pithy comments he made never made the git log. 

Over the years following the codebase got better and better, but adoption has been slow.

An intro to that woefully underfunded project:

NTPsec project - a secure, hardened, and improved implementation of Network Time Protocol derived from NTP Classic, Dave Mills’s original.
NTPsec, as its name implies, is a more secure NTP. Our goal is to deliver code that can be used with confidence in deployments with the most stringent security, availability, and assurance requirements.
Towards that end we apply best practices and state-of-the art technology in code auditing, verification, and testing. We begin with the most important best practice: true open-source code review. The NTPsec code is available in a public git repository. One of our goals is to support broader community participation.

> 
>> Has nobody attacked the JTAG connector on a Dishy yet?

I reached out to one of the teardown folk (mike (mikeonsoftware?)) months ago to get the debris but the rightest answer was to drill down into it on a still-alive ones.

>> 
>> Best,
>> 
>> Mike
>> On Jun 12, 2021, 00:14 +0200, David Collier-Brown <davecb.42 at gmail.com <mailto:davecb.42 at gmail.com>>, wrote:
>>> OK, Oh Smarter Colleagues, the challenge to you is to say if there is a "natural" place to capture state changes to get the data we want, and if so, is it common or similar enough between drivers to be worthy of attention?
>>> 
>>> --dave
>>> 
>>> On 2021-06-09 9:15 a.m., Dave Taht wrote:
>>>> 
>>>> 
>>>>> Begin forwarded message:
>>>>> 
>>>>> From: David Collier-Brown <davecb.42 at gmail.com <mailto:davecb.42 at gmail.com>>
>>>>> Subject: Microstate Accounting and the Nyquist problem
>>>>> Date: June 9, 2021 at 4:44:14 AM PDT
>>>>> To: Dave Taht <davet at teklibre.net <mailto:davet at teklibre.net>>
>>>>> Cc: Dave Collier-Brown <dave.collier-brown at indexexchange.com <mailto:dave.collier-brown at indexexchange.com>>
>>>>> Reply-To: davecb at spamcop.net <mailto:davecb at spamcop.net>
>>>>> 
>>>>> A million years ago (roughly around Solaris 9), Sun was suffering from the same problems in measuring their dispatcher as you are with "sloshing".
>>>>> 
>>>>> A CPU would be 100% busy in one microsecond, 10% busy in the next gazillion, and the average CPU utilization for our sample period would be maybe 10.1, if the sampler happened to sample right when the spike was happening.
>>>>> 
>>>>> This was utterly useless for things like the fair-share scheduler, so it got fixed in Solaris 10, by having the dispatcher record the time a process (well, kernel thread) had spent in a state when the state changed.
>>>>> 
>>>>> Initially "microstate accounting" could be toggled on and off, but the branch-around cost more time than always doing the calculation (as discovered by my mad friend Fred) and the kernel folks left it on. It's on to this day.
>>>>> 
>>>>> In Simon Sundberg's talk, the opportunity to measure occurs every 1,000 packets, when a suitable timestamp is provided. While the eBPF program can look at every packet and do after-the-fact book-keeping in a map, that's only good if the phenomenon you're measuring is persistent enough that it's around for ~2,000 packets.
>>>>> 
>>>>> I'm going to suggest that the right place to record the information you want is right where the event happens.  Preferably in c code, as performance is easy to mess up, but perhaps with an eBPF mechanism to export it.
>>>>> 
>>>>> In previous Solaris work, I reliably found that exporting kstats was a darn sight harder than collecting them, and in Eric's blog post[1] he notes that converting time is expensive and best done long after collecting, when someone wanted to read the data.
>>>>> 
>>>>> There was an effort to do kstats in Linux[2], but it had supposedly poor performance, and actual trouble when the clock frequency changed.
>>>>> 
>>>>> Is there, in your opinion, a "natural" place to capture state changes to get the data you want, and if so, is it common or similar enough between drivers to be worthy of attention?
>>>>> 
>>>>> --dave
>>>>> 
>>>>> 
>>>>> 
>>>>> References:
>>>>> 
>>>>> Solaris: http://dtrace.org/blogs/eschrock/2004/10/13/microstate-accounting-in-solaris-10/ <https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdtrace.org%2Fblogs%2Feschrock%2F2004%2F10%2F13%2Fmicrostate-accounting-in-solaris-10%2F&data=04%7C01%7C%7C7f7cd5aab2ca42e2e7e908d92d25e27f%7Cb07c069022b843668d8d7b845d088e18%7C1%7C0%7C637590463000477252%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=fdZDOtRCcBk%2BO1ksiTOSU%2FltR8IMwueHyj0kQG4UkHw%3D&reserved=0>
>>>>> A failing Linux effort: https://lwn.net/Articles/127296/ <https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F127296%2F&data=04%7C01%7C%7C7f7cd5aab2ca42e2e7e908d92d25e27f%7Cb07c069022b843668d8d7b845d088e18%7C1%7C0%7C637590463000487248%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=uN0gq8vi0GJHMPpjKVYRjX6G5nQOc%2BugxUwUEk3%2BWJ8%3D&reserved=0>, https://sourceforge.net/projects/microstate/ <https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsourceforge.net%2Fprojects%2Fmicrostate%2F&data=04%7C01%7C%7C7f7cd5aab2ca42e2e7e908d92d25e27f%7Cb07c069022b843668d8d7b845d088e18%7C1%7C0%7C637590463000497242%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=iMNi40Pl9hMmd1h7WrLFP5jmHQ60mJl7zehhO8miJv4%3D&reserved=0>
>>>>> --  
>>>>> David Collier-Brown,         | Always do right. This will gratify
>>>>> System Programmer and Author | some people and astonish the rest
>>>>> davecb at spamcop.net <mailto:davecb at spamcop.net>           |                      -- Mark Twain
>>>> 
>>> _______________________________________________
>>> Starlink mailing list
>>> Starlink at lists.bufferbloat.net <mailto:Starlink at lists.bufferbloat.net>
>>> https://lists.bufferbloat.net/listinfo/starlink <https://lists.bufferbloat.net/listinfo/starlink>
>> _______________________________________________
>> Starlink mailing list
>> Starlink at lists.bufferbloat.net <mailto:Starlink at lists.bufferbloat.net>
>> https://lists.bufferbloat.net/listinfo/starlink
> 
> _______________________________________________
> Starlink mailing list
> Starlink at lists.bufferbloat.net <mailto:Starlink at lists.bufferbloat.net>
> https://lists.bufferbloat.net/listinfo/starlink <https://lists.bufferbloat.net/listinfo/starlink>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/starlink/attachments/20210611/fbe465e7/attachment.html>