[Starlink] Fwd: Microstate Accounting and the Nyquist problem

[Starlink] Fwd: Microstate Accounting and the Nyquist problem

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 2963 bytes --]



> Begin forwarded message:
> 
> From: David Collier-Brown <davecb.42@gmail.com>
> Subject: Microstate Accounting and the Nyquist problem
> Date: June 9, 2021 at 4:44:14 AM PDT
> To: Dave Taht <davet@teklibre.net>
> Cc: Dave Collier-Brown <dave.collier-brown@indexexchange.com>
> Reply-To: davecb@spamcop.net
> 
> A million years ago (roughly around Solaris 9), Sun was suffering from the same problems in measuring their dispatcher as you are with "sloshing".
> 
> A CPU would be 100% busy in one microsecond, 10% busy in the next gazillion, and the average CPU utilization for our sample period would be maybe 10.1, if the sampler happened to sample right when the spike was happening.
> 
> This was utterly useless for things like the fair-share scheduler, so it got fixed in Solaris 10, by having the dispatcher record the time a process (well, kernel thread) had spent in a state when the state changed.
> 
> Initially "microstate accounting" could be toggled on and off, but the branch-around cost more time than always doing the calculation (as discovered by my mad friend Fred) and the kernel folks left it on. It's on to this day.
> 
> In Simon Sundberg's talk, the opportunity to measure occurs every 1,000 packets, when a suitable timestamp is provided. While the eBPF program can look at every packet and do after-the-fact book-keeping in a map, that's only good if the phenomenon you're measuring is persistent enough that it's around for ~2,000 packets.
> 
> I'm going to suggest that the right place to record the information you want is right where the event happens.  Preferably in c code, as performance is easy to mess up, but perhaps with an eBPF mechanism to export it.
> 
> In previous Solaris work, I reliably found that exporting kstats was a darn sight harder than collecting them, and in Eric's blog post[1] he notes that converting time is expensive and best done long after collecting, when someone wanted to read the data.
> 
> There was an effort to do kstats in Linux[2], but it had supposedly poor performance, and actual trouble when the clock frequency changed.
> 
> Is there, in your opinion, a "natural" place to capture state changes to get the data you want, and if so, is it common or similar enough between drivers to be worthy of attention?
> 
> --dave
> 
> 
> 
> References:
> 
> Solaris: http://dtrace.org/blogs/eschrock/2004/10/13/microstate-accounting-in-solaris-10/ <http://dtrace.org/blogs/eschrock/2004/10/13/microstate-accounting-in-solaris-10/> 
> A failing Linux effort: https://lwn.net/Articles/127296/ <https://lwn.net/Articles/127296/>,https://sourceforge.net/projects/microstate/ <https://sourceforge.net/projects/microstate/>
> -- 
> David Collier-Brown,         | Always do right. This will gratify
> System Programmer and Author | some people and astonish the rest
> davecb@spamcop.net <mailto:davecb@spamcop.net>           |                      -- Mark Twain


[-- Attachment #2: Type: text/html, Size: 6586 bytes --]

Re: [Starlink] Fwd: Microstate Accounting and the Nyquist problem

[David Collier-Brown]

[-- Attachment #1: Type: text/plain, Size: 3311 bytes --]

OK, /Oh Smarter Colleagues/, the challenge to you is to say if there is 
a "natural" place to capture state changes to get the data we want, and 
if so, is it common or similar enough between drivers to be worthy of 
attention?

--dave

On 2021-06-09 9:15 a.m., Dave Taht wrote:
>
>
>> Begin forwarded message:
>>
>> *From: *David Collier-Brown <davecb.42@gmail.com 
>> <mailto:davecb.42@gmail.com>>
>> *Subject: **Microstate Accounting and the Nyquist problem*
>> *Date: *June 9, 2021 at 4:44:14 AM PDT
>> *To: *Dave Taht <davet@teklibre.net <mailto:davet@teklibre.net>>
>> *Cc: *Dave Collier-Brown <dave.collier-brown@indexexchange.com 
>> <mailto:dave.collier-brown@indexexchange.com>>
>> *Reply-To: *davecb@spamcop.net <mailto:davecb@spamcop.net>
>>
>> A million years ago (roughly around Solaris 9), Sun was suffering 
>> from the same problems in measuring their dispatcher as you are with 
>> "sloshing".
>>
>> A CPU would be 100% busy in one microsecond, 10% busy in the next 
>> gazillion, and the average CPU utilization for our sample period 
>> would be /maybe/ 10.1, if the sampler happened to sample right when 
>> the spike was happening.
>>
>> This was utterly useless for things like the fair-share scheduler, so 
>> it got fixed in Solaris 10, by having the dispatcher record the time 
>> a process (well, kernel thread) had spent in a state when the state 
>> changed.
>>
>> Initially "microstate accounting" could be toggled on and off, but 
>> the branch-around cost more time than always doing the calculation 
>> (as discovered by my mad friend Fred) and the kernel folks left it 
>> on. It's on to this day.
>>
>> In Simon Sundberg's talk, the opportunity to measure occurs every 
>> 1,000 packets, when a suitable timestamp is provided. While the eBPF 
>> program can look at every packet and do after-the-fact book-keeping 
>> in a map, that's only good if the phenomenon you're measuring is 
>> persistent enough that it's around for ~2,000 packets.
>>
>> I'm going to suggest that the right place to record the information 
>> you want is right where the event happens.  Preferably in c code, as 
>> performance is easy to mess up, but perhaps with an eBPF mechanism to 
>> export it.
>>
>> In previous Solaris work, I reliably found that exporting kstats was 
>> a darn sight harder than collecting them, and in Eric's blog post[1] 
>> he notes that converting time is expensive and best done long after 
>> collecting, when someone wanted to read the data.
>>
>> There was an effort to do kstats in Linux[2], but it had supposedly 
>> poor performance, and actual trouble when the clock frequency changed.
>>
>> Is there, in your opinion, a "natural" place to capture state changes 
>> to get the data you want, and if so, is it common or similar enough 
>> between drivers to be worthy of attention?
>>
>> --dave
>>
>>
>> References:
>>
>>  1. Solaris:
>>     http://dtrace.org/blogs/eschrock/2004/10/13/microstate-accounting-in-solaris-10/
>>
>>  2. A failing Linux effort: https://lwn.net/Articles/127296/,
>>     https://sourceforge.net/projects/microstate/
>>
>> -- 
>> David Collier-Brown,         | Always do right. This will gratify
>> System Programmer and Author | some people and astonish the rest
>> davecb@spamcop.net            |                      -- Mark Twain
>

[-- Attachment #2: Type: text/html, Size: 11179 bytes --]

Re: [Starlink] Fwd: Microstate Accounting and the Nyquist problem

[Mike Puchol]

[-- Attachment #1: Type: text/plain, Size: 4048 bytes --]

We know that Starlink recalculates topology every 15 seconds (this guy, who obviously has way too much spare time, came up with an indirect observation of this interval: https://blog.beerriot.com/2021/02/14/starlink-raster-scan/ )

If we could align with this, we could at least know when potential changes in path delays happen, and try to observe other changes that happen at a similar cadence.

Other thoughts, try to plug more details out of the gRPC data, setup GPS-synced probes with a device at the exit PoP, measure differences between time-sync probes to an array of endpoints.

Has nobody attacked the JTAG connector on a Dishy yet?

Best,

Mike
On Jun 12, 2021, 00:14 +0200, David Collier-Brown <davecb.42@gmail.com>, wrote:
> OK, Oh Smarter Colleagues, the challenge to you is to say if there is a "natural" place to capture state changes to get the data we want, and if so, is it common or similar enough between drivers to be worthy of attention?
> --dave
> On 2021-06-09 9:15 a.m., Dave Taht wrote:
> >
> >
> > > Begin forwarded message:
> > >
> > > From: David Collier-Brown <davecb.42@gmail.com>
> > > Subject: Microstate Accounting and the Nyquist problem
> > > Date: June 9, 2021 at 4:44:14 AM PDT
> > > To: Dave Taht <davet@teklibre.net>
> > > Cc: Dave Collier-Brown <dave.collier-brown@indexexchange.com>
> > > Reply-To: davecb@spamcop.net
> > >
> > > A million years ago (roughly around Solaris 9), Sun was suffering from the same problems in measuring their dispatcher as you are with "sloshing".
> > > A CPU would be 100% busy in one microsecond, 10% busy in the next gazillion, and the average CPU utilization for our sample period would be maybe 10.1, if the sampler happened to sample right when the spike was happening.
> > > This was utterly useless for things like the fair-share scheduler, so it got fixed in Solaris 10, by having the dispatcher record the time a process (well, kernel thread) had spent in a state when the state changed.
> > > Initially "microstate accounting" could be toggled on and off, but the branch-around cost more time than always doing the calculation (as discovered by my mad friend Fred) and the kernel folks left it on. It's on to this day.
> > > In Simon Sundberg's talk, the opportunity to measure occurs every 1,000 packets, when a suitable timestamp is provided. While the eBPF program can look at every packet and do after-the-fact book-keeping in a map, that's only good if the phenomenon you're measuring is persistent enough that it's around for ~2,000 packets.
> > > I'm going to suggest that the right place to record the information you want is right where the event happens.  Preferably in c code, as performance is easy to mess up, but perhaps with an eBPF mechanism to export it.
> > > In previous Solaris work, I reliably found that exporting kstats was a darn sight harder than collecting them, and in Eric's blog post[1] he notes that converting time is expensive and best done long after collecting, when someone wanted to read the data.
> > > There was an effort to do kstats in Linux[2], but it had supposedly poor performance, and actual trouble when the clock frequency changed.
> > > Is there, in your opinion, a "natural" place to capture state changes to get the data you want, and if so, is it common or similar enough between drivers to be worthy of attention?
> > > --dave
> > >
> > > References:
> > >
> > > 1. Solaris: http://dtrace.org/blogs/eschrock/2004/10/13/microstate-accounting-in-solaris-10/
> > > 2. A failing Linux effort: https://lwn.net/Articles/127296/, https://sourceforge.net/projects/microstate/
> > >
> > > --
> > > David Collier-Brown,         | Always do right. This will gratify
> > > System Programmer and Author | some people and astonish the rest
> > > davecb@spamcop.net           |                      -- Mark Twain
> >
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink

[-- Attachment #2: Type: text/html, Size: 10062 bytes --]

Re: [Starlink] Microstate Accounting and the Nyquist problem

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 6172 bytes --]



> On Jun 11, 2021, at 3:34 PM, Mike Puchol <mike@starlink.sx> wrote:
> 
> We know that Starlink recalculates topology every 15 seconds (this guy, who obviously has way too much spare time, came up with an indirect observation of this interval: https://blog.beerriot.com/2021/02/14/starlink-raster-scan/ <https://blog.beerriot.com/2021/02/14/starlink-raster-scan/> )
> 
> If we could align with this, we could at least know when potential changes in path delays happen, and try to observe other changes that happen at a similar cadence.
> 
> Other thoughts, try to plug more details out of the gRPC data, setup GPS-synced probes with a device at the exit PoP, measure differences between time-sync probes to an array of endpoints.
> 

It’s ironic that the device has to have gps in it, and thus should be  able to provide perfect time to clients directly behind it, isn’t.

I haven’t captured a dhcp or dhcpv6 transaction yet myself,
do they have a ntp option?

What gps software or driver might they have used? (esr’s gpsd is quite popular, but there are others) 

What’s the gps chip?


> Has nobody attacked the JTAG connector on a Dishy yet?
> 
> Best,
> 
> Mike
> On Jun 12, 2021, 00:14 +0200, David Collier-Brown <davecb.42@gmail.com>, wrote:
>> OK, Oh Smarter Colleagues, the challenge to you is to say if there is a "natural" place to capture state changes to get the data we want, and if so, is it common or similar enough between drivers to be worthy of attention?
>> 
>> --dave
>> 
>> On 2021-06-09 9:15 a.m., Dave Taht wrote:
>>> 
>>> 
>>>> Begin forwarded message:
>>>> 
>>>> From: David Collier-Brown <davecb.42@gmail.com <mailto:davecb.42@gmail.com>>
>>>> Subject: Microstate Accounting and the Nyquist problem
>>>> Date: June 9, 2021 at 4:44:14 AM PDT
>>>> To: Dave Taht <davet@teklibre.net <mailto:davet@teklibre.net>>
>>>> Cc: Dave Collier-Brown <dave.collier-brown@indexexchange.com <mailto:dave.collier-brown@indexexchange.com>>
>>>> Reply-To: davecb@spamcop.net <mailto:davecb@spamcop.net>
>>>> 
>>>> A million years ago (roughly around Solaris 9), Sun was suffering from the same problems in measuring their dispatcher as you are with "sloshing".
>>>> 
>>>> A CPU would be 100% busy in one microsecond, 10% busy in the next gazillion, and the average CPU utilization for our sample period would be maybe 10.1, if the sampler happened to sample right when the spike was happening.
>>>> 
>>>> This was utterly useless for things like the fair-share scheduler, so it got fixed in Solaris 10, by having the dispatcher record the time a process (well, kernel thread) had spent in a state when the state changed.
>>>> 
>>>> Initially "microstate accounting" could be toggled on and off, but the branch-around cost more time than always doing the calculation (as discovered by my mad friend Fred) and the kernel folks left it on. It's on to this day.
>>>> 
>>>> In Simon Sundberg's talk, the opportunity to measure occurs every 1,000 packets, when a suitable timestamp is provided. While the eBPF program can look at every packet and do after-the-fact book-keeping in a map, that's only good if the phenomenon you're measuring is persistent enough that it's around for ~2,000 packets.
>>>> 
>>>> I'm going to suggest that the right place to record the information you want is right where the event happens.  Preferably in c code, as performance is easy to mess up, but perhaps with an eBPF mechanism to export it.
>>>> 
>>>> In previous Solaris work, I reliably found that exporting kstats was a darn sight harder than collecting them, and in Eric's blog post[1] he notes that converting time is expensive and best done long after collecting, when someone wanted to read the data.
>>>> 
>>>> There was an effort to do kstats in Linux[2], but it had supposedly poor performance, and actual trouble when the clock frequency changed.
>>>> 
>>>> Is there, in your opinion, a "natural" place to capture state changes to get the data you want, and if so, is it common or similar enough between drivers to be worthy of attention?
>>>> 
>>>> --dave
>>>> 
>>>> 
>>>> 
>>>> References:
>>>> 
>>>> Solaris: http://dtrace.org/blogs/eschrock/2004/10/13/microstate-accounting-in-solaris-10/ <https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdtrace.org%2Fblogs%2Feschrock%2F2004%2F10%2F13%2Fmicrostate-accounting-in-solaris-10%2F&data=04%7C01%7C%7C7f7cd5aab2ca42e2e7e908d92d25e27f%7Cb07c069022b843668d8d7b845d088e18%7C1%7C0%7C637590463000477252%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=fdZDOtRCcBk%2BO1ksiTOSU%2FltR8IMwueHyj0kQG4UkHw%3D&reserved=0>
>>>> A failing Linux effort: https://lwn.net/Articles/127296/ <https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F127296%2F&data=04%7C01%7C%7C7f7cd5aab2ca42e2e7e908d92d25e27f%7Cb07c069022b843668d8d7b845d088e18%7C1%7C0%7C637590463000487248%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=uN0gq8vi0GJHMPpjKVYRjX6G5nQOc%2BugxUwUEk3%2BWJ8%3D&reserved=0>, https://sourceforge.net/projects/microstate/ <https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsourceforge.net%2Fprojects%2Fmicrostate%2F&data=04%7C01%7C%7C7f7cd5aab2ca42e2e7e908d92d25e27f%7Cb07c069022b843668d8d7b845d088e18%7C1%7C0%7C637590463000497242%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=iMNi40Pl9hMmd1h7WrLFP5jmHQ60mJl7zehhO8miJv4%3D&reserved=0>
>>>> --  
>>>> David Collier-Brown,         | Always do right. This will gratify
>>>> System Programmer and Author | some people and astonish the rest
>>>> davecb@spamcop.net <mailto:davecb@spamcop.net>           |                      -- Mark Twain
>>> 
>> _______________________________________________
>> Starlink mailing list
>> Starlink@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/starlink
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink


[-- Attachment #2: Type: text/html, Size: 11756 bytes --]

Re: [Starlink] Microstate Accounting and the Nyquist problem

[Nathan Owens]

[-- Attachment #1: Type: text/plain, Size: 6517 bytes --]

Ha, that’s a great point about the GPS/NTP thing.

GPS chip, not sure, there’s only one tear down I’ve seen, here’s the gps
chip:
https://cdn.arstechnica.net/wp-content/uploads/2020/12/starlink-28-gps-receiver.jpg


No idea about the software, maybe I’ll try to JTAG mine.



On Fri, Jun 11, 2021 at 3:39 PM Dave Taht <davet@teklibre.net> wrote:

>
>
> On Jun 11, 2021, at 3:34 PM, Mike Puchol <mike@starlink.sx> wrote:
>
> We know that Starlink recalculates topology every 15 seconds (this guy,
> who obviously has way too much spare time, came up with an indirect
> observation of this interval:
> https://blog.beerriot.com/2021/02/14/starlink-raster-scan/ )
>
> If we could align with this, we could at least know when potential changes
> in path delays happen, and try to observe other changes that happen at a
> similar cadence.
>
> Other thoughts, try to plug more details out of the gRPC data, setup
> GPS-synced probes with a device at the exit PoP, measure differences
> between time-sync probes to an array of endpoints.
>
>
> It’s ironic that the device has to have gps in it, and thus should be
>  able to provide perfect time to clients directly behind it, isn’t.
>
> I haven’t captured a dhcp or dhcpv6 transaction yet myself,
> do they have a ntp option?
>
> What gps software or driver might they have used? (esr’s gpsd is quite
> popular, but there are others)
>
> What’s the gps chip?
>
>
> Has nobody attacked the JTAG connector on a Dishy yet?
>
> Best,
>
> Mike
> On Jun 12, 2021, 00:14 +0200, David Collier-Brown <davecb.42@gmail.com>,
> wrote:
>
> OK, *Oh Smarter Colleagues*, the challenge to you is to say if there is a
> "natural" place to capture state changes to get the data we want, and if
> so, is it common or similar enough between drivers to be worthy of
> attention?
>
> --dave
> On 2021-06-09 9:15 a.m., Dave Taht wrote:
>
>
>
> Begin forwarded message:
>
> *From:* David Collier-Brown <davecb.42@gmail.com>
> *Subject:* *Microstate Accounting and the Nyquist problem*
> *Date:* June 9, 2021 at 4:44:14 AM PDT
> *To:* Dave Taht <davet@teklibre.net>
> *Cc:* Dave Collier-Brown <dave.collier-brown@indexexchange.com>
> *Reply-To:* davecb@spamcop.net
>
> A million years ago (roughly around Solaris 9), Sun was suffering from the
> same problems in measuring their dispatcher as you are with "sloshing".
>
> A CPU would be 100% busy in one microsecond, 10% busy in the next
> gazillion, and the average CPU utilization for our sample period would be
> *maybe* 10.1, if the sampler happened to sample right when the spike was
> happening.
>
> This was utterly useless for things like the fair-share scheduler, so it
> got fixed in Solaris 10, by having the dispatcher record the time a process
> (well, kernel thread) had spent in a state when the state changed.
>
> Initially "microstate accounting" could be toggled on and off, but the
> branch-around cost more time than always doing the calculation (as
> discovered by my mad friend Fred) and the kernel folks left it on. It's on
> to this day.
>
> In Simon Sundberg's talk, the opportunity to measure occurs every 1,000
> packets, when a suitable timestamp is provided. While the eBPF program can
> look at every packet and do after-the-fact book-keeping in a map, that's
> only good if the phenomenon you're measuring is persistent enough that it's
> around for ~2,000 packets.
>
> I'm going to suggest that the right place to record the information you
> want is right where the event happens.  Preferably in c code, as
> performance is easy to mess up, but perhaps with an eBPF mechanism to
> export it.
>
> In previous Solaris work, I reliably found that exporting kstats was a
> darn sight harder than collecting them, and in Eric's blog post[1] he notes
> that converting time is expensive and best done long after collecting, when
> someone wanted to read the data.
>
> There was an effort to do kstats in Linux[2], but it had supposedly poor
> performance, and actual trouble when the clock frequency changed.
>
> Is there, in your opinion, a "natural" place to capture state changes to
> get the data you want, and if so, is it common or similar enough between
> drivers to be worthy of attention?
>
> --dave
>
>
> References:
>
>    1. Solaris:
>    http://dtrace.org/blogs/eschrock/2004/10/13/microstate-accounting-in-solaris-10/
>    <https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdtrace.org%2Fblogs%2Feschrock%2F2004%2F10%2F13%2Fmicrostate-accounting-in-solaris-10%2F&data=04%7C01%7C%7C7f7cd5aab2ca42e2e7e908d92d25e27f%7Cb07c069022b843668d8d7b845d088e18%7C1%7C0%7C637590463000477252%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=fdZDOtRCcBk%2BO1ksiTOSU%2FltR8IMwueHyj0kQG4UkHw%3D&reserved=0>
>    2. A failing Linux effort: https://lwn.net/Articles/127296/
>    <https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F127296%2F&data=04%7C01%7C%7C7f7cd5aab2ca42e2e7e908d92d25e27f%7Cb07c069022b843668d8d7b845d088e18%7C1%7C0%7C637590463000487248%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=uN0gq8vi0GJHMPpjKVYRjX6G5nQOc%2BugxUwUEk3%2BWJ8%3D&reserved=0>,
>    https://sourceforge.net/projects/microstate/
>    <https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsourceforge.net%2Fprojects%2Fmicrostate%2F&data=04%7C01%7C%7C7f7cd5aab2ca42e2e7e908d92d25e27f%7Cb07c069022b843668d8d7b845d088e18%7C1%7C0%7C637590463000497242%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=iMNi40Pl9hMmd1h7WrLFP5jmHQ60mJl7zehhO8miJv4%3D&reserved=0>
>
> --
> David Collier-Brown,         | Always do right. This will gratify
> System Programmer and Author | some people and astonish the restdavecb@spamcop.net           |                      -- Mark Twain
>
>
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
>
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
>
>
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
>

[-- Attachment #2: Type: text/html, Size: 11635 bytes --]

Re: [Starlink] Microstate Accounting and the Nyquist problem

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 8308 bytes --]



> On Jun 11, 2021, at 3:39 PM, Dave Taht <davet@teklibre.net> wrote:
> 
> 
> 
>> On Jun 11, 2021, at 3:34 PM, Mike Puchol <mike@starlink.sx <mailto:mike@starlink.sx>> wrote:
>> 
>> We know that Starlink recalculates topology every 15 seconds (this guy, who obviously has way too much spare time, came up with an indirect observation of this interval: https://blog.beerriot.com/2021/02/14/starlink-raster-scan/ <https://blog.beerriot.com/2021/02/14/starlink-raster-scan/> )
>> 
>> If we could align with this, we could at least know when potential changes in path delays happen, and try to observe other changes that happen at a similar cadence.
>> 
>> Other thoughts, try to plug more details out of the gRPC data, setup GPS-synced probes with a device at the exit PoP, measure differences between time-sync probes to an array of endpoints.
>> 
> 
> It’s ironic that the device has to have gps in it, and thus should be  able to provide perfect time to clients directly behind it, isn’t.
> 
> I haven’t captured a dhcp or dhcpv6 transaction yet myself,
> do they have a ntp option?
> 
> What gps software or driver might they have used? (esr’s gpsd is quite popular, but there are others) 
> 
> What’s the gps chip?
> 

It would be good to have solid time everywhere, as I am seeing clocks not synced even close to 40ms accuracy of late.

BTW, Eric Raymond (esr) is also one of the driving forces behind ntpsec, along with gary and a few other people now on our list.

For more details, see:
https://www.ntpsec.org/

Once upon a time, I sat in esr's basement hearing him rip much crud out of the old ntpd codebase over the course of a very few days. The shouts “What? WHAAAT?” and most of the other pithy comments he made never made the git log. 

Over the years following the codebase got better and better, but adoption has been slow.

An intro to that woefully underfunded project:

NTPsec project - a secure, hardened, and improved implementation of Network Time Protocol derived from NTP Classic, Dave Mills’s original.
NTPsec, as its name implies, is a more secure NTP. Our goal is to deliver code that can be used with confidence in deployments with the most stringent security, availability, and assurance requirements.
Towards that end we apply best practices and state-of-the art technology in code auditing, verification, and testing. We begin with the most important best practice: true open-source code review. The NTPsec code is available in a public git repository. One of our goals is to support broader community participation.

> 
>> Has nobody attacked the JTAG connector on a Dishy yet?

I reached out to one of the teardown folk (mike (mikeonsoftware?)) months ago to get the debris but the rightest answer was to drill down into it on a still-alive ones.

>> 
>> Best,
>> 
>> Mike
>> On Jun 12, 2021, 00:14 +0200, David Collier-Brown <davecb.42@gmail.com <mailto:davecb.42@gmail.com>>, wrote:
>>> OK, Oh Smarter Colleagues, the challenge to you is to say if there is a "natural" place to capture state changes to get the data we want, and if so, is it common or similar enough between drivers to be worthy of attention?
>>> 
>>> --dave
>>> 
>>> On 2021-06-09 9:15 a.m., Dave Taht wrote:
>>>> 
>>>> 
>>>>> Begin forwarded message:
>>>>> 
>>>>> From: David Collier-Brown <davecb.42@gmail.com <mailto:davecb.42@gmail.com>>
>>>>> Subject: Microstate Accounting and the Nyquist problem
>>>>> Date: June 9, 2021 at 4:44:14 AM PDT
>>>>> To: Dave Taht <davet@teklibre.net <mailto:davet@teklibre.net>>
>>>>> Cc: Dave Collier-Brown <dave.collier-brown@indexexchange.com <mailto:dave.collier-brown@indexexchange.com>>
>>>>> Reply-To: davecb@spamcop.net <mailto:davecb@spamcop.net>
>>>>> 
>>>>> A million years ago (roughly around Solaris 9), Sun was suffering from the same problems in measuring their dispatcher as you are with "sloshing".
>>>>> 
>>>>> A CPU would be 100% busy in one microsecond, 10% busy in the next gazillion, and the average CPU utilization for our sample period would be maybe 10.1, if the sampler happened to sample right when the spike was happening.
>>>>> 
>>>>> This was utterly useless for things like the fair-share scheduler, so it got fixed in Solaris 10, by having the dispatcher record the time a process (well, kernel thread) had spent in a state when the state changed.
>>>>> 
>>>>> Initially "microstate accounting" could be toggled on and off, but the branch-around cost more time than always doing the calculation (as discovered by my mad friend Fred) and the kernel folks left it on. It's on to this day.
>>>>> 
>>>>> In Simon Sundberg's talk, the opportunity to measure occurs every 1,000 packets, when a suitable timestamp is provided. While the eBPF program can look at every packet and do after-the-fact book-keeping in a map, that's only good if the phenomenon you're measuring is persistent enough that it's around for ~2,000 packets.
>>>>> 
>>>>> I'm going to suggest that the right place to record the information you want is right where the event happens.  Preferably in c code, as performance is easy to mess up, but perhaps with an eBPF mechanism to export it.
>>>>> 
>>>>> In previous Solaris work, I reliably found that exporting kstats was a darn sight harder than collecting them, and in Eric's blog post[1] he notes that converting time is expensive and best done long after collecting, when someone wanted to read the data.
>>>>> 
>>>>> There was an effort to do kstats in Linux[2], but it had supposedly poor performance, and actual trouble when the clock frequency changed.
>>>>> 
>>>>> Is there, in your opinion, a "natural" place to capture state changes to get the data you want, and if so, is it common or similar enough between drivers to be worthy of attention?
>>>>> 
>>>>> --dave
>>>>> 
>>>>> 
>>>>> 
>>>>> References:
>>>>> 
>>>>> Solaris: http://dtrace.org/blogs/eschrock/2004/10/13/microstate-accounting-in-solaris-10/ <https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdtrace.org%2Fblogs%2Feschrock%2F2004%2F10%2F13%2Fmicrostate-accounting-in-solaris-10%2F&data=04%7C01%7C%7C7f7cd5aab2ca42e2e7e908d92d25e27f%7Cb07c069022b843668d8d7b845d088e18%7C1%7C0%7C637590463000477252%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=fdZDOtRCcBk%2BO1ksiTOSU%2FltR8IMwueHyj0kQG4UkHw%3D&reserved=0>
>>>>> A failing Linux effort: https://lwn.net/Articles/127296/ <https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F127296%2F&data=04%7C01%7C%7C7f7cd5aab2ca42e2e7e908d92d25e27f%7Cb07c069022b843668d8d7b845d088e18%7C1%7C0%7C637590463000487248%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=uN0gq8vi0GJHMPpjKVYRjX6G5nQOc%2BugxUwUEk3%2BWJ8%3D&reserved=0>, https://sourceforge.net/projects/microstate/ <https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsourceforge.net%2Fprojects%2Fmicrostate%2F&data=04%7C01%7C%7C7f7cd5aab2ca42e2e7e908d92d25e27f%7Cb07c069022b843668d8d7b845d088e18%7C1%7C0%7C637590463000497242%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=iMNi40Pl9hMmd1h7WrLFP5jmHQ60mJl7zehhO8miJv4%3D&reserved=0>
>>>>> --  
>>>>> David Collier-Brown,         | Always do right. This will gratify
>>>>> System Programmer and Author | some people and astonish the rest
>>>>> davecb@spamcop.net <mailto:davecb@spamcop.net>           |                      -- Mark Twain
>>>> 
>>> _______________________________________________
>>> Starlink mailing list
>>> Starlink@lists.bufferbloat.net <mailto:Starlink@lists.bufferbloat.net>
>>> https://lists.bufferbloat.net/listinfo/starlink <https://lists.bufferbloat.net/listinfo/starlink>
>> _______________________________________________
>> Starlink mailing list
>> Starlink@lists.bufferbloat.net <mailto:Starlink@lists.bufferbloat.net>
>> https://lists.bufferbloat.net/listinfo/starlink
> 
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net <mailto:Starlink@lists.bufferbloat.net>
> https://lists.bufferbloat.net/listinfo/starlink <https://lists.bufferbloat.net/listinfo/starlink>

[-- Attachment #2: Type: text/html, Size: 22100 bytes --]

Re: [Starlink] Microstate Accounting and the Nyquist problem

[Mike Puchol]

[-- Attachment #1: Type: text/plain, Size: 7262 bytes --]

So the GPS is an STA8090 automotive-grade, makes sense as Dishy gets proper hot! Also supports sensor fusion for eg pointing in the right azimuth. https://www.st.com/en/automotive-infotainment-and-telematics/gnss-ics.html

However, the beauty of GPS sync is we can choose our own time base external to Dishy :-)

Best,

Mike
On Jun 12, 2021, 01:00 +0200, Dave Taht <davet@teklibre.net>, wrote:
>
>
> > On Jun 11, 2021, at 3:39 PM, Dave Taht <davet@teklibre.net> wrote:
> >
> >
> >
> > > On Jun 11, 2021, at 3:34 PM, Mike Puchol <mike@starlink.sx> wrote:
> > >
> > > We know that Starlink recalculates topology every 15 seconds (this guy, who obviously has way too much spare time, came up with an indirect observation of this interval: https://blog.beerriot.com/2021/02/14/starlink-raster-scan/ )
> > >
> > > If we could align with this, we could at least know when potential changes in path delays happen, and try to observe other changes that happen at a similar cadence.
> > >
> > > Other thoughts, try to plug more details out of the gRPC data, setup GPS-synced probes with a device at the exit PoP, measure differences between time-sync probes to an array of endpoints.
> > >
> >
> > It’s ironic that the device has to have gps in it, and thus should be  able to provide perfect time to clients directly behind it, isn’t.
> >
> > I haven’t captured a dhcp or dhcpv6 transaction yet myself,
> > do they have a ntp option?
> >
> > What gps software or driver might they have used? (esr’s gpsd is quite popular, but there are others)
> >
> > What’s the gps chip?
> >
>
> It would be good to have solid time everywhere, as I am seeing clocks not synced even close to 40ms accuracy of late.
>
> BTW, Eric Raymond (esr) is also one of the driving forces behind ntpsec, along with gary and a few other people now on our list.
>
> For more details, see:
> https://www.ntpsec.org/
>
> Once upon a time, I sat in esr's basement hearing him rip much crud out of the old ntpd codebase over the course of a very few days. The shouts “What? WHAAAT?” and most of the other pithy comments he made never made the git log.
>
> Over the years following the codebase got better and better, but adoption has been slow.
>
> An intro to that woefully underfunded project:
>
> NTPsec project - a secure, hardened, and improved implementation of Network Time Protocol derived from NTP Classic, Dave Mills’s original.
> NTPsec, as its name implies, is a more secure NTP. Our goal is to deliver code that can be used with confidence in deployments with the most stringent security, availability, and assurance requirements.
> Towards that end we apply best practices and state-of-the art technology in code auditing, verification, and testing. We begin with the most important best practice: true open-source code review. The NTPsec code is available in a public git repository. One of our goals is to support broader community participation.
>
> >
> > > Has nobody attacked the JTAG connector on a Dishy yet?
>
> I reached out to one of the teardown folk (mike (mikeonsoftware?)) months ago to get the debris but the rightest answer was to drill down into it on a still-alive ones.
>
> > >
> > > Best,
> > >
> > > Mike
> > > On Jun 12, 2021, 00:14 +0200, David Collier-Brown <davecb.42@gmail.com>, wrote:
> > > > OK, Oh Smarter Colleagues, the challenge to you is to say if there is a "natural" place to capture state changes to get the data we want, and if so, is it common or similar enough between drivers to be worthy of attention?
> > > > --dave
> > > > On 2021-06-09 9:15 a.m., Dave Taht wrote:
> > > > >
> > > > >
> > > > > > Begin forwarded message:
> > > > > >
> > > > > > From: David Collier-Brown <davecb.42@gmail.com>
> > > > > > Subject: Microstate Accounting and the Nyquist problem
> > > > > > Date: June 9, 2021 at 4:44:14 AM PDT
> > > > > > To: Dave Taht <davet@teklibre.net>
> > > > > > Cc: Dave Collier-Brown <dave.collier-brown@indexexchange.com>
> > > > > > Reply-To: davecb@spamcop.net
> > > > > >
> > > > > > A million years ago (roughly around Solaris 9), Sun was suffering from the same problems in measuring their dispatcher as you are with "sloshing".
> > > > > > A CPU would be 100% busy in one microsecond, 10% busy in the next gazillion, and the average CPU utilization for our sample period would be maybe 10.1, if the sampler happened to sample right when the spike was happening.
> > > > > > This was utterly useless for things like the fair-share scheduler, so it got fixed in Solaris 10, by having the dispatcher record the time a process (well, kernel thread) had spent in a state when the state changed.
> > > > > > Initially "microstate accounting" could be toggled on and off, but the branch-around cost more time than always doing the calculation (as discovered by my mad friend Fred) and the kernel folks left it on. It's on to this day.
> > > > > > In Simon Sundberg's talk, the opportunity to measure occurs every 1,000 packets, when a suitable timestamp is provided. While the eBPF program can look at every packet and do after-the-fact book-keeping in a map, that's only good if the phenomenon you're measuring is persistent enough that it's around for ~2,000 packets.
> > > > > > I'm going to suggest that the right place to record the information you want is right where the event happens.  Preferably in c code, as performance is easy to mess up, but perhaps with an eBPF mechanism to export it.
> > > > > > In previous Solaris work, I reliably found that exporting kstats was a darn sight harder than collecting them, and in Eric's blog post[1] he notes that converting time is expensive and best done long after collecting, when someone wanted to read the data.
> > > > > > There was an effort to do kstats in Linux[2], but it had supposedly poor performance, and actual trouble when the clock frequency changed.
> > > > > > Is there, in your opinion, a "natural" place to capture state changes to get the data you want, and if so, is it common or similar enough between drivers to be worthy of attention?
> > > > > > --dave
> > > > > >
> > > > > > References:
> > > > > >
> > > > > > 1. Solaris: http://dtrace.org/blogs/eschrock/2004/10/13/microstate-accounting-in-solaris-10/
> > > > > > 2. A failing Linux effort: https://lwn.net/Articles/127296/, https://sourceforge.net/projects/microstate/
> > > > > >
> > > > > > --
> > > > > > David Collier-Brown,         | Always do right. This will gratify
> > > > > > System Programmer and Author | some people and astonish the rest
> > > > > > davecb@spamcop.net           |                      -- Mark Twain
> > > > >
> > > > _______________________________________________
> > > > Starlink mailing list
> > > > Starlink@lists.bufferbloat.net
> > > > https://lists.bufferbloat.net/listinfo/starlink
> > > _______________________________________________
> > > Starlink mailing list
> > > Starlink@lists.bufferbloat.net
> > > https://lists.bufferbloat.net/listinfo/starlink
> >
> > _______________________________________________
> > Starlink mailing list
> > Starlink@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/starlink
>

[-- Attachment #2: Type: text/html, Size: 23297 bytes --]

[Starlink] NTP implementations [was: Microstate Accounting and the Nyquist problem]

[Juliusz Chroboczek]

> Once upon a time, I sat in esr's basement hearing him rip much crud out of the
> old ntpd codebase over the course of a very few days. The shouts “What?
> WHAAAT?” and most of the other pithy comments he made never made the git log.

Yeah, it's a university project -- generations after generations of
students who hack on the codebase during summer projects.  I have some
experience with such ;-)

> Over the years following the codebase got better and better, but adoption has
> been slow.

The potential userbase might be using chrony, which has set a rather high
bar to compete against (I've seen it behave beautifully on some pretty
broken hardware).

-- Juliusz

Re: [Starlink] Microstate Accounting and the Nyquist problem

[Michael Richardson]

Dave Taht <davet@teklibre.net> wrote:
    > It’s ironic that the device has to have gps in it, and thus should be
    > able to provide perfect time to clients directly behind it, isn’t.

Couldn't starlink satellites *also* provide a GPS reference?
They are lower and way more of them...

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [ 
	

Re: [Starlink] Microstate Accounting and the Nyquist problem

[Mike Puchol]

[-- Attachment #1: Type: text/plain, Size: 746 bytes --]

The basis for the GPS time sync is an atomic clock on board each satellite. I very much doubt Starlink is placing one of those in their birds :-)

Best,

Mike
On Jun 12, 2021, 16:00 +0200, Michael Richardson <mcr@sandelman.ca>, wrote:
>
> Dave Taht <davet@teklibre.net> wrote:
> > It’s ironic that the device has to have gps in it, and thus should be
> > able to provide perfect time to clients directly behind it, isn’t.
>
> Couldn't starlink satellites *also* provide a GPS reference?
> They are lower and way more of them...
>
> --
> ] Never tell me the odds! | ipv6 mesh networks [
> ] Michael Richardson, Sandelman Software Works | network architect [
> ] mcr@sandelman.ca http://www.sandelman.ca/ | ruby on rails [
>

[-- Attachment #2: Type: text/html, Size: 1278 bytes --]

Re: [Starlink] Microstate Accounting and the Nyquist problem

[Nathan Owens]

[-- Attachment #1: Type: text/plain, Size: 1091 bytes --]

You can buy a coin sized chip-scale atomic clock, so it wouldn’t surprise
me.

On Sat, Jun 12, 2021 at 9:15 AM Mike Puchol <mike@starlink.sx> wrote:

> The basis for the GPS time sync is an atomic clock on board each
> satellite. I very much doubt Starlink is placing one of those in their
> birds :-)
>
> Best,
>
> Mike
> On Jun 12, 2021, 16:00 +0200, Michael Richardson <mcr@sandelman.ca>,
> wrote:
>
>
> Dave Taht <davet@teklibre.net> wrote:
>
> It’s ironic that the device has to have gps in it, and thus should be
> able to provide perfect time to clients directly behind it, isn’t.
>
>
> Couldn't starlink satellites *also* provide a GPS reference?
> They are lower and way more of them...
>
> --
> ] Never tell me the odds! | ipv6 mesh networks [
> ] Michael Richardson, Sandelman Software Works | network architect [
> ] mcr@sandelman.ca http://www.sandelman.ca/ | ruby on rails [
>
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
>

[-- Attachment #2: Type: text/html, Size: 2201 bytes --]

Re: [Starlink] Microstate Accounting and the Nyquist problem

[Mike Puchol]

[-- Attachment #1: Type: text/plain, Size: 1561 bytes --]

Duh… you are correct, I was still on the desktop-sizes ones… but would you need one? What I can find is not cheap. It would be interesting to work the math eg for Doppler shift compensation which could require better accuracy than a “normal” GPS synced timing source.

Best,

Mike
On Jun 12, 2021, 18:43 +0200, Nathan Owens <nathan@nathan.io>, wrote:
> You can buy a coin sized chip-scale atomic clock, so it wouldn’t surprise me.
>
> > On Sat, Jun 12, 2021 at 9:15 AM Mike Puchol <mike@starlink.sx> wrote:
> > > The basis for the GPS time sync is an atomic clock on board each satellite. I very much doubt Starlink is placing one of those in their birds :-)
> > >
> > > Best,
> > >
> > > Mike
> > > On Jun 12, 2021, 16:00 +0200, Michael Richardson <mcr@sandelman.ca>, wrote:
> > > >
> > > > Dave Taht <davet@teklibre.net> wrote:
> > > > > It’s ironic that the device has to have gps in it, and thus should be
> > > > > able to provide perfect time to clients directly behind it, isn’t.
> > > >
> > > > Couldn't starlink satellites *also* provide a GPS reference?
> > > > They are lower and way more of them...
> > > >
> > > > --
> > > > ] Never tell me the odds! | ipv6 mesh networks [
> > > > ] Michael Richardson, Sandelman Software Works | network architect [
> > > > ] mcr@sandelman.ca http://www.sandelman.ca/ | ruby on rails [
> > > >
> > > _______________________________________________
> > > Starlink mailing list
> > > Starlink@lists.bufferbloat.net
> > > https://lists.bufferbloat.net/listinfo/starlink

[-- Attachment #2: Type: text/html, Size: 3059 bytes --]

Re: [Starlink] Microstate Accounting and the Nyquist problem

[Gary E. Miller]

[-- Attachment #1: Type: text/plain, Size: 1621 bytes --]

Yo Mike!

On Sat, 12 Jun 2021 21:05:19 +0200
Mike Puchol <mike@starlink.sx> wrote:

> Duh… you are correct, I was still on the desktop-sizes ones… but
> would you need one? What I can find is not cheap. It would be
> interesting to work the math eg for Doppler shift compensation which
> could require better accuracy than a “normal” GPS synced timing
> source.

GNSS satellites have many atomic clocks.  Usually three, to start
with.  Some Ceasium, some Rubidium.  They know what time it is, but not
where they are.  Solar winds psuh them around by several meters.  So
ground stations bounce lasers off the sats to reduce the positiion
uncertainty.

After that, the Ionosphere and Troposphere uncertainty dominates.

The doppler can be +/- 10Khz, but the receiver bandwidth is about 250
hz.  So you need to know the doppler before you can lock on the signal
from the bird.

Starlink must have even greater doppler, compensated by a wider receiver
bandwidth.

No way you are improving on what the GNSS folks do.  A lot of black
magic invovled.

For well under $100 you can get a receiver solution that will asily
surpass the clock resolution in a Linux host (<50 ns).  The actual
PPS will be around 15ns.  That can be improved with a bit of external
hardware.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem@rellim.com  Tel:+1 541 382 8588

	    Veritas liberabit vos. -- Quid est veritas?
    "If you can't measure it, you can't improve it." - Lord Kelvin

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 851 bytes --]

Re: [Starlink] Fwd: Microstate Accounting and the Nyquist problem

[Karl Auerbach]

Measuring things across network is indeed made a lot easier if we have a 
solid, precise, synchronized clock at the end points.

(I would have thought that Starlink would have had a built-in clock 
based on the resonance of the heavy isotope of Elonmuskium.)

HP once upon a time had a patent on using GPS to get that time sync for 
doing network measurements.  But that was in the 1980's. One hopes that 
that patent has long since faded into dust.

(Not long ago I wrote a fairly simple tool that runs on Linux on cheap 
hardware, I call it "C-Clamp" that wraps around a device or network 
(such as a satellite uplink and downlink) and sends packets to itself 
and collects numbers so it can generate statistics about the amount of 
delay, its stability (are there bursts, and if so, how big and how 
often), losses, etc etc.  Since it has a single clock it can get some 
fairly high precision time measurements.)  I thought it would be fun to 
slap one of these around two Starlink dishes located at the same 
location (or within a distance I could cross with a single Ethernet cable.)

I've long been intrigued by all the interesting stuff that astronomers 
derive by watching deviations of periodic or predictable events.  
There's an astrophysicist at UC Santa Cruz who apparently is interested, 
as am I, whether we can apply those techniques to measuring paths on 
networks.

For one project I was going to buy a big bag of ESP32s (fun little 
systems-on-a-chip) and use them as network pulsars that would emit a 
predictable pattern of packets of various size, times (and intervals, 
sometimes bunched together sometimes widely separated.)  A receiver 
could lock onto the pattern and from the dissonance between the emission 
times and the reception times could intuit things about the path.

I understand that RIPE has done something like this.

     --karl--