From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io1-xd29.google.com (mail-io1-xd29.google.com [IPv6:2607:f8b0:4864:20::d29]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 9767A3B2A4 for ; Fri, 11 Jun 2021 19:00:08 -0400 (EDT) Received: by mail-io1-xd29.google.com with SMTP id a6so32905813ioe.0 for ; Fri, 11 Jun 2021 16:00:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nathan.io; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=3lwdxCiT7im2Sqi5yOboTg66pKyVHv+Xclo88Y1lzCo=; b=b3NaQXYgPKywzcIW208gJMmobEc9hUiyJT3w+oJc6wvvwJZOSbPeDuYEJrZ5OxIaI+ Hlco/MIEIHkS899iv1d7iGQ/TX5m+2hi3P1mHg6J9WQe/2e2rTSZAao4Ti2Cxot1pefx M6OVGjvUxQjMyH7mU4HSdomffnWFfZ5h+ZhoDFnto9ItPa1Iv8HuTAHoXTkJdE4G5Kll CH4m5/ZI/01briRREe8JDRpLDBDvGbF0HTVce0iCyx7azAxqug8F0+p5CMY50wBWSaYJ wEvDrd3gTvZUUqXH539X602mzoSaBTyk7YEQYhh3PorArU6i8XSOmk7/PAHUr4Dy4Izj wmnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=3lwdxCiT7im2Sqi5yOboTg66pKyVHv+Xclo88Y1lzCo=; b=uYBbM2hw+bHVaQfCMyqHplFaIMKLb/Z95Q1Fqz+cZwpFL8H0QHj7loBDItOfonQd8o 0iFuiCWhuZ4W+ptQoEDq8y9nwIX+C1Nd2J02s0XuKe+TiDDDWJiZVJSkBJW7vD4NCqhD 8xxxWKWY3ZDj3RNkzOH1ZeO+2hwMTaxzvV2TG4XXrQmY/1Voks+zxrzWlmPl9eXciqze oSRUaLDMKinLvlNAKW0Rx0n2KkCYJT1hu1PAbwS0bQBzs8PHTFGI5tj5tHRAqPQdTil8 bAAB4LSBJZWRkENBNtEHMfqc8xrInrsVhNldQabcdjEQb2+r5urAzSGgnoLT162hnrVE SqjQ== X-Gm-Message-State: AOAM531fL2KUMxEXmSkOTS6xnTmpqy7KYLOsG+yBYFit5u6O3Y1RQjS5 3Vx0U3yQVt9UG0xL5S4vZG9oCB+Zp8IfU+kR/KdP/MC+gnxAQQ== X-Google-Smtp-Source: ABdhPJwRBBjLS3eXbXlvwIfpa5cidDdNpBZb8yW65En07lHGCPh0jfTpDfkPICIjA4H/y1jtj8DgOAG07jnmmoLLZzc= X-Received: by 2002:a6b:6a0c:: with SMTP id x12mr4971168iog.81.1623452407694; Fri, 11 Jun 2021 16:00:07 -0700 (PDT) MIME-Version: 1.0 References: <950B8EAF-90B9-41A6-951D-91821F591D41@teklibre.net> <01a7bed2-6f49-3d7d-eb5a-209031ee8070@gmail.com> <391A8897-5A1F-424A-9DD0-01B66824887B@teklibre.net> In-Reply-To: <391A8897-5A1F-424A-9DD0-01B66824887B@teklibre.net> From: Nathan Owens Date: Fri, 11 Jun 2021 15:59:57 -0700 Message-ID: To: Dave Taht Cc: Mike Puchol , starlink@lists.bufferbloat.net Content-Type: multipart/alternative; boundary="00000000000012221805c4857818" Subject: Re: [Starlink] Microstate Accounting and the Nyquist problem X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Jun 2021 23:00:08 -0000 --00000000000012221805c4857818 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ha, that=E2=80=99s a great point about the GPS/NTP thing. GPS chip, not sure, there=E2=80=99s only one tear down I=E2=80=99ve seen, h= ere=E2=80=99s the gps chip: https://cdn.arstechnica.net/wp-content/uploads/2020/12/starlink-28-gps-rece= iver.jpg No idea about the software, maybe I=E2=80=99ll try to JTAG mine. On Fri, Jun 11, 2021 at 3:39 PM Dave Taht wrote: > > > On Jun 11, 2021, at 3:34 PM, Mike Puchol wrote: > > We know that Starlink recalculates topology every 15 seconds (this guy, > who obviously has way too much spare time, came up with an indirect > observation of this interval: > https://blog.beerriot.com/2021/02/14/starlink-raster-scan/ ) > > If we could align with this, we could at least know when potential change= s > in path delays happen, and try to observe other changes that happen at a > similar cadence. > > Other thoughts, try to plug more details out of the gRPC data, setup > GPS-synced probes with a device at the exit PoP, measure differences > between time-sync probes to an array of endpoints. > > > It=E2=80=99s ironic that the device has to have gps in it, and thus shoul= d be > able to provide perfect time to clients directly behind it, isn=E2=80=99= t. > > I haven=E2=80=99t captured a dhcp or dhcpv6 transaction yet myself, > do they have a ntp option? > > What gps software or driver might they have used? (esr=E2=80=99s gpsd is = quite > popular, but there are others) > > What=E2=80=99s the gps chip? > > > Has nobody attacked the JTAG connector on a Dishy yet? > > Best, > > Mike > On Jun 12, 2021, 00:14 +0200, David Collier-Brown , > wrote: > > OK, *Oh Smarter Colleagues*, the challenge to you is to say if there is a > "natural" place to capture state changes to get the data we want, and if > so, is it common or similar enough between drivers to be worthy of > attention? > > --dave > On 2021-06-09 9:15 a.m., Dave Taht wrote: > > > > Begin forwarded message: > > *From:* David Collier-Brown > *Subject:* *Microstate Accounting and the Nyquist problem* > *Date:* June 9, 2021 at 4:44:14 AM PDT > *To:* Dave Taht > *Cc:* Dave Collier-Brown > *Reply-To:* davecb@spamcop.net > > A million years ago (roughly around Solaris 9), Sun was suffering from th= e > same problems in measuring their dispatcher as you are with "sloshing". > > A CPU would be 100% busy in one microsecond, 10% busy in the next > gazillion, and the average CPU utilization for our sample period would be > *maybe* 10.1, if the sampler happened to sample right when the spike was > happening. > > This was utterly useless for things like the fair-share scheduler, so it > got fixed in Solaris 10, by having the dispatcher record the time a proce= ss > (well, kernel thread) had spent in a state when the state changed. > > Initially "microstate accounting" could be toggled on and off, but the > branch-around cost more time than always doing the calculation (as > discovered by my mad friend Fred) and the kernel folks left it on. It's o= n > to this day. > > In Simon Sundberg's talk, the opportunity to measure occurs every 1,000 > packets, when a suitable timestamp is provided. While the eBPF program ca= n > look at every packet and do after-the-fact book-keeping in a map, that's > only good if the phenomenon you're measuring is persistent enough that it= 's > around for ~2,000 packets. > > I'm going to suggest that the right place to record the information you > want is right where the event happens. Preferably in c code, as > performance is easy to mess up, but perhaps with an eBPF mechanism to > export it. > > In previous Solaris work, I reliably found that exporting kstats was a > darn sight harder than collecting them, and in Eric's blog post[1] he not= es > that converting time is expensive and best done long after collecting, wh= en > someone wanted to read the data. > > There was an effort to do kstats in Linux[2], but it had supposedly poor > performance, and actual trouble when the clock frequency changed. > > Is there, in your opinion, a "natural" place to capture state changes to > get the data you want, and if so, is it common or similar enough between > drivers to be worthy of attention? > > --dave > > > References: > > 1. Solaris: > http://dtrace.org/blogs/eschrock/2004/10/13/microstate-accounting-in-s= olaris-10/ > > 2. A failing Linux effort: https://lwn.net/Articles/127296/ > , > https://sourceforge.net/projects/microstate/ > > > -- > David Collier-Brown, | Always do right. This will gratify > System Programmer and Author | some people and astonish the restdavecb@sp= amcop.net | -- Mark Twain > > > _______________________________________________ > Starlink mailing list > Starlink@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/starlink > > _______________________________________________ > Starlink mailing list > Starlink@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/starlink > > > _______________________________________________ > Starlink mailing list > Starlink@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/starlink > --00000000000012221805c4857818 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Ha, that=E2=80=99s a great point about the GPS/NTP thing.= =C2=A0

GPS chip, not sur= e, there=E2=80=99s only one tear down I=E2=80=99ve seen, here=E2=80=99s the= gps chip:=C2=A0
https://cdn.arstechn= ica.net/wp-content/uploads/2020/12/starlink-28-gps-receiver.jpg=C2=A0

No idea about the softwar= e, maybe I=E2=80=99ll try to JTAG mine.=C2=A0
=



On Fri, Jun 11, 2021 at 3:3= 9 PM Dave Taht <davet@teklibre.net= > wrote:


On Jun 11, 2021, at 3:34 PM, Mike Puchol <mike@starlink.sx> wrote:

We know that Starlink recalculates topology every 15 seco= nds (this guy, who obviously has way too much spare time, came up with an i= ndirect observation of this interval:=C2=A0https://blog.beerr= iot.com/2021/02/14/starlink-raster-scan/=C2=A0)

If we could align with this, we could at least know when potential changes = in path delays happen, and try to observe other changes that happen at a si= milar cadence.

Other thoughts, try to plug more details out of the gRPC data, setup GPS-sy= nced probes with a device at the exit PoP, measure differences between time= -sync probes to an array of endpoints.


It=E2=80=99s ironic= that the device has to have gps in it, and thus should be =C2=A0able to pr= ovide perfect time to clients directly behind it, isn=E2=80=99t.
=
I haven=E2=80=99t captured a dhcp or dhcpv6 transaction yet = myself,
do they have a ntp option?

What = gps software or driver might they have used? (esr=E2=80=99s gpsd is quite p= opular, but there are others)=C2=A0

What=E2=80=99s= the gps chip?


Has nobody attacked the JTAG connector on a Dishy yet?

Best,

Mike
On Jun 12, 2021, 00:14 +0200, David Colli= er-Brown <davec= b.42@gmail.com>, wrote:

OK, Oh Smarter Colleagues, the challenge to you is to say if= there is a "natural" place to capture state changes to get the d= ata we want, and if so, is it common or similar enough between drivers to b= e worthy of attention?

--dave

On 2021-06-09 9:15 a.m., Dave Taht wrote:


Begin forwarded message:

From:<= /span> David Collier-Brown <davecb.42@gmail.com>= ;
Subject: Microstate Accounting and = the Nyquist problem
Date:<= /span> June 9, 2021 at 4:44:14 AM PDT
To: Dave Taht <davet@teklibre.net>
Reply-To:<= /b> davecb@spamcop.net

A million years ago (roughly around Solaris 9), Sun was suffering f= rom the same problems in measuring their dispatcher as you are with "s= loshing".

A CPU would be 100% busy in one microsecond, 10% busy = in the next gazillion, and the average CPU utilization for our sample perio= d would be maybe 10.1, if the sampler happened to sample right when = the spike was happening.

This was utterly useless for things like the= fair-share scheduler, so it got fixed in Solaris 10, by having the dispatc= her record the time a process (well, kernel thread) had spent in a state wh= en the state changed.

Initially "microstate accounting"= could be toggled on and off, but the branch-around cost more time than alw= ays doing the calculation (as discovered by my mad friend Fred) and the ker= nel folks left it on. It's on to this day.

In Simon Sundberg'= s talk, the opportunity to measure occurs every 1,000 packets, when a suita= ble timestamp is provided. While the eBPF program can look at every packet = and do after-the-fact book-keeping in a map, that's only good if the ph= enomenon you're measuring is persistent enough that it's around for= ~2,000 packets.

I'm going to suggest that the right place to rec= ord the information you want is right where the event happens.=C2=A0 Prefer= ably in c code, as performance is easy to mess up, but perhaps with an eBPF= mechanism to export it.

In previous Solaris work, I reliably found t= hat exporting kstats was a darn sight harder than collecting them, and in E= ric's blog post[1] he notes that converting time is expensive and best = done long after collecting, when someone wanted to read the data.

The= re was an effort to do kstats in Linux[2], but it had supposedly poor perfo= rmance, and actual trouble when the clock frequency changed.

Is t= here, in your opinion, a "natural" place to capture state changes= to get the data you want, and if so, is it common or similar enough betwee= n drivers to be worthy of attention?

--dave


Refe= rences:

  1. Solaris: http://dtrace.org/blogs/esch= rock/2004/10/13/microstate-accounting-in-solaris-10/
  2. A failing Linux effort: https://lwn.net/Articles/127296/, https://sourceforge.net/projects/microstate/<= /a>
-- =20
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
davecb@spamcop.net           |                      -- Mar=
k Twain

_______________________________________________
Starlink mailing list
Starlin= k@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink
_______________________________________________
Starlink mailing listStarli= nk@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/s= tarlink

_________________________= ______________________
Starlink mailing list
Starlin= k@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink --00000000000012221805c4857818--