<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]--><o:SmartTagType
namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="country-region"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="place"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="PersonName"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:Arial;
color:navy;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 77.95pt 1.0in 77.95pt;}
div.Section1
{page:Section1;}
-->
</style>
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<div>
<div class=MsoNormal align=center style='text-align:center'><font size=3
face="Times New Roman"><span style='font-size:12.0pt'>
<hr size=3 width="100%" align=center tabindex=-1>
</span></font></div>
<p class=MsoNormal><b><font size=2 face=Tahoma><span style='font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font size=2
face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'> Sebastian
Moeller [mailto:moeller0@gmx.de] <br>
<b><span style='font-weight:bold'>Sent:</span></b> Thursday, January 12, 2023
11:33 PM<br>
<b><span style='font-weight:bold'>To:</span></b> <st1:PersonName w:st="on">dickroy@alum.mit.edu</st1:PersonName>;
Dick Roy<br>
<b><span style='font-weight:bold'>Cc:</span></b> 'Rodney W. Grimes';
mike.reynolds@netforecast.com; 'libreqos'; '<st1:PersonName w:st="on">David P.
Reed</st1:PersonName>'; 'Rpm'; 'rjmcmahon'; 'bloat'<br>
<b><span style='font-weight:bold'>Subject:</span></b> RE: [Starlink] [Rpm]
Researchers Seeking Probe Volunteers in <st1:country-region w:st="on"><st1:place
w:st="on">USA</st1:place></st1:country-region></span></font><o:p></o:p></p>
</div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><font size=3
face="Times New Roman"><span style='font-size:12.0pt'>Hi RR,<br>
<br>
Thanks for the detailed response below, since my point is somewhat orthogonal I
opted for top-posting.<br>
Let me take a step back here and rephrase, synchronising clocks within an
acceptable range to be useful is not rocket science nor witchcraft. For
measuring internet traffic 'millisecond' range seems acceptable, local networks
can probably profit from finer time resolution. So I am not after e.g. clock
synchronisation to participate in SDH/SONET. Heck in the toy project I am
active in, we operate on load dependent delay deltas so we even ignore
different time offsets and are tolerant to (mildly) different tickrates and
clock skew, but it would certainly be nice to have some acceptable measure of
UTC from endpoints to be able to interpret timestamps as 'absolute'. Mind you I
am fine with them not being veridical absolute, but just good enough for my
measurement purpose and I guess that should be within the range of the
achievable. Heck, if all servers we query timestamps of would be
NTP-'synchronized' and would follow the RFC recommendation to report timestamps
in milliseconds past midnight UTC I would be happy.<font color=navy><span
style='color:navy'><o:p></o:p></span></font></span></font></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><b><i><font size=2 color=navy
face=Arial><span style='font-size:10.0pt;font-family:Arial;color:navy;
font-weight:bold;font-style:italic'>[RR] Yup! All true. Hence my post that
obviously passed this one in the ether! </span></font></i></b><b><i><font
size=2 color=navy face=Wingdings><span style='font-size:10.0pt;font-family:
Wingdings;color:navy;font-weight:bold;font-style:italic'>J</span></font></i></b><b><i><font
size=2 color=navy face=Arial><span style='font-size:10.0pt;font-family:Arial;
color:navy;font-weight:bold;font-style:italic'> </span></font></i></b><b><i><font
size=2 color=navy face=Wingdings><span style='font-size:10.0pt;font-family:
Wingdings;color:navy;font-weight:bold;font-style:italic'>J</span></font></i></b><b><i><font
size=2 color=navy face=Arial><span style='font-size:10.0pt;font-family:Arial;
color:navy;font-weight:bold;font-style:italic'> <o:p></o:p></span></font></i></b></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><font size=3
face="Times New Roman"><span style='font-size:12.0pt'><br>
<br>
Regards<br>
Sebsstian<o:p></o:p></span></font></p>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>On 12 January 2023 21:39:21 CET, Dick Roy <<st1:PersonName w:st="on">dickroy@alum.mit.edu</st1:PersonName>>
wrote:<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>Hi Sebastian (et. al.),<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>[I’ll comment up here instead of inline.] <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>Let me start by saying that I have not been intimately involved with
the IEEE 1588 effort (PTP), however I was involved in the 802.11 efforts along
a similar vein, just adding the wireless first hop component and it’s effects
on PTP. <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>What was apparent from the outset was that there was a lack of
understanding what the terms “to synchronize” or “to be synchronized” actually
mean. It’s not trivial … because we live in a (approximately, that’s
another story!) 4-D space-time continuum where the Lorentz metric plays a
critical role. Therein, simultaneity (aka “things happening at the same
time”) means the “distance” between two such events is zero and that distance
is given by sqrt(x^2 + y^2 + z^2 <span style='background:yellow'>–</span>
(ct)^2) and the “thing happening” can be the tick of a clock somewhere. Now
since everything is relative (time with respect to what? / location with
respect to where?) it’s pretty easy to see that “if you don’t know where you
are, you can’t know what time it is!” (English sailors of the 18<sup>th</sup>
century knew this well!) Add to this the fact that if everything were
stationary, nothing would happen (as Einstein said “Nothing happens until
something moves!”), special relativity also pays a role. Clocks on GPS
satellites run approx. 7usecs/day slower than those on earth due to their
“speed” (8700 mph roughly)! Then add the consequence that without mass we
wouldn’t exist (in these forms at least</span></font><font face=Wingdings><span
style='font-family:Wingdings'>J</span></font>), and gravitational effects (aka
General Relativity) come into play. Those turn out to make clocks on GPS
satellites run 45usec/day faster than those on earth! The net effect is
that GPS clocks run about 38usec/day faster than clocks on earth. So what
does it mean to “synchronize to GPS”? Point is: it’s a non-trivial
question with a very complicated answer. The reason it is important to
get all this right is that the “what that ties time and space together” is the
speed of light and that turns out to be a “foot-per-nanosecond” in a vacuum
(roughly 300m/usec). This means if I am uncertain about my location to
say 300 meters, then I also am not sure what time it is to a usec AND
vice-versa! <o:p></o:p></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>All that said, the simplest explanation of synchronization is probably:
Two clocks are synchronized if, when they are brought (slowly) into physical
proximity (“sat next to each other”) in the same (quasi-)inertial frame and the
same gravitational potential (not so obvious BTW … see the FYI below!), an
observer of both would say “they are keeping time identically”. Since this
experiment is rarely possible, one can never be “sure” that his clock is
synchronized to any other clock elsewhere. And what does it mean to say they
“were synchronized” when brought together, but now they are not because they
are now in different gravitational potentials! (FYI, there are land mine
detectors being developed on this very principle! I know someone who actually
worked on such a project!) <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>This all gets even more complicated when dealing with large networks of
networks in which the “speed of information transmission” can vary depending on
the medium (cf. coaxial cables versus fiber versus microwave links!) In fact,
the atmosphere is one of those media and variations therein result in the need
for “GPS corrections” (cf. RTCM GPS correction messages, RTK, etc.) in order to
get to sub-nsec/cm accuracy. Point is if you have a set of nodes
distributed across the country all with GPS and all “synchronized to GPS time”,
and a second identical set of nodes (with no GPS) instead connected with a
network of cables and fiber links, all of different lengths and composition
using different carrier frequencies (dielectric constants vary with frequency!)
“synchronized” to some clock somewhere using NTP or PTP), the synchronization
of the two sets will be different unless a common reference clock is used AND
all the above effects are taken into account, and good luck with that! </span></font><font
face=Wingdings><span style='font-family:Wingdings'>J</span></font> <o:p></o:p></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>In conclusion, if anyone tells you that clock synchronization in
communication networks is simple (“Just use GPS!”), you should feel free to
chuckle (under your breath if necessary</span></font><font face=Wingdings><span
style='font-family:Wingdings'>J</span></font>) <o:p></o:p></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>Cheers,<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>RR<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>-----Original Message-----<br>
From: Sebastian Moeller [mailto:moeller0@gmx.de] <br>
Sent: Thursday, January 12, 2023 12:23 AM<br>
To: Dick Roy<br>
Cc: Rodney W. Grimes; mike.reynolds@netforecast.com; libreqos; <st1:personname w:st="on"><st1:PersonName
w:st="on">David P. Reed</st1:personname></st1:PersonName>; Rpm; rjmcmahon;
bloat<br>
Subject: Re: [Starlink] [Rpm] Researchers Seeking Probe Volunteers in <st1:place
w:st="on"><st1:country-region w:st="on">USA</st1:country-region></st1:place><o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>Hi RR,<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> On Jan 11, 2023, at 22:46, Dick Roy <<st1:personname w:st="on"><st1:PersonName
w:st="on">dickroy@alum.mit.edu</st1:personname></st1:PersonName>> wrote:<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> -----Original Message-----<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> From: Starlink [mailto:starlink-bounces@lists.bufferbloat.net] On
Behalf Of Sebastian Moeller via Starlink<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> Sent: Wednesday, January 11, 2023 12:01 PM<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> To: Rodney W. Grimes<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> Cc: Dave Taht via Starlink; mike.reynolds@netforecast.com;
libreqos; <st1:personname w:st="on"><st1:PersonName w:st="on">David P. Reed</st1:personname></st1:PersonName>;
Rpm; rjmcmahon; bloat<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> Subject: Re: [Starlink] [Rpm] Researchers Seeking Probe Volunteers
in <st1:place w:st="on"><st1:country-region w:st="on">USA</st1:country-region></st1:place><o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> Hi Rodney,<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> > On Jan 11, 2023, at 19:32, Rodney W. Grimes
<starlink@gndrsh.dnsmgr.net> wrote:<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> > <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> > Hello,<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> > <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> > Yall can call me crazy if you want..
but... see below [RWG]<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >> Hi Bib,<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>> On Jan 9, 2023, at 20:13, rjmcmahon via Starlink
<starlink@lists.bufferbloat.net> wrote:<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>> My biggest barrier is the lack of clock sync by the
devices, i.e. very limited support for PTP in data centers and in end devices.
This limits the ability to measure one way delays (OWD) and most assume that
OWD is 1/2 and RTT which typically is a mistake. We know this intuitively with
airplane flight times or even car commute times where the one way time is not
1/2 a round trip time. Google maps & directions provide a time estimate for
the one way link. It doesn't compute a round trip and divide by two.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>> For those that can get clock sync working, the iperf 2
--trip-times options is useful.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >> [SM] +1; and yet even with
unsynchronized clocks one can try to measure how latency changes under load and
that can be done per direction. Sure this is far inferior to real reliably
measured OWDs, but if life/the internet deals you lemons....<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> > <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> > [RWG] iperf2/iperf3, etc are already moving large amounts of
data back and forth, for that matter any rate test, why not abuse some of that
data and add the fundemental NTP clock sync data and bidirectionally pass each
others concept of "current time". IIRC (its been 25 years since
I worked on NTP at this level) you *should* be able to get a fairly accurate
clock delta between each end, and then use that info and time stamps in the
data stream to compute OWD's. You need to put 4 time stamps in the
packet, and with that you can compute "offset".<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> [RR] For this to work at a reasonable level of accuracy, the
timestamping circuits on both ends need to be deterministic and repeatable as I
recall. Any uncertainty in that process adds to synchronization
errors/uncertainties.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> [SM] Nice idea. I would guess
that all timeslot based access technologies (so starlink, docsis, GPON, LTE?)
all distribute "high quality time" carefully to the
"modems", so maybe all that would be needed is to expose that high
quality time to the LAN side of those modems, dressed up as NTP server?<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> [RR] It’s not that simple! Distributing “high-quality time”,
i.e. “synchronizing all clocks” does not solve the communication problem in synchronous
slotted MAC/PHYs!<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'> [SM] I happily believe you, but the same
idea of "time slot" needs to be shared by all nodes, no? So the
clockss need to be reasonably similar rate, aka synchronized (see below).<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> All the technologies you mentioned above are essentially
P2P, not intended for broadcast. Point is, there is a point controller
(aka PoC) often called a base station (eNodeB, gNodeB, …) that actually
“controls everything that is necessary to control” at the UE including time,
frequency and sampling time offsets, and these are critical to get right if you
want to communicate, and they are ALL subject to the laws of physics (cf. the
speed of light)! Turns out that what is necessary for the system to function
anywhere near capacity, is for all the clocks governing transmissions from the
UEs to be “unsynchronized” such that all the UE transmissions arrive at the PoC
at the same (prescribed) time!<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'> [SM] Fair enough. I would call clocks
that are "in sync" albeit with individual offsets as synchronized,
but I am a layman and that might sound offensively wrong to experts in the
field. But even without the naming my point is that all systems that depend on
some idea of shared time-base are halfway there of exposing that time to end
users, by "translating it into an NTP time source at the modem.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> For some technologies, in particular 5G!, these considerations are
ESSENTIAL. Feel free to scour the 3GPP LTE 5G RLC and PHY specs if you don’t
believe me! J <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'> [SM Far be it from me not to believe
you, so thanks for the pointers. Yet, I still think that unless different nodes
of a shared segment move at significantly different speeds, that there should
be a common "tick-duration" for all clocks even if each clock runs at
an offset... (I naively would try to implement something like that by trying to
fully synchronize clocks and maintain a local offset value to convert from
"absolute" time to "network" time, but likely because
coming from the outside I am blissfully unaware of the detail challenges that
need to be solved).<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>Regards & Thanks<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'> Sebastian<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> > <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>> --trip-times<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>> enable the measurement of end to end write to read
latencies (client and server clocks must be synchronized)<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> > [RWG] --clock-skew<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> > enable the measurement of the wall
clock difference between sender and receiver<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> > <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >> [SM] Sweet!<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >> Regards<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >> Sebastian<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>> Bob<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> I have many kvetches about the new latency under
load tests being<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> designed and distributed over the past year. I am
delighted! that they<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> are happening, but most really need third party
evaluation, and<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> calibration, and a solid explanation of what
network pathologies they<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> do and don't cover. Also a RED team attitude
towards them, as well as<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> thinking hard about what you are not measuring
(operations research).<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> I actually rather love the new cloudflare
speedtest, because it tests<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> a single TCP connection, rather than dozens, and
at the same time folk<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> are complaining that it doesn't find the actual
"speed!". yet... the<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> test itself more closely emulates a user
experience than speedtest.net<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> does. I am personally pretty convinced that the
fewer numbers of flows<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> that a web page opens improves the likelihood of
a good user<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> experience, but lack data on it.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> To try to tackle the evaluation and calibration
part, I've reached out<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> to all the new test designers in the hope that we
could get together<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> and produce a report of what each new test is
actually doing. I've<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> tweeted, linked in, emailed, and spammed every
measurement list I know<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> of, and only to some response, please reach out
to other test designer<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> folks and have them join the rpm email list?<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> My principal kvetches in the new tests so far
are:<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> 0) None of the tests last long enough.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> Ideally there should be a mode where they at
least run to "time of<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> first loss", or periodically, just run
longer than the<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> industry-stupid^H^H^H^H^H^Hstandard 20 seconds.
There be dragons<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> there! It's really bad science to optimize the
internet for 20<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> seconds. It's like optimizing a car, to handle
well, for just 20<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> seconds.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> 1) Not testing up + down + ping at the same time<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> None of the new tests actually test the same
thing that the infamous<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> rrul test does - all the others still test up,
then down, and ping. It<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> was/remains my hope that the simpler parts of the
flent test suite -<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> such as the tcp_up_squarewave tests, the rrul
test, and the rtt_fair<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> tests would provide calibration to the test
designers.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> we've got zillions of flent results in the
archive published here:<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> https://blog.cerowrt.org/post/found_in_flent/<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> ps. Misinformation about iperf 2 impacts my
ability to do this.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> The new tests have all added up + ping and down +
ping, but not up +<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> down + ping. Why??<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> The behaviors of what happens in that case are
really non-intuitive, I<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> know, but... it's just one more phase to add to
any one of those new<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> tests. I'd be deliriously happy if someone(s) new
to the field<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> started doing that, even optionally, and boggled
at how it defeated<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> their assumptions.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> Among other things that would show...<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> It's the home router industry's dirty secret than
darn few "gigabit"<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> home routers can actually forward in both
directions at a gigabit. I'd<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> like to smash that perception thoroughly, but
given our starting point<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> is a gigabit router was a "gigabit
switch" - and historically been<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> something that couldn't even forward at 200Mbit -
we have a long way<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> to go there.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> Only in the past year have non-x86 home routers
appeared that could<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> actually do a gbit in both directions.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> 2) Few are actually testing within-stream latency<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> Apple's rpm project is making a stab in that
direction. It looks<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> highly likely, that with a little more work, crusader
and<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> go-responsiveness can finally start sampling the
tcp RTT, loss and<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> markings, more directly. As for the rest...
sampling TCP_INFO on<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> windows, and Linux, at least, always appeared
simple to me, but I'm<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> discovering how hard it is by delving deep into
the rust behind<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> crusader.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> the goresponsiveness thing is also IMHO running
WAY too many streams<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> at the same time, I guess motivated by an attempt
to have the test<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> complete quickly?<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> B) To try and tackle the validation problem:ps.
Misinformation about iperf 2 impacts my ability to do this.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> In the libreqos.io project we've established a
testbed where tests can<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> be plunked through various ISP plan network
emulations. It's here:<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> https://payne.taht.net (run bandwidth test for
what's currently hooked<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> up)<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> We could rather use an AS number and at least a
ipv4/24 and ipv6/48 to<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> leverage with that, so I don't have to nat the
various emulations.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> (and funding, anyone got funding?) Or, as the
code is GPLv2 licensed,<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> to see more test designers setup a testbed like
this to calibrate<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> their own stuff.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> Presently we're able to test:<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> flent<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> netperf<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> iperf2<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> iperf3<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> speedtest-cli<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> crusader<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> the broadband forum udp based test:<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> https://github.com/BroadbandForum/obudpst<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> trexx<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> There's also a virtual machine setup that we can
remotely drive a web<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> browser from (but I didn't want to nat the
results to the world) to<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> test other web services.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> _______________________________________________<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> Rpm mailing list<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> Rpm@lists.bufferbloat.net<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>>> https://lists.bufferbloat.net/listinfo/rpm<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>> _______________________________________________<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>> Starlink mailing list<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>> Starlink@lists.bufferbloat.net<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >>> https://lists.bufferbloat.net/listinfo/starlink<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >> _______________________________________________<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >> Starlink mailing list<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >> Starlink@lists.bufferbloat.net<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> >> https://lists.bufferbloat.net/listinfo/starlink<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> _______________________________________________<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> Starlink mailing list<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> Starlink@lists.bufferbloat.net<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>> https://lists.bufferbloat.net/listinfo/starlink<o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>-- <br>
Sent from my Android device with K-9 Mail. Please excuse my brevity.<o:p></o:p></span></font></p>
</div>
</div>
</body>
</html>