[Starlink] a puzzling starlink uplink trace

[Starlink] a puzzling starlink uplink trace

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 1280 bytes --]

In the attached 5 minute plot from a few days ago (I can supply the
flent.gz files if anyone wants them), I see a puzzling spike at T+155s
to nearly 90ms of baseline latency, then down to 20ms. No degree of
orbital mechanics can apply to this change, even factoring in an over
the horizon connection, routing packets on the ground through LA to
seattle, and back, or using a couple ISLs, can make this add up for
me. A combination of all that, kind of does make sense.

The trace otherwise shows the sawtooth pattern of a single tcp flow ,
a loss (sometimes catastrophic) at every downward bandwidth change.

An assumption I have long been making is the latency staircase effect
(see T+170) forward is achieving the best encoding rate at the
distance then seen, the distance growing and the encoding rate falling
in distinct steps, with a fixed amount of buffering, until finally
that sat starts falling out of range, and it choses another at T+240s.

But jeeze, a 70ms baseline latency swing? What gives?  I imagine
somehow correlating this with a mpls enabled traceroute might begin to
make some sense of it, correlated by orbital positions....


-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

[-- Attachment #2: thestarlinkmystery.png --]
[-- Type: image/png, Size: 317233 bytes --]

Re: [Starlink] a puzzling starlink uplink trace

[Sebastian Moeller]

Hi Dave,


you probably know already, but
mtr -ezb4 www.heise.de
will report compliant MPLS segments:

                                                                                          My traceroute  [v0.95]
123-1234567.local (192.168.42.229) -> www.heise.de (193.99.144.85)                                                                                                               2023-08-30T21:39:48+0200
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                                                                                                                 Packets               Pings
 Host                                                                                                                                                          Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. AS???    192.168.42.1 (192.168.42.1)                                                                                                                        0.0%  1067    0.9   0.6   0.5   9.1   0.4
 2. AS6805   loopback1.0003.acln.06.ham.de.net.telefonica.de (62.52.201.200)                                                                                    0.0%  1067   12.3  22.5   9.9 172.7  17.5
 3. AS6805   bundle-ether10.0002.dbrx.06.ham.de.net.telefonica.de (62.53.2.98)                                                                                  0.7%  1067   12.4  12.0  10.3  16.5   0.7
 4. AS6805   ae7-0.0001.corx.06.ham.de.net.telefonica.de (62.53.15.0)                                                                                           0.0%  1067   18.7  19.2  17.2  48.9   2.4
    [MPLS: Lbl 16624 TC 0 S 1 TTL 1]
 5. AS6805   ae6-0.0002.corx.02.fra.de.net.telefonica.de (62.53.0.49)                                                                                           0.0%  1067   41.6  19.6  17.2  51.6   3.9
    [MPLS: Lbl 16624 TC 0 S 1 TTL 1]
 6. AS6805   bundle-ether2.0001.cord.01.off.de.net.telefonica.de (62.53.0.199)                                                                                  0.0%  1067   19.0  19.3  17.7  87.1   2.2
    [MPLS: Lbl 16624 TC 0 S 1 TTL 1]
 7. AS6805   bundle-ether1.0002.corp.01.off.de.net.telefonica.de (62.53.28.171)                                                                                 0.0%  1067   22.9  19.2  17.5  29.7   0.9
 8. AS???    ipv4.de-cix.fra.de.as12306.plusline.net (80.81.192.132)                                                                                            0.0%  1066   18.5  19.7  18.2  93.0   2.5
 9. AS12306  82.98.102.71 (82.98.102.71)                                                                                                                        0.0%  1066   19.6  19.4  17.7  69.5   1.7
    [MPLS: Lbl 24002 TC 0 S 1 TTL 1]
10. AS12306  82.98.102.23 (82.98.102.23)                                                                                                                        0.0%  1066   17.8  19.8  17.4 197.7   9.9
11. AS12306  212.19.61.13 (212.19.61.13)                                                                                                                        0.0%  1066   19.8  19.6  17.5 155.7   6.1
12. AS12306  www.heise.de (193.99.144.85)                                                                                                                       0.1%  1066   19.6  19.1  17.6  63.7   1.5

Not that these do not already stick out as hops at different locations (here Hamburg (ham), Frankfurt (fra) and Offenbach)) all with more or less identical distance-independent RTTs.

Other than that quite intriguing puzzle, I wonder whether we could teach flent to run a bidirectional traceroute/mtr as part of its measurements..?

Regards
	Sebastian


> On Aug 30, 2023, at 20:07, Dave Taht via Starlink <starlink@lists.bufferbloat.net> wrote:
> 
> In the attached 5 minute plot from a few days ago (I can supply the
> flent.gz files if anyone wants them), I see a puzzling spike at T+155s
> to nearly 90ms of baseline latency, then down to 20ms. No degree of
> orbital mechanics can apply to this change, even factoring in an over
> the horizon connection, routing packets on the ground through LA to
> seattle, and back, or using a couple ISLs, can make this add up for
> me. A combination of all that, kind of does make sense.
> 
> The trace otherwise shows the sawtooth pattern of a single tcp flow ,
> a loss (sometimes catastrophic) at every downward bandwidth change.
> 
> An assumption I have long been making is the latency staircase effect
> (see T+170) forward is achieving the best encoding rate at the
> distance then seen, the distance growing and the encoding rate falling
> in distinct steps, with a fixed amount of buffering, until finally
> that sat starts falling out of range, and it choses another at T+240s.
> 
> But jeeze, a 70ms baseline latency swing? What gives?  I imagine
> somehow correlating this with a mpls enabled traceroute might begin to
> make some sense of it, correlated by orbital positions....
> 
> 
> -- 
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos
> <thestarlinkmystery.png>_______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink

Re: [Starlink] a puzzling starlink uplink trace

[Dave Taht]

On Wed, Aug 30, 2023 at 12:43 PM Sebastian Moeller <moeller0@gmx.de> wrote:
>
> Hi Dave,
>
>
> you probably know already, but
> mtr -ezb4 www.heise.de
> will report compliant MPLS segments:

Despite any reputation for omniscience I may have acquired, I was an
ipv6 bigot in the era when MPLS was being specced, and somewhat
willfully ignored it, and did not know traceroute and mtr had gained
the ability to see it.

It would be kind of neat (for business services at least), if starlink
offered MPLS to the router. It would make some things easier. Nobody
to this day seems to know what the dishy to ground station actually
is, although a lot of folk seem to be speculating it is mpls, my bet
is still something custom that has a destination and encrypted
payload....



>                                                                                           My traceroute  [v0.95]
> 123-1234567.local (192.168.42.229) -> www.heise.de (193.99.144.85)                                                                                                               2023-08-30T21:39:48+0200
> Keys:  Help   Display mode   Restart statistics   Order of fields   quit
>                                                                                                                                                                  Packets               Pings
>  Host                                                                                                                                                          Loss%   Snt   Last   Avg  Best  Wrst StDev
>  1. AS???    192.168.42.1 (192.168.42.1)                                                                                                                        0.0%  1067    0.9   0.6   0.5   9.1   0.4
>  2. AS6805   loopback1.0003.acln.06.ham.de.net.telefonica.de (62.52.201.200)                                                                                    0.0%  1067   12.3  22.5   9.9 172.7  17.5
>  3. AS6805   bundle-ether10.0002.dbrx.06.ham.de.net.telefonica.de (62.53.2.98)                                                                                  0.7%  1067   12.4  12.0  10.3  16.5   0.7
>  4. AS6805   ae7-0.0001.corx.06.ham.de.net.telefonica.de (62.53.15.0)                                                                                           0.0%  1067   18.7  19.2  17.2  48.9   2.4
>     [MPLS: Lbl 16624 TC 0 S 1 TTL 1]
>  5. AS6805   ae6-0.0002.corx.02.fra.de.net.telefonica.de (62.53.0.49)                                                                                           0.0%  1067   41.6  19.6  17.2  51.6   3.9
>     [MPLS: Lbl 16624 TC 0 S 1 TTL 1]
>  6. AS6805   bundle-ether2.0001.cord.01.off.de.net.telefonica.de (62.53.0.199)                                                                                  0.0%  1067   19.0  19.3  17.7  87.1   2.2
>     [MPLS: Lbl 16624 TC 0 S 1 TTL 1]
>  7. AS6805   bundle-ether1.0002.corp.01.off.de.net.telefonica.de (62.53.28.171)                                                                                 0.0%  1067   22.9  19.2  17.5  29.7   0.9
>  8. AS???    ipv4.de-cix.fra.de.as12306.plusline.net (80.81.192.132)                                                                                            0.0%  1066   18.5  19.7  18.2  93.0   2.5
>  9. AS12306  82.98.102.71 (82.98.102.71)                                                                                                                        0.0%  1066   19.6  19.4  17.7  69.5   1.7
>     [MPLS: Lbl 24002 TC 0 S 1 TTL 1]
> 10. AS12306  82.98.102.23 (82.98.102.23)                                                                                                                        0.0%  1066   17.8  19.8  17.4 197.7   9.9
> 11. AS12306  212.19.61.13 (212.19.61.13)                                                                                                                        0.0%  1066   19.8  19.6  17.5 155.7   6.1
> 12. AS12306  www.heise.de (193.99.144.85)                                                                                                                       0.1%  1066   19.6  19.1  17.6  63.7   1.5
>
> Not that these do not already stick out as hops at different locations (here Hamburg (ham), Frankfurt (fra) and Offenbach)) all with more or less identical distance-independent RTTs.
>
> Other than that quite intriguing puzzle, I wonder whether we could teach flent to run a bidirectional traceroute/mtr as part of its measurements..?
>
> Regards
>         Sebastian
>
>
> > On Aug 30, 2023, at 20:07, Dave Taht via Starlink <starlink@lists.bufferbloat.net> wrote:
> >
> > In the attached 5 minute plot from a few days ago (I can supply the
> > flent.gz files if anyone wants them), I see a puzzling spike at T+155s
> > to nearly 90ms of baseline latency, then down to 20ms. No degree of
> > orbital mechanics can apply to this change, even factoring in an over
> > the horizon connection, routing packets on the ground through LA to
> > seattle, and back, or using a couple ISLs, can make this add up for
> > me. A combination of all that, kind of does make sense.
> >
> > The trace otherwise shows the sawtooth pattern of a single tcp flow ,
> > a loss (sometimes catastrophic) at every downward bandwidth change.
> >
> > An assumption I have long been making is the latency staircase effect
> > (see T+170) forward is achieving the best encoding rate at the
> > distance then seen, the distance growing and the encoding rate falling
> > in distinct steps, with a fixed amount of buffering, until finally
> > that sat starts falling out of range, and it choses another at T+240s.
> >
> > But jeeze, a 70ms baseline latency swing? What gives?  I imagine
> > somehow correlating this with a mpls enabled traceroute might begin to
> > make some sense of it, correlated by orbital positions....
> >
> >
> > --
> > Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> > Dave Täht CSO, LibreQos
> > <thestarlinkmystery.png>_______________________________________________
> > Starlink mailing list
> > Starlink@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/starlink
>


-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: [Starlink] a puzzling starlink uplink trace

[Sebastian Moeller]

Hi Dave,

On 31 August 2023 15:42:45 CEST, Dave Taht <dave.taht@gmail.com> wrote:
>On Wed, Aug 30, 2023 at 12:43 PM Sebastian Moeller <moeller0@gmx.de> wrote:
>>
>> Hi Dave,
>>
>>
>> you probably know already, but
>> mtr -ezb4 www.heise.de
>> will report compliant MPLS segments:
>
>Despite any reputation for omniscience I may have acquired, 

;) I was not sure which level of MPLS information you are after...

> I was an
>ipv6 bigot in the era when MPLS was being specced, and somewhat
>willfully ignored it,

It probably scratches an itch, but ine I do not have in my home network, so I am pretty ignorant of it as well, short of the little treatment it got in the nanog steenbergen roisman traceroute explainer over here:
https://archive.nanog.org/sites/default/files/10_Roisman_Traceroute.pdf

> and did not know traceroute and mtr had gained
>the ability to see it.

I think this is not so much seeing it and more aking politely for the MPLS segments to report faithfully. I have no idea how 'true' the reported hops are, but I also have no reason to doubt them...

>
>It would be kind of neat (for business services at least), if starlink
>offered MPLS to the router. It would make some things easier. Nobody
>to this day seems to know what the dishy to ground station actually
>is, although a lot of folk seem to be speculating it is mpls, my bet
>is still something custom that has a destination and encrypted
>payload....

Encryption certainly seems in order given that the whole transmissions are pretty much out in the open... (I would guess to passively snoop download traffic an adversary would need to be close enough to the dishy and for snooping uploads close enough to the respective base station makes sense, with close enough being within a few miles? with similar view to the sky).


>
>
>
>>                                                                                           My traceroute  [v0.95]
>> 123-1234567.local (192.168.42.229) -> www.heise.de (193.99.144.85)                                                                                                               2023-08-30T21:39:48+0200
>> Keys:  Help   Display mode   Restart statistics   Order of fields   quit
>>                                                                                                                                                                  Packets               Pings
>>  Host                                                                                                                                                          Loss%   Snt   Last   Avg  Best  Wrst StDev
>>  1. AS???    192.168.42.1 (192.168.42.1)                                                                                                                        0.0%  1067    0.9   0.6   0.5   9.1   0.4
>>  2. AS6805   loopback1.0003.acln.06.ham.de.net.telefonica.de (62.52.201.200)                                                                                    0.0%  1067   12.3  22.5   9.9 172.7  17.5
>>  3. AS6805   bundle-ether10.0002.dbrx.06.ham.de.net.telefonica.de (62.53.2.98)                                                                                  0.7%  1067   12.4  12.0  10.3  16.5   0.7
>>  4. AS6805   ae7-0.0001.corx.06.ham.de.net.telefonica.de (62.53.15.0)                                                                                           0.0%  1067   18.7  19.2  17.2  48.9   2.4
>>     [MPLS: Lbl 16624 TC 0 S 1 TTL 1]
>>  5. AS6805   ae6-0.0002.corx.02.fra.de.net.telefonica.de (62.53.0.49)                                                                                           0.0%  1067   41.6  19.6  17.2  51.6   3.9
>>     [MPLS: Lbl 16624 TC 0 S 1 TTL 1]
>>  6. AS6805   bundle-ether2.0001.cord.01.off.de.net.telefonica.de (62.53.0.199)                                                                                  0.0%  1067   19.0  19.3  17.7  87.1   2.2
>>     [MPLS: Lbl 16624 TC 0 S 1 TTL 1]
>>  7. AS6805   bundle-ether1.0002.corp.01.off.de.net.telefonica.de (62.53.28.171)                                                                                 0.0%  1067   22.9  19.2  17.5  29.7   0.9
>>  8. AS???    ipv4.de-cix.fra.de.as12306.plusline.net (80.81.192.132)                                                                                            0.0%  1066   18.5  19.7  18.2  93.0   2.5
>>  9. AS12306  82.98.102.71 (82.98.102.71)                                                                                                                        0.0%  1066   19.6  19.4  17.7  69.5   1.7
>>     [MPLS: Lbl 24002 TC 0 S 1 TTL 1]
>> 10. AS12306  82.98.102.23 (82.98.102.23)                                                                                                                        0.0%  1066   17.8  19.8  17.4 197.7   9.9
>> 11. AS12306  212.19.61.13 (212.19.61.13)                                                                                                                        0.0%  1066   19.8  19.6  17.5 155.7   6.1
>> 12. AS12306  www.heise.de (193.99.144.85)                                                                                                                       0.1%  1066   19.6  19.1  17.6  63.7   1.5
>>
>> Not that these do not already stick out as hops at different locations (here Hamburg (ham), Frankfurt (fra) and Offenbach)) all with more or less identical distance-independent RTTs.
>>
>> Other than that quite intriguing puzzle, I wonder whether we could teach flent to run a bidirectional traceroute/mtr as part of its measurements..?
>>
>> Regards
>>         Sebastian
>>
>>
>> > On Aug 30, 2023, at 20:07, Dave Taht via Starlink <starlink@lists.bufferbloat.net> wrote:
>> >
>> > In the attached 5 minute plot from a few days ago (I can supply the
>> > flent.gz files if anyone wants them), I see a puzzling spike at T+155s
>> > to nearly 90ms of baseline latency, then down to 20ms. No degree of
>> > orbital mechanics can apply to this change, even factoring in an over
>> > the horizon connection, routing packets on the ground through LA to
>> > seattle, and back, or using a couple ISLs, can make this add up for
>> > me. A combination of all that, kind of does make sense.
>> >
>> > The trace otherwise shows the sawtooth pattern of a single tcp flow ,
>> > a loss (sometimes catastrophic) at every downward bandwidth change.
>> >
>> > An assumption I have long been making is the latency staircase effect
>> > (see T+170) forward is achieving the best encoding rate at the
>> > distance then seen, the distance growing and the encoding rate falling
>> > in distinct steps, with a fixed amount of buffering, until finally
>> > that sat starts falling out of range, and it choses another at T+240s.
>> >
>> > But jeeze, a 70ms baseline latency swing? What gives?  I imagine
>> > somehow correlating this with a mpls enabled traceroute might begin to
>> > make some sense of it, correlated by orbital positions....
>> >
>> >
>> > --
>> > Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
>> > Dave Täht CSO, LibreQos
>> > <thestarlinkmystery.png>_______________________________________________
>> > Starlink mailing list
>> > Starlink@lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/starlink
>>
>
>

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [Starlink] a puzzling starlink uplink trace

[Alexandre Petrescu]

Le 30/08/2023 à 20:07, Dave Taht via Starlink a écrit :
> In the attached 5 minute plot from a few days ago (I can supply the
> flent.gz files if anyone wants them), I see a puzzling spike at T+155s
> to nearly 90ms of baseline latency, then down to 20ms.

20ms?

A latency of 20ms might come if these low altitude starlink sats (70km 
or so) pass by there?

Or maybe I dont see quite well these sat altitudes.

Alex


>   No degree of
> orbital mechanics can apply to this change, even factoring in an over
> the horizon connection, routing packets on the ground through LA to
> seattle, and back, or using a couple ISLs, can make this add up for
> me. A combination of all that, kind of does make sense.
>
> The trace otherwise shows the sawtooth pattern of a single tcp flow ,
> a loss (sometimes catastrophic) at every downward bandwidth change.
>
> An assumption I have long been making is the latency staircase effect
> (see T+170) forward is achieving the best encoding rate at the
> distance then seen, the distance growing and the encoding rate falling
> in distinct steps, with a fixed amount of buffering, until finally
> that sat starts falling out of range, and it choses another at T+240s.
>
> But jeeze, a 70ms baseline latency swing? What gives?  I imagine
> somehow correlating this with a mpls enabled traceroute might begin to
> make some sense of it, correlated by orbital positions....
>
>
>
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink

Re: [Starlink] a puzzling starlink uplink trace

[Alexandre Petrescu]

to clarify: I some times look with satmap.space and with n2yo.com at sat 
altitudes.

Some times I see some starlink sats at lower altitudes than LEO (some 
times at 70km).

Right now I see sat STARLINK-6065 at 360 km altitude, which is way below 
LEO and typical 550km altitude of starlink sats.

https://www.n2yo.com/satellite/?s=56812

If these altitude reports are correct, then I think it is hard to say 
Starlink is anymore simply a LEO constellation.  It is much lower than that.

Further, if the 20ms latency report is due to that 365 km altitude then 
it is very easy to imagine what lower altitudes would give.

If one reports a great ms latency then it would be great to tell which 
sat at which altitude was there above at that timestamp.

Alex

Le 31/08/2023 à 10:56, Alexandre Petrescu via Starlink a écrit :
>
> Le 30/08/2023 à 20:07, Dave Taht via Starlink a écrit :
>> In the attached 5 minute plot from a few days ago (I can supply the
>> flent.gz files if anyone wants them), I see a puzzling spike at T+155s
>> to nearly 90ms of baseline latency, then down to 20ms.
>
> 20ms?
>
> A latency of 20ms might come if these low altitude starlink sats (70km 
> or so) pass by there?
>
> Or maybe I dont see quite well these sat altitudes.
>
> Alex
>
>
>>   No degree of
>> orbital mechanics can apply to this change, even factoring in an over
>> the horizon connection, routing packets on the ground through LA to
>> seattle, and back, or using a couple ISLs, can make this add up for
>> me. A combination of all that, kind of does make sense.
>>
>> The trace otherwise shows the sawtooth pattern of a single tcp flow ,
>> a loss (sometimes catastrophic) at every downward bandwidth change.
>>
>> An assumption I have long been making is the latency staircase effect
>> (see T+170) forward is achieving the best encoding rate at the
>> distance then seen, the distance growing and the encoding rate falling
>> in distinct steps, with a fixed amount of buffering, until finally
>> that sat starts falling out of range, and it choses another at T+240s.
>>
>> But jeeze, a 70ms baseline latency swing? What gives?  I imagine
>> somehow correlating this with a mpls enabled traceroute might begin to
>> make some sense of it, correlated by orbital positions....
>>
>>
>>
>> _______________________________________________
>> Starlink mailing list
>> Starlink@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/starlink
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink

Re: [Starlink] a puzzling starlink uplink trace

[David Lang]

[-- Attachment #1: Type: text/plain, Size: 3577 bytes --]

satellites below the allocated orbital shells are in transit, either going up or 
coming down. It takes a LOT of energy to change the orbital altitude.

anything at 70km is in the process of re-entering (at launch they are still on 
the rocket at that altitude), so if you are getting reports of satellites being 
operational at that altitude, your source of reports is faulty.

David Lang

  On Thu, 31 Aug 
2023, Alexandre Petrescu via Starlink wrote:

> Date: Thu, 31 Aug 2023 11:13:07 +0200
> From: Alexandre Petrescu via Starlink <starlink@lists.bufferbloat.net>
> Reply-To: Alexandre Petrescu <alexandre.petrescu@gmail.com>
> To: starlink@lists.bufferbloat.net
> Subject: Re: [Starlink] a puzzling starlink uplink trace
> 
> to clarify: I some times look with satmap.space and with n2yo.com at sat 
> altitudes.
>
> Some times I see some starlink sats at lower altitudes than LEO (some 
> times at 70km).
>
> Right now I see sat STARLINK-6065 at 360 km altitude, which is way below 
> LEO and typical 550km altitude of starlink sats.
>
> https://www.n2yo.com/satellite/?s=56812
>
> If these altitude reports are correct, then I think it is hard to say 
> Starlink is anymore simply a LEO constellation.  It is much lower than that.
>
> Further, if the 20ms latency report is due to that 365 km altitude then 
> it is very easy to imagine what lower altitudes would give.
>
> If one reports a great ms latency then it would be great to tell which 
> sat at which altitude was there above at that timestamp.
>
> Alex
>
> Le 31/08/2023 à 10:56, Alexandre Petrescu via Starlink a écrit :
>>
>> Le 30/08/2023 à 20:07, Dave Taht via Starlink a écrit :
>>> In the attached 5 minute plot from a few days ago (I can supply the
>>> flent.gz files if anyone wants them), I see a puzzling spike at T+155s
>>> to nearly 90ms of baseline latency, then down to 20ms.
>>
>> 20ms?
>>
>> A latency of 20ms might come if these low altitude starlink sats (70km 
>> or so) pass by there?
>>
>> Or maybe I dont see quite well these sat altitudes.
>>
>> Alex
>>
>>
>>>   No degree of
>>> orbital mechanics can apply to this change, even factoring in an over
>>> the horizon connection, routing packets on the ground through LA to
>>> seattle, and back, or using a couple ISLs, can make this add up for
>>> me. A combination of all that, kind of does make sense.
>>>
>>> The trace otherwise shows the sawtooth pattern of a single tcp flow ,
>>> a loss (sometimes catastrophic) at every downward bandwidth change.
>>>
>>> An assumption I have long been making is the latency staircase effect
>>> (see T+170) forward is achieving the best encoding rate at the
>>> distance then seen, the distance growing and the encoding rate falling
>>> in distinct steps, with a fixed amount of buffering, until finally
>>> that sat starts falling out of range, and it choses another at T+240s.
>>>
>>> But jeeze, a 70ms baseline latency swing? What gives?  I imagine
>>> somehow correlating this with a mpls enabled traceroute might begin to
>>> make some sense of it, correlated by orbital positions....
>>>
>>>
>>>
>>> _______________________________________________
>>> Starlink mailing list
>>> Starlink@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/starlink
>> _______________________________________________
>> Starlink mailing list
>> Starlink@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/starlink
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
>

Re: [Starlink] a puzzling starlink uplink trace

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 4661 bytes --]

Zooming in on the anomalous behavior (attached, I also include the
flent.gz file for those of you that have the flent-gui installed,
which contains a timestamp for those of you plugged into orbit
calculation software), and plotting ping, tcp rtt and throughput, the
ping does indeed hit 20ms, but the tcp rtt only falls to 40ms. Could
they be synthesizing ping responses? I can think of a few ways to
detect that (putting some data in the payload, trying a different ping
type, like 13 (it tickled me that the sqm-autorate project discovered
that the ping timestamp option actually worked over some devices). Or
perhaps ping was taking a different sat entirely?

I like the idea of starlink finding some way to leverage satellites
moving into the right orbits. (the timestamp of this trace was
2023-08-27 06:59:35 UTC and I am in half moon bay, ca) This would
imply that they can transmit data at full thrust, and also, perhaps,
VLEO (I am not sure if that is an established acronym - very low earth
orbit operations) -  feasible. Dropping the things even lower, making
the shell more aerodynamic, and burning fuel to stay there might well
be an interesting option. It seems to me the current altitude(s) are
pretty conservative and given the fuel consumption reported on the
maneuvering side, they can last at 530km much longer than 5 years, and
with the pace of technology and launch rates, moving them lower would
be a huge win for bandwidth and latency.

On Thu, Aug 31, 2023 at 2:13 AM Alexandre Petrescu via Starlink
<starlink@lists.bufferbloat.net> wrote:
>
> to clarify: I some times look with satmap.space and with n2yo.com at sat
> altitudes.
>
> Some times I see some starlink sats at lower altitudes than LEO (some
> times at 70km).
>
> Right now I see sat STARLINK-6065 at 360 km altitude, which is way below
> LEO and typical 550km altitude of starlink sats.
>
> https://www.n2yo.com/satellite/?s=56812
>
> If these altitude reports are correct, then I think it is hard to say
> Starlink is anymore simply a LEO constellation.  It is much lower than that.
>
> Further, if the 20ms latency report is due to that 365 km altitude then
> it is very easy to imagine what lower altitudes would give.
>
> If one reports a great ms latency then it would be great to tell which
> sat at which altitude was there above at that timestamp.
>
> Alex
>
> Le 31/08/2023 à 10:56, Alexandre Petrescu via Starlink a écrit :
> >
> > Le 30/08/2023 à 20:07, Dave Taht via Starlink a écrit :
> >> In the attached 5 minute plot from a few days ago (I can supply the
> >> flent.gz files if anyone wants them), I see a puzzling spike at T+155s
> >> to nearly 90ms of baseline latency, then down to 20ms.
> >
> > 20ms?
> >
> > A latency of 20ms might come if these low altitude starlink sats (70km
> > or so) pass by there?
> >
> > Or maybe I dont see quite well these sat altitudes.
> >
> > Alex
> >
> >
> >>   No degree of
> >> orbital mechanics can apply to this change, even factoring in an over
> >> the horizon connection, routing packets on the ground through LA to
> >> seattle, and back, or using a couple ISLs, can make this add up for
> >> me. A combination of all that, kind of does make sense.
> >>
> >> The trace otherwise shows the sawtooth pattern of a single tcp flow ,
> >> a loss (sometimes catastrophic) at every downward bandwidth change.
> >>
> >> An assumption I have long been making is the latency staircase effect
> >> (see T+170) forward is achieving the best encoding rate at the
> >> distance then seen, the distance growing and the encoding rate falling
> >> in distinct steps, with a fixed amount of buffering, until finally
> >> that sat starts falling out of range, and it choses another at T+240s.
> >>
> >> But jeeze, a 70ms baseline latency swing? What gives?  I imagine
> >> somehow correlating this with a mpls enabled traceroute might begin to
> >> make some sense of it, correlated by orbital positions....
> >>
> >>
> >>
> >> _______________________________________________
> >> Starlink mailing list
> >> Starlink@lists.bufferbloat.net
> >> https://lists.bufferbloat.net/listinfo/starlink
> > _______________________________________________
> > Starlink mailing list
> > Starlink@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/starlink
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink



-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

[-- Attachment #2: mystery_tcp_rtt_ping.png --]
[-- Type: image/png, Size: 368976 bytes --]

[-- Attachment #3: tcp_nup-2023-08-27T065935.741922.starlink-long-ipv4-1-flows.flent.gz --]
[-- Type: application/gzip, Size: 196415 bytes --]

Re: [Starlink] a puzzling starlink uplink trace

[David Lang]

[-- Attachment #1: Type: text/plain, Size: 1821 bytes --]

On Thu, 31 Aug 2023, Dave Taht via Starlink wrote:

> I like the idea of starlink finding some way to leverage satellites
> moving into the right orbits. (the timestamp of this trace was
> 2023-08-27 06:59:35 UTC and I am in half moon bay, ca) This would
> imply that they can transmit data at full thrust, and also, perhaps,
> VLEO (I am not sure if that is an established acronym - very low earth
> orbit operations) -  feasible. Dropping the things even lower, making
> the shell more aerodynamic, and burning fuel to stay there might well
> be an interesting option. It seems to me the current altitude(s) are
> pretty conservative and given the fuel consumption reported on the
> maneuvering side, they can last at 530km much longer than 5 years, and
> with the pace of technology and launch rates, moving them lower would
> be a huge win for bandwidth and latency.

It's not just a fuel issue, it's a matter of their assigned orbits (not getting 
in the way of other satellites)

Here are the altitudes they are authorized to operate in and the current count 
per orbit (from wikipedia so I don't know how up to date they are given the 
frequent launches, but I suspect pretty close)

Altitude	Authorized	Active	Decaying/deorbited
550 km		1584[313]	1457	268
570 km		720		404	4
560 km		348		233	10
540 km		1584		1567	70
560 km		172
335.9 km	2493
340.8 km	2478
345.6 km	2547

it's not a matter of being able to operate while under thrust from a technical 
point of view, it's a matter of them being authorized by the FCC to do so and 
anti-collision rules.

the US considers 'space' to start at a round number of 50 miles (just over 
80km), the metric world picked 100km (another nice round number, just over 62 
miles), so any satellite at 70km is technically not 'in space' and will die in a 
few orbits

David Lang

[-- Attachment #2: Type: image/png, Size: 368976 bytes --]

[-- Attachment #3: Type: application/gzip, Size: 196415 bytes --]

[-- Attachment #4: Type: text/plain, Size: 149 bytes --]

_______________________________________________
Starlink mailing list
Starlink@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink

Re: [Starlink] a puzzling starlink uplink trace

[Alexandre Petrescu]

Le 31/08/2023 à 15:36, Dave Taht a écrit :
[...]
> VLEO (I am not sure if that is an established acronym - very low
> earth orbit operations)

I saw very often the term vLEO (very low Earth orbit).  Sometimes it was
said to range between 100k and 400km, at which LEO would start.

However, it is indeed a surprise to see starlink sats below 500km, right
now.

I think the things advance so fast that even the terminology is
fluctuating a lot.  Add to that that the definition of these orbit
altitudes are very approximate, Space is approximate, smaller and
smaller devices are easier and easier to control and describe
trajectories of arbitrary forms.

> -  feasible. Dropping the things even lower, making the shell more
> aerodynamic, and burning fuel to stay there might well be an
> interesting option. It seems to me the current altitude(s) are pretty
> conservative and given the fuel consumption reported on the 
> maneuvering side, they can last at 530km much longer than 5 years,
> and with the pace of technology and launch rates, moving them lower
> would be a huge win for bandwidth and latency.

Yes, I agree.

I would add to that also the new characteristic of being practically
'disposable' - launch, orbit, re-entry - all during maybe 2 to 5 years,
maybe 4 times per year.  Compare that with other sats with lifespans in
the order of 20, 30 years and even more.

With that kind of short lifetime, the quantity of energy needed to
maintain in an 'orbit' at, say 50km altitude, might not be impossible.

If one asks Starlink, I think they'd surely want to make it that way.
But there could be another way in which somebody else than Starlink puts
these sats at lower altitudes, and make constellation-to-constellation
communications.

Also, one inconvenient of these lower altitudes (say, below 300km) might
be (if I understand correctly) that they necessitate smaller sats, i.e.
smaller computers, i.e. lower bandwidths.  These small sats might offer
lower latencies, but lower bandwidths too.  There might be a need to mix
the lower latency of lower altitude sats with the higher bandwidths of
bigger sats at higher altitudes, onto the end user (not onto an
intermediary aggregator).

Alex

> 
> On Thu, Aug 31, 2023 at 2:13 AM Alexandre Petrescu via Starlink 
> <starlink@lists.bufferbloat.net> wrote:
>> 
>> to clarify: I some times look with satmap.space and with n2yo.com
>> at sat altitudes.
>> 
>> Some times I see some starlink sats at lower altitudes than LEO
>> (some times at 70km).
>> 
>> Right now I see sat STARLINK-6065 at 360 km altitude, which is way
>> below LEO and typical 550km altitude of starlink sats.
>> 
>> https://www.n2yo.com/satellite/?s=56812
>> 
>> If these altitude reports are correct, then I think it is hard to
>> say Starlink is anymore simply a LEO constellation.  It is much
>> lower than that.
>> 
>> Further, if the 20ms latency report is due to that 365 km altitude
>> then it is very easy to imagine what lower altitudes would give.
>> 
>> If one reports a great ms latency then it would be great to tell
>> which sat at which altitude was there above at that timestamp.
>> 
>> Alex
>> 
>> Le 31/08/2023 à 10:56, Alexandre Petrescu via Starlink a écrit :
>>> 
>>> Le 30/08/2023 à 20:07, Dave Taht via Starlink a écrit :
>>>> In the attached 5 minute plot from a few days ago (I can supply
>>>> the flent.gz files if anyone wants them), I see a puzzling
>>>> spike at T+155s to nearly 90ms of baseline latency, then down
>>>> to 20ms.
>>> 
>>> 20ms?
>>> 
>>> A latency of 20ms might come if these low altitude starlink sats
>>> (70km or so) pass by there?
>>> 
>>> Or maybe I dont see quite well these sat altitudes.
>>> 
>>> Alex
>>> 
>>> 
>>>> No degree of orbital mechanics can apply to this change, even
>>>> factoring in an over the horizon connection, routing packets on
>>>> the ground through LA to seattle, and back, or using a couple
>>>> ISLs, can make this add up for me. A combination of all that,
>>>> kind of does make sense.
>>>> 
>>>> The trace otherwise shows the sawtooth pattern of a single tcp
>>>> flow , a loss (sometimes catastrophic) at every downward
>>>> bandwidth change.
>>>> 
>>>> An assumption I have long been making is the latency staircase
>>>> effect (see T+170) forward is achieving the best encoding rate
>>>> at the distance then seen, the distance growing and the
>>>> encoding rate falling in distinct steps, with a fixed amount of
>>>> buffering, until finally that sat starts falling out of range,
>>>> and it choses another at T+240s.
>>>> 
>>>> But jeeze, a 70ms baseline latency swing? What gives?  I
>>>> imagine somehow correlating this with a mpls enabled traceroute
>>>> might begin to make some sense of it, correlated by orbital
>>>> positions....
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________ Starlink
>>>> mailing list Starlink@lists.bufferbloat.net 
>>>> https://lists.bufferbloat.net/listinfo/starlink
>>> _______________________________________________ Starlink mailing
>>> list Starlink@lists.bufferbloat.net 
>>> https://lists.bufferbloat.net/listinfo/starlink
>> _______________________________________________ Starlink mailing
>> list Starlink@lists.bufferbloat.net 
>> https://lists.bufferbloat.net/listinfo/starlink
> 
> 
>