Re: [Starlink] insanely great waveform result for starlink

[Dave Taht <dave.taht@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 19271 bytes --]

Lovely pics, thank you, and the sawtooths are obvious, for a change, so in
my mind they've cut the buffering back quite a bit since we last checked.
The amount of TCP related loss remains low, at least according to the
packet captures (not enough loss for tcp! a BDP should settle in at 40ms
with a fifo, not 160+!).

Kind of related to my "glitches per minute" metric described here:
https://blog.cerowrt.org/post/speedtests/

What I'm interested in is the distribution pattern of the irtt udp loss. A
way to plot that might be along the -1 vertical axis, but incremented down
one further for every consecutive loss, and dots, colored,
blue->green->yellow->orange->red - voip can survive 3/5 losses in a row
fairly handily. Two ways

 + axis as it is now
----------------------------------------got a packet
---------------------------------------------
B  loss 1                                                       loss 1

G         2 loss in a row                                             loss 2
Y                                 3rd loss
         los 3
O
                        4
R
                           5
R
                              ....
R
R
R

and/or the same as above but plotting the rx vs tx loss


On Sat, Jan 14, 2023 at 6:20 AM Nathan Owens <nathan@nathan.io> wrote:

> Sorry, was rejected from the listserv, here's the google drive link for
> all 3 IRTT's visualized
>
>
> https://drive.google.com/drive/folders/1SbUyvdlfdllgqrEcSSxvGdn3yN40vSx-?usp=share_link
>
> On Sat, Jan 14, 2023 at 6:14 AM Nathan Owens <nathan@nathan.io> wrote:
>
>> I realized I goofed up my visualizations, here's all of them again:
>> I see a ton of loss on all of these!
>>
>> While Luis was downloading/uploading... oof.
>> [image: Screenshot 2023-01-14 at 6.13.50 AM.png]
>>
>> [image: Screenshot 2023-01-14 at 6.12.53 AM.png]
>>
>>
>>
>>
>> On Fri, Jan 13, 2023 at 3:44 PM Dave Taht <dave.taht@gmail.com> wrote:
>>
>>> I am forced to conclude that waveform's upload test is broken in some
>>> cases and has been for some time. All the joy many have been feeling about
>>> their uplink speeds has to be cast away. Felt good, though, didn't it?
>>>
>>> There is a *slight* possibility that there is some fq in the starlink
>>> network on tcp port 433. The first showing normal rtt growth and loss for
>>> cubic, the second, a low rate flow that is holding the line... but I didn't
>>> check to see if these were sequential or parallel.
>>>
>>> The last is a cwnd plot, clearly showing the cubic sawtooth on the
>>> upload.
>>>
>>> It's weirdly nice to be able to follow a port 433 stream, see the tls
>>> handshake put the website en'clair, and the rest go dark, and still trace
>>> the behaviors.
>>>
>>> we're so going to lose this analytical ability with quic. I'm enjoying
>>> it while we still can.
>>>
>>>
>>> On Fri, Jan 13, 2023 at 2:10 PM Jonathan Bennett via Starlink <
>>> starlink@lists.bufferbloat.net> wrote:
>>>
>>>> The irtt run finished a few seconds before the flent run, but here are
>>>> the results:
>>>>
>>>>
>>>> https://drive.google.com/file/d/1FKve13ssUMW1LLWOXLM2931Yx6uMHw8K/view?usp=share_link
>>>>
>>>> https://drive.google.com/file/d/1ZXd64A0pfUedLr3FyhDNTHA7vxv8S2Gk/view?usp=share_link
>>>>
>>>> https://drive.google.com/file/d/1rx64UPQHHz3IMNiJtb1oFqtqw2DvhvEE/view?usp=share_link
>>>>
>>>>
>>>> [image: image.png]
>>>> [image: image.png]
>>>>
>>>>
>>>> Jonathan Bennett
>>>> Hackaday.com
>>>>
>>>>
>>>> On Fri, Jan 13, 2023 at 3:30 PM Nathan Owens via Starlink <
>>>> starlink@lists.bufferbloat.net> wrote:
>>>>
>>>>> Here's Luis's run -- the top line below the edge of the graph is 200ms
>>>>> [image: Screenshot 2023-01-13 at 1.30.03 PM.png]
>>>>>
>>>>>
>>>>> On Fri, Jan 13, 2023 at 1:25 PM Luis A. Cornejo <
>>>>> luis.a.cornejo@gmail.com> wrote:
>>>>>
>>>>>> Dave,
>>>>>>
>>>>>> Here is a run the way I think you wanted it.
>>>>>>
>>>>>> irtt running for 5 min to your dallas server, followed by a waveform
>>>>>> test, then a few seconds of inactivity, cloudflare test, a few more secs of
>>>>>> nothing, flent test to dallas. Packet capture is currently uploading (will
>>>>>> be done in 20 min or so), irtt JSON also in there (.zip file):
>>>>>>
>>>>>>
>>>>>> https://drive.google.com/drive/folders/1FLWqrzNcM8aK-ZXQywNkZGFR81Fnzn-F?usp=share_link
>>>>>>
>>>>>> -Luis
>>>>>>
>>>>>> On Fri, Jan 13, 2023 at 2:50 PM Dave Taht via Starlink <
>>>>>> starlink@lists.bufferbloat.net> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jan 13, 2023 at 12:30 PM Nathan Owens <nathan@nathan.io>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Here's the data visualization for Johnathan's Data
>>>>>>>>
>>>>>>>> [image: Screenshot 2023-01-13 at 12.29.15 PM.png]
>>>>>>>>
>>>>>>>> You can see the path change at :12, :27, :42, :57 after the minute.
>>>>>>>> Some paths are clearly busier than others with increased loss, latency, and
>>>>>>>> jitter.
>>>>>>>>
>>>>>>>
>>>>>>> I am so glad to see loss and bounded delay here. Also a bit of rigor
>>>>>>> regarding what traffic was active locally vs on the path would be nice,
>>>>>>> although it seems to line up with the known 15s starlink switchover thing
>>>>>>> (need a name for this), in this case, doing a few speedtests
>>>>>>> while that irtt is running would show the impact(s) of whatever else
>>>>>>> they are up to.
>>>>>>>
>>>>>>> It would also be my hope that the loss distribution in the middle
>>>>>>> portion of this data is good, not bursty, but we don't have a tool to take
>>>>>>> apart that. (I am so hopeless at json)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jan 13, 2023 at 10:09 AM Nathan Owens <nathan@nathan.io>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I’ll run my visualization code on this result this afternoon and
>>>>>>>>> report back!
>>>>>>>>>
>>>>>>>>> On Fri, Jan 13, 2023 at 9:41 AM Jonathan Bennett via Starlink <
>>>>>>>>> starlink@lists.bufferbloat.net> wrote:
>>>>>>>>>
>>>>>>>>>> The irtt command, run with normal, light usage:
>>>>>>>>>> https://drive.google.com/file/d/1SiVCiUYnx7nDTxIVOY5w-z20S2O059rA/view?usp=share_link
>>>>>>>>>>
>>>>>>>>>> Jonathan Bennett
>>>>>>>>>> Hackaday.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Jan 13, 2023 at 11:26 AM Dave Taht <dave.taht@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> packet caps would be nice... all this is very exciting news.
>>>>>>>>>>>
>>>>>>>>>>> I'd so love for one or more of y'all reporting such great uplink
>>>>>>>>>>> results nowadays to duplicate and re-plot the original irtt
>>>>>>>>>>> tests we
>>>>>>>>>>> did:
>>>>>>>>>>>
>>>>>>>>>>> irtt client -i3ms -d300s myclosestservertoyou.starlink.taht.net
>>>>>>>>>>> -o whatever.json
>>>>>>>>>>>
>>>>>>>>>>> They MUST have changed their scheduling to get such amazing
>>>>>>>>>>> uplink
>>>>>>>>>>> results, in addition to better queue management.
>>>>>>>>>>>
>>>>>>>>>>> (for the record, my servers are de, london, fremont, sydney,
>>>>>>>>>>> dallas,
>>>>>>>>>>> newark, atlanta, singapore, mumbai)
>>>>>>>>>>>
>>>>>>>>>>> There's an R and gnuplot script for plotting that output around
>>>>>>>>>>> here
>>>>>>>>>>> somewhere (I have largely personally put down the starlink
>>>>>>>>>>> project,
>>>>>>>>>>> loaning out mine) - that went by on this list... I should have
>>>>>>>>>>> written
>>>>>>>>>>> a blog entry so I can find that stuff again.
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jan 13, 2023 at 9:02 AM Jonathan Bennett via Starlink
>>>>>>>>>>> <starlink@lists.bufferbloat.net> wrote:
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > On Fri, Jan 13, 2023 at 6:28 AM Ulrich Speidel via Starlink <
>>>>>>>>>>> starlink@lists.bufferbloat.net> wrote:
>>>>>>>>>>> >>
>>>>>>>>>>> >> On 13/01/2023 6:13 pm, Ulrich Speidel wrote:
>>>>>>>>>>> >> >
>>>>>>>>>>> >> > From Auckland, New Zealand, using a roaming subscription,
>>>>>>>>>>> it puts me
>>>>>>>>>>> >> > in touch with a server 2000 km away. OK then:
>>>>>>>>>>> >> >
>>>>>>>>>>> >> >
>>>>>>>>>>> >> > IP address: nix six.
>>>>>>>>>>> >> >
>>>>>>>>>>> >> > My thoughts shall follow later.
>>>>>>>>>>> >>
>>>>>>>>>>> >> OK, so here we go.
>>>>>>>>>>> >>
>>>>>>>>>>> >> I'm always a bit skeptical when it comes to speed tests -
>>>>>>>>>>> they're really
>>>>>>>>>>> >> laden with so many caveats that it's not funny. I took our
>>>>>>>>>>> new work
>>>>>>>>>>> >> Starlink kit home in December to give it a try and the other
>>>>>>>>>>> day finally
>>>>>>>>>>> >> got around to set it up. It's on a roaming subscription
>>>>>>>>>>> because our
>>>>>>>>>>> >> badly built-up campus really isn't ideal in terms of a clear
>>>>>>>>>>> view of the
>>>>>>>>>>> >> sky. Oh - and did I mention that I used the Starlink Ethernet
>>>>>>>>>>> adapter,
>>>>>>>>>>> >> not the WiFi?
>>>>>>>>>>> >>
>>>>>>>>>>> >> Caveat 1: Location, location. I live in a place where the
>>>>>>>>>>> best Starlink
>>>>>>>>>>> >> promises is about 1/3 in terms of data rate you can actually
>>>>>>>>>>> get from
>>>>>>>>>>> >> fibre to the home at under half of Starlink's price. Read:
>>>>>>>>>>> There are few
>>>>>>>>>>> >> Starlink users around. I might be the only one in my suburb.
>>>>>>>>>>> >>
>>>>>>>>>>> >> Caveat 2: Auckland has three Starlink gateways close by:
>>>>>>>>>>> Clevedon (which
>>>>>>>>>>> >> is at a stretch daytrip cycling distance from here), Te Hana
>>>>>>>>>>> and Puwera,
>>>>>>>>>>> >> the most distant of the three and about 130 km away from me
>>>>>>>>>>> as the crow
>>>>>>>>>>> >> flies. Read: My dishy can use any satellite that any of these
>>>>>>>>>>> three can
>>>>>>>>>>> >> see, and then depending on where I put it and how much of the
>>>>>>>>>>> southern
>>>>>>>>>>> >> sky it can see, maybe also the one in Hinds, 840 km away,
>>>>>>>>>>> although that
>>>>>>>>>>> >> is obviously stretching it a bit. Either way, that's plenty
>>>>>>>>>>> of options
>>>>>>>>>>> >> for my bits to travel without needing a lot of handovers.
>>>>>>>>>>> Why? Easy: If
>>>>>>>>>>> >> your nearest teleport is close by, then the set of satellites
>>>>>>>>>>> that the
>>>>>>>>>>> >> teleport can see and the set that you can see is almost the
>>>>>>>>>>> same, so you
>>>>>>>>>>> >> can essentially stick with the same satellite while it's in
>>>>>>>>>>> view for you
>>>>>>>>>>> >> because it'll also be in view for the teleport. Pretty much
>>>>>>>>>>> any bird
>>>>>>>>>>> >> above you will do.
>>>>>>>>>>> >>
>>>>>>>>>>> >> And because I don't get a lot of competition from other users
>>>>>>>>>>> in my area
>>>>>>>>>>> >> vying for one of the few available satellites that can see
>>>>>>>>>>> both us and
>>>>>>>>>>> >> the teleport, this is about as good as it gets at 37S
>>>>>>>>>>> latitude. If I'd
>>>>>>>>>>> >> want it any better, I'd have to move a lot further south.
>>>>>>>>>>> >>
>>>>>>>>>>> >> It'd be interesting to hear from Jonathan what the
>>>>>>>>>>> availability of home
>>>>>>>>>>> >> broadband is like in the Dallas area. I note that it's at a
>>>>>>>>>>> lower
>>>>>>>>>>> >> latitude (33N) than Auckland, but the difference isn't huge.
>>>>>>>>>>> I notice
>>>>>>>>>>> >> two teleports each about 160 km away, which is also not too
>>>>>>>>>>> bad. I also
>>>>>>>>>>> >> note Starlink availability in the area is restricted at the
>>>>>>>>>>> moment -
>>>>>>>>>>> >> oversubscribed? But if Jonathan gets good data rates, then
>>>>>>>>>>> that means
>>>>>>>>>>> >> that competition for bird capacity can't be too bad - for
>>>>>>>>>>> whatever reason.
>>>>>>>>>>> >
>>>>>>>>>>> > I'm in Southwest Oklahoma, but Dallas is the nearby Starlink
>>>>>>>>>>> gateway. In cities, like Dallas, and Lawton where I live, there are good
>>>>>>>>>>> broadband options. But there are also many people that live outside cities,
>>>>>>>>>>> and the options are much worse. The low density userbase in rural Oklahoma
>>>>>>>>>>> and Texas is probably ideal conditions for Starlink.
>>>>>>>>>>> >>
>>>>>>>>>>> >>
>>>>>>>>>>> >> Caveat 3: Backhaul. There isn't just one queue between me and
>>>>>>>>>>> whatever I
>>>>>>>>>>> >> talk to in terms of my communications. Traceroute shows about
>>>>>>>>>>> 10 hops
>>>>>>>>>>> >> between me and the University of Auckland via Starlink.
>>>>>>>>>>> That's 10
>>>>>>>>>>> >> queues, not one. Many of them will have cross traffic. So
>>>>>>>>>>> it's a bit
>>>>>>>>>>> >> hard to tell where our packets really get to wait or where
>>>>>>>>>>> they get
>>>>>>>>>>> >> dropped. The insidious bit here is that a lot of them will be
>>>>>>>>>>> between 1
>>>>>>>>>>> >> Gb/s and 10 Gb/s links, and with a bit of cross traffic, they
>>>>>>>>>>> can all
>>>>>>>>>>> >> turn into bottlenecks. This isn't like a narrowband GEO link
>>>>>>>>>>> of a few
>>>>>>>>>>> >> Mb/s where it's obvious where the dominant long latency
>>>>>>>>>>> bottleneck in
>>>>>>>>>>> >> your TCP connection's path is. Read: It's pretty hard to tell
>>>>>>>>>>> whether a
>>>>>>>>>>> >> drop in "speed" is due to a performance issue in the Starlink
>>>>>>>>>>> system or
>>>>>>>>>>> >> somewhere between Starlink's systems and the target system.
>>>>>>>>>>> >>
>>>>>>>>>>> >> I see RTTs here between 20 ms and 250 ms, where the physical
>>>>>>>>>>> latency
>>>>>>>>>>> >> should be under 15 ms. So there's clearly a bit of buffer
>>>>>>>>>>> here along the
>>>>>>>>>>> >> chain that occasionally fills up.
>>>>>>>>>>> >>
>>>>>>>>>>> >> Caveat 4: Handovers. Handover between birds and teleports is
>>>>>>>>>>> inevitably
>>>>>>>>>>> >> associated with a change in RTT and in most cases also
>>>>>>>>>>> available
>>>>>>>>>>> >> bandwidth. Plus your packets now arrive at a new queue on a
>>>>>>>>>>> new
>>>>>>>>>>> >> satellite while your TCP is still trying to respond to
>>>>>>>>>>> whatever it
>>>>>>>>>>> >> thought the queue on the previous bird was doing. Read:
>>>>>>>>>>> Whatever your
>>>>>>>>>>> >> cwnd is immediately after a handover, it's probably not what
>>>>>>>>>>> it should be.
>>>>>>>>>>> >>
>>>>>>>>>>> >> I ran a somewhat hamstrung (sky view restricted) set of four
>>>>>>>>>>> Ookla
>>>>>>>>>>> >> speedtest.net tests each to five local servers. Average
>>>>>>>>>>> upload rate was
>>>>>>>>>>> >> 13 Mb/s, average down 75.5 Mb/s. Upload to the server of the
>>>>>>>>>>> ISP that
>>>>>>>>>>> >> Starlink seems to be buying its local connectivity from
>>>>>>>>>>> (Vocus Group)
>>>>>>>>>>> >> varied between 3.04 and 14.38 Mb/s, download between 23.33
>>>>>>>>>>> and 52.22
>>>>>>>>>>> >> Mb/s, with RTTs between 37 and 56 ms not correlating well to
>>>>>>>>>>> rates
>>>>>>>>>>> >> observed. In fact, they were the ISP with consistently the
>>>>>>>>>>> worst rates.
>>>>>>>>>>> >>
>>>>>>>>>>> >> Another ISP (MyRepublic) scored between 11.81 and 21.81 Mb/s
>>>>>>>>>>> up and
>>>>>>>>>>> >> between 106.5 and 183.8 Mb/s down, again with RTTs badly
>>>>>>>>>>> correlating
>>>>>>>>>>> >> with rates. Average RTT was the same as for Vocus.
>>>>>>>>>>> >>
>>>>>>>>>>> >> Note the variation though: More or less a factor of two
>>>>>>>>>>> between highest
>>>>>>>>>>> >> and lowest rates for each ISP. Did MyRepublic just get lucky
>>>>>>>>>>> in my
>>>>>>>>>>> >> tests? Or is there something systematic behind this? Way too
>>>>>>>>>>> few tests
>>>>>>>>>>> >> to tell.
>>>>>>>>>>> >>
>>>>>>>>>>> >> What these tests do is establish a ballpark.
>>>>>>>>>>> >>
>>>>>>>>>>> >> I'm currently repeating tests with dish placed on a trestle
>>>>>>>>>>> closer to
>>>>>>>>>>> >> the heavens. This seems to have translated into fewer outages
>>>>>>>>>>> / ping
>>>>>>>>>>> >> losses (around 1/4 of what I had yesterday with dishy on the
>>>>>>>>>>> ground on
>>>>>>>>>>> >> my deck). Still good enough for a lengthy video Skype call
>>>>>>>>>>> with my folks
>>>>>>>>>>> >> in Germany, although they did comment about reduced video
>>>>>>>>>>> quality. But
>>>>>>>>>>> >> maybe that was the lighting or the different background as I
>>>>>>>>>>> wasn't in
>>>>>>>>>>> >> my usual spot with my laptop when I called them.
>>>>>>>>>>> >
>>>>>>>>>>> > Clear view of the sky is king for Starlink reliability. I've
>>>>>>>>>>> got my dishy mounted on the back fence, looking up over an empty field, so
>>>>>>>>>>> it's pretty much best-case scenario here.
>>>>>>>>>>> >>
>>>>>>>>>>> >>
>>>>>>>>>>> >> --
>>>>>>>>>>> >>
>>>>>>>>>>> >>
>>>>>>>>>>> ****************************************************************
>>>>>>>>>>> >> Dr. Ulrich Speidel
>>>>>>>>>>> >>
>>>>>>>>>>> >> School of Computer Science
>>>>>>>>>>> >>
>>>>>>>>>>> >> Room 303S.594 (City Campus)
>>>>>>>>>>> >>
>>>>>>>>>>> >> The University of Auckland
>>>>>>>>>>> >> u.speidel@auckland.ac.nz
>>>>>>>>>>> >> http://www.cs.auckland.ac.nz/~ulrich/
>>>>>>>>>>> >>
>>>>>>>>>>> ****************************************************************
>>>>>>>>>>> >>
>>>>>>>>>>> >>
>>>>>>>>>>> >>
>>>>>>>>>>> >> _______________________________________________
>>>>>>>>>>> >> Starlink mailing list
>>>>>>>>>>> >> Starlink@lists.bufferbloat.net
>>>>>>>>>>> >> https://lists.bufferbloat.net/listinfo/starlink
>>>>>>>>>>> >
>>>>>>>>>>> > _______________________________________________
>>>>>>>>>>> > Starlink mailing list
>>>>>>>>>>> > Starlink@lists.bufferbloat.net
>>>>>>>>>>> > https://lists.bufferbloat.net/listinfo/starlink
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> This song goes out to all the folk that thought Stadia would
>>>>>>>>>>> work:
>>>>>>>>>>>
>>>>>>>>>>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
>>>>>>>>>>> Dave Täht CEO, TekLibre, LLC
>>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Starlink mailing list
>>>>>>>>>> Starlink@lists.bufferbloat.net
>>>>>>>>>> https://lists.bufferbloat.net/listinfo/starlink
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> This song goes out to all the folk that thought Stadia would work:
>>>>>>>
>>>>>>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
>>>>>>> Dave Täht CEO, TekLibre, LLC
>>>>>>> _______________________________________________
>>>>>>> Starlink mailing list
>>>>>>> Starlink@lists.bufferbloat.net
>>>>>>> https://lists.bufferbloat.net/listinfo/starlink
>>>>>>>
>>>>>> _______________________________________________
>>>>> Starlink mailing list
>>>>> Starlink@lists.bufferbloat.net
>>>>> https://lists.bufferbloat.net/listinfo/starlink
>>>>>
>>>> _______________________________________________
>>>> Starlink mailing list
>>>> Starlink@lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/starlink
>>>>
>>>
>>>
>>> --
>>> This song goes out to all the folk that thought Stadia would work:
>>>
>>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
>>> Dave Täht CEO, TekLibre, LLC
>>>
>>

-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
Dave Täht CEO, TekLibre, LLC

[-- Attachment #1.2: Type: text/html, Size: 25327 bytes --]

[-- Attachment #2: Screenshot 2023-01-13 at 12.29.15 PM.png --]
[-- Type: image/png, Size: 1380606 bytes --]

[-- Attachment #3: Screenshot 2023-01-13 at 1.30.03 PM.png --]
[-- Type: image/png, Size: 855840 bytes --]

[-- Attachment #4: image.png --]
[-- Type: image/png, Size: 296989 bytes --]

[-- Attachment #5: image.png --]
[-- Type: image/png, Size: 222655 bytes --]

[-- Attachment #6: Screenshot 2023-01-14 at 6.12.53 AM.png --]
[-- Type: image/png, Size: 921914 bytes --]

[-- Attachment #7: Screenshot 2023-01-14 at 6.13.50 AM.png --]
[-- Type: image/png, Size: 337863 bytes --]