Lovely pics, thank you, and the sawtooths are obvious, for a change, so in my mind they've cut the buffering back quite a bit since we last checked. The amount of TCP related loss remains low, at least according to the packet captures (not enough loss for tcp! a BDP should settle in at 40ms with a fifo, not 160+!). Kind of related to my "glitches per minute" metric described here: https://blog.cerowrt.org/post/speedtests/ What I'm interested in is the distribution pattern of the irtt udp loss. A way to plot that might be along the -1 vertical axis, but incremented down one further for every consecutive loss, and dots, colored, blue->green->yellow->orange->red - voip can survive 3/5 losses in a row fairly handily. Two ways + axis as it is now ----------------------------------------got a packet --------------------------------------------- B loss 1 loss 1 G 2 loss in a row loss 2 Y 3rd loss los 3 O 4 R 5 R .... R R R and/or the same as above but plotting the rx vs tx loss On Sat, Jan 14, 2023 at 6:20 AM Nathan Owens wrote: > Sorry, was rejected from the listserv, here's the google drive link for > all 3 IRTT's visualized > > > https://drive.google.com/drive/folders/1SbUyvdlfdllgqrEcSSxvGdn3yN40vSx-?usp=share_link > > On Sat, Jan 14, 2023 at 6:14 AM Nathan Owens wrote: > >> I realized I goofed up my visualizations, here's all of them again: >> I see a ton of loss on all of these! >> >> While Luis was downloading/uploading... oof. >> [image: Screenshot 2023-01-14 at 6.13.50 AM.png] >> >> [image: Screenshot 2023-01-14 at 6.12.53 AM.png] >> >> >> >> >> On Fri, Jan 13, 2023 at 3:44 PM Dave Taht wrote: >> >>> I am forced to conclude that waveform's upload test is broken in some >>> cases and has been for some time. All the joy many have been feeling about >>> their uplink speeds has to be cast away. Felt good, though, didn't it? >>> >>> There is a *slight* possibility that there is some fq in the starlink >>> network on tcp port 433. The first showing normal rtt growth and loss for >>> cubic, the second, a low rate flow that is holding the line... but I didn't >>> check to see if these were sequential or parallel. >>> >>> The last is a cwnd plot, clearly showing the cubic sawtooth on the >>> upload. >>> >>> It's weirdly nice to be able to follow a port 433 stream, see the tls >>> handshake put the website en'clair, and the rest go dark, and still trace >>> the behaviors. >>> >>> we're so going to lose this analytical ability with quic. I'm enjoying >>> it while we still can. >>> >>> >>> On Fri, Jan 13, 2023 at 2:10 PM Jonathan Bennett via Starlink < >>> starlink@lists.bufferbloat.net> wrote: >>> >>>> The irtt run finished a few seconds before the flent run, but here are >>>> the results: >>>> >>>> >>>> https://drive.google.com/file/d/1FKve13ssUMW1LLWOXLM2931Yx6uMHw8K/view?usp=share_link >>>> >>>> https://drive.google.com/file/d/1ZXd64A0pfUedLr3FyhDNTHA7vxv8S2Gk/view?usp=share_link >>>> >>>> https://drive.google.com/file/d/1rx64UPQHHz3IMNiJtb1oFqtqw2DvhvEE/view?usp=share_link >>>> >>>> >>>> [image: image.png] >>>> [image: image.png] >>>> >>>> >>>> Jonathan Bennett >>>> Hackaday.com >>>> >>>> >>>> On Fri, Jan 13, 2023 at 3:30 PM Nathan Owens via Starlink < >>>> starlink@lists.bufferbloat.net> wrote: >>>> >>>>> Here's Luis's run -- the top line below the edge of the graph is 200ms >>>>> [image: Screenshot 2023-01-13 at 1.30.03 PM.png] >>>>> >>>>> >>>>> On Fri, Jan 13, 2023 at 1:25 PM Luis A. Cornejo < >>>>> luis.a.cornejo@gmail.com> wrote: >>>>> >>>>>> Dave, >>>>>> >>>>>> Here is a run the way I think you wanted it. >>>>>> >>>>>> irtt running for 5 min to your dallas server, followed by a waveform >>>>>> test, then a few seconds of inactivity, cloudflare test, a few more secs of >>>>>> nothing, flent test to dallas. Packet capture is currently uploading (will >>>>>> be done in 20 min or so), irtt JSON also in there (.zip file): >>>>>> >>>>>> >>>>>> https://drive.google.com/drive/folders/1FLWqrzNcM8aK-ZXQywNkZGFR81Fnzn-F?usp=share_link >>>>>> >>>>>> -Luis >>>>>> >>>>>> On Fri, Jan 13, 2023 at 2:50 PM Dave Taht via Starlink < >>>>>> starlink@lists.bufferbloat.net> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Jan 13, 2023 at 12:30 PM Nathan Owens >>>>>>> wrote: >>>>>>> >>>>>>>> Here's the data visualization for Johnathan's Data >>>>>>>> >>>>>>>> [image: Screenshot 2023-01-13 at 12.29.15 PM.png] >>>>>>>> >>>>>>>> You can see the path change at :12, :27, :42, :57 after the minute. >>>>>>>> Some paths are clearly busier than others with increased loss, latency, and >>>>>>>> jitter. >>>>>>>> >>>>>>> >>>>>>> I am so glad to see loss and bounded delay here. Also a bit of rigor >>>>>>> regarding what traffic was active locally vs on the path would be nice, >>>>>>> although it seems to line up with the known 15s starlink switchover thing >>>>>>> (need a name for this), in this case, doing a few speedtests >>>>>>> while that irtt is running would show the impact(s) of whatever else >>>>>>> they are up to. >>>>>>> >>>>>>> It would also be my hope that the loss distribution in the middle >>>>>>> portion of this data is good, not bursty, but we don't have a tool to take >>>>>>> apart that. (I am so hopeless at json) >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Jan 13, 2023 at 10:09 AM Nathan Owens >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I’ll run my visualization code on this result this afternoon and >>>>>>>>> report back! >>>>>>>>> >>>>>>>>> On Fri, Jan 13, 2023 at 9:41 AM Jonathan Bennett via Starlink < >>>>>>>>> starlink@lists.bufferbloat.net> wrote: >>>>>>>>> >>>>>>>>>> The irtt command, run with normal, light usage: >>>>>>>>>> https://drive.google.com/file/d/1SiVCiUYnx7nDTxIVOY5w-z20S2O059rA/view?usp=share_link >>>>>>>>>> >>>>>>>>>> Jonathan Bennett >>>>>>>>>> Hackaday.com >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Jan 13, 2023 at 11:26 AM Dave Taht >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> packet caps would be nice... all this is very exciting news. >>>>>>>>>>> >>>>>>>>>>> I'd so love for one or more of y'all reporting such great uplink >>>>>>>>>>> results nowadays to duplicate and re-plot the original irtt >>>>>>>>>>> tests we >>>>>>>>>>> did: >>>>>>>>>>> >>>>>>>>>>> irtt client -i3ms -d300s myclosestservertoyou.starlink.taht.net >>>>>>>>>>> -o whatever.json >>>>>>>>>>> >>>>>>>>>>> They MUST have changed their scheduling to get such amazing >>>>>>>>>>> uplink >>>>>>>>>>> results, in addition to better queue management. >>>>>>>>>>> >>>>>>>>>>> (for the record, my servers are de, london, fremont, sydney, >>>>>>>>>>> dallas, >>>>>>>>>>> newark, atlanta, singapore, mumbai) >>>>>>>>>>> >>>>>>>>>>> There's an R and gnuplot script for plotting that output around >>>>>>>>>>> here >>>>>>>>>>> somewhere (I have largely personally put down the starlink >>>>>>>>>>> project, >>>>>>>>>>> loaning out mine) - that went by on this list... I should have >>>>>>>>>>> written >>>>>>>>>>> a blog entry so I can find that stuff again. >>>>>>>>>>> >>>>>>>>>>> On Fri, Jan 13, 2023 at 9:02 AM Jonathan Bennett via Starlink >>>>>>>>>>> wrote: >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > On Fri, Jan 13, 2023 at 6:28 AM Ulrich Speidel via Starlink < >>>>>>>>>>> starlink@lists.bufferbloat.net> wrote: >>>>>>>>>>> >> >>>>>>>>>>> >> On 13/01/2023 6:13 pm, Ulrich Speidel wrote: >>>>>>>>>>> >> > >>>>>>>>>>> >> > From Auckland, New Zealand, using a roaming subscription, >>>>>>>>>>> it puts me >>>>>>>>>>> >> > in touch with a server 2000 km away. OK then: >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > IP address: nix six. >>>>>>>>>>> >> > >>>>>>>>>>> >> > My thoughts shall follow later. >>>>>>>>>>> >> >>>>>>>>>>> >> OK, so here we go. >>>>>>>>>>> >> >>>>>>>>>>> >> I'm always a bit skeptical when it comes to speed tests - >>>>>>>>>>> they're really >>>>>>>>>>> >> laden with so many caveats that it's not funny. I took our >>>>>>>>>>> new work >>>>>>>>>>> >> Starlink kit home in December to give it a try and the other >>>>>>>>>>> day finally >>>>>>>>>>> >> got around to set it up. It's on a roaming subscription >>>>>>>>>>> because our >>>>>>>>>>> >> badly built-up campus really isn't ideal in terms of a clear >>>>>>>>>>> view of the >>>>>>>>>>> >> sky. Oh - and did I mention that I used the Starlink Ethernet >>>>>>>>>>> adapter, >>>>>>>>>>> >> not the WiFi? >>>>>>>>>>> >> >>>>>>>>>>> >> Caveat 1: Location, location. I live in a place where the >>>>>>>>>>> best Starlink >>>>>>>>>>> >> promises is about 1/3 in terms of data rate you can actually >>>>>>>>>>> get from >>>>>>>>>>> >> fibre to the home at under half of Starlink's price. Read: >>>>>>>>>>> There are few >>>>>>>>>>> >> Starlink users around. I might be the only one in my suburb. >>>>>>>>>>> >> >>>>>>>>>>> >> Caveat 2: Auckland has three Starlink gateways close by: >>>>>>>>>>> Clevedon (which >>>>>>>>>>> >> is at a stretch daytrip cycling distance from here), Te Hana >>>>>>>>>>> and Puwera, >>>>>>>>>>> >> the most distant of the three and about 130 km away from me >>>>>>>>>>> as the crow >>>>>>>>>>> >> flies. Read: My dishy can use any satellite that any of these >>>>>>>>>>> three can >>>>>>>>>>> >> see, and then depending on where I put it and how much of the >>>>>>>>>>> southern >>>>>>>>>>> >> sky it can see, maybe also the one in Hinds, 840 km away, >>>>>>>>>>> although that >>>>>>>>>>> >> is obviously stretching it a bit. Either way, that's plenty >>>>>>>>>>> of options >>>>>>>>>>> >> for my bits to travel without needing a lot of handovers. >>>>>>>>>>> Why? Easy: If >>>>>>>>>>> >> your nearest teleport is close by, then the set of satellites >>>>>>>>>>> that the >>>>>>>>>>> >> teleport can see and the set that you can see is almost the >>>>>>>>>>> same, so you >>>>>>>>>>> >> can essentially stick with the same satellite while it's in >>>>>>>>>>> view for you >>>>>>>>>>> >> because it'll also be in view for the teleport. Pretty much >>>>>>>>>>> any bird >>>>>>>>>>> >> above you will do. >>>>>>>>>>> >> >>>>>>>>>>> >> And because I don't get a lot of competition from other users >>>>>>>>>>> in my area >>>>>>>>>>> >> vying for one of the few available satellites that can see >>>>>>>>>>> both us and >>>>>>>>>>> >> the teleport, this is about as good as it gets at 37S >>>>>>>>>>> latitude. If I'd >>>>>>>>>>> >> want it any better, I'd have to move a lot further south. >>>>>>>>>>> >> >>>>>>>>>>> >> It'd be interesting to hear from Jonathan what the >>>>>>>>>>> availability of home >>>>>>>>>>> >> broadband is like in the Dallas area. I note that it's at a >>>>>>>>>>> lower >>>>>>>>>>> >> latitude (33N) than Auckland, but the difference isn't huge. >>>>>>>>>>> I notice >>>>>>>>>>> >> two teleports each about 160 km away, which is also not too >>>>>>>>>>> bad. I also >>>>>>>>>>> >> note Starlink availability in the area is restricted at the >>>>>>>>>>> moment - >>>>>>>>>>> >> oversubscribed? But if Jonathan gets good data rates, then >>>>>>>>>>> that means >>>>>>>>>>> >> that competition for bird capacity can't be too bad - for >>>>>>>>>>> whatever reason. >>>>>>>>>>> > >>>>>>>>>>> > I'm in Southwest Oklahoma, but Dallas is the nearby Starlink >>>>>>>>>>> gateway. In cities, like Dallas, and Lawton where I live, there are good >>>>>>>>>>> broadband options. But there are also many people that live outside cities, >>>>>>>>>>> and the options are much worse. The low density userbase in rural Oklahoma >>>>>>>>>>> and Texas is probably ideal conditions for Starlink. >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> Caveat 3: Backhaul. There isn't just one queue between me and >>>>>>>>>>> whatever I >>>>>>>>>>> >> talk to in terms of my communications. Traceroute shows about >>>>>>>>>>> 10 hops >>>>>>>>>>> >> between me and the University of Auckland via Starlink. >>>>>>>>>>> That's 10 >>>>>>>>>>> >> queues, not one. Many of them will have cross traffic. So >>>>>>>>>>> it's a bit >>>>>>>>>>> >> hard to tell where our packets really get to wait or where >>>>>>>>>>> they get >>>>>>>>>>> >> dropped. The insidious bit here is that a lot of them will be >>>>>>>>>>> between 1 >>>>>>>>>>> >> Gb/s and 10 Gb/s links, and with a bit of cross traffic, they >>>>>>>>>>> can all >>>>>>>>>>> >> turn into bottlenecks. This isn't like a narrowband GEO link >>>>>>>>>>> of a few >>>>>>>>>>> >> Mb/s where it's obvious where the dominant long latency >>>>>>>>>>> bottleneck in >>>>>>>>>>> >> your TCP connection's path is. Read: It's pretty hard to tell >>>>>>>>>>> whether a >>>>>>>>>>> >> drop in "speed" is due to a performance issue in the Starlink >>>>>>>>>>> system or >>>>>>>>>>> >> somewhere between Starlink's systems and the target system. >>>>>>>>>>> >> >>>>>>>>>>> >> I see RTTs here between 20 ms and 250 ms, where the physical >>>>>>>>>>> latency >>>>>>>>>>> >> should be under 15 ms. So there's clearly a bit of buffer >>>>>>>>>>> here along the >>>>>>>>>>> >> chain that occasionally fills up. >>>>>>>>>>> >> >>>>>>>>>>> >> Caveat 4: Handovers. Handover between birds and teleports is >>>>>>>>>>> inevitably >>>>>>>>>>> >> associated with a change in RTT and in most cases also >>>>>>>>>>> available >>>>>>>>>>> >> bandwidth. Plus your packets now arrive at a new queue on a >>>>>>>>>>> new >>>>>>>>>>> >> satellite while your TCP is still trying to respond to >>>>>>>>>>> whatever it >>>>>>>>>>> >> thought the queue on the previous bird was doing. Read: >>>>>>>>>>> Whatever your >>>>>>>>>>> >> cwnd is immediately after a handover, it's probably not what >>>>>>>>>>> it should be. >>>>>>>>>>> >> >>>>>>>>>>> >> I ran a somewhat hamstrung (sky view restricted) set of four >>>>>>>>>>> Ookla >>>>>>>>>>> >> speedtest.net tests each to five local servers. Average >>>>>>>>>>> upload rate was >>>>>>>>>>> >> 13 Mb/s, average down 75.5 Mb/s. Upload to the server of the >>>>>>>>>>> ISP that >>>>>>>>>>> >> Starlink seems to be buying its local connectivity from >>>>>>>>>>> (Vocus Group) >>>>>>>>>>> >> varied between 3.04 and 14.38 Mb/s, download between 23.33 >>>>>>>>>>> and 52.22 >>>>>>>>>>> >> Mb/s, with RTTs between 37 and 56 ms not correlating well to >>>>>>>>>>> rates >>>>>>>>>>> >> observed. In fact, they were the ISP with consistently the >>>>>>>>>>> worst rates. >>>>>>>>>>> >> >>>>>>>>>>> >> Another ISP (MyRepublic) scored between 11.81 and 21.81 Mb/s >>>>>>>>>>> up and >>>>>>>>>>> >> between 106.5 and 183.8 Mb/s down, again with RTTs badly >>>>>>>>>>> correlating >>>>>>>>>>> >> with rates. Average RTT was the same as for Vocus. >>>>>>>>>>> >> >>>>>>>>>>> >> Note the variation though: More or less a factor of two >>>>>>>>>>> between highest >>>>>>>>>>> >> and lowest rates for each ISP. Did MyRepublic just get lucky >>>>>>>>>>> in my >>>>>>>>>>> >> tests? Or is there something systematic behind this? Way too >>>>>>>>>>> few tests >>>>>>>>>>> >> to tell. >>>>>>>>>>> >> >>>>>>>>>>> >> What these tests do is establish a ballpark. >>>>>>>>>>> >> >>>>>>>>>>> >> I'm currently repeating tests with dish placed on a trestle >>>>>>>>>>> closer to >>>>>>>>>>> >> the heavens. This seems to have translated into fewer outages >>>>>>>>>>> / ping >>>>>>>>>>> >> losses (around 1/4 of what I had yesterday with dishy on the >>>>>>>>>>> ground on >>>>>>>>>>> >> my deck). Still good enough for a lengthy video Skype call >>>>>>>>>>> with my folks >>>>>>>>>>> >> in Germany, although they did comment about reduced video >>>>>>>>>>> quality. But >>>>>>>>>>> >> maybe that was the lighting or the different background as I >>>>>>>>>>> wasn't in >>>>>>>>>>> >> my usual spot with my laptop when I called them. >>>>>>>>>>> > >>>>>>>>>>> > Clear view of the sky is king for Starlink reliability. I've >>>>>>>>>>> got my dishy mounted on the back fence, looking up over an empty field, so >>>>>>>>>>> it's pretty much best-case scenario here. >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> -- >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> **************************************************************** >>>>>>>>>>> >> Dr. Ulrich Speidel >>>>>>>>>>> >> >>>>>>>>>>> >> School of Computer Science >>>>>>>>>>> >> >>>>>>>>>>> >> Room 303S.594 (City Campus) >>>>>>>>>>> >> >>>>>>>>>>> >> The University of Auckland >>>>>>>>>>> >> u.speidel@auckland.ac.nz >>>>>>>>>>> >> http://www.cs.auckland.ac.nz/~ulrich/ >>>>>>>>>>> >> >>>>>>>>>>> **************************************************************** >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> _______________________________________________ >>>>>>>>>>> >> Starlink mailing list >>>>>>>>>>> >> Starlink@lists.bufferbloat.net >>>>>>>>>>> >> https://lists.bufferbloat.net/listinfo/starlink >>>>>>>>>>> > >>>>>>>>>>> > _______________________________________________ >>>>>>>>>>> > Starlink mailing list >>>>>>>>>>> > Starlink@lists.bufferbloat.net >>>>>>>>>>> > https://lists.bufferbloat.net/listinfo/starlink >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> This song goes out to all the folk that thought Stadia would >>>>>>>>>>> work: >>>>>>>>>>> >>>>>>>>>>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz >>>>>>>>>>> Dave Täht CEO, TekLibre, LLC >>>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Starlink mailing list >>>>>>>>>> Starlink@lists.bufferbloat.net >>>>>>>>>> https://lists.bufferbloat.net/listinfo/starlink >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> This song goes out to all the folk that thought Stadia would work: >>>>>>> >>>>>>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz >>>>>>> Dave Täht CEO, TekLibre, LLC >>>>>>> _______________________________________________ >>>>>>> Starlink mailing list >>>>>>> Starlink@lists.bufferbloat.net >>>>>>> https://lists.bufferbloat.net/listinfo/starlink >>>>>>> >>>>>> _______________________________________________ >>>>> Starlink mailing list >>>>> Starlink@lists.bufferbloat.net >>>>> https://lists.bufferbloat.net/listinfo/starlink >>>>> >>>> _______________________________________________ >>>> Starlink mailing list >>>> Starlink@lists.bufferbloat.net >>>> https://lists.bufferbloat.net/listinfo/starlink >>>> >>> >>> >>> -- >>> This song goes out to all the folk that thought Stadia would work: >>> >>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz >>> Dave Täht CEO, TekLibre, LLC >>> >> -- This song goes out to all the folk that thought Stadia would work: https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz Dave Täht CEO, TekLibre, LLC