From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 8F6203CB45 for ; Mon, 15 May 2023 02:36:20 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de; s=s31663417; t=1684132570; i=moeller0@gmx.de; bh=hHXTtN93ykEreFQ8y2VIDNjtvfYb4CuifrCKifpzzVA=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=bAAbyclv6ZdCD3Er2mALObohIp1rSMxKwaOxKfZxGsPOaJf5WNzOfrfpczU5UInOJ UBSHkDeSwIgtecNwHwOiHyOwt/oPJRtkI2wqTNQZOuxXgsEyNF2+IWmRVpG7WyQFB5 ZaqeOQ0XfPSEt+NvAJVLuxxrbSgkEYVW9UOEX8y2V5Cmc3Q/AWJjIcwaaKoLfshZST 3wGwZ0y8PCw13N0NzYXBO6Mn+YckUde8tuXVbiVYUCvIwQha3uqHZFHvZdt9p8PjHI 4tCuMoXXE9oT4u9muv/JmjN0+JSMo27igULYGapTmvVBefvuvG8nTO8p0RJpIdCPV2 CS6h+30P+pE1w== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Received: from smtpclient.apple ([134.76.241.253]) by mail.gmx.net (mrgmx104 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MXp5Q-1pfi4s0beu-00YEHb; Mon, 15 May 2023 08:36:10 +0200 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.3\)) From: Sebastian Moeller In-Reply-To: <9q29o7n2-69rs-3os1-s93q-0795262qs3o1@ynat.uz> Date: Mon, 15 May 2023 08:36:08 +0200 Cc: Ulrich Speidel , "starlink@lists.bufferbloat.net" Content-Transfer-Encoding: quoted-printable Message-Id: References: <0no84q43-s4n6-45n8-50or-12o3rq104n99@ynat.uz> <48b00469-0dbb-54c4-bedb-3aecbf714a1a@auckland.ac.nz> <728orr66-1432-751p-263q-sqopr12s20sq@ynat.uz> <077e6ad1-d7cc-2d57-39f8-e9646bea32a5@auckland.ac.nz> <09552rq0-0n24-0pqo-4085-n918r0n71138@ynat.uz> <9q29o7n2-69rs-3os1-s93q-0795262qs3o1@ynat.uz> To: David Lang X-Mailer: Apple Mail (2.3696.120.41.1.3) X-Provags-ID: V03:K1:IhuUigq9g9ZZxIU1CBOdmAi+GFQY/jmeXURgLv1wytWsi5+DDkH /pIuhEo7OfnDAcF7z+D5LZCU8rw0otSpi1hqG1ESABTIxHpaJ41q52cL1/d41RmK2lkaQ5j 70nIOfPSvUXi8v0dqel6+qHKCT7V2M4/r2gzdZb0tzIvt1eCPf/vToHaF5nW1YVsjwLlkF+ YQKymfLweAutiIOr6xE3A== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:rETA6WbboF4=;T1P+LyDOB9Q05S15lnsFneVILyB cLNXnZiXKFsn+Z2evNr3y/64WSlD7YjChvK4YnpQgP7zf45TUNVgs+hhXgq9Wd6fAY0+ycAQn QcchCn/ztoyza/nw3tHi3Z5OiP/UCj6dOCdPXD3mHWzinvQ7lwJKL9Ruf5gYKdSmc4X+vTTiE IWF/dxuCzZNA4VOWb4nVfqCuvBfqd/rNIb7dBi2/UVnYwZSp2RGNbJKDf/6LI+g3i3V4LOfFW uRP+/UiDgOAwNzzAa/IM+fvCoeDAi5uUVSlZtRDkNUWF2iBTsgkRO2A1YUJ7av69ILTA7ramc q7kTV2QhPN6UCB8rTH4PhWdo5IgqrCC7v1KoLhl5I71Wj0NCoL7gyJHUPicyIdQ4U21jId+34 6pGyYSmPHLrFQr14Wjcs4KI4ip27lhfalW5c+t+H0nRg+uWWqBwC83gneIhoLbtRVEzB8SA/A 5BiVS7jHQTV5dkFRnQYv3F44q29gG5VkpUimLdjgAjtRy2LiWas6yMAe11r0dW9mimd1/jz87 tZ9k2IWZ6V+VvqkZRKi82YXf0D7EVb1DnEmM9Yker22YHSH06yzum3IAHNpMrUu7AmiUzHMVK tumxhMxvUjEgrXRWR6/gp5/zpCHyrgMx3Gv4EwVVUfVdUi2JOG/cY8RPDEZ3kXR+yNGIirX0o Cc+kIQ3hVU60058SMg/3LXU9u9eMF1XBpysbFRNSSzWVyhDp3lff0/X0a6bY12+uElXDQYc5Z EeEuMy3xdRcRe52tB1Ve2Wp5jhAPsPRSS+8KpkJw8XXvZezRufVsrlmVQ5Myd5LbIdkyv3uD7 0KcPaJZfAw5591TbbSZal9HbHhHhKiWlBfIu5jbXFbkXdCc1o0O8Fu91jkjVzaKbf8G0gTEcz v7BaEwuzYh0qIyGJLzURDVmRe/biPVR/WusQSAKLyJz5oGfzK9i5/bLoNNed7AvESPiJdfvhA pn6rmMesQSmABs8GJ78qXnrcuJo= Subject: Re: [Starlink] Starlink hidden buffers X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 May 2023 06:36:20 -0000 Hi David, please see [SM] below. > On May 15, 2023, at 05:33, David Lang via Starlink = wrote: >=20 > On Mon, 15 May 2023, Ulrich Speidel wrote: >=20 >> On 14/05/2023 9:00 pm, David Lang wrote: >>> On Sun, 14 May 2023, Ulrich Speidel wrote: >>> >> I just discovered that someone is manufacturing an adapter so you = no >> longer have to cut the cable >>> >> >>> >> = https://www.amazon.com/YAOSHENG-Rectangular-Adapter-Connect-Injector/dp/B0= BYJTHX4P >=20 >>> >> >>> > I'll see whether I can get hold of one of these. Cutting a cable = on a > university IT asset as an academic is not allowed here, except if = it > doesn't meet electrical safety standards. >>> > >>> > Alternatively, has anyone tried the standard Starlink Ethernet = adapter with > a PoE injector instead of the WiFi box? The adapter above = seems to be like > the Starlink one (which also inserts into the cable = between Dishy and > router). >>> that connects you a 2nd ethernet port on the router, not on the = dishy >>> I just ordered one of those adapters, it will take a few weeks to = arrive. >> How do we know that the Amazon version doesn't do the same? >=20 > because it doesn't involve the router at all. It allows you to replace = the router with anything you want. >=20 > People have documented how to cut the cable and crimp on a RJ45 = connector, use a standard PoE injector, and connect to any router you = want. I was preparing to do that (and probably still will for one cable = to use a different locations to avoid having a 75 ft cable from the dish = mounted on the roof of my van to the router a couple feet away), This = appears to allow me to do the same functional thing, but without cutting = the cable. >=20 >>> >> > I suspect they're handing over whole cells, not individual = users, at a >> > time. >>> >> >>> >> I would guess the same (remember, in spite of them having = launched >4000 >> satellites, this is still the early days, with the = network changing as >> more launching) >>> >> >>> >> We've seen that it seems that there is only one satellite serving = any cell >> one time. >>> > But the reverse is almost certainly not true: Each satellite must = serve > multiple cells. >>> true, but while the satellite over a given area will change, the = usage in that area isn't changing that much >=20 >> Exactly. But your underlying queue sits on the satellite, not in the = area. >=20 > only if the satellite is where you have more input than output. That = may be the case for users uploading, but for users downloading, I would = expect that the bandwidth bottleneck would be from the Internet = connected ground station to the satellite, with the satellite serving = many cells but only having one uplink. >=20 >>> >> But remember that the system does know how much usage there is in = the cell >> before they do the handoff. It's unknown if they do anything = with that, or >> if they are just relaying based on geography. We also = don't know what the >> bandwidth to the ground stations is compared to = the dishy. >>> > Well, we do know for NZ, sort of, based on the licences Starlink = has here. >>> what is the ground station bandwith? >>=20 >> https://rrf.rsm.govt.nz/ui/search/licence - seach for "Starlink" >>=20 >> ...all NZ licences in all their glory. Looking at Starlink SES = (satellite earth station) TX (which is the interesting direction I = guess): >>=20 >> - Awarua, Puwera, Hinds, Clevedon, Cromwell, Te Hana: 29750.000000 TX = (BW =3D 500 MHz) >> - Awarua, Puwera, Hinds, Clevedon, Cromwell, Te Hana: 28850.000000 TX = (BW =3D 500 MHz) >> - Awarua, Puwera, Hinds, Clevedon, Cromwell, Te Hana: 28350.000000 TX = (BW =3D 500 MHz) >> - Awarua, Puwera, Hinds, Clevedon, Cromwell, Te Hana: 28250.000000 TX = (BW =3D 500 MHz) >> - Awarua, Puwera, Hinds, Clevedon, Cromwell, Te Hana: 27750.000000 TX = (BW =3D 500 MHz) >>=20 >> So 2.5 GHz up, licensed from 6 ground stations. Now I'm not convinced = that they would use all of those from all locations simultaneously = because of the risk of off-beam interference. They'll all be = transmitting south, ballpark. If there was full re-use at all ground = stations, we'd be looking at 15 GHz. If they are able to re-use on all = antennas at each ground station, then we're looking at 9 golf balls each = in Puwera, Te Hana, Clevedon, Hinds and Cromwell, and an unknown number = at Awarua. Assuming 9 there, we'd be looking at 135 GHz all up max. >>=20 >> Awarua and Cromwell are 175 km apart, Hinds another 220 km from = Cromwell, then it's a hop of about 830 km to Clevedon, and from there = another 100 km to Te Hana, which is another 53 km from Puwera, so = keeping them all out of each other's hair all the time might be a bit = difficult. >>=20 >> Lots of other interesting info in the licenses, such as EIRP, in case = you're wanting to do link budgets. >=20 > I was asking more in terms of Gb/s rather than MHz of bandwidth. = Dedicated ground stations with bigger antennas, better filters, more = processing and overall a much higher budget can get much better data = rates out of a given amount of bandwidth than the user end stations = will. >=20 > it's also possible (especially with bigger antennas) for one ground = station location to talk to multiple different satellites at once (the = aiming of the antennas can isolate the signals from each other) >=20 >>> As latency changes, figuring out if it's extra distance that must be = traveled, or buffering is hard. does the latency stay roughly the same = until the next satellite change? or does it taper off? >=20 >> Good question. You would expect step changes in physical latency = between satellites, but also gradual change related to satellite = movement. Plus of course any rubble thrown into any queue by something = suddenly turning up on that path. Don't forget that it's not just cells = now, we're also talking up- and downlink for the laser ISLs, at least in = some places. >=20 > how far do the satellites move in 15 min and what effect would that = have on latency (I would assume that most of the time, the satellites = are switched to as they are getting nearer the two stations, so most of = the time, I would expect a slight reduction in latency for ~7 min and = then a slight increase for ~7 min, but I would not expect that this = would be a large variation >=20 >>> If it stays the same, I would suspect that you are actually hitting = a different ground station and there is a VPN backhaul to your egress = point to the regular Internet (which doesn't support mobile IP = addresses) for that cycle. If it tapers off, then I could buy = bufferbloat that gets resolved as TCP backs off. >>=20 >> Yes, quite sorting out which part of your latency is what is the = million dollar question here... >>=20 >> We saw significant RTT changes here during the recent cyclone over = periods of several hours, and these came in steps (see below), with the = initial change being a downward one. Averages are over 60 pings (the = time scale isn't 100% true as we used "one ping, one second" timing) = here. >>=20 >>=20 >> We're still not sure whether to attribute this to load change or = ground station changes. There were a lot of power outages, especially in = Auckland's lifestyle block belt, which teems with Starlink users, but = all three North Island ground stations were also in areas affected by = power outages (although the power companies concerned don't provide the = level of detail to establish whether they were affected). It's also not = clear what, if any, backup power arrangements they have). At ~25 ms, the = step changes in RTT are too large be the result of a switch in ground = stations, though, the path differences just aren't that large. You'd = also expect a ground station outage to result in longer RTTs, not = shorter ones, if you need to re-route via another ground station. One = explanation might be users getting cut off if they relied on one = particular ground station for bent pipe ops - but that would not explain = this order of magnitude effect as I'd expect that number to be small. So = maybe power outages at the user end after all. But that would then tell = us that these are load-dependent queuing delays. Moreover, since those = load changes wouldn't have involved the router at our site, we can = conclude that these are queue sojourn times in the Starlink network. >=20 > I have two starlink dishes in the southern california area, I'm going = to put one on the low-priority mobile plan shortly. These are primarily = used for backup communication, so I would be happy to add something to = them to do latency monitoring. [SM] I would consider using irtt for that (like running in for = 15 minutes with say 5ms spacing a few times a day, sometimes together = with a saturating load sometimes without), this is a case where OWDs are = especially interesting and irtt will also report the direction in which = packets were lost. Maybe Dave (once back from his time-off) has an idea = about which irtt reflector to use? > In looking at what geo-location reports my location as, I see it = wander up and down the west coast, from the Los Angeles area all the way = up to Canada. [SM] Demonstrating once more that geoIP is just a heuristic ;) >=20 >>> I think that active queue management on the sending side of the = bottleneck will handle it fairly well. It doesn't have to do = calculations based on what the bandwidth is, it just needs to know what = it has pending to go out. >=20 >> Understood - but your customer for AQM is the sending TCP client, and = there are two questions here: (a) Does your AQM handle rapid load = changes and (b) how do your TCP clients actually respond to your AQM's = handling? >=20 > AQM allocates the available bandwidth between different connections = (usually different users) [SM] Not sure AQM is actually defined that stringently, I was = under the impression anything other that FIFO with tail drop would = already count as AQM, no? > When it does this indirectly for inbound traffic by delaying acks, the = results depend on the senders handling of these indirect signals that = were never intended for this purpose. [SM] Hmm, ACKs where intended to be a feed-back mechanism for = the sender to use to asses the in-flight data, so I am not sure whether = delaying ACKs is something that was never envisaged by TCP's creators? >=20 > But when it does this directly on the sending side, it doesn't matter = what the senders want, their data WILL be managed to the = priority/bandwidth that the AQM sets, and eventually their feedback is = dropped packets, which everyone who is legitimate responds to. [SM] Some more quickly than others though, looking at you BBR ;) > But even if they don't respond (say a ping flood or DoS attack), the = AQM will limit the damage to that connection, allowing the other = connections trying to use that link to continue to function. [SM] Would that not require an AQM with effectively a = multi-queue scheduler? I think it seems clear that starlink uses some = form of AQM (potentially ARED), but on the scheduler/queue side there = see to be competing claims ranging from single-queue FIFO (with ARED) to = FQ-scheduler. Not having a starlink-link I can not test any of this so = all I can report is competing reports from starlink users... Regards Sebastian >=20 > David Lang > _______________________________________________ > Starlink mailing list > Starlink@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/starlink