From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.15.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id CB83C3B29E; Wed, 11 Jan 2023 15:01:33 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de; s=s31663417; t=1673467284; bh=3hrW0O1CS93jBmQbd2vGJI2/mdUW868hWSlEj9N5NqU=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=pVSzNOJDSMCJnNxWoaRCN1BZmR+GsnVgGykUaQvaB6Om0sBQ6d9rhzdkkoy8nqDQM C43oj1cI+AaxZf1DVlkzgcem+zRIG1S86LIYQMHB5wf1OnKfCTErLcdaQrqklCAHdu cbJ3Gg60DO3SEtqwkkuT+WJnf2Yp+HVt5/LwVrkqtuYjOO52sMCiAaeYcCIEGC6f29 MvPfjSs5jRBw6Z3k0wxPLdEz4DPoENZU6FH65c0AwXR67I2BlCIJhaUjAH99pO6B30 6ixvh30lfNpM6aT6WBKrhOZCQ3MiTlaPV/HJ0ZSiEHTdfbzMHONs/qChrM+HjYjJyP Ch8OjmijZ1DkA== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Received: from smtpclient.apple ([95.116.136.244]) by mail.gmx.net (mrgmx004 [212.227.17.190]) with ESMTPSA (Nemesis) id 1MtOGa-1ouSdu1kTO-00utuC; Wed, 11 Jan 2023 21:01:24 +0100 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) From: Sebastian Moeller In-Reply-To: <202301111832.30BIWevV030127@gndrsh.dnsmgr.net> Date: Wed, 11 Jan 2023 21:01:22 +0100 Cc: rjmcmahon , Rpm , mike.reynolds@netforecast.com, "David P. Reed" , libreqos , Dave Taht via Starlink , bloat Content-Transfer-Encoding: quoted-printable Message-Id: References: <202301111832.30BIWevV030127@gndrsh.dnsmgr.net> To: "Rodney W. Grimes" X-Mailer: Apple Mail (2.3696.120.41.1.1) X-Provags-ID: V03:K1:kpRpoS3uubhmfNIprKgVDUu1vT+e2vGLtKcKwdasiYF3QUC89Md 0MFLYQmf9UH5CnL+wut+Uf3zQPZd/Ocm2JG3EGVxodxogeaq1hsFLVH24gRu440Xz7Bxb2X CMuXy1oiUwha4NMQL+JVWUQOnZwYsCHBCHvpmNyhYAySDoqb47K/EsDjhVfnhL4fbnEpEk+ VljBuiA7y8cSkAIKe0r/g== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:gSoIvJOjfO0=;tTw9PflbLki9n26HqAUWvlLcTGL 3vOlq++Azrf05Bj48KuWoLztCkCZADgOng8eS4/h8vdkWUC5B26BE6A17OpEpHnCJC5G0fJEi kHVckMmJAvwiMpLFJ03qa1PWI05CUgg+xVp+GGkrM0KEeg/zqHEZOOULDg7b+mw2bbVkqRATI wGY/Nlooy5LJZ7lXWdZOvjMai35i9FWL9i5OIQUubHDUJ58Dw3rW1E3h+t7D6qUCrJTQcRWZe 8o1vB0OTa+aK1oHFUQuccCd0va1jwNABwA64l8stX6BgwrDRmt5MepwlmuRZNla5XV9yLP9Qu ZyrXI2Z9jTbapeBq8U6e997sY8UFivB0bjb2H7QRcEYzl0VPQIEeAeA4wz5rUx7TTokzYznMs 4s4APcRFnIGe1me+DgPmCmdX9JZ7wrwzgQ6HlR1g76ToJrOFyu4BOUFo+9k7+unFIYPAUqgfC nnGUsgjXeRqVxIWSeNioaP0IOtVLAhsZs5FUZOFM/HxbMSMSIpS5a0Kl3vCDHmlXimH1tSfDp OToOyWj4BZjmTOKwwORw2DR/S1lO6sOy3/vh9YKuNW4FTtWuyJZu9uVZTX21teArarxcthrEk eyzDqVb1lEJHezQTvooSp68LEa9ruwjPRcsteVQa/khc/WGOeS84+0yiVnaH5bdTBg0cYHcAB jKTBoAYrwbW8mXs1KiLGbdySohlolNVGUL7pDiqeDl1WtmbTlMwejr7Sqld+yRB8ChR/Q3GNX koxhhkzdLlJjSFkL+2gScTclHkpK7OE+NsaAHPouiWU2/Xl8KMPBSOgKH7BCdyAgRAGhNYFQ2 Z6RzyVi+7M6SefzB8Zn1ijnviRWtauKoK2o1AZ9042SWgnNgYZQcmUi0GE0X/3ndQWkLiolX3 fPe/RPCwGwRMKgyz5TgKnDbsg54LjNd1pYXrx+K9rqua1S5370oyA70SX87Hg4jBKe2u4B7GK 9virZfIEHhA/wpGwLN7FqFAgyxw= Subject: Re: [Starlink] [Rpm] Researchers Seeking Probe Volunteers in USA X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jan 2023 20:01:34 -0000 Hi Rodney, > On Jan 11, 2023, at 19:32, Rodney W. Grimes = wrote: >=20 > Hello, >=20 > Yall can call me crazy if you want.. but... see below [RWG] >> Hi Bib, >>=20 >>=20 >>> On Jan 9, 2023, at 20:13, rjmcmahon via Starlink = wrote: >>>=20 >>> My biggest barrier is the lack of clock sync by the devices, i.e. = very limited support for PTP in data centers and in end devices. This = limits the ability to measure one way delays (OWD) and most assume that = OWD is 1/2 and RTT which typically is a mistake. We know this = intuitively with airplane flight times or even car commute times where = the one way time is not 1/2 a round trip time. Google maps & directions = provide a time estimate for the one way link. It doesn't compute a round = trip and divide by two. >>>=20 >>> For those that can get clock sync working, the iperf 2 --trip-times = options is useful. >>=20 >> [SM] +1; and yet even with unsynchronized clocks one can try to = measure how latency changes under load and that can be done per = direction. Sure this is far inferior to real reliably measured OWDs, but = if life/the internet deals you lemons.... >=20 > [RWG] iperf2/iperf3, etc are already moving large amounts of data back = and forth, for that matter any rate test, why not abuse some of that = data and add the fundemental NTP clock sync data and bidirectionally = pass each others concept of "current time". IIRC (its been 25 years = since I worked on NTP at this level) you *should* be able to get a = fairly accurate clock delta between each end, and then use that info and = time stamps in the data stream to compute OWD's. You need to put 4 time = stamps in the packet, and with that you can compute "offset". [SM] Nice idea. I would guess that all timeslot based access = technologies (so starlink, docsis, GPON, LTE?) all distribute "high = quality time" carefully to the "modems", so maybe all that would be = needed is to expose that high quality time to the LAN side of those = modems, dressed up as NTP server? >=20 >>=20 >>=20 >>>=20 >>> --trip-times >>> enable the measurement of end to end write to read latencies (client = and server clocks must be synchronized) > [RWG] --clock-skew > enable the measurement of the wall clock difference between = sender and receiver >=20 >>=20 >> [SM] Sweet! >>=20 >> Regards >> Sebastian >>=20 >>>=20 >>> Bob >>>> I have many kvetches about the new latency under load tests being >>>> designed and distributed over the past year. I am delighted! that = they >>>> are happening, but most really need third party evaluation, and >>>> calibration, and a solid explanation of what network pathologies = they >>>> do and don't cover. Also a RED team attitude towards them, as well = as >>>> thinking hard about what you are not measuring (operations = research). >>>> I actually rather love the new cloudflare speedtest, because it = tests >>>> a single TCP connection, rather than dozens, and at the same time = folk >>>> are complaining that it doesn't find the actual "speed!". yet... = the >>>> test itself more closely emulates a user experience than = speedtest.net >>>> does. I am personally pretty convinced that the fewer numbers of = flows >>>> that a web page opens improves the likelihood of a good user >>>> experience, but lack data on it. >>>> To try to tackle the evaluation and calibration part, I've reached = out >>>> to all the new test designers in the hope that we could get = together >>>> and produce a report of what each new test is actually doing. I've >>>> tweeted, linked in, emailed, and spammed every measurement list I = know >>>> of, and only to some response, please reach out to other test = designer >>>> folks and have them join the rpm email list? >>>> My principal kvetches in the new tests so far are: >>>> 0) None of the tests last long enough. >>>> Ideally there should be a mode where they at least run to "time of >>>> first loss", or periodically, just run longer than the >>>> industry-stupid^H^H^H^H^H^Hstandard 20 seconds. There be dragons >>>> there! It's really bad science to optimize the internet for 20 >>>> seconds. It's like optimizing a car, to handle well, for just 20 >>>> seconds. >>>> 1) Not testing up + down + ping at the same time >>>> None of the new tests actually test the same thing that the = infamous >>>> rrul test does - all the others still test up, then down, and ping. = It >>>> was/remains my hope that the simpler parts of the flent test suite = - >>>> such as the tcp_up_squarewave tests, the rrul test, and the = rtt_fair >>>> tests would provide calibration to the test designers. >>>> we've got zillions of flent results in the archive published here: >>>> https://blog.cerowrt.org/post/found_in_flent/ >>>> ps. Misinformation about iperf 2 impacts my ability to do this. >>>=20 >>>> The new tests have all added up + ping and down + ping, but not up = + >>>> down + ping. Why?? >>>> The behaviors of what happens in that case are really = non-intuitive, I >>>> know, but... it's just one more phase to add to any one of those = new >>>> tests. I'd be deliriously happy if someone(s) new to the field >>>> started doing that, even optionally, and boggled at how it defeated >>>> their assumptions. >>>> Among other things that would show... >>>> It's the home router industry's dirty secret than darn few = "gigabit" >>>> home routers can actually forward in both directions at a gigabit. = I'd >>>> like to smash that perception thoroughly, but given our starting = point >>>> is a gigabit router was a "gigabit switch" - and historically been >>>> something that couldn't even forward at 200Mbit - we have a long = way >>>> to go there. >>>> Only in the past year have non-x86 home routers appeared that could >>>> actually do a gbit in both directions. >>>> 2) Few are actually testing within-stream latency >>>> Apple's rpm project is making a stab in that direction. It looks >>>> highly likely, that with a little more work, crusader and >>>> go-responsiveness can finally start sampling the tcp RTT, loss and >>>> markings, more directly. As for the rest... sampling TCP_INFO on >>>> windows, and Linux, at least, always appeared simple to me, but I'm >>>> discovering how hard it is by delving deep into the rust behind >>>> crusader. >>>> the goresponsiveness thing is also IMHO running WAY too many = streams >>>> at the same time, I guess motivated by an attempt to have the test >>>> complete quickly? >>>> B) To try and tackle the validation problem:ps. Misinformation = about iperf 2 impacts my ability to do this. >>>=20 >>>> In the libreqos.io project we've established a testbed where tests = can >>>> be plunked through various ISP plan network emulations. It's here: >>>> https://payne.taht.net (run bandwidth test for what's currently = hooked >>>> up) >>>> We could rather use an AS number and at least a ipv4/24 and ipv6/48 = to >>>> leverage with that, so I don't have to nat the various emulations. >>>> (and funding, anyone got funding?) Or, as the code is GPLv2 = licensed, >>>> to see more test designers setup a testbed like this to calibrate >>>> their own stuff. >>>> Presently we're able to test: >>>> flent >>>> netperf >>>> iperf2 >>>> iperf3 >>>> speedtest-cli >>>> crusader >>>> the broadband forum udp based test: >>>> https://github.com/BroadbandForum/obudpst >>>> trexx >>>> There's also a virtual machine setup that we can remotely drive a = web >>>> browser from (but I didn't want to nat the results to the world) to >>>> test other web services. >>>> _______________________________________________ >>>> Rpm mailing list >>>> Rpm@lists.bufferbloat.net >>>> https://lists.bufferbloat.net/listinfo/rpm >>> _______________________________________________ >>> Starlink mailing list >>> Starlink@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/starlink >>=20 >> _______________________________________________ >> Starlink mailing list >> Starlink@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/starlink