From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id DD2513B2A4; Fri, 13 Jan 2023 02:45:11 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de; s=s31663417; t=1673595902; bh=5qyTuL+mn4oec2sSmy+peGAZ/J7v+L+hmrIR8kAA5/w=; h=X-UI-Sender-Class:Date:From:To:CC:Subject:In-Reply-To:References; b=k8VUbOXkVdgrD+AEaW5zZVnmalkp3V6gkoAnVtZ7YsaH0UmyshHJD423LQyd3DHOP iOn4lFxkybYKI19D6p316BYgj7aViX9+vJoMIq3aZC/KOxWi7P33FPzCFfHFiACpE1 iIgAExCW9YCpRJ0wi+M63EySHL3v1bY1h4xHlK2BTNKXLKKDOVUdR8NvF9T6kO9o88 7bULQsqjTdYDkiPcM6X4L2OgnE7bVKTw6BFj0c1c8E4jj5sYMAn/rLLv8wqIqYgO7G PdJ7/eDV2vie8jFNpd44vNsrCB8jGHaZsb1DGHpXxEagJYlYSL7Od8tRuMcYNgNJgN JDTm1kXXqI1PA== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Received: from [127.0.0.1] ([80.187.108.196]) by mail.gmx.net (mrgmx105 [212.227.17.168]) with ESMTPSA (Nemesis) id 1M4JmT-1pFxd634Gf-000IE6; Fri, 13 Jan 2023 08:45:01 +0100 Date: Fri, 13 Jan 2023 08:44:57 +0100 From: Sebastian Moeller To: dickroy@alum.mit.edu, Dick Roy , 'Robert McMahon' CC: mike.reynolds@netforecast.com, 'libreqos' , "'David P. Reed'" , 'Rpm' , 'bloat' User-Agent: K-9 Mail for Android In-Reply-To: <177980E5906F412288C57A60C82DF9D1@SRA6> References: <202301111832.30BIWevV030127@gndrsh.dnsmgr.net> <44D8ADEB-8F89-4477-BCBD-C597A888A83E@gmx.de> <752678c9-9fdf-4abe-9915-564e3989f4d8@rjmcmahon.com> <177980E5906F412288C57A60C82DF9D1@SRA6> Message-ID: <45F6F0F0-CADA-4E63-9347-A526A0117F52@gmx.de> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K1:238zp1ts4JlfL0psdYOrLUqDlHjWbQX5NHKDJVhJfF+ZnwGsSzT YhFdn2m4PUWZXxgKwtWHR1cU2WmHfGbUp3BpqkF8RVesEUs4cp/0Sg2emPh4gEUgOeJzctY t7CvavMbgeR3t/RZyXQtnICngbHL7sxAH2AuLaebC2AiBFr/5LgJvtYcaAeRUjRT4YprtG+ 5lXXniAOmvvafYyuMbVwA== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:o2OI8ZHZMdQ=;mrGcUb0RcmHonETNRRhIv49tFvk 7iWdLvTw87FgGHgrVWd+51M8e5NY9HtD3uARHyZPA5pzk37ykOjleyXYZpWr5GleawLy3Eo6Z qNdM9E7/dWDBRsDhTyQqsXlVsKg7Rauw1RcIu+G3PxM6HvcNk0wyhVRD6p8+GnIHQRQA4gmH+ +2xRWtfWVi/wXpozcE4t+umIRj7GIrA/VpzQiFKNqRKfhVEPvmAlEWPFFM1WYZ/NPjTnjazHR bIjEftycRnaKczcXkr98n3HScZibiDof9VrKOtV5STXcY7JPP7smBkkygDFLOwCfhQ9nt4xGg q3ywwR/30797mFH/fpcNHDxFNW7PrYj1lQoxlgaDODC+KkSEq89wFOpKkkmaz8c50T3k3g1cA R5fuOFIe+v31eQ1gAXxdCT7Iex2oKLwUUH/rYwboH7MZfJhqnUoYpT7UYPlzreDLOOwKhmIjS e3UQKAYuFYjh4IkLrByjoz6QM3NqtQ/Vz4dnf/Z0mi1xsw6hs22nHdl9JTLzy3mxqLsg/FmAV ekbqkfX+3xb6QMpWmDzuIjjh3ld3913NBlH0BT8UGRjLEbvxbztlfuHxMLc5xQuP3b79f/eOQ jWiOlLjxkCszS3ucXA9L0Z8kCKlOhdTILIPfF/wCWzqH5ExluwoZ8Pa+1wHf7YZZtPvfftQKh Ec2OY87EWcdjVglOoTbzZ5dPRmvMBnGUFk6jo57SX4NzBQ6ohr8FTKbRjMJgwHrZG2zEhhmBq ZcxiBzzm+9b9m/XVTFF2IPNzDP0TfwK95/sPtDz1FQCLfw6nngCBAZONsnCr0ZlT48QZ4sWL2 0CFI/pGV4JevgepkVkucEpiHGNIusD1AnB07oYvS3TXbXirXKP1QvTUd/9+f7wC8+b3/wN/xW RTbOq/VRwcqUbIaaIFZp+DYXnqVn067L3WzbKpWpXCH2XCbtHhmmhhGP1XldtsxweJ5r8w+T4 QCqgOb0LzDfkH1YnxSCkmxMdZbM= Subject: Re: [LibreQoS] [Starlink] [Rpm] Researchers Seeking Probe Volunteers in USA X-BeenThere: libreqos@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Many ISPs need the kinds of quality shaping cake can do List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2023 07:45:12 -0000 Hi RR On 12 January 2023 22:57:32 CET, Dick Roy wrote= : >FYI =2E > >=20 > >https://www=2Efiercewireless=2Ecom/tech/cbrs-based-fwa-beats-starlink-per= formanc >e-madden > [SM] He is so close: 'Speed tests don=E2=80=99t tell us much about the capacity of the network,= or the reliability of the network, or the true latency with larger packet = sizes=2E Packet loss testing can help to fill in key missing information to= give the end customer the smooth experience they=E2=80=99re looking for=2E= ' and 'Packets received over 250 ms latency are considered too late to be useful= for video conferencing=2E' He actually reports both loss numbers and delay > 250ms, so in spite argui= ng that loss is the relevant metric he already dips his toes into the laten= cy issue=2E=2E=2E I wonder whether his view will refine over time now that = he apparently moved from a link with 8% packet loss to one with a more sane= 0=2E1% loss rate (no idea how he measured lossrate though, or latency)=2E = I guess this shows that there is no single solution for all links, it reall= y matters where one starts which of throughput, delay, loss is the most pai= nful and hence the dimension in need of a fix first=2E Regards Sebastian >=20 > >Nothing earth-shaking :-) > > >RR > >=20 > > _____ =20 > >From: Starlink [mailto:starlink-bounces@lists=2Ebufferbloat=2Enet] On Beh= alf Of >Robert McMahon via Starlink >Sent: Thursday, January 12, 2023 9:50 AM >To: Sebastian Moeller >Cc: Dave Taht via Starlink; mike=2Ereynolds@netforecast=2Ecom; libreqos; = David >P=2E Reed; Rpm; bloat >Subject: Re: [Starlink] [Rpm] Researchers Seeking Probe Volunteers in USA > >=20 > >Hi Sebastien, > >You make a good point=2E What I did was issue a warning if the tool found= it >was being CPU limited vs i/o limited=2E This indicates the i/o test likel= y is >inaccurate from an i/o perspective, and the results are suspect=2E It doe= s >this crudely by comparing the cpu thread doing stats against the traffic >threads doing i/o, which thread is waiting on the others=2E There is no >attempt to assess the cpu load itself=2E So it's designed with a singular >purpose of making sure i/o threads only block on syscalls of write and re= ad=2E > >I probably should revisit this both in design and implementation=2E Thank= s for >bringing it up and all input is truly appreciated=2E=20 > >Bob > >On Jan 12, 2023, at 12:14 AM, Sebastian Moeller wrote= : > >Hi Bob, > > > > > > > On Jan 11, 2023, at 21:09, rjmcmahon wrote: > > >=20 > > > Iperf 2 is designed to measure network i/o=2E Note: It doesn't have to m= ove >large amounts of data=2E It can support data profiles that don't drive TC= P's >CCA as an example=2E > > >=20 > > > Two things I've been asked for and avoided: > > >=20 > > > 1) Integrate clock sync into iperf's test traffic > > > > [SM] This I understand, measurement conditions can be unsuited for tight >time synchronization=2E=2E=2E > > > > > > > 2) Measure and output CPU usages > > > > [SM] This one puzzles me, as far as I understand the only way to properl= y >diagnose network issues is to rule out other things like CPU overload tha= t >can have symptoms similar to network issues=2E As an example, the cake qd= isc >will if CPU cycles become tight first increases its internal queueing and >jitter (not consciously, it is just an observation that once cake does no= t >get access to the CPU as timely as it wants, queuing latency and variabil= ity >increases) and then later also shows reduced throughput, so similar thing= s >that can happen along an e2e network path for completely different reason= s, >e=2Eg=2E lower level retransmissions or a variable rate link=2E So i woul= d think >that checking the CPU load at least coarse would be within the scope of >network testing tools, no? > > > > > >Regards > > > Sebastian > > > > > > > > > > > > > I think both of these are outside the scope of a tool designed to test >network i/o over sockets, rather these should be developed & validated >independently of a network i/o tool=2E > > >=20 > > > Clock error really isn't about amount/frequency of traffic but rather >getting a periodic high-quality reference=2E I tend to use GPS pulse per >second to lock the local system oscillator to=2E As David says, most ever= y >modern handheld computer has the GPS chips to do this already=2E So to me= it >seems more of a policy choice between data center operators and device mf= gs >and less of a technical issue=2E > > >=20 > > > Bob > Hello, > > > Yall can call me crazy if you want=2E=2E but=2E=2E=2E see below [RWG] > Hi Bib, > On Jan 9, 2023, at 20:13, rjmcmahon via Starlink > wrote: > > > > > > My biggest barrier is the lack of clock sync by the devices, i=2Ee=2E ve= ry >limited support for PTP in data centers and in end devices=2E This limits= the >ability to measure one way delays (OWD) and most assume that OWD is 1/2 a= nd >RTT which typically is a mistake=2E We know this intuitively with airplan= e >flight times or even car commute times where the one way time is not 1/2 = a >round trip time=2E Google maps & directions provide a time estimate for t= he >one way link=2E It doesn't compute a round trip and divide by two=2E > > > > > > For those that can get clock sync working, the iperf 2 --trip-times opti= ons >is useful=2E > [SM] +1; and yet even with unsynchronized clocks one can try to measure >how latency changes under load and that can be done per direction=2E Sure= this >is far inferior to real reliably measured OWDs, but if life/the internet >deals you lemons=2E=2E=2E=2E > [RWG] iperf2/iperf3, etc are already moving large amounts of data > > > back and forth, for that matter any rate test, why not abuse some of > > > that data and add the fundemental NTP clock sync data and > > > bidirectionally pass each others concept of "current time"=2E IIRC (its > > > been 25 years since I worked on NTP at this level) you *should* be > > > able to get a fairly accurate clock delta between each end, and then > > > use that info and time stamps in the data stream to compute OWD's=2E > > > You need to put 4 time stamps in the packet, and with that you can > > > compute "offset"=2E > > > > > --trip-times > > > enable the measurement of end to end write to read latencies (client an= d >server clocks must be synchronized) > > [RWG] --clock-skew > > > enable the measurement of the wall clock difference between sender and >receiver > [SM] Sweet! > > > Regards > > > Sebastian > > > > Bob > I have many kvetches about the new latency under load tests being > > > designed and distributed over the past year=2E I am delighted! that they > > > are happening, but most really need third party evaluation, and > > > calibration, and a solid explanation of what network pathologies they > > > do and don't cover=2E Also a RED team attitude towards them, as well as > > > thinking hard about what you are not measuring (operations research)=2E > > > I actually rather love the new cloudflare speedtest, because it tests > > > a single TCP connection, rather than dozens, and at the same time folk > > > are complaining that it doesn't find the actual "speed!"=2E yet=2E=2E=2E= the > > > test itself more closely emulates a user experience than speedtest=2Enet > > > does=2E I am personally pretty convinced that the fewer numbers of flows > > > that a web page opens improves the likelihood of a good user > > > experience, but lack data on it=2E > > > To try to tackle the evaluation and calibration part, I've reached out > > > to all the new test designers in the hope that we could get together > > > and produce a report of what each new test is actually doing=2E I've > > > tweeted, linked in, emailed, and spammed every measurement list I know > > > of, and only to some response, please reach out to other test designer > > > folks and have them join the rpm email list? > > > My principal kvetches in the new tests so far are: > > > 0) None of the tests last long enough=2E > > > Ideally there should be a mode where they at least run to "time of > > > first loss", or periodically, just run longer than the > > > industry-stupid^H^H^H^H^H^Hstandard 20 seconds=2E There be dragons > > > there! It's really bad science to optimize the internet for 20 > > > seconds=2E It's like optimizing a car, to handle well, for just 20 > > > seconds=2E > > > 1) Not testing up + down + ping at the same time > > > None of the new tests actually test the same thing that the infamous > > > rrul test does - all the others still test up, then down, and ping=2E It > > > was/remains my hope that the simpler parts of the flent test suite - > > > such as the tcp_up_squarewave tests, the rrul test, and the rtt_fair > > > tests would provide calibration to the test designers=2E > > > we've got zillions of flent results in the archive published here: > > > https://blog=2Ecerowrt=2Eorg/post/found_in_flent/ > > > ps=2E Misinformation about iperf 2 impacts my ability to do this=2E > >=20 > The new tests have all added up + ping and down + ping, but not up + > > > down + ping=2E Why?? > > > The behaviors of what happens in that case are really non-intuitive, I > > > know, but=2E=2E=2E it's just one more phase to add to any one of those n= ew > > > tests=2E I'd be deliriously happy if someone(s) new to the field > > > started doing that, even optionally, and boggled at how it defeated > > > their assumptions=2E > > > Among other things that would show=2E=2E=2E > > > It's the home router industry's dirty secret than darn few "gigabit" > > > home routers can actually forward in both directions at a gigabit=2E I'd > > > like to smash that perception thoroughly, but given our starting point > > > is a gigabit router was a "gigabit switch" - and historically been > > > something that couldn't even forward at 200Mbit - we have a long way > > > to go there=2E > > > Only in the past year have non-x86 home routers appeared that could > > > actually do a gbit in both directions=2E > > > 2) Few are actually testing within-stream latency > > > Apple's rpm project is making a stab in that direction=2E It looks > > > highly likely, that with a little more work, crusader and > > > go-responsiveness can finally start sampling the tcp RTT, loss and > > > markings, more directly=2E As for the rest=2E=2E=2E sampling TCP_INFO on > > > windows, and Linux, at least, always appeared simple to me, but I'm > > > discovering how hard it is by delving deep into the rust behind > > > crusader=2E > > > the goresponsiveness thing is also IMHO running WAY too many streams > > > at the same time, I guess motivated by an attempt to have the test > > > complete quickly? > > > B) To try and tackle the validation problem:ps=2E Misinformation about i= perf >2 impacts my ability to do this=2E > >=20 > In the libreqos=2Eio project we've established a testbed where tests can > > > be plunked through various ISP plan network emulations=2E It's here: > > > https://payne=2Etaht=2Enet (run bandwidth test for what's currently hook= ed > > > up) > > > We could rather use an AS number and at least a ipv4/24 and ipv6/48 to > > > leverage with that, so I don't have to nat the various emulations=2E > > > (and funding, anyone got funding?) Or, as the code is GPLv2 licensed, > > > to see more test designers setup a testbed like this to calibrate > > > their own stuff=2E > > > Presently we're able to test: > > > flent > > > netperf > > > iperf2 > > > iperf3 > > > speedtest-cli > > > crusader > > > the broadband forum udp based test: > > > https://github=2Ecom/BroadbandForum/obudpst > > > trexx > > > There's also a virtual machine setup that we can remotely drive a web > > > browser from (but I didn't want to nat the results to the world) to > > > test other web services=2E > > > > > > _____ =20 > > > > > > > Rpm mailing list > > > Rpm@lists=2Ebufferbloat=2Enet > > > https://lists=2Ebufferbloat=2Enet/listinfo/rpm > > > > > > > _____ =20 > > > > > > > Starlink mailing list > > > Starlink@lists=2Ebufferbloat=2Enet > > > https://lists=2Ebufferbloat=2Enet/listinfo/starlink > > > > > > _____ =20 > > > > > > > Starlink mailing list > > > Starlink@lists=2Ebufferbloat=2Enet > > > https://lists=2Ebufferbloat=2Enet/listinfo/starlink >=20 --=20 Sent from my Android device with K-9 Mail=2E Please excuse my brevity=2E