From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yw1-x1131.google.com (mail-yw1-x1131.google.com [IPv6:2607:f8b0:4864:20::1131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 178B13B29E; Mon, 13 Mar 2023 13:26:44 -0400 (EDT) Received: by mail-yw1-x1131.google.com with SMTP id 00721157ae682-53916ab0c6bso255342387b3.7; Mon, 13 Mar 2023 10:26:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1678728403; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=SUoK1TDvem0lDuVKAIAoIO+xpO/BSIbN/XKNNQkjLtA=; b=KuY5IHsFgbtYcpFJM2sSewX60i/u/sSTUMd5L3sbCxCOBSmJXNxkfsts1PQxjBwDih BMIwH57nTrTU0wSJPsnM0Ki9B66PgMJyDHv06W/35rT834unZcuJYp8odbC4yhip7pBt kA+dVoVtb921TcDtEgHPUIRB8vRtEitCx023y0z3Lp8/NLbOMn8MZVnXTYPma9hstlnc yrt1vFs78tmV8HoWlfflXkPainhK7MXYKERAc6MIgLDTqCXCYAzm7EVBN+Yk0v7TjxZ+ QVqC8RAUq3TJGXRSf+UFCNIhCF3aiRtSQQI2wEUtzo1x8UCOdlW9t+pmlIHw+I3kNgqG GOcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678728403; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SUoK1TDvem0lDuVKAIAoIO+xpO/BSIbN/XKNNQkjLtA=; b=WVUE8W5HAGqqNYODYIyahZgJhBKtv8L17UmY/+oOerKj1KmDX6MHQ9yFtNHe4LuCC9 5SSjEFHkr63Utayw+747ZxQ+K21e2Mx/Ufx0h7/CdqfoHbm4kOW1nemU55MRiKZjo+in m/LGD9KK2uIRWFCOomscg0B+pAQULwCS9xc1vWyzdhSsXp7b0uz1ThnowKsm3XunmVmf kFEkAe55HXfoptPrfM6hkTCcYfhNa7uv44iHM9ewgIZ/oQ7ejXHqg7W5czFdO793Y8Rp Oq1d/XPPu1uUwJaoa5/9Lw5wakhQQTgIl0f9E0suWINbfjf50qB0DBgY/avz5FFTwEd7 7KHA== X-Gm-Message-State: AO0yUKWixlKllXy7ci1GNOPtqOvXUvu4I092x7SYbAeqsNbyKXTPLK2H nfluFNKYGLmTNC5OpjZXNr/B/hVsIm+Iwf5LKvk= X-Google-Smtp-Source: AK7set/6h+0+NBd9wMsjc5bn8kg5g/h9qTnd/NjeOd/k1qvtRHwU4kfMavgmipYASS00EpAJcGJTzkAAOG6nvEsi28o= X-Received: by 2002:a81:a784:0:b0:541:6dce:50d6 with SMTP id e126-20020a81a784000000b005416dce50d6mr5329958ywh.1.1678728403303; Mon, 13 Mar 2023 10:26:43 -0700 (PDT) MIME-Version: 1.0 References: <1672786712.106922180@apps.rackspace.com> <77CCAD19-07E0-4F9E-88C1-D207CF7BF376@cable.comcast.com> <83ffc0dad19e3343e49271889369cefc@rjmcmahon.com> <3CD0B9E6-0B2A-4A70-8F53-ED0822DF77A6@gmx.de> <13DE6E53-665F-4C20-BBE2-70E685421E9D@gmx.de> In-Reply-To: <13DE6E53-665F-4C20-BBE2-70E685421E9D@gmx.de> From: dan Date: Mon, 13 Mar 2023 11:26:31 -0600 Message-ID: To: Sebastian Moeller Cc: Jeremy Austin , Rpm , libreqos , Dave Taht via Starlink , rjmcmahon , bloat Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Bloat] [Starlink] [Rpm] [LibreQoS] [EXTERNAL] Re: Researchers Seeking Probe Volunteers in USA X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Mar 2023 17:26:44 -0000 On Mon, Mar 13, 2023 at 10:36=E2=80=AFAM Sebastian Moeller wrote: > > Hi Dan, > > > > On Mar 13, 2023, at 17:12, dan wrote: > >... > > > > High water mark on their router. > > [SM] Nope, my router is connected to my (bridged) modem via gigab= it ethernet, with out a traffic shaper there is never going to be any notic= eable water mark on the router side... sure the modem will built up a queue= , but alas it does not expose the length of that DSL queue to me... A high = water mark on my traffic shaped router informs me about my shaper setting (= which I already know, after al I set it) but little about the capacity over= the bottleneck link. And we are still talking about the easy egress direct= ion, in the download direction Jeremy's question aplied is the achieved tho= ughput I measure limited by the link's capacity of are there simply not eno= iugh packet available/sent to fill the pipe. > And yet it can still see the flow of data on it's ports. The queue is irelevant to the measurement of data across a port. turn off the shaper and run anything. run your speed test. don't look at the speed test results, just use it to generate some traffic. you'll find your peak and where you hit the buffers on the DSL modem by measuring on the interface and measuring latency. That speed test isn't giving you this data and more than Disney+, other than you get to pick when it runs. > > Highwater mark on our CPE, on our > > shaper, etc. Modern services are very happy to burst traffic. > > [SM] Yes, this is also one of the readons, why too-little-bufferi= ng is problematic, I like the Nichols/Jacobsen analogy of buffers as shiock= (burst) absorbers. > > > Nearly > > every customer we have will hit the top of their service place each > > day, even if only briefly and even if their average usage is quite > > low. Customers on 600Mbps mmwave services have a usage charge that is > > flat lines and ~600Mbps blips. > > [SM] Fully agree. most links are essentially idle most of the tim= e, but that does not answer what instantaneous capacity is actually availab= le, no? yes, because most services burst. That Roku Ultra or Apple TV is going to running a 'speed test' every time it goes to fill it's buffer. Windows and Apple updates are asking for everything. Again, I'm measuring even the lowly grandma's house as consuming the entire connection for a few seconds before it sits idle for a minute. That instantaneous capacity is getting used up so long as there is a device/connection on the network capable of using it up. > > > > > " [SM] No ISP I know of publishes which periods are low, mid, high > > congestion so end-users will need to make some assumptions here (e.g. > > by looking at per day load graphs of big traffic exchanges like DE-CIX > > here https://www.de-cix.net/en/locations/frankfurt/statistics )" > > > > You read this wrong. Consumer routers run their daily speeds tests in > > the middle of the night. > > [SM] So on my turris omnia I run a speedtest roughly every 2 hour= s exactly so I get coverage through low and high demand epochs. The only co= nsumer router I know that does repeated tests is the IQrouter, which as far= as I know schedules them regularly so it can adjust the traffic shaper to = still deliver acceptale responsiveness even during peak hour. Consider this. Customer under load, using their plan to the maximum, speed test fires up adding more constraint. Speed test is a stress test, not a capacity test. Speed test cannot return actual capacity because it's being used by other services AND the rest of the internet is in the way of accuracy as well, unless of course you prioritize the speed test and then you cause an effective outage or you're running a speed test on-net which isn't an 'internet' test, it's a network test. Guess what the only way to get an actual measure of the capacity is? my way. measure what's passing the interface and measure what happens to a reliable latency test during that time. > > > > Eero at 3am for example. Netgear 230-430am. > > [SM] That sounds"specisl" not a useless daa point per se, but of = limited utility during normal usage times. In practical terms, useless. Like measuring how freeway congestion affects commutes at 3am. > > > THAT is a bad measurement of the experience the consumer will have. > > [SM] Sure, but it still gives a usable reference for "what is the= best my ISP actually delivers" if if the odds are stacked in his direction= . ehh...... what the ISP could deliver if all other considerations are removed. I mean, it'd be a synthetic test in any other scenario and the only reason it's not is because it's on real hardware. I don't have a single subscriber on network that can't get 2-3x their plan speed at 3am if I opened up their shaper. Very narrow use case here from a consumer point of view. Eero runs speed tests at 3am every single day on a few hundred subs on my network and they look AMAZING every time. no surprise. > > > It's essentially useless data for ... > > [SM] There is no single "service latency" it really depends on he= specific network paths to the remote end and back. Unless you are talking = about the latency over the access link only tere we have a single number bu= t one of limited utility. The intermediate hops are still useless to the consumer. Only the latency to their door so to speak. again, hop 2 to hop 3 on my network gives them absolutely nothing. > > > > My (ISP) latency from hop 2 to 3 on the network has > ...> > hops are which because they'll hidden in a tunnel/MPLS/etc. > > [SM] Yes, end-users can do little, but not nothing, e.g. one can = often work-around shitty peering by using a VPN to route one's packets into= an AS that is both well connected with one's ISP as well as with one's rem= ote ASs. And I accept your point of one-way testing, getting a remote site = at the ight location to do e.g. reverse tracerputes mtrs is tricky (sometim= es RIPE ATLAS can help) to impossible (like my ISP that does not offer even= simple lookingglas servers at all)). This is a REALLY narrow use case. Also, irrelevant. Consumer can test to their target, to the datacenter, and datacenter to target and compare, and do that in reverse to get bi-directional latency. per hop latency is still of zero benefit to them because they can't use that in any practical way. Like 1 in maybe a few thousand consumers might be able to use this data to identify the slow hop and find a datacenter before that hop to route around it and they get about 75% of the way with a traditional trace router. and then of course they've added VPN overheads so are they really getting an improvement? I'm not saying that testing is bad in any way, I'm saying that 'speed tests' as they are generally understood in this industry are a bad method. Run 10 speed tests, get 10 results. Run a speed test while netflix buffers, get a bad result. Run a speed test from a weak wifi connection, get a bad result. A tool that is inherently flawed because it's methodology is flawed is of no use to find the truth. If you troubleshoot your ISP based on speed tests you will be chasing your tail. Meanwhile, that internet facing interface can see the true numbers the entire time. The consumer is pulling their full capacity on almost all links routinely even if briefly and can be nudged into pushing more a dozen ways (including a speed test). The only thing lacking is a latency measurement of some sort. Preseem and Libreqos's TCP measurements on the head end are awesome, but that's not available on the subscriber's side but if it were, there's the full testing suite. how much peak data, what happened to latency. If you could get data from the ISP's head end to diff you'd have internet vs isp latencies. 'speed test' is a stress test or a burn in test in effect.