From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [IPv6:2a00:1450:4864:20::631]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id CF4F03B2A4 for ; Fri, 26 Feb 2021 03:21:01 -0500 (EST) Received: by mail-ej1-x631.google.com with SMTP id lr13so13286562ejb.8 for ; Fri, 26 Feb 2021 00:21:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=waveform.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=/LDuh9ItyTXm2dPrGBciNkb5YjsyZejpzbGxcYoQXlU=; b=jWa5x2qyBOBuagHxbLhKhFDH/bJz8EYEfv2eCjdZ6BML6PxmXmDlGNX2N/DsfmID4k JgMQgKvyEQrPfz928nWILOavuQl/dj+6dGaUj+H6N94TGaDPDjoSsqL9ltC/+c0+vmzQ eorNXruvz94b6+Hc9lElNvm19gqWj+8V86pYFLCzY9w5fukWmhT77B1RLvRqsiKXQ6yP Ef1d0cWGH9lixXiBoCuOA8/3ZUQ1mtUZyB6DO8a41ioZhL70vEZKTVPanseKhjdeU0j1 hYMKcX8KoNjJX1ceGX5Fs+fFNKB0Xm1ArrsmanseplAgfcMvH7KUuIrrJHce382WnW3C U/4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=/LDuh9ItyTXm2dPrGBciNkb5YjsyZejpzbGxcYoQXlU=; b=T9V+G8JNsS7qd1Z9m4wM1ORHVM3UkrjzJUeg7PcxTV2B+Kb+bAWtSpoaNdrGmuRwn+ 9GKpebd/DXCZnSzAAIMREgxsAkwv5ls3YZik8Yus+deeM/c9JhA1RsmWlbIYIAbPvtIz 1WMV9y3gQNWfGIkUE/SC5x/L2lD3oyj03PyEEeOnlDrL9ijUez5ARHqo6DJLVCbfMNSS er+Bekxc6E5/rS4Sek7Q/AjuXt0q8xa6VdnJFwdPalhOYVifjUhkJAGHwdMACYbHR0dT xsxgqFL6gK1lXWRWqh7HTXuGSPqTfwI53hRixPL1J4syHkWu2e2GRlzUjIfveG64sVMe 7nzA== X-Gm-Message-State: AOAM531jc2Z32fcg6pTPkj49UB+O3708V3WncwQ/hcEpvhM780GltR8e hQnnlcdD6bqQ+8YICbbqFmPV5qqXVn7AHtwhVxv9mAlGQ3rsi6NpF40Jhz842HQJ1GZw2+laBI9 +QfmKH3NYEridmpaTMw7sfi7GU/Sxz/KT4UIdCLsAe/KL3HWQciksMVGKQKz3U8TLG3B6TN7Vvy 7Ie54ulL0= X-Google-Smtp-Source: ABdhPJwICGnCG5KPAERLobYEiq5qxp263ejnt2LVJy35zBimoBrXekcq9eoRUyT1dnJdjj5NOJvcMQ== X-Received: by 2002:a17:907:1b02:: with SMTP id mp2mr2112010ejc.419.1614327660257; Fri, 26 Feb 2021 00:21:00 -0800 (PST) Received: from mail-ej1-f54.google.com (mail-ej1-f54.google.com. [209.85.218.54]) by smtp.gmail.com with ESMTPSA id la24sm4784278ejb.18.2021.02.26.00.20.59 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 26 Feb 2021 00:20:59 -0800 (PST) Received: by mail-ej1-f54.google.com with SMTP id n20so13327759ejb.5 for ; Fri, 26 Feb 2021 00:20:59 -0800 (PST) X-Received: by 2002:a17:906:b4c:: with SMTP id v12mr2130005ejg.330.1614327659479; Fri, 26 Feb 2021 00:20:59 -0800 (PST) MIME-Version: 1.0 References: <5CB9F40C-5FFB-4ABC-A184-A3C3A29D2413@gmx.de> <7635C4DC-33F0-44C5-A909-6B81A463FE3F@gmx.de> <8b25a2b4-e158-3291-4f1d-b27ddf7b55d5@lakelandappliedsciences.com> In-Reply-To: <8b25a2b4-e158-3291-4f1d-b27ddf7b55d5@lakelandappliedsciences.com> From: Sina Khanifar Date: Fri, 26 Feb 2021 00:20:47 -0800 X-Gmail-Original-Message-ID: Message-ID: To: contact@lakelandappliedsciences.com Cc: Sebastian Moeller , =?UTF-8?Q?Dave_T=C3=A4ht?= , sam@waveform.com, bloat Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Bloat] Updated Bufferbloat Test X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Feb 2021 08:21:02 -0000 Hi Daniel, and thanks for chiming in! > I have used the waveform.com test myself, and found that it didn't do a g= ood job of measuring latency in terms of having rather high outlying tails,= which were not realistic for the actual traffic of interest. I think the reason that this is happening may be due to CPU throttling, and potentially a limitation of the fact that ours is a browser-based test, rather than anything else. I am on a cable modem with about 900 Mbps down and 35 Mbps up. If I use Chrome's developer tools to I limit my connection to 700 Mbps down and 25 Mbps up, I end up with no bufferbloat as the tool can no longer saturate my connection. I don't see those unusual latency spikes. This CPU-throttling issue is one of the biggest problems with browser-based tests. We spent a lot of time trying to minimize CPU usage, but this was the best we could manage (and it's quite a bit better than earlier versions of the test). I think that given this limitation of browser-based tests, maybe the users you have in mind are not our target audience. Our goal is to make an easy-to-use bufferbloat test "for the rest of us." For technically-minded gamers, running flent will likely give much better results. I'd love for it to be otherwise, so we may try to tinker with things to see if we can get CPU usage down and simultaneously deal with the UDP question below. > I think the ideal test would use webRTC connection using UDP. I definitely agree that UDP traffic is the most latency-sensitive. But bufferbloat also affects TCP traffic. Our thinking went something like this: QoS features on some routers prioritize UDP traffic. But bufferbloat affects all traffic. If we only tested UDP, QoS prioritization of UDP would mean that some routers would show no bufferbloat in our test, but nevertheless exhibit increased latency on TCP traffic. By testing TCP, we could ensure that both TCP and UDP traffic weren't affected by bufferbloat. That being said: the ideal case would be to test both TCP and UDP traffic, and then combine the results from both to rate bufferbloat and grade the "real-world impact" section. I believe Cloudflare is adding support for WebSockets and UDP traffic to the workers, which should mean that we can add a UDP test when that happens. We'll revisit this in the future, but given the CPU throttling issue discussed above, I'm not sure it's going to get our test to exactly where you'd like it to be. > Supposing your test runs for say 10 seconds, you'll have 640 RTT samples.= Resampling 64 of them randomly say 100 times and calculate how many of the= se resamples have less than 1/64 =3D 15ms of increased latency. Then expres= s something like 97% chance of lag free each second, or 45% chance of lag f= ree each second or whatever. I'm not quite sure I understand this part of your suggested protocol. What do you mean by resampling? Why is the resamplling necessary? Would like to grok this in case in the future we're able to deal with the other issues and can implement an updated set of criteria for tech-savvy gamers. Best, Sina. On Thu, Feb 25, 2021 at 5:06 PM Daniel Lakeland wrote: > > On 2/25/21 12:14 PM, Sebastian Moeller wrote: > > Hi Sina, > > > > let me try to invite Daniel Lakeland (cc'd) into this discussion. He is= doing tremendous work in the OpenWrt forum to single handedly help gamers = getting the most out of their connections. I think he might have some opini= on and data on latency requirements for modern gaming. > > @Daniel, this discussion is about a new and really nice speedtest= (that I already plugging in the OpenWrt forum, as you probably recall) tha= t has a strong focus on latency under load increases. We are currently disc= ussion what latency increase limits to use to rate a connection for on-line= gaming > > > > Now, as a non-gamer, I would assume that gaming has at least similarly = strict latency requirements as VoIP, as in most games even a slight delay a= t the wrong time can direct;y translate into a "game-over". But as I said, = I stopped reflex gaming pretty much when I realized how badly I was doing i= n doom/quake. > > > > Best Regards > > Sebastian > > > Thanks Sebastian, > > I have used the waveform.com test myself, and found that it didn't do a > good job of measuring latency in terms of having rather high outlying > tails, which were not realistic for the actual traffic of interest. > > Here's a test. I ran this test while doing a ping flood, with attached > txt file > > https://www.waveform.com/tools/bufferbloat?test-id=3De2fb822d-458b-43c6-9= 84a-92694333ae92 > > > Now this is with a QFQ on my desktop, and a custom HFSC shaper on my WAN > router. This is somewhat more relaxed than I used to run things (I used > to HFSC shape my desktop too, but now I just reschedule with QFQ). Pings > get highest priority along with interactive traffic in the QFQ system, > and they get low-latency but not realtime treatment at the WAN boundary. > Basically you can see this never went above 44ms ping time, whereas the > waveform test had outliers up to 228/231 ms > > Almost all latency sensitive traffic will be UDP. Specifically the voice > in VOIP, and the control packets in games, those are all UDP. However it > seems like the waveform test measures http connection open/close, and > I'm thinking something about that is causing extreme outliers. From the > test description: > > "We are using HTTP requests instead of WebSockets for our latency and > speed test. This has both advantages and disadvantages in terms of > measuring bufferbloat. We hope to improve this test in the future to > measure latency both via WebSockets, HTTP requests, and WebRTC Data > Channels." > > I think the ideal test would use webRTC connection using UDP. > > A lot of the games I've seen packet captures from have a fixed clock > tick in the vicinity of 60-65Hz with a UDP packet sent every tick. Even > 1 packet lost per second would generally feel not that good to a player, > and it doesn't even need to be lost, just delayed by a large fraction of > 1/64 =3D 15.6ms... High performance games will use closer to 120Hz. > > So to guarantee good network performance for a game, you really want to > ensure less than say 10ms of increasing latency at the say 99.5%tile > (that's 1 packet delayed 67% or more of the between-packet time every > 200 packets or so, or every ~3 seconds at 64Hz) > > To measure this would really require WebRTC sending ~ 300 byte packets > at say 64Hz to a reflector that would send them back, and then compare > the send time to the receive time. > > Rather than picking a strict percentile, I'd recommend trying to give a > "probability of no noticeable lag in a 1 second interval" or something > like that. > > Supposing your test runs for say 10 seconds, you'll have 640 RTT > samples. Resampling 64 of them randomly say 100 times and calculate how > many of these resamples have less than 1/64 =3D 15ms of increased latency= . > Then express something like 97% chance of lag free each second, or 45% > chance of lag free each second or whatever. > > This is a quite stringent requirement. But you know what? The message > boards are FLOODED with people unhappy with their gaming performance, so > I think it's realistic. Honestly to tell a technically minded gamer "Hey > 5% of your packets will be delayed more than 400ms" they'd say THAT"S > HORRIBLE not "oh good". > > If a gamer can keep their round trip time below 10ms of latency increase > at the 99.5%tile level they'll probably be feeling good. If they get > above 20ms of increase for more than 1% of the time, they'll be really > irritated (this is more or less equivalent to 1% packet drop) > >