From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.15.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id C9E373BA8E for ; Fri, 29 Jun 2018 10:43:23 -0400 (EDT) Received: from [172.16.11.169] ([134.76.241.253]) by mail.gmx.com (mrgmx001 [212.227.17.190]) with ESMTPSA (Nemesis) id 0M86Cn-1gLfWS46WP-00vg2H; Fri, 29 Jun 2018 16:43:22 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.4 \(3445.8.2\)) From: Sebastian Moeller In-Reply-To: Date: Fri, 29 Jun 2018 16:42:26 +0200 Cc: Jonathan Morton , bloat Content-Transfer-Encoding: quoted-printable Message-Id: References: To: =?utf-8?Q?Dave_T=C3=A4ht?= X-Mailer: Apple Mail (2.3445.8.2) X-Provags-ID: V03:K1:vDlaodgCkfeKUb6szB10CvLfkL0K9d2keHmdfOfyBJ23drQzsT6 Ecm+KaVrPOeJqQLPJ8Q1+9BdOjQQKlUeTYh4F5HEA8HWaltWvK7qw30JO7uldTwUNpkGt0T CLxnx70IS6z9uTosxBBRYnqrP/c3gPHjEhdq/CHx1Vk3nYBgr460sUovzgXB9NodnXW3M9p NyT5kQhr+issIObjAz6fw== X-UI-Out-Filterresults: notjunk:1;V01:K0:Nk4WCa951zg=:B0OhPND4CAkv5FtrLe2WR6 OvQwiillPy5b+MiEoBxnGdWD6g9BEP0AImWcS1DwfwbZLRtWeFoMIdk0klYQAW6zdNo4utlX2 G3IFr3OV7XR+Z9PVZLCb93/F4pHo2GLa25AwadLYYVC15Z7GT4YHDKSxbqdSMnTw8bMucwXQK vha/5XBX9r8Fklz1jgal6JAkMY7wcdW9ACkFG/bTRElLF6VEESS3p6NG5Q7zqmNVgmaXX5lKX uyrkD9rdtUE/70zp7SUj0VXPcNMDDOZaUErhSoGR0qZH3f9xnaAcNjT1JuKgyxJuPqJ5LcutE OYuloKb7yCgqp2+FivJHq8qG7f15zSnEpyFO9PNottkx/myTZiSJrenn2/zG1PagRrIano7Ot K9gatCZ8xA9sk+900gJ5z4di4i+zFG/AqFZt4RLoraCMHxMPai6LFGL08w6+eh4vpNsjCva04 189cbS++6/g9UDHlpAxrRP/umYETSvg1x+0kEAjnm20mvtzsllomyIQMd5qfc9OYvpYMdDl/G VpXoeBqGwWEbCmGnlBJ+LLKNZTK/2pCAYlyuM6ltQn2MEAe3oLbQfEc9Ol6caQ3gRzJz2ceiS J4EcuKmhKJ+oF/l/CQ2BAFkhkBBA5RjU0EmHs1W2KUB3nkw+PCVcu+arntSvE2yILMfonGt6m JdfhmQtiifxss6yg5cOpJlx2/WXvVj25mpDq+A5BeleZtF52enE/xbNA7EDs7xo/9DyBLmt23 7865CB5zx6fdlz+HpwseIrxKhHzU9qESYgqk6flhOYutECs7rVrmmLbqfdk9HkK1lw7cNkE6Y eqUTM73 Subject: Re: [Bloat] powerboost and sqm X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jun 2018 14:43:24 -0000 Hi Dave, > On Jun 29, 2018, at 16:00, Dave Taht wrote: >=20 > That is a remarkably good explanation and should end up on a website > or blog somewhere. +1 > A piece targetted > at gamers with the hz values.... I do not believe that using Hz will make things easier to digest = and understand; after all everybody understands time, but few people = intuitively understand reciprocals. At that point it might be more = intuitive to give the distance light will travel in fiber in the latency = increase under load ("you could be playing your FPS against an opponent = on the moon at that bufferbloat level"). Best Regards Sebastian > On Fri, Jun 29, 2018 at 8:22 AM Jonathan Morton = wrote: >>=20 >>> On 29 Jun, 2018, at 1:56 pm, Jonas M=C3=A5rtensson = wrote: >>>=20 >>> So, what do you think: >>>=20 >>> - Are the latency spikes real? The fact that they disappear with sqm = suggests so but what could cause such short spikes? Is it related to the = powerboost? >>=20 >> I think I can explain what's going on here. TL;DR - the ISP is still = using a dumb FIFO, but have resized it to something reasonable. The = latency spikes result from an interaction between several systems. >>=20 >> "PowerBoost" is just a trade name for the increasingly-common = practice of configuring a credit-mode shaper (typically a token bucket = filter, or TBF for short) with a very large bucket. It refills that = bucket at a fixed rate (100Mbps in your case), up to a specified = maximum, and drains it proportionately for every byte sent over the = link. Packets are sent when there are enough tokens in the bucket to = transmit them in full, and not before. There may be a second TBF with a = much smaller bucket, to enforce a limit on the burst rate (say 300Mbps) = - but let's assume that isn't used here, so the true limit is the 1Gbps = link rate. >>=20 >> Until the bucket empties, packets are sent over the link as soon as = they arrive, so the observed latency is minimal and the throughput = converges on a high value. The moment the bucket empties, however, = packets are queued and throughput instantaneously drops to 100Mbps. The = queue fills up quickly and overflows; packets are then dropped. TCP = rightly interprets this as its cue to back off. >>=20 >> The maximum inter-flow induced latency is consistently about 125ms. = This is roughly what you'd expect from a dumb FIFO that's been sized to = 1x BDP, and *much* better than typical ISP configurations to date. I'd = still much rather have the sub-millisecond induced latencies that Cake = achieves, but this is a win for the average Joe Oblivious. >>=20 >> So why the spikes? Well, TCP backs off when it sees the packet = losses, and it continues to do so until the queue drains enough to stop = losing packets. However, this leaves TCP transmitting at less than the = shaped rate, so the TBF starts filling its bucket with the leftovers. = Latency returns to minimum because the queue is empty. TCP then = gradually grows its congestion window again to probe the path; different = TCPs do this in different patterns, but it'll generally take much more = than one RTT at this bandwidth. Windows, I believe, increases cwnd by = one packet per RTT (ie. is Reno-like). >>=20 >> So by the time TCP gets back up to 100Mbps, the TBF has stored quite = a few spare tokens in its bucket. These now start to be spent, while = TCP continues probing *past* 100Mbps, still not seeing the true limit. = By the time the bucket empties, TCP is transmitting considerably faster = than 100Mbps, such that the *average* throughput since the last loss is = *exactly* 100Mbps - this last is what TBF is designed to enforce; it = technically doesn't start with a full bucket but an empty one, but the = bucket usually fills up before the user gets around to measuring it. >>=20 >> So then the TBF runs out of spare tokens and slams on the brakes, the = queue rapidly fills and overflows, packets are lost, TCP reels, = retransmits, and backs off again. Rinse, repeat. >>=20 >>> - Would you enable sqm on this connection? By doing so I miss out on = the higher rate for the first few seconds. >>=20 >> Yes, I would absolutely use SQM here. It'll both iron out those = latency spikes and reduce packet loss, and what's more it'll prevent = congestion-related latency and loss from affecting any but the provoking = flow(s). >>=20 >> IMHO, the benefits of PowerBoost are illusory. When you've got = 100Mbps steady-state, tripling that for a short period is simply not = perceptible in most applications. Even Web browsing, which typically = involves transfers smaller than the size of the bucket, is limited by = latency not bandwidth, once you get above a couple of Mbps. For a real = performance benefit - for example, speeding up large software updates - = bandwidth increases need to be available for minutes, not seconds, so = that a gigabyte or more can be transferred at the higher speed. >>=20 >>> What are the actual downsides of not enabling sqm in this case? >>=20 >>=20 >> Those latency spikes would be seen by latency-sensitive applications = as jitter, which is one of the most insidious problems to cope with in a = realtime interactive system. They coincide with momentary spikes in = packet loss (which unfortunately is not represented in dslreports' = graphs) which are also difficult to cope with. >>=20 >> That means your VoIP or videoconference session will glitch and drop = out periodically, unless it has deliberately increased its own latency = with internal buffering and redundant transmissions to compensate, if a = bulk transfer is started up by some background application (Steam, = Windows Update) or by someone else in your house (visiting niece = bulk-uploads an SD card full of holiday photos to Instagram, wife cues = up Netflix for the evening). >>=20 >> That also means your online game session, under similar = circumstances, will occasionally fail to show you an enemy move in time = for you to react to it, or even delay or fail to register your own = actions because the packets notifying the server of them were queued = and/or lost. Sure, 125ms is a far cry from the multiple seconds we = often see, but it's still problematic for gamers - it corresponds to = just 8Hz, when they're running their monitors at 144Hz and their mice at = 1000Hz. >>=20 >> - Jonathan Morton >>=20 >> _______________________________________________ >> Bloat mailing list >> Bloat@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/bloat >=20 >=20 >=20 > --=20 >=20 > Dave T=C3=A4ht > CEO, TekLibre, LLC > http://www.teklibre.com > Tel: 1-669-226-2619 > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat