From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qt0-x22a.google.com (mail-qt0-x22a.google.com [IPv6:2607:f8b0:400d:c0d::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 8DD883BA8E for ; Fri, 29 Jun 2018 10:00:05 -0400 (EDT) Received: by mail-qt0-x22a.google.com with SMTP id q12-v6so3948819qtp.6 for ; Fri, 29 Jun 2018 07:00:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=hjo+Sq1e5NJ2+hqIyO0eFOzHmQvQbjAgJeeTl1XQgyA=; b=nUuyJOTulK5Lo5kHJJIohAeb0zLL+Wulgyei/usLQL2DJgAO7W1RWEzKtxazpQEA7/ QXZB5aZRzsnDOZSD6Eq9+49vA2FJCEvadrd2+fkExDbI/5wW0LAPVsuJ4Mb/94F/81XB 03mQdLpBJUuQTvuuBtoZBPUjRH7hUJFjA+UPjttvQnms6qushoHIHj/A5odix3o0hB0G ErOeqcJwIZpWBe9GuZOV7+owloXT90E1EFpZsRO1LejlQphCamUtcVqNCBF0YHGBNqp3 VyCH/zVb7X9r715aESPvQuDEX/fwckGQolUnuf5zM2Py1H2Hdq7DGREsYjQVRnkGqo7a qx0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=hjo+Sq1e5NJ2+hqIyO0eFOzHmQvQbjAgJeeTl1XQgyA=; b=Iup26tGbBkF7otm6GxYRCUngiq0WBGpxM/r96V2Sg82lTPIkbxyHQLwUGlfa7kUzEn 2raF6l17S+XTcKyM0PiCGmux2xeMsLdQrINC22yFXRB2Kureop/AsoizpWdahWDcnjvx pKQB48RQ/mNozBDikI1OJ0rOtcAtxtRk17RqRm8xibsa77OBap7690KCRhuApJFrWPgV mGszYRn6E10a6+ZIMeI25wM5dw2cSW9YpMLyMt2P+b3+axQLhaND7a5lRt6W7fXmUy2i dTwMXeJALY5dGIMaFyQd4mBFwwwaRe/G0bqc4LgdSuo3gMzyVW/uOdOXp2klzcv8meCu oB+g== X-Gm-Message-State: APt69E2odtLOw25+OlZOqDThWKL8wI57EiN+U0ibz4yMeXwAckYhGADf /m/b0pGPK3uTZSqhreWyPW7zgSS5h7ehtzdycKs= X-Google-Smtp-Source: AAOMgpdcAvzzA9gS8kqaqWhatmRwocyZlXYrxq6PHsOOjLH5xYlyWnThoVDM0NR2X+D4cEOqUjvgipMjJa9LhVNgzhU= X-Received: by 2002:ac8:683:: with SMTP id f3-v6mr13893404qth.104.1530280805046; Fri, 29 Jun 2018 07:00:05 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Dave Taht Date: Fri, 29 Jun 2018 10:00:13 -0400 Message-ID: To: Jonathan Morton Cc: =?UTF-8?Q?Jonas_M=C3=A5rtensson?= , bloat Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Bloat] powerboost and sqm X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jun 2018 14:00:05 -0000 That is a remarkably good explanation and should end up on a website or blog somewhere. A piece targetted at gamers with the hz values.... On Fri, Jun 29, 2018 at 8:22 AM Jonathan Morton wro= te: > > > On 29 Jun, 2018, at 1:56 pm, Jonas M=C3=A5rtensson wrote: > > > > So, what do you think: > > > > - Are the latency spikes real? The fact that they disappear with sqm su= ggests so but what could cause such short spikes? Is it related to the powe= rboost? > > I think I can explain what's going on here. TL;DR - the ISP is still usi= ng a dumb FIFO, but have resized it to something reasonable. The latency s= pikes result from an interaction between several systems. > > "PowerBoost" is just a trade name for the increasingly-common practice of= configuring a credit-mode shaper (typically a token bucket filter, or TBF = for short) with a very large bucket. It refills that bucket at a fixed rat= e (100Mbps in your case), up to a specified maximum, and drains it proporti= onately for every byte sent over the link. Packets are sent when there are= enough tokens in the bucket to transmit them in full, and not before. The= re may be a second TBF with a much smaller bucket, to enforce a limit on th= e burst rate (say 300Mbps) - but let's assume that isn't used here, so the = true limit is the 1Gbps link rate. > > Until the bucket empties, packets are sent over the link as soon as they = arrive, so the observed latency is minimal and the throughput converges on = a high value. The moment the bucket empties, however, packets are queued a= nd throughput instantaneously drops to 100Mbps. The queue fills up quickly= and overflows; packets are then dropped. TCP rightly interprets this as i= ts cue to back off. > > The maximum inter-flow induced latency is consistently about 125ms. This= is roughly what you'd expect from a dumb FIFO that's been sized to 1x BDP,= and *much* better than typical ISP configurations to date. I'd still much= rather have the sub-millisecond induced latencies that Cake achieves, but = this is a win for the average Joe Oblivious. > > So why the spikes? Well, TCP backs off when it sees the packet losses, a= nd it continues to do so until the queue drains enough to stop losing packe= ts. However, this leaves TCP transmitting at less than the shaped rate, so= the TBF starts filling its bucket with the leftovers. Latency returns to = minimum because the queue is empty. TCP then gradually grows its congestio= n window again to probe the path; different TCPs do this in different patte= rns, but it'll generally take much more than one RTT at this bandwidth. Wi= ndows, I believe, increases cwnd by one packet per RTT (ie. is Reno-like). > > So by the time TCP gets back up to 100Mbps, the TBF has stored quite a fe= w spare tokens in its bucket. These now start to be spent, while TCP conti= nues probing *past* 100Mbps, still not seeing the true limit. By the time = the bucket empties, TCP is transmitting considerably faster than 100Mbps, s= uch that the *average* throughput since the last loss is *exactly* 100Mbps = - this last is what TBF is designed to enforce; it technically doesn't star= t with a full bucket but an empty one, but the bucket usually fills up befo= re the user gets around to measuring it. > > So then the TBF runs out of spare tokens and slams on the brakes, the que= ue rapidly fills and overflows, packets are lost, TCP reels, retransmits, a= nd backs off again. Rinse, repeat. > > > - Would you enable sqm on this connection? By doing so I miss out on th= e higher rate for the first few seconds. > > Yes, I would absolutely use SQM here. It'll both iron out those latency = spikes and reduce packet loss, and what's more it'll prevent congestion-rel= ated latency and loss from affecting any but the provoking flow(s). > > IMHO, the benefits of PowerBoost are illusory. When you've got 100Mbps s= teady-state, tripling that for a short period is simply not perceptible in = most applications. Even Web browsing, which typically involves transfers s= maller than the size of the bucket, is limited by latency not bandwidth, on= ce you get above a couple of Mbps. For a real performance benefit - for ex= ample, speeding up large software updates - bandwidth increases need to be = available for minutes, not seconds, so that a gigabyte or more can be trans= ferred at the higher speed. > > > What are the actual downsides of not enabling sqm in this case? > > > Those latency spikes would be seen by latency-sensitive applications as j= itter, which is one of the most insidious problems to cope with in a realti= me interactive system. They coincide with momentary spikes in packet loss = (which unfortunately is not represented in dslreports' graphs) which are al= so difficult to cope with. > > That means your VoIP or videoconference session will glitch and drop out = periodically, unless it has deliberately increased its own latency with int= ernal buffering and redundant transmissions to compensate, if a bulk transf= er is started up by some background application (Steam, Windows Update) or = by someone else in your house (visiting niece bulk-uploads an SD card full = of holiday photos to Instagram, wife cues up Netflix for the evening). > > That also means your online game session, under similar circumstances, wi= ll occasionally fail to show you an enemy move in time for you to react to = it, or even delay or fail to register your own actions because the packets = notifying the server of them were queued and/or lost. Sure, 125ms is a far= cry from the multiple seconds we often see, but it's still problematic for= gamers - it corresponds to just 8Hz, when they're running their monitors a= t 144Hz and their mice at 1000Hz. > > - Jonathan Morton > > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat --=20 Dave T=C3=A4ht CEO, TekLibre, LLC http://www.teklibre.com Tel: 1-669-226-2619