From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qk1-x72f.google.com (mail-qk1-x72f.google.com [IPv6:2607:f8b0:4864:20::72f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 58AAE3CB37 for ; Sat, 9 Jan 2021 18:01:34 -0500 (EST) Received: by mail-qk1-x72f.google.com with SMTP id 22so11732248qkf.9 for ; Sat, 09 Jan 2021 15:01:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:subject:reply-to:to:cc:message-id:date:user-agent:mime-version :content-language; bh=AXiVBTelVh1fhNFj/b07N1qI+QSXTEMDsxMfy+6PEMQ=; b=WNXMdPlh6FSoitsrcslbdxkJ8tdNsiSvmIeencuWke57AM0T1GeVTk95mzQhJQ+gPa mDw4A8dWEOychbqWBFcv6tGAIve6XsW3kl/LVuv4Ct7I2Q162kAAIFW3u4AQZo84LFU1 VDKD4OS+iAXlscazpT58BSKT2AF9mlm/4aNfCntf5hZ0iYq63DKhRovEuZzb1SgzS6+y TQUd77RnlnlXzG/28idBT2RbgeOVfi3tVgNjPOzYFakIlz9RDEOWvhVYuhjZRyiYFGeh e1qPiJSQYZkfO+NENh5o/ZMVAeJkTXaKJK7Y4o0hdbn9PW8fN2zaeNgpbzchJgvYlvNS eXnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:subject:reply-to:to:cc:message-id:date :user-agent:mime-version:content-language; bh=AXiVBTelVh1fhNFj/b07N1qI+QSXTEMDsxMfy+6PEMQ=; b=kYtf185VbB9fILi9UWb2k6kY+70uv6tqf+RQuzxNzbhQvqvOM9lQl2DS/kABOcZrxN 4PxCFOsXZS/kESv2DxnLNuTY+POIfoZKmVQFmbo45irtQdv8ee22y3vSiTF0czG+Yl1h YuwZUFqo4cqODr7KbPDuPzE6V4QT4hzHk8neopVauKXrKcRXLe9RbiNnC+H+irI4xrK+ uLStSVXY/qUqLlsG6cUZYRjp1MjHY5KyhBMIIZL/1lw4dhgF+VXRaaD/wZYVqRjPdgQH XyTS1G7BK7MICAxdK1K4BUI1GpZNopDp+vYCwJfP/lSnX58lry3bMvQGh1PTjCNT0P61 TzRA== X-Gm-Message-State: AOAM532LSLSWoeWcZBQzBMiQWotDhsdCBbWZ+hUPc5OBIzkRl/jGZKM5 x2Z5tLIU/3iG9SbpLKyav68= X-Google-Smtp-Source: ABdhPJwEbeXDNtngVEuyj3Dn0qQh8eQ2jC+M6bRnP0rUDEiEesFig77K0qkQ+hr2WMmJC5+VNT1d2Q== X-Received: by 2002:a37:b08:: with SMTP id 8mr10406674qkl.230.1610233293836; Sat, 09 Jan 2021 15:01:33 -0800 (PST) Received: from [192.168.7.123] (cpe5896306dfa2e-cm5896306dfa2c.cpe.net.cable.rogers.com. [99.240.238.19]) by smtp.gmail.com with ESMTPSA id q37sm6225282qte.10.2021.01.09.15.01.33 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 09 Jan 2021 15:01:33 -0800 (PST) From: David Collier-Brown X-Google-Original-From: David Collier-Brown Reply-To: davecb@spamcop.net To: bloat Cc: dave.collier-brown@indexexchange.com Message-ID: <6d393265-cae1-6108-ed90-58ef53abe46a@rogers.com> Date: Sat, 9 Jan 2021 18:01:32 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.4.0 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="------------6593F0FA7D9D0E337414CDAD" Content-Language: en-US Subject: [Bloat] Rebecca Drucker's talk sounds like it exposes an addressable bloat issue in Ciscos X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Jan 2021 23:01:34 -0000 This is a multi-part message in MIME format. --------------6593F0FA7D9D0E337414CDAD Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit At work, I recently had a database outage due to network saturation and timeouts, which we proposed to address by setting up a QOS policy for the machines in question. However, from the discussion in Ms Drucker's BBR talk, that could lead us to doing /A Bad Thing/ (;-)) Let's start at the beginning, though.  The talk, mentioned before in the list[1], was about the interaction of BBR and large values of buffering, specifically for video traffic.  I attended it, and listened with interest to the questions from the committee. She subsequently gave me a copy of the paper and presentation, which I appreciate: it's very good work. She reported the severity of the effect of large buffers on BBR. I've attached a screenshot, but the list probably won't take it, so I'll describe it. After the first few packets with large buffers, RTT rises, throughput plummets and then throughput stays low for about 200,000 ms. Then it rises to about half the initial throughput for about 50,000 ms as RTT falls, then throughput plummets once more. This pattern repeats throughout the test. Increasing the buffering in the test environment turns perfectly reasonable performance into a real disappointment, even though BBR is trying to estimate /the network’s bandwidth-delay product, BDP, and regulating its //sending rate to maximize throughput while attempting to maintain BDP worth of packets in the //buffer, irrespective of the size of the buffer/. One of the interesting questions was about the token-bucket algorithm used in the router to limit performance. In her paper, she discusses the token bucket filter used by OpenWRT 19.07.1 on a Linksys WRT1900ACS router. Allowing more than the actual bandwidth of the interface as the /burst rate/ can exacerbate the buffering problem, so the listener was concerned that routers "in the wild" might also be contributing to the poor performance by using token-bucket algorithms with "excess burst size" parameters. The very first Cisco manual I found in a Google search explained how to */set/* excess burst size (!) https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/qos_plcshp/configuration/12-4/qos-plcshp-12-4-book.pdf defined excess burst size as /Traffic that falls between the normal burst size and the Excess Burst size/ and specifies it will be sent regardless, /with a probability that increases as the burst size increases./ A little later, it explains that the excess or "extended" burst size /exists so as to avoid tail-drop behavior, and, instead, engage behavior like that of Random Early Detection (RED)./ In order to avoid tail drop, they suggest the "extended burst" be set to twice the burst size, where the burst size by definition is the capacity of the interface, per unit time. So, folks, am I right in thinking that Cisco's recommendation just might be a /terrible/ piece of advice? As a capacity planner, it sounds a lot like they're praying for a conveniently timed lull after every time they let too many bytes through. As a follower of the discussion here, the reference to tail drop and RED sound faintly ... antique. --dave c-b [1. https://www.cs.stonybrook.edu/Rebecca-Drucker-Research-Proficiency-Presentation-Investigating-BBR-Bufferbloat-Problem-DASH-Video ] -- David Collier-Brown, | Always do right. This will gratify System Programmer and Author | some people and astonish the rest davecb@spamcop.net | -- Mark Twain --------------6593F0FA7D9D0E337414CDAD Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit

At work, I recently had a database outage due to network saturation and timeouts, which we proposed to address by setting up a QOS policy for the machines in question. However, from the discussion in Ms Drucker's BBR talk, that could lead us to doing A Bad Thing (;-))


Let's start at the beginning, though.  The talk, mentioned before in the list[1], was about the interaction of BBR and large values of buffering, specifically for video traffic.  I attended it, and listened with interest to the questions from the committee. She subsequently gave me a copy of the paper and presentation, which I appreciate: it's very good work.

She reported the severity of the effect of large buffers on BBR. I've attached a screenshot, but the list probably won't take it, so I'll describe it. After the first few packets with large buffers, RTT rises, throughput plummets and then throughput stays low for about 200,000 ms. Then it rises to about half the initial throughput for about 50,000 ms as RTT falls, then throughput plummets once more. This pattern repeats throughout the test.

Increasing the buffering in the test environment turns perfectly reasonable performance into a real disappointment, even though BBR is trying to estimate the network’s bandwidth-delay product, BDP, and regulating its sending rate to maximize throughput while attempting to maintain BDP worth of packets in the buffer, irrespective of the size of the buffer.

One of the interesting questions was about the token-bucket algorithm used in the router to limit performance. In her paper, she discusses the token bucket filter used by OpenWRT 19.07.1 on a Linksys WRT1900ACS router. Allowing more than the actual bandwidth of the interface as the burst rate can exacerbate the buffering problem, so the listener was concerned that routers "in the wild" might also be contributing to the poor performance by using token-bucket algorithms with "excess burst size" parameters.

The very first Cisco manual I found in a Google search explained how to set excess burst size (!)

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/qos_plcshp/configuration/12-4/qos-plcshp-12-4-book.pdf defined excess burst size as Traffic that falls between the normal burst size and the Excess Burst size and specifies it will be sent regardless, with a probability that increases as the burst size increases.

A little later, it explains that the excess or "extended" burst size exists so as to avoid tail-drop behavior, and, instead,
engage behavior like that of Random Early Detection (RED).

In order to avoid tail drop, they suggest the "extended burst" be set to twice the burst size, where the burst size by definition is the capacity of the interface, per unit time.


So, folks, am I right in thinking that Cisco's recommendation just might be a terrible piece of advice? 

As a capacity planner, it sounds a lot like they're praying for a conveniently timed lull after every time they let too many bytes through.

As a follower of the discussion here, the reference to tail drop and RED sound faintly ... antique.

--dave c-b

[1.  https://www.cs.stonybrook.edu/Rebecca-Drucker-Research-Proficiency-Presentation-Investigating-BBR-Bufferbloat-Problem-DASH-Video ] 

-- 
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
davecb@spamcop.net           |                      -- Mark Twain
--------------6593F0FA7D9D0E337414CDAD--