From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mailgw1.uni-kl.de (mailgw1.uni-kl.de [IPv6:2001:638:208:120::220]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 13D543B29E for ; Sun, 15 Aug 2021 09:39:38 -0400 (EDT) Received: from sushi.unix-ag.uni-kl.de (sushi.unix-ag.uni-kl.de [IPv6:2001:638:208:ef34:0:ff:fe00:65]) by mailgw1.uni-kl.de (8.14.4/8.14.4/Debian-8+deb8u2) with ESMTP id 17FDdNcC089000 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 15 Aug 2021 15:39:23 +0200 Received: from sushi.unix-ag.uni-kl.de (ip6-localhost [IPv6:::1]) by sushi.unix-ag.uni-kl.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id 17FDdMA9018804 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 15 Aug 2021 15:39:22 +0200 Received: (from auerswal@localhost) by sushi.unix-ag.uni-kl.de (8.14.4/8.14.4/Submit) id 17FDdMxW018803; Sun, 15 Aug 2021 15:39:22 +0200 Date: Sun, 15 Aug 2021 15:39:22 +0200 From: Erik Auerswald To: bloat@lists.bufferbloat.net Cc: draft-cpaasch-ippm-responsiveness@ietf.org, ippm@ietf.org Message-ID: <20210815133922.GA18118@unix-ag.uni-kl.de> References: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="opJtzjQTFsWo+cga" Content-Disposition: inline In-Reply-To: Author: Erik Auerswald User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, hits=-1, tests=ALL_TRUSTED=-1 X-Spam-Score: (-1) X-Spam-Flag: NO Subject: Re: [Bloat] Fwd: New Version Notification for draft-cpaasch-ippm-responsiveness-00.txt X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 15 Aug 2021 13:39:39 -0000 --opJtzjQTFsWo+cga Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi, I'd like to thank you for working on a nice I-D describing an interesting and IMHO useful network measurement metric. Since feedback was asked for, I'd like to try and provide constructive feedback. In general, I like the idea of "Round-trips per Minute" (RPM) as a metric used to characterize (one aspect of) a network. I do think that introducing this would improve the status quo. Since this RPM definition comprises a specific way of adding load to the network and measuring a complex metric, I think it is useful to "standardize" it. I do not think RPM can replace all other metrics. This is, in a way, mentioned in the introduction, where it is suggested to add RPM to existing measurement platforms. As such I just want to point this out more explicitely, but do not intend to diminish the RPM idea by this. In short, I'd say it's complicated. Bandwidth matters for bulk data transfer, e.g., downloading a huge update required for playing a multiplayer game online. Minimum latency matters for the feasibility of interactive applications, e.g., controlling a toy car in your room vs. a robotic arm on the ISS from Earth vs. orbital insertion around Mars from Earth. For a more mundane use case consider a voice conference. (A good decade ago I experienced a voice conferencing system running over IP that introduced over one second of (minimum) latency and therefore was awkward to use.) Expressing 'bufferbloat as a measure of "Round-trips per Minute" (RPM)' exhibits (at least) two problems: 1. A high RPM value is associated with little bufferbloat problems. 2. A low RPM value may be caused by high minimum delay instead of bufferbloat. I think that RPM (i.e., under working conditions) measures a network's usefulness for interactive applications, but not necessarily bufferbloat. I do think that RPM is in itself more generally useful than minimum latency or bandwidth. A combination of low minimum latency with low RPM value strongly hints at bufferbloat. Other combinations are less easily characterized. Bufferbloat can still lie in hiding, e.g., when a link with bufferbloat is not yet the bottleneck, or if the communications end-points are not yet able to saturate the network inbetween. Thus high bandwidth can result in high RPM values despite (hidden) bufferbloat. The "Measuring is Hard" section mentions additional complications. All in all, I do think that "measuring bufferbloat" and "measuring RPM" should not be used synonymously. The I-D title clearly shows this: RPM is measuring "Responsiveness under Working Conditions" which may be affected by bufferbloat, among other potential factors, but is not in itself bufferbloat. Under the assumption that only a single value (performance score) is considered, I do think that RPM is more generally useful than bandwidth or idle latency. On a meta-level, I think that the word "bufferbloat" is not used according to a single self-consistent definition in the I-D. Additionally, I think that the I-D should reference DNS, HTTP/2, and TLS 1.3, since these protocols are required for implementing the RPM measurement. The same for JSON, I think. Possibly URL. Using "rpm.example" instead of "example.apple.com" would result in shorter lines for the example JSON. "host123.cdn.example" instead of "hostname123.cdnprovider.com" might be a more appropriate example DNS name. Adding an informative reference to RFC 2606 / BCP 32 might raise awareness of the existence of a BCP on example DNS names. Please find both a unified diff against the text rendering of the I-D, and a word diff produced from the unified diff, attached to this email in order to suggest editorial changes that are intended to improve the reading experience. They are intended for reading and (possibly partial) manual application, since the text rendering of an I-D is usually not the preferred form of editing it. Thanks, Erik -- Always use the right tool for the job. -- Rob Pike On Fri, Aug 13, 2021 at 02:41:05PM -0700, Christoph Paasch via Bloat wrote: > I already posted this to the RPM-list, but the audience here on bloat should > be interested as well. > > > This is the specification of Apple's responsiveness/RPM test. We believe that it > would be good for the bufferbloat-effort to have a specification of how to > quantify the extend of bufferbloat from a user's perspective. Our > Internet-draft is a first step in that direction and we hope that it will > kick off some collaboration. > > > Feedback is very welcome! > > > Cheers, > Christoph > > > ----- Forwarded message from internet-drafts@ietf.org ----- > > From: internet-drafts@ietf.org > To: Christoph Paasch , Omer Shapira , Randall Meyer , Stuart Cheshire > > Date: Fri, 13 Aug 2021 09:43:40 -0700 > Subject: New Version Notification for draft-cpaasch-ippm-responsiveness-00.txt > > > A new version of I-D, draft-cpaasch-ippm-responsiveness-00.txt > has been successfully submitted by Christoph Paasch and posted to the > IETF repository. > > Name: draft-cpaasch-ippm-responsiveness > Revision: 00 > Title: Responsiveness under Working Conditions > Document date: 2021-08-13 > Group: Individual Submission > Pages: 12 > URL: https://www.ietf.org/archive/id/draft-cpaasch-ippm-responsiveness-00.txt > Status: https://datatracker.ietf.org/doc/draft-cpaasch-ippm-responsiveness/ > Htmlized: https://datatracker.ietf.org/doc/html/draft-cpaasch-ippm-responsiveness > > > Abstract: > Bufferbloat has been a long-standing problem on the Internet with > more than a decade of work on standardizing technical solutions, > implementations and testing. However, to this date, bufferbloat is > still a very common problem for the end-users. Everyone "knows" that > it is "normal" for a video conference to have problems when somebody > else on the same home-network is watching a 4K movie. > > The reason for this problem is not the lack of technical solutions, > but rather a lack of awareness of the problem-space, and a lack of > tooling to accurately measure the problem. We believe that exposing > the problem of bufferbloat to the end-user by measuring the end- > users' experience at a high level will help to create the necessary > awareness. > > This document is a first attempt at specifying a measurement > methodology to evaluate bufferbloat the way common users are > experiencing it today, using today's most frequently used protocols > and mechanisms to accurately measure the user-experience. We also > provide a way to express the bufferbloat as a measure of "Round-trips > per minute" (RPM) to have a more intuitive way for the users to > understand the notion of bufferbloat. > > > > > The IETF Secretariat > > > > ----- End forwarded message ----- > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat --opJtzjQTFsWo+cga Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="editorial-suggestions-2021-08-15-unified.diff" --- draft-cpaasch-ippm-responsiveness-00.txt 2021-08-15 12:01:01.213813125 +0200 +++ draft-cpaasch-ippm-responsiveness-00-ea.txt 2021-08-15 15:08:08.013416074 +0200 @@ -17,7 +17,7 @@ Bufferbloat has been a long-standing problem on the Internet with more than a decade of work on standardizing technical solutions, - implementations and testing. However, to this date, bufferbloat is + implementations, and testing. However, to this date, bufferbloat is still a very common problem for the end-users. Everyone "knows" that it is "normal" for a video conference to have problems when somebody else on the same home-network is watching a 4K movie. @@ -33,8 +33,8 @@ methodology to evaluate bufferbloat the way common users are experiencing it today, using today's most frequently used protocols and mechanisms to accurately measure the user-experience. We also - provide a way to express the bufferbloat as a measure of "Round-trips - per minute" (RPM) to have a more intuitive way for the users to + provide a way to express bufferbloat as a measure of "Round-trips + per Minute" (RPM) to have a more intuitive way for the users to understand the notion of bufferbloat. Status of This Memo @@ -81,14 +81,14 @@ Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 - 2. Measuring is hard . . . . . . . . . . . . . . . . . . . . . . 3 + 2. Measuring is Hard . . . . . . . . . . . . . . . . . . . . . . 3 3. Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 4. Measuring Responsiveness . . . . . . . . . . . . . . . . . . 5 4.1. Working Conditions . . . . . . . . . . . . . . . . . . . 5 4.1.1. Parallel vs Sequential Uplink and Downlink . . . . . 6 - 4.1.2. From single-flow to multi-flow . . . . . . . . . . . 7 - 4.1.3. Reaching saturation . . . . . . . . . . . . . . . . . 7 - 4.1.4. Final algorithm . . . . . . . . . . . . . . . . . . . 7 + 4.1.2. From Single-flow to Multi-flow . . . . . . . . . . . 7 + 4.1.3. Reaching Saturation . . . . . . . . . . . . . . . . . 7 + 4.1.4. Final Algorithm . . . . . . . . . . . . . . . . . . . 7 4.2. Measuring Responsiveness . . . . . . . . . . . . . . . . 8 4.2.1. Aggregating Round-trips per Minute . . . . . . . . . 9 4.2.2. Statistical Confidence . . . . . . . . . . . . . . . 10 @@ -103,8 +103,8 @@ For many years, bufferbloat has been known as an unfortunately common issue in todays networks [Bufferbloat]. Solutions like FQ-codel - [RFC8289] or PIE [RFC8033] have been standardized and are to some - extend widely implemented. Nevertheless, users still suffer from + [RFC8290] or PIE [RFC8033] have been standardized and are to some + extent widely implemented. Nevertheless, users still suffer from bufferbloat. @@ -129,7 +129,7 @@ bufferbloat problem. We believe that it is necessary to create a standardized way for - measuring the extend of bufferbloat in a network and express it to + measuring the extent of bufferbloat in a network and express it to the user in a user-friendly way. This should help existing measurement tools to add a bufferbloat measurement to their set of metrics. It will also allow to raise the awareness to the problem @@ -144,10 +144,10 @@ classification for those protocols is very common. It is thus very important to use those protocols for the measurements to avoid focusing on use-cases that are not actually affecting the end-user. - Finally, we propose to use "round-trips per minute" as a metric to - express the extend of bufferbloat. + Finally, we propose to use "Round-trips per Minute" as a metric to + express the extent of bufferbloat. -2. Measuring is hard +2. Measuring is Hard There are several challenges around measuring bufferbloat accurately on the Internet. These challenges are due to different factors. @@ -155,7 +155,7 @@ problem space, and the reproducibility of the measurement. It is well-known that transparent TCP proxies are widely deployed on - port 443 and/or port 80, while less common on other ports. Thus, + port 443 and/or port 80, while less commonly on other ports. Thus, choice of the port-number to measure bufferbloat has a significant influence on the result. Other factors are the protocols being used. TCP and UDP traffic may take a largely different path on the Internet @@ -186,17 +186,17 @@ measurement. It seems that it's best to avoid extending the duration of the test beyond what's needed. - The problem space around the bufferbloat is huge. Traditionally, one + The problem space around bufferbloat is huge. Traditionally, one thinks of bufferbloat happening on the routers and switches of the Internet. Thus, simply measuring bufferbloat at the transport layer would be sufficient. However, the networking stacks of the clients and servers can also experience huge amounts of bufferbloat. Data sitting in TCP sockets or waiting in the application to be scheduled for sending causes artificial latency, which affects user-experience - the same way the "traditional" bufferbloat does. + the same way "traditional" bufferbloat does. Finally, measuring bufferbloat requires us to fill the buffers of the - bottleneck and when buffer occupancy is at its peak, the latency + bottleneck, and when buffer occupancy is at its peak, the latency measurement needs to be done. Achieving this in a reliable and reproducible way is not easy. First, one needs to ensure that buffers are actually full for a sustained period of time to allow for @@ -250,15 +250,15 @@ bufferbloat. 4. Finally, in order for this measurement to be user-friendly to a - wide audience it is important that such a measurement finishes - within a short time-frame and short being anything below 20 + wide audience, it is important that such a measurement finishes + within a short time-frame with short being anything below 20 seconds. 4. Measuring Responsiveness The ability to reliably measure the responsiveness under typical working conditions is predicated by the ability to reliably put the - network in a state representative of the said conditions. Once the + network in a state representative of said conditions. Once the network has reached the required state, its responsiveness can be measured. The following explains how the former and the latter are achieved. @@ -270,7 +270,7 @@ experiencing ingress and egress flows that are similar to those when used by humans in the typical day-to-day pattern. - While any network can be put momentarily into working condition by + While any network can be put momentarily into working conditions by the means of a single HTTP transaction, taking measurements requires maintaining such conditions over sufficient time. Thus, measuring the network responsiveness in a consistent way depends on our ability @@ -286,7 +286,7 @@ way to achieve this is by creating multiple large bulk data-transfers in either downstream or upstream direction. Similar to conventional speed-test applications that also create a varying number of streams - to measure throughput. Working-conditions does the same. It also + to measure throughput. Working conditions does the same. It also requires a way to detect when the network is in a persistent working condition, called "saturation". This can be achieved by monitoring the instantaneous goodput over time. When the goodput stops @@ -298,7 +298,7 @@ o Should not waste traffic, since the user may be paying for it o Should finish within a short time-frame to avoid impacting other - users on the same network and/or experience varying conditions + users on the same network and/or experiencing varying conditions 4.1.1. Parallel vs Sequential Uplink and Downlink @@ -308,8 +308,8 @@ upstream) or the routing in the ISPs. Users sending data to an Internet service will fill the bottleneck on the upstream path to the server and thus expose a potential for bufferbloat to happen at this - bottleneck. On the downlink direction any download from an Internet - service will encounter a bottleneck and thus exposes another + bottleneck. In the downlink direction any download from an Internet + service will encounter a bottleneck and thus expose another potential for bufferbloat. Thus, when measuring responsiveness under working conditions it is important to consider both, the upstream and the downstream bufferbloat. This opens the door to measure both @@ -322,13 +322,16 @@ seconds of test per direction, while parallel measurement will allow for 20 seconds of testing in both directions. - However, a number caveats come with measuring in parallel: - Half- - duplex links may not expose uplink and downlink bufferbloat: A half- - duplex link may not allow during parallel measurement to saturate - both the uplink and the downlink direction. Thus, bufferbloat in - either of the directions may not be exposed during parallel - measurement. - Debuggability of the results becomes more obscure: - During parallel measurement it is impossible to differentiate on + However, a number caveats come with measuring in parallel: + + - Half-duplex links may not expose uplink and downlink bufferbloat: + A half-duplex link may not allow to saturate both the uplink + and the downlink direction during parallel measurement. Thus, + bufferbloat in either of the directions may not be exposed during + parallel measurement. + + - Debuggability of the results becomes more obscure: + During parallel measurement it is impossible to differentiate on @@ -338,26 +341,26 @@ Internet-Draft Responsiveness under Working Conditions August 2021 - whether the bufferbloat happens in the uplink or the downlink - direction. + whether the bufferbloat happens in the uplink or the downlink + direction. -4.1.2. From single-flow to multi-flow +4.1.2. From Single-flow to Multi-flow - As described in RFC 6349, a single TCP connection may not be + As described in [RFC6349], a single TCP connection may not be sufficient to saturate a path between a client and a server. On a high-BDP network, traditional TCP window-size constraints of 4MB are often not sufficient to fill the pipe. Additionally, traditional - loss-based TCP congestion control algorithms aggressively reacts to + loss-based TCP congestion control algorithms aggressively react to packet-loss by reducing the congestion window. This reaction will - reduce the queuing in the network, and thus "artificially" make the - bufferbloat appear lesser. + reduce the queuing in the network, and thus "artificially" make + bufferbloat appear less of a problem. - The goal of the measurement is to keep the network as busy as - possible in a sustained and persistent way. Thus, using multiple TCP + The goal is to keep the network as busy as possible in a sustained + and persistent way during the measurement. Thus, using multiple TCP connections is needed for a sustained bufferbloat by gradually adding - TCP flows until saturation is needed. + TCP flows until saturation is reached. -4.1.3. Reaching saturation +4.1.3. Reaching Saturation It is best to detect when saturation has been reached so that the measurement of responsiveness can start with the confidence that the @@ -367,8 +370,8 @@ buffers are completely filled. Thus, this depends highly on the congestion control that is being deployed on the sender-side. Congestion control algorithms like BBR may reach high throughput - without causing bufferbloat. (because the bandwidth-detection portion - of BBR is effectively seeking the bottleneck capacity) + without causing bufferbloat (because the bandwidth-detection portion + of BBR is effectively seeking the bottleneck capacity). It is advised to rather use loss-based congestion controls like Cubic to "reliably" ensure that the buffers are filled. @@ -379,7 +382,7 @@ packet-loss or ECN-marks signaling a congestion or even a full buffer of the bottleneck link. -4.1.4. Final algorithm +4.1.4. Final Algorithm The following is a proposal for an algorithm to reach saturation of a network by using HTTP/2 upload (POST) or download (GET) requests of @@ -404,7 +407,7 @@ throughput will remain stable. In the latter case, this means that saturation has been reached and - more importantly - is stable. - In detail, the steps of the algorithm are the following + In detail, the steps of the algorithm are the following: o Create 4 load-bearing connections @@ -453,7 +456,7 @@ the different stages of a separate network transaction as well as measuring on the load-bearing connections themselves. - Two aspects are being measured with this approach : + Two aspects are being measured with this approach: 1. How the network handles new connections and their different stages (DNS-request, TCP-handshake, TLS-handshake, HTTP/2 @@ -463,19 +466,19 @@ 2. How the network and the client/server networking stack handles the latency on the load-bearing connections themselves. E.g., - Smart queuing techniques on the bottleneck will allow to keep the + smart queuing techniques on the bottleneck will allow to keep the latency within a reasonable limit in the network and buffer- - reducing techniques like TCP_NOTSENT_LOWAT makes sure the client + reducing techniques like TCP_NOTSENT_LOWAT make sure the client and server TCP-stack is not a source of significant latency. To measure the former, we send a DNS-request, establish a TCP- connection on port 443, establish a TLS-context using TLS1.3 and send - an HTTP2 GET request for an object of a single byte large. This + an HTTP/2 GET request for an object the size of a single byte. This measurement will be repeated multiple times for accuracy. Each of these stages allows to collect a single latency measurement that can then be factored into the responsiveness computation. - To measure the latter, on the load-bearing connections (that uses + To measure the latter, on the load-bearing connections (that use HTTP/2) a GET request is multiplexed. This GET request is for a 1-byte object. This allows to measure the end-to-end latency on the connections that are using the network at full speed. @@ -492,10 +495,10 @@ an equal weight to each of these measurements. Finally, the resulting latency needs to be exposed to the users. - Users have been trained to accept metrics that have a notion of "The + Users have been trained to accept metrics that have a notion of "the higher the better". Latency measuring in units of seconds however is "the lower the better". Thus, converting the latency measurement to - a frequency allows using the familiar notion of "The higher the + a frequency allows using the familiar notion of "the higher the better". The term frequency has a very technical connotation. What we are effectively measuring is the number of round-trips from the @@ -513,7 +516,7 @@ which is a wink to the "revolutions per minute" that we are used to in cars. - Thus, our unit of measure is "Round-trip per Minute" (RPM) that + Thus, our unit of measure is "Round-trips per Minute" (RPM) that expresses responsiveness under working conditions. 4.2.2. Statistical Confidence @@ -527,13 +530,13 @@ 5. Protocol Specification By using standard protocols that are most commonly used by end-users, - no new protocol needs to be specified. However, both client and + no new protocol needs to be specified. However, both clients and servers need capabilities to execute this kind of measurement as well - as a standard to flow to provision the client with the necessary + as a standard to follow to provision the client with the necessary information. First, the capabilities of both the client and the server: It is - expected that both hosts support HTTP/2 over TLS 1.3. That the + expected that both hosts support HTTP/2 over TLS 1.3, and that the client is able to send a GET-request and a POST. The server needs the ability to serve both of these HTTP commands. Further, the server endpoint is accessible through a hostname that can be resolved @@ -546,13 +549,13 @@ 1. A config URL/response: This is the configuration file/format used by the client. It's a simple JSON file format that points the client at the various URLs mentioned below. All of the fields - are required except "test_endpoint". If the service-procier can + are required except "test_endpoint". If the service-provider can pin all of the requests for a test run to a specific node in the service (for a particular run), they can specify that node's name in the "test_endpoint" field. It's preferred that pinning of some sort is available. This is to ensure the measurement is against the same paths and not switching hosts during a test run - (ie moving from near POP A to near POP B) Sample content of this + (i.e., moving from near POP A to near POP B). Sample content of this JSON would be: @@ -577,7 +580,7 @@ 3. A "large" URL/response: This needs to serve a status code of 200 and a body size of at least 8GB. The body can be bigger, and - will need to grow as network speeds increases over time. The + will need to grow as network speeds increase over time. The actual body content is irrelevant. The client will probably never completely download the object. @@ -618,16 +621,19 @@ Internet-Draft Responsiveness under Working Conditions August 2021 + [RFC6349] ... + [RFC8033] Pan, R., Natarajan, P., Baker, F., and G. White, "Proportional Integral Controller Enhanced (PIE): A Lightweight Control Scheme to Address the Bufferbloat Problem", RFC 8033, DOI 10.17487/RFC8033, February 2017, . - [RFC8289] Nichols, K., Jacobson, V., McGregor, A., Ed., and J. - Iyengar, Ed., "Controlled Delay Active Queue Management", - RFC 8289, DOI 10.17487/RFC8289, January 2018, - . + [RFC8290] Hoeiland-Joergensen, T., McKenney, P., Taht, D., Ed., and + Gettys, J., "The Flow Queue CoDel Packet Scheduler and + Active Queue Management Algorithm", RFC 8290, + DOI 10.17487/RFC8290, January 2018, + . Authors' Addresses --opJtzjQTFsWo+cga Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="editorial-suggestions-2021-08-15-word.diff" [--- draft-cpaasch-ippm-responsiveness-00.txt-]{+++ draft-cpaasch-ippm-responsiveness-00-ea.txt+} 2021-08-15 [-12:01:01.213813125-] {+15:08:08.013416074+} +0200 @@ -17,7 +17,7 @@ Bufferbloat has been a long-standing problem on the Internet with more than a decade of work on standardizing technical solutions, [-implementations-] {+implementations,+} and testing. However, to this date, bufferbloat is still a very common problem for the end-users. Everyone "knows" that it is "normal" for a video conference to have problems when somebody else on the same home-network is watching a 4K movie. @@ -33,8 +33,8 @@ methodology to evaluate bufferbloat the way common users are experiencing it today, using today's most frequently used protocols and mechanisms to accurately measure the user-experience. We also provide a way to express [-the-] bufferbloat as a measure of "Round-trips per [-minute"-] {+Minute"+} (RPM) to have a more intuitive way for the users to understand the notion of bufferbloat. Status of This Memo @@ -81,14 +81,14 @@ Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Measuring is [-hard-] {+Hard+} . . . . . . . . . . . . . . . . . . . . . . 3 3. Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 4. Measuring Responsiveness . . . . . . . . . . . . . . . . . . 5 4.1. Working Conditions . . . . . . . . . . . . . . . . . . . 5 4.1.1. Parallel vs Sequential Uplink and Downlink . . . . . 6 4.1.2. From [-single-flow-] {+Single-flow+} to [-multi-flow-] {+Multi-flow+} . . . . . . . . . . . 7 4.1.3. Reaching [-saturation-] {+Saturation+} . . . . . . . . . . . . . . . . . 7 4.1.4. Final [-algorithm-] {+Algorithm+} . . . . . . . . . . . . . . . . . . . 7 4.2. Measuring Responsiveness . . . . . . . . . . . . . . . . 8 4.2.1. Aggregating Round-trips per Minute . . . . . . . . . 9 4.2.2. Statistical Confidence . . . . . . . . . . . . . . . 10 @@ -103,8 +103,8 @@ For many years, bufferbloat has been known as an unfortunately common issue in todays networks [Bufferbloat]. Solutions like FQ-codel [-[RFC8289]-] {+[RFC8290]+} or PIE [RFC8033] have been standardized and are to some [-extend-] {+extent+} widely implemented. Nevertheless, users still suffer from bufferbloat. @@ -129,7 +129,7 @@ bufferbloat problem. We believe that it is necessary to create a standardized way for measuring the [-extend-] {+extent+} of bufferbloat in a network and express it to the user in a user-friendly way. This should help existing measurement tools to add a bufferbloat measurement to their set of metrics. It will also allow to raise the awareness to the problem @@ -144,10 +144,10 @@ classification for those protocols is very common. It is thus very important to use those protocols for the measurements to avoid focusing on use-cases that are not actually affecting the end-user. Finally, we propose to use [-"round-trips-] {+"Round-trips+} per [-minute"-] {+Minute"+} as a metric to express the [-extend-] {+extent+} of bufferbloat. 2. Measuring is [-hard-] {+Hard+} There are several challenges around measuring bufferbloat accurately on the Internet. These challenges are due to different factors. @@ -155,7 +155,7 @@ problem space, and the reproducibility of the measurement. It is well-known that transparent TCP proxies are widely deployed on port 443 and/or port 80, while less [-common-] {+commonly+} on other ports. Thus, choice of the port-number to measure bufferbloat has a significant influence on the result. Other factors are the protocols being used. TCP and UDP traffic may take a largely different path on the Internet @@ -186,17 +186,17 @@ measurement. It seems that it's best to avoid extending the duration of the test beyond what's needed. The problem space around [-the-] bufferbloat is huge. Traditionally, one thinks of bufferbloat happening on the routers and switches of the Internet. Thus, simply measuring bufferbloat at the transport layer would be sufficient. However, the networking stacks of the clients and servers can also experience huge amounts of bufferbloat. Data sitting in TCP sockets or waiting in the application to be scheduled for sending causes artificial latency, which affects user-experience the same way [-the-] "traditional" bufferbloat does. Finally, measuring bufferbloat requires us to fill the buffers of the [-bottleneck-] {+bottleneck,+} and when buffer occupancy is at its peak, the latency measurement needs to be done. Achieving this in a reliable and reproducible way is not easy. First, one needs to ensure that buffers are actually full for a sustained period of time to allow for @@ -250,15 +250,15 @@ bufferbloat. 4. Finally, in order for this measurement to be user-friendly to a wide [-audience-] {+audience,+} it is important that such a measurement finishes within a short time-frame [-and-] {+with+} short being anything below 20 seconds. 4. Measuring Responsiveness The ability to reliably measure the responsiveness under typical working conditions is predicated by the ability to reliably put the network in a state representative of [-the-] said conditions. Once the network has reached the required state, its responsiveness can be measured. The following explains how the former and the latter are achieved. @@ -270,7 +270,7 @@ experiencing ingress and egress flows that are similar to those when used by humans in the typical day-to-day pattern. While any network can be put momentarily into working [-condition-] {+conditions+} by the means of a single HTTP transaction, taking measurements requires maintaining such conditions over sufficient time. Thus, measuring the network responsiveness in a consistent way depends on our ability @@ -286,7 +286,7 @@ way to achieve this is by creating multiple large bulk data-transfers in either downstream or upstream direction. Similar to conventional speed-test applications that also create a varying number of streams to measure throughput. [-Working-conditions-] {+Working conditions+} does the same. It also requires a way to detect when the network is in a persistent working condition, called "saturation". This can be achieved by monitoring the instantaneous goodput over time. When the goodput stops @@ -298,7 +298,7 @@ o Should not waste traffic, since the user may be paying for it o Should finish within a short time-frame to avoid impacting other users on the same network and/or [-experience-] {+experiencing+} varying conditions 4.1.1. Parallel vs Sequential Uplink and Downlink @@ -308,8 +308,8 @@ upstream) or the routing in the ISPs. Users sending data to an Internet service will fill the bottleneck on the upstream path to the server and thus expose a potential for bufferbloat to happen at this bottleneck. [-On-] {+In+} the downlink direction any download from an Internet service will encounter a bottleneck and thus [-exposes-] {+expose+} another potential for bufferbloat. Thus, when measuring responsiveness under working conditions it is important to consider both, the upstream and the downstream bufferbloat. This opens the door to measure both @@ -322,13 +322,16 @@ seconds of test per direction, while parallel measurement will allow for 20 seconds of testing in both directions. However, a number caveats come with measuring in parallel: - [-Half- duplex-] {+Half-duplex+} links may not expose uplink and downlink bufferbloat: A [-half- duplex-] {+half-duplex+} link may not allow [-during parallel measurement-] to saturate both the uplink and the downlink [-direction.-] {+direction during parallel measurement.+} Thus, bufferbloat in either of the directions may not be exposed during parallel measurement. - Debuggability of the results becomes more obscure: During parallel measurement it is impossible to differentiate on @@ -338,26 +341,26 @@ Internet-Draft Responsiveness under Working Conditions August 2021 whether the bufferbloat happens in the uplink or the downlink direction. 4.1.2. From [-single-flow-] {+Single-flow+} to [-multi-flow-] {+Multi-flow+} As described in [-RFC 6349,-] {+[RFC6349],+} a single TCP connection may not be sufficient to saturate a path between a client and a server. On a high-BDP network, traditional TCP window-size constraints of 4MB are often not sufficient to fill the pipe. Additionally, traditional loss-based TCP congestion control algorithms aggressively [-reacts-] {+react+} to packet-loss by reducing the congestion window. This reaction will reduce the queuing in the network, and thus "artificially" make [-the-] bufferbloat appear [-lesser.-] {+less of a problem.+} The goal [-of the measurement-] is to keep the network as busy as possible in a sustained and persistent [-way.-] {+way during the measurement.+} Thus, using multiple TCP connections is needed for a sustained bufferbloat by gradually adding TCP flows until saturation is [-needed.-] {+reached.+} 4.1.3. Reaching [-saturation-] {+Saturation+} It is best to detect when saturation has been reached so that the measurement of responsiveness can start with the confidence that the @@ -367,8 +370,8 @@ buffers are completely filled. Thus, this depends highly on the congestion control that is being deployed on the sender-side. Congestion control algorithms like BBR may reach high throughput without causing [-bufferbloat.-] {+bufferbloat+} (because the bandwidth-detection portion of BBR is effectively seeking the bottleneck [-capacity)-] {+capacity).+} It is advised to rather use loss-based congestion controls like Cubic to "reliably" ensure that the buffers are filled. @@ -379,7 +382,7 @@ packet-loss or ECN-marks signaling a congestion or even a full buffer of the bottleneck link. 4.1.4. Final [-algorithm-] {+Algorithm+} The following is a proposal for an algorithm to reach saturation of a network by using HTTP/2 upload (POST) or download (GET) requests of @@ -404,7 +407,7 @@ throughput will remain stable. In the latter case, this means that saturation has been reached and - more importantly - is stable. In detail, the steps of the algorithm are the [-following-] {+following:+} o Create 4 load-bearing connections @@ -453,7 +456,7 @@ the different stages of a separate network transaction as well as measuring on the load-bearing connections themselves. Two aspects are being measured with this [-approach :-] {+approach:+} 1. How the network handles new connections and their different stages (DNS-request, TCP-handshake, TLS-handshake, HTTP/2 @@ -463,19 +466,19 @@ 2. How the network and the client/server networking stack handles the latency on the load-bearing connections themselves. E.g., [-Smart-] {+smart+} queuing techniques on the bottleneck will allow to keep the latency within a reasonable limit in the network and buffer- reducing techniques like TCP_NOTSENT_LOWAT [-makes-] {+make+} sure the client and server TCP-stack is not a source of significant latency. To measure the former, we send a DNS-request, establish a TCP- connection on port 443, establish a TLS-context using TLS1.3 and send an [-HTTP2-] {+HTTP/2+} GET request for an object {+the size+} of a single [-byte large.-] {+byte.+} This measurement will be repeated multiple times for accuracy. Each of these stages allows to collect a single latency measurement that can then be factored into the responsiveness computation. To measure the latter, on the load-bearing connections (that [-uses-] {+use+} HTTP/2) a GET request is multiplexed. This GET request is for a 1-byte object. This allows to measure the end-to-end latency on the connections that are using the network at full speed. @@ -492,10 +495,10 @@ an equal weight to each of these measurements. Finally, the resulting latency needs to be exposed to the users. Users have been trained to accept metrics that have a notion of [-"The-] {+"the+} higher the better". Latency measuring in units of seconds however is "the lower the better". Thus, converting the latency measurement to a frequency allows using the familiar notion of [-"The-] {+"the+} higher the better". The term frequency has a very technical connotation. What we are effectively measuring is the number of round-trips from the @@ -513,7 +516,7 @@ which is a wink to the "revolutions per minute" that we are used to in cars. Thus, our unit of measure is [-"Round-trip-] {+"Round-trips+} per Minute" (RPM) that expresses responsiveness under working conditions. 4.2.2. Statistical Confidence @@ -527,13 +530,13 @@ 5. Protocol Specification By using standard protocols that are most commonly used by end-users, no new protocol needs to be specified. However, both [-client-] {+clients+} and servers need capabilities to execute this kind of measurement as well as a standard to [-flow-] {+follow+} to provision the client with the necessary information. First, the capabilities of both the client and the server: It is expected that both hosts support HTTP/2 over TLS [-1.3. That-] {+1.3, and that+} the client is able to send a GET-request and a POST. The server needs the ability to serve both of these HTTP commands. Further, the server endpoint is accessible through a hostname that can be resolved @@ -546,13 +549,13 @@ 1. A config URL/response: This is the configuration file/format used by the client. It's a simple JSON file format that points the client at the various URLs mentioned below. All of the fields are required except "test_endpoint". If the [-service-procier-] {+service-provider+} can pin all of the requests for a test run to a specific node in the service (for a particular run), they can specify that node's name in the "test_endpoint" field. It's preferred that pinning of some sort is available. This is to ensure the measurement is against the same paths and not switching hosts during a test run [-(ie-] {+(i.e.,+} moving from near POP A to near POP [-B)-] {+B).+} Sample content of this JSON would be: @@ -577,7 +580,7 @@ 3. A "large" URL/response: This needs to serve a status code of 200 and a body size of at least 8GB. The body can be bigger, and will need to grow as network speeds [-increases-] {+increase+} over time. The actual body content is irrelevant. The client will probably never completely download the object. @@ -618,16 +621,19 @@ Internet-Draft Responsiveness under Working Conditions August 2021 {+[RFC6349] ...+} [RFC8033] Pan, R., Natarajan, P., Baker, F., and G. White, "Proportional Integral Controller Enhanced (PIE): A Lightweight Control Scheme to Address the Bufferbloat Problem", RFC 8033, DOI 10.17487/RFC8033, February 2017, . [-[RFC8289] Nichols, K., Jacobson, V., McGregor, A.,-] {+[RFC8290] Hoeiland-Joergensen, T., McKenney, P., Taht, D.,+} Ed., and [-J. Iyengar, Ed., "Controlled Delay-] {+Gettys, J., "The Flow Queue CoDel Packet Scheduler and+} Active Queue [-Management",-] {+Management Algorithm",+} RFC [-8289,-] {+8290,+} DOI [-10.17487/RFC8289,-] {+10.17487/RFC8290,+} January 2018, [-.-] {+.+} Authors' Addresses --opJtzjQTFsWo+cga--