--- draft-cpaasch-ippm-responsiveness-00.txt 2021-08-15 12:01:01.213813125 +0200 +++ draft-cpaasch-ippm-responsiveness-00-ea.txt 2021-08-15 15:08:08.013416074 +0200 @@ -17,7 +17,7 @@ Bufferbloat has been a long-standing problem on the Internet with more than a decade of work on standardizing technical solutions, - implementations and testing. However, to this date, bufferbloat is + implementations, and testing. However, to this date, bufferbloat is still a very common problem for the end-users. Everyone "knows" that it is "normal" for a video conference to have problems when somebody else on the same home-network is watching a 4K movie. @@ -33,8 +33,8 @@ methodology to evaluate bufferbloat the way common users are experiencing it today, using today's most frequently used protocols and mechanisms to accurately measure the user-experience. We also - provide a way to express the bufferbloat as a measure of "Round-trips - per minute" (RPM) to have a more intuitive way for the users to + provide a way to express bufferbloat as a measure of "Round-trips + per Minute" (RPM) to have a more intuitive way for the users to understand the notion of bufferbloat. Status of This Memo @@ -81,14 +81,14 @@ Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 - 2. Measuring is hard . . . . . . . . . . . . . . . . . . . . . . 3 + 2. Measuring is Hard . . . . . . . . . . . . . . . . . . . . . . 3 3. Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 4. Measuring Responsiveness . . . . . . . . . . . . . . . . . . 5 4.1. Working Conditions . . . . . . . . . . . . . . . . . . . 5 4.1.1. Parallel vs Sequential Uplink and Downlink . . . . . 6 - 4.1.2. From single-flow to multi-flow . . . . . . . . . . . 7 - 4.1.3. Reaching saturation . . . . . . . . . . . . . . . . . 7 - 4.1.4. Final algorithm . . . . . . . . . . . . . . . . . . . 7 + 4.1.2. From Single-flow to Multi-flow . . . . . . . . . . . 7 + 4.1.3. Reaching Saturation . . . . . . . . . . . . . . . . . 7 + 4.1.4. Final Algorithm . . . . . . . . . . . . . . . . . . . 7 4.2. Measuring Responsiveness . . . . . . . . . . . . . . . . 8 4.2.1. Aggregating Round-trips per Minute . . . . . . . . . 9 4.2.2. Statistical Confidence . . . . . . . . . . . . . . . 10 @@ -103,8 +103,8 @@ For many years, bufferbloat has been known as an unfortunately common issue in todays networks [Bufferbloat]. Solutions like FQ-codel - [RFC8289] or PIE [RFC8033] have been standardized and are to some - extend widely implemented. Nevertheless, users still suffer from + [RFC8290] or PIE [RFC8033] have been standardized and are to some + extent widely implemented. Nevertheless, users still suffer from bufferbloat. @@ -129,7 +129,7 @@ bufferbloat problem. We believe that it is necessary to create a standardized way for - measuring the extend of bufferbloat in a network and express it to + measuring the extent of bufferbloat in a network and express it to the user in a user-friendly way. This should help existing measurement tools to add a bufferbloat measurement to their set of metrics. It will also allow to raise the awareness to the problem @@ -144,10 +144,10 @@ classification for those protocols is very common. It is thus very important to use those protocols for the measurements to avoid focusing on use-cases that are not actually affecting the end-user. - Finally, we propose to use "round-trips per minute" as a metric to - express the extend of bufferbloat. + Finally, we propose to use "Round-trips per Minute" as a metric to + express the extent of bufferbloat. -2. Measuring is hard +2. Measuring is Hard There are several challenges around measuring bufferbloat accurately on the Internet. These challenges are due to different factors. @@ -155,7 +155,7 @@ problem space, and the reproducibility of the measurement. It is well-known that transparent TCP proxies are widely deployed on - port 443 and/or port 80, while less common on other ports. Thus, + port 443 and/or port 80, while less commonly on other ports. Thus, choice of the port-number to measure bufferbloat has a significant influence on the result. Other factors are the protocols being used. TCP and UDP traffic may take a largely different path on the Internet @@ -186,17 +186,17 @@ measurement. It seems that it's best to avoid extending the duration of the test beyond what's needed. - The problem space around the bufferbloat is huge. Traditionally, one + The problem space around bufferbloat is huge. Traditionally, one thinks of bufferbloat happening on the routers and switches of the Internet. Thus, simply measuring bufferbloat at the transport layer would be sufficient. However, the networking stacks of the clients and servers can also experience huge amounts of bufferbloat. Data sitting in TCP sockets or waiting in the application to be scheduled for sending causes artificial latency, which affects user-experience - the same way the "traditional" bufferbloat does. + the same way "traditional" bufferbloat does. Finally, measuring bufferbloat requires us to fill the buffers of the - bottleneck and when buffer occupancy is at its peak, the latency + bottleneck, and when buffer occupancy is at its peak, the latency measurement needs to be done. Achieving this in a reliable and reproducible way is not easy. First, one needs to ensure that buffers are actually full for a sustained period of time to allow for @@ -250,15 +250,15 @@ bufferbloat. 4. Finally, in order for this measurement to be user-friendly to a - wide audience it is important that such a measurement finishes - within a short time-frame and short being anything below 20 + wide audience, it is important that such a measurement finishes + within a short time-frame with short being anything below 20 seconds. 4. Measuring Responsiveness The ability to reliably measure the responsiveness under typical working conditions is predicated by the ability to reliably put the - network in a state representative of the said conditions. Once the + network in a state representative of said conditions. Once the network has reached the required state, its responsiveness can be measured. The following explains how the former and the latter are achieved. @@ -270,7 +270,7 @@ experiencing ingress and egress flows that are similar to those when used by humans in the typical day-to-day pattern. - While any network can be put momentarily into working condition by + While any network can be put momentarily into working conditions by the means of a single HTTP transaction, taking measurements requires maintaining such conditions over sufficient time. Thus, measuring the network responsiveness in a consistent way depends on our ability @@ -286,7 +286,7 @@ way to achieve this is by creating multiple large bulk data-transfers in either downstream or upstream direction. Similar to conventional speed-test applications that also create a varying number of streams - to measure throughput. Working-conditions does the same. It also + to measure throughput. Working conditions does the same. It also requires a way to detect when the network is in a persistent working condition, called "saturation". This can be achieved by monitoring the instantaneous goodput over time. When the goodput stops @@ -298,7 +298,7 @@ o Should not waste traffic, since the user may be paying for it o Should finish within a short time-frame to avoid impacting other - users on the same network and/or experience varying conditions + users on the same network and/or experiencing varying conditions 4.1.1. Parallel vs Sequential Uplink and Downlink @@ -308,8 +308,8 @@ upstream) or the routing in the ISPs. Users sending data to an Internet service will fill the bottleneck on the upstream path to the server and thus expose a potential for bufferbloat to happen at this - bottleneck. On the downlink direction any download from an Internet - service will encounter a bottleneck and thus exposes another + bottleneck. In the downlink direction any download from an Internet + service will encounter a bottleneck and thus expose another potential for bufferbloat. Thus, when measuring responsiveness under working conditions it is important to consider both, the upstream and the downstream bufferbloat. This opens the door to measure both @@ -322,13 +322,16 @@ seconds of test per direction, while parallel measurement will allow for 20 seconds of testing in both directions. - However, a number caveats come with measuring in parallel: - Half- - duplex links may not expose uplink and downlink bufferbloat: A half- - duplex link may not allow during parallel measurement to saturate - both the uplink and the downlink direction. Thus, bufferbloat in - either of the directions may not be exposed during parallel - measurement. - Debuggability of the results becomes more obscure: - During parallel measurement it is impossible to differentiate on + However, a number caveats come with measuring in parallel: + + - Half-duplex links may not expose uplink and downlink bufferbloat: + A half-duplex link may not allow to saturate both the uplink + and the downlink direction during parallel measurement. Thus, + bufferbloat in either of the directions may not be exposed during + parallel measurement. + + - Debuggability of the results becomes more obscure: + During parallel measurement it is impossible to differentiate on @@ -338,26 +341,26 @@ Internet-Draft Responsiveness under Working Conditions August 2021 - whether the bufferbloat happens in the uplink or the downlink - direction. + whether the bufferbloat happens in the uplink or the downlink + direction. -4.1.2. From single-flow to multi-flow +4.1.2. From Single-flow to Multi-flow - As described in RFC 6349, a single TCP connection may not be + As described in [RFC6349], a single TCP connection may not be sufficient to saturate a path between a client and a server. On a high-BDP network, traditional TCP window-size constraints of 4MB are often not sufficient to fill the pipe. Additionally, traditional - loss-based TCP congestion control algorithms aggressively reacts to + loss-based TCP congestion control algorithms aggressively react to packet-loss by reducing the congestion window. This reaction will - reduce the queuing in the network, and thus "artificially" make the - bufferbloat appear lesser. + reduce the queuing in the network, and thus "artificially" make + bufferbloat appear less of a problem. - The goal of the measurement is to keep the network as busy as - possible in a sustained and persistent way. Thus, using multiple TCP + The goal is to keep the network as busy as possible in a sustained + and persistent way during the measurement. Thus, using multiple TCP connections is needed for a sustained bufferbloat by gradually adding - TCP flows until saturation is needed. + TCP flows until saturation is reached. -4.1.3. Reaching saturation +4.1.3. Reaching Saturation It is best to detect when saturation has been reached so that the measurement of responsiveness can start with the confidence that the @@ -367,8 +370,8 @@ buffers are completely filled. Thus, this depends highly on the congestion control that is being deployed on the sender-side. Congestion control algorithms like BBR may reach high throughput - without causing bufferbloat. (because the bandwidth-detection portion - of BBR is effectively seeking the bottleneck capacity) + without causing bufferbloat (because the bandwidth-detection portion + of BBR is effectively seeking the bottleneck capacity). It is advised to rather use loss-based congestion controls like Cubic to "reliably" ensure that the buffers are filled. @@ -379,7 +382,7 @@ packet-loss or ECN-marks signaling a congestion or even a full buffer of the bottleneck link. -4.1.4. Final algorithm +4.1.4. Final Algorithm The following is a proposal for an algorithm to reach saturation of a network by using HTTP/2 upload (POST) or download (GET) requests of @@ -404,7 +407,7 @@ throughput will remain stable. In the latter case, this means that saturation has been reached and - more importantly - is stable. - In detail, the steps of the algorithm are the following + In detail, the steps of the algorithm are the following: o Create 4 load-bearing connections @@ -453,7 +456,7 @@ the different stages of a separate network transaction as well as measuring on the load-bearing connections themselves. - Two aspects are being measured with this approach : + Two aspects are being measured with this approach: 1. How the network handles new connections and their different stages (DNS-request, TCP-handshake, TLS-handshake, HTTP/2 @@ -463,19 +466,19 @@ 2. How the network and the client/server networking stack handles the latency on the load-bearing connections themselves. E.g., - Smart queuing techniques on the bottleneck will allow to keep the + smart queuing techniques on the bottleneck will allow to keep the latency within a reasonable limit in the network and buffer- - reducing techniques like TCP_NOTSENT_LOWAT makes sure the client + reducing techniques like TCP_NOTSENT_LOWAT make sure the client and server TCP-stack is not a source of significant latency. To measure the former, we send a DNS-request, establish a TCP- connection on port 443, establish a TLS-context using TLS1.3 and send - an HTTP2 GET request for an object of a single byte large. This + an HTTP/2 GET request for an object the size of a single byte. This measurement will be repeated multiple times for accuracy. Each of these stages allows to collect a single latency measurement that can then be factored into the responsiveness computation. - To measure the latter, on the load-bearing connections (that uses + To measure the latter, on the load-bearing connections (that use HTTP/2) a GET request is multiplexed. This GET request is for a 1-byte object. This allows to measure the end-to-end latency on the connections that are using the network at full speed. @@ -492,10 +495,10 @@ an equal weight to each of these measurements. Finally, the resulting latency needs to be exposed to the users. - Users have been trained to accept metrics that have a notion of "The + Users have been trained to accept metrics that have a notion of "the higher the better". Latency measuring in units of seconds however is "the lower the better". Thus, converting the latency measurement to - a frequency allows using the familiar notion of "The higher the + a frequency allows using the familiar notion of "the higher the better". The term frequency has a very technical connotation. What we are effectively measuring is the number of round-trips from the @@ -513,7 +516,7 @@ which is a wink to the "revolutions per minute" that we are used to in cars. - Thus, our unit of measure is "Round-trip per Minute" (RPM) that + Thus, our unit of measure is "Round-trips per Minute" (RPM) that expresses responsiveness under working conditions. 4.2.2. Statistical Confidence @@ -527,13 +530,13 @@ 5. Protocol Specification By using standard protocols that are most commonly used by end-users, - no new protocol needs to be specified. However, both client and + no new protocol needs to be specified. However, both clients and servers need capabilities to execute this kind of measurement as well - as a standard to flow to provision the client with the necessary + as a standard to follow to provision the client with the necessary information. First, the capabilities of both the client and the server: It is - expected that both hosts support HTTP/2 over TLS 1.3. That the + expected that both hosts support HTTP/2 over TLS 1.3, and that the client is able to send a GET-request and a POST. The server needs the ability to serve both of these HTTP commands. Further, the server endpoint is accessible through a hostname that can be resolved @@ -546,13 +549,13 @@ 1. A config URL/response: This is the configuration file/format used by the client. It's a simple JSON file format that points the client at the various URLs mentioned below. All of the fields - are required except "test_endpoint". If the service-procier can + are required except "test_endpoint". If the service-provider can pin all of the requests for a test run to a specific node in the service (for a particular run), they can specify that node's name in the "test_endpoint" field. It's preferred that pinning of some sort is available. This is to ensure the measurement is against the same paths and not switching hosts during a test run - (ie moving from near POP A to near POP B) Sample content of this + (i.e., moving from near POP A to near POP B). Sample content of this JSON would be: @@ -577,7 +580,7 @@ 3. A "large" URL/response: This needs to serve a status code of 200 and a body size of at least 8GB. The body can be bigger, and - will need to grow as network speeds increases over time. The + will need to grow as network speeds increase over time. The actual body content is irrelevant. The client will probably never completely download the object. @@ -618,16 +621,19 @@ Internet-Draft Responsiveness under Working Conditions August 2021 + [RFC6349] ... + [RFC8033] Pan, R., Natarajan, P., Baker, F., and G. White, "Proportional Integral Controller Enhanced (PIE): A Lightweight Control Scheme to Address the Bufferbloat Problem", RFC 8033, DOI 10.17487/RFC8033, February 2017, . - [RFC8289] Nichols, K., Jacobson, V., McGregor, A., Ed., and J. - Iyengar, Ed., "Controlled Delay Active Queue Management", - RFC 8289, DOI 10.17487/RFC8289, January 2018, - . + [RFC8290] Hoeiland-Joergensen, T., McKenney, P., Taht, D., Ed., and + Gettys, J., "The Flow Queue CoDel Packet Scheduler and + Active Queue Management Algorithm", RFC 8290, + DOI 10.17487/RFC8290, January 2018, + . Authors' Addresses