From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ma1-aaemail-dr-lapp02.apple.com (ma1-aaemail-dr-lapp02.apple.com [17.171.2.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 4B0963CB37 for ; Mon, 28 Jun 2021 18:54:07 -0400 (EDT) Received: from pps.filterd (ma1-aaemail-dr-lapp02.apple.com [127.0.0.1]) by ma1-aaemail-dr-lapp02.apple.com (8.16.0.42/8.16.0.42) with SMTP id 15SMmL51042956; Mon, 28 Jun 2021 15:54:06 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=mime-version : content-transfer-encoding : content-type : date : from : to : cc : subject : message-id : references : in-reply-to; s=20180706; bh=DJ96oAqvm6KspdHSMqUE5blKXruE2uT9YABmmPyK7ew=; b=nyK7T9j+PQ26afCqom3wIdJnw/ZQdxj6VtCdNhdy5q4aveuf+uXypNlxvisXT/evi+Gh DzZf9jJajgel6S4eJB0iu6Ve3qQLHhyktzqiyUv6VuDQQdoGb7YRIoMb8bhFrxsNEFXR GXJISb6G1iZ0XewnGMJJeD+bwhEcVUzpLU6hoMoopuuCUgCjCsdQ7AxTT/Pd03RoAdui tfzV7YbbMPT6Tm7wouTPMU/GnxAhN1jiirlIf1p/ZpTsN7/eBkua92HiJE4Tz2Tujhlo BNSHr8AK76vN8ju7uNfi3QvOYWqScyZxqGughtUzYPNPn0OOCWGbmmPdTkxe6HQ8yuVu fA== Received: from rn-mailsvcp-mta-lapp04.rno.apple.com (rn-mailsvcp-mta-lapp04.rno.apple.com [10.225.203.152]) by ma1-aaemail-dr-lapp02.apple.com with ESMTP id 39e0ws6k31-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Mon, 28 Jun 2021 15:54:06 -0700 Received: from rn-mailsvcp-mmp-lapp02.rno.apple.com (rn-mailsvcp-mmp-lapp02.rno.apple.com [17.179.253.15]) by rn-mailsvcp-mta-lapp04.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) with ESMTPS id <0QVF00FDEOY4W090@rn-mailsvcp-mta-lapp04.rno.apple.com>; Mon, 28 Jun 2021 15:54:04 -0700 (PDT) Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp02.rno.apple.com by rn-mailsvcp-mmp-lapp02.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) id <0QVF00N00OP1OC00@rn-mailsvcp-mmp-lapp02.rno.apple.com>; Mon, 28 Jun 2021 15:54:04 -0700 (PDT) X-Va-A: X-Va-T-CD: 0784ebc13a2136f668ccea2ff57975fe X-Va-E-CD: bcb3ed6ab20966b139d3f01c1a94bc42 X-Va-R-CD: c61df103aa9e6bfe62892afaffca7067 X-Va-CD: 0 X-Va-ID: 6dde7d2c-c8ff-4b28-8725-1b042f1f151e X-V-A: X-V-T-CD: 0784ebc13a2136f668ccea2ff57975fe X-V-E-CD: bcb3ed6ab20966b139d3f01c1a94bc42 X-V-R-CD: c61df103aa9e6bfe62892afaffca7067 X-V-CD: 0 X-V-ID: ac6d6812-33af-480d-907d-e283933b98ca X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-06-28_14:2021-06-25, 2021-06-28 signatures=0 MIME-version: 1.0 Content-transfer-encoding: 8BIT Content-disposition: inline Content-type: text/plain; charset=utf-8 Received: from localhost ([17.149.238.112]) by rn-mailsvcp-mmp-lapp02.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) with ESMTPSA id <0QVF009R8OY4JT00@rn-mailsvcp-mmp-lapp02.rno.apple.com>; Mon, 28 Jun 2021 15:54:04 -0700 (PDT) Date: Mon, 28 Jun 2021 15:54:04 -0700 From: Christoph Paasch To: Matt Mathis Cc: bloat , Randall Meyer Message-id: References: <30546A13-7887-47C6-9104-E2D7E20DA38F@apple.com> In-reply-to: X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-06-28_14:2021-06-25, 2021-06-28 signatures=0 Subject: Re: [Bloat] Apple WWDC Talks on Latency/Bufferbloat X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jun 2021 22:54:07 -0000 +Randall On 06/17/21 - 20:33, Matt Mathis wrote: > Also consider ippm. intarea might be a good choice for joint sponsorship, > but they probably won't want to be the lead. Indeed, ippm might be a good candidate. Thanks! > > BTW by using two TCP connections you potentially give a free pass to many > types of networks (e.g. ECMP, SFQ, etc) and certain OS mis features. Yes, we are aware of that. Which is why we are looking into how to minimize the effects you mentioned in your previous email so that we can accurately measure the latency under load on the load-bearing connections. For the curious ones here. If you run on macOS the networkQuality command-line with option "-c", you get more verbose output. In particular, you will see the latency for H2-pings on the load-bearing connections (labeled lud_self_dl_h2 and lud_self_ul_h2). You can see in the below how the download load-bearing flows are suffering from too much data in the TCP-socket buffer and data queued in the server's process behind the bulk-data transfer (in this case it is Apache Traffic Server - we are looking into how other server implementations behave). MacBook-Pro:~ cpaasch$ networkQuality -c { "lud_self_ul_h2" : [ 71.202995300292969, 89.105010986328125, 51.216960906982422, 581.09698486328125, 155.85601806640625, 304.031982421875, 271.76202392578125, 202.48997497558594, 139.15800476074219, 160.45701599121094, 247.11001586914062, 626.049072265625, 399.29306030273438, 335.45803833007812, 164.31092834472656 ], "responsiveness" : 1075, "ul_throughput" : 28645646, "lud_foreign_tcp_handshake_443" : [ 34, 42, 34, 37, 39, 36, 36, 31 ], "lud_self_dl_h2" : [ 313.34603881835938, 359.79306030273438, 699.39007568359375, 929.51605224609375, 1653.333984375, 2466.970947265625, 2981.800048828125, 2969.277099609375, 3595.947021484375, 3785.244873046875, 3572.677001953125, 2802.677978515625 ], "dl_flows" : 20, "dl_throughput" : 396551712, "ul_flows" : 12, "lud_foreign_h2_req_resp" : [ 63, 75, 80, 73, 70, 69, 78, 96 ] } Christoph > Thanks, > --MM-- > The best way to predict the future is to create it. - Alan Kay > > We must not tolerate intolerance; > however our response must be carefully measured: > too strong would be hypocritical and risks spiraling out of > control; > too weak risks being mistaken for tacit approval. > > > On Thu, Jun 17, 2021 at 6:04 PM Christoph Paasch wrote: > > > Not sure yet - there isn’t a good one that would really fit. Maybe tsvwg > > or intarea. > > > > Suggestions? > > > > Cheers, > > Christoph > > > > On Jun 17, 2021, at 5:17 PM, Matt Mathis wrote: > > > >  > > Which WG are you targeting? > > > > Thanks, > > --MM-- > > The best way to predict the future is to create it. - Alan Kay > > > > We must not tolerate intolerance; > > however our response must be carefully measured: > > too strong would be hypocritical and risks spiraling out of > > control; > > too weak risks being mistaken for tacit approval. > > > > > > On Thu, Jun 17, 2021 at 4:43 PM Christoph Paasch > > wrote: > > > >> Hello, > >> > >> On 06/17/21 - 11:16, Matt Mathis via Bloat wrote: > >> > Is there a paper or spec for RPM? > >> > >> we try to publish an IETF-draft on the methodology before the upcoming > >> IETF > >> in July. > >> > >> But, in the mean-time please see inline: > >> > >> > There are at least two different ways to define RPM, both of which > >> might be > >> > relevant. > >> > > >> > At the TCP layer: it can be directly computed from a packet capture. > >> The > >> > trick is to time reverse a trace and compute the critical path backwards > >> > through the trace: what event triggered each segment or ACK, and count > >> > round trips. This would be super robust but does not include the > >> queueing > >> > required in the kernel socket buffers. I need to think some more about > >> > computing TCP RPM from tcp_info or other kernel instrumentation - it > >> might > >> > be possible. > >> > >> We explicitly opted against measuring purely TCP-level round-trip times. > >> Because > >> there are countless transparent TCP-proxies out there that would skew > >> these > >> numbers. Our goal with RPM/Responsiveness is to measure how an end-user > >> would > >> experience the network. Which means, DNS-resolution, TCP handshake-time, > >> TLS-handshake, HTTP/2 Request/response. Because, at the end, that's what > >> actually matters to the users. > >> > >> > A different RPM can be done in the application, above TCP, for example > >> by > >> > ping-ponging messages. This would include the delays traversing the > >> kernel > >> > socket buffers which have to be at least as large as a full network RTT. > >> > > >> > This is perhaps an important point: due to the retransmit and > >> > reassuebly queues (which are required to implement robust data delivery) > >> > TCP must be able hold at least a full RTT of data in it's own buffers, > >> > which means that under some conditions the RTT as seen by the > >> application > >> > has be be at least twice the network's RTT, including any bloat in the > >> > network. > >> > >> Currently, we measure RPM on separate connections (not the load-bearing > >> ones). We are also measuring on the load-bearing connections themselves > >> through H2 Ping frames. But for the reasons you described we haven't yet > >> factored it into the RPM-number. > >> > >> One way may be to inspect with TCP_INFO whether or not the connections had > >> retransmissions and then throw away the number. On the other hand, if the > >> network becomes extremely lossy under working conditions, it does impact > >> the > >> user-experience and so it could make sense to take this into account. > >> > >> > >> In the end, we realized how hard it is to accurately measure bufferbloat > >> within a reasonable time-frame (our goal is to finish the test within ~15 > >> seconds). > >> > >> We hope that with the IETF-draft we can get the right people together to > >> iterate over it and squash out a very accurate measurement that represents > >> what users would experience. > >> > >> > >> Cheers, > >> Christoph > >> > >> > >> > > >> > Thanks, > >> > --MM-- > >> > The best way to predict the future is to create it. - Alan Kay > >> > > >> > We must not tolerate intolerance; > >> > however our response must be carefully measured: > >> > too strong would be hypocritical and risks spiraling out of > >> > control; > >> > too weak risks being mistaken for tacit approval. > >> > > >> > > >> > On Sat, Jun 12, 2021 at 9:11 AM Rich Brown > >> wrote: > >> > > >> > > > On Jun 12, 2021, at 12:00 PM, bloat-request@lists.bufferbloat.net > >> wrote: > >> > > > > >> > > > Some relevant talks / publicity at WWDC -- the first mentioning > >> CoDel, > >> > > > queueing, etc. Featuring Stuart Cheshire. iOS 15 adds a developer > >> test > >> > > for > >> > > > loaded latency, reported in "RPM" or round-trips per minute. > >> > > > > >> > > > I ran it on my machine: > >> > > > nowens@mac1015 ~ % /usr/bin/networkQuality > >> > > > ==== SUMMARY ==== > >> > > > Upload capacity: 90.867 Mbps > >> > > > Download capacity: 93.616 Mbps > >> > > > Upload flows: 16 > >> > > > Download flows: 20 > >> > > > Responsiveness: Medium (840 RPM) > >> > > > >> > > Does anyone know how to get the command-line version for current (not > >> > > upcoming) macOS? Thanks. > >> > > > >> > > Rich > >> > > _______________________________________________ > >> > > Bloat mailing list > >> > > Bloat@lists.bufferbloat.net > >> > > https://lists.bufferbloat.net/listinfo/bloat > >> > > > >> > >> > _______________________________________________ > >> > Bloat mailing list > >> > Bloat@lists.bufferbloat.net > >> > https://lists.bufferbloat.net/listinfo/bloat > >> > >>