From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ma1-aaemail-dr-lapp03.apple.com (ma1-aaemail-dr-lapp03.apple.com [17.171.2.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id C1F8F3B2A4; Mon, 25 Oct 2021 23:11:18 -0400 (EDT) Received: from pps.filterd (ma1-aaemail-dr-lapp03.apple.com [127.0.0.1]) by ma1-aaemail-dr-lapp03.apple.com (8.16.0.42/8.16.0.42) with SMTP id 19Q38DJW051612; Mon, 25 Oct 2021 20:11:16 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=content-type : mime-version : subject : from : in-reply-to : date : cc : content-transfer-encoding : message-id : references : to; s=20180706; bh=gruT590pdVcnGwGkUwXWBI2UhLiDU9+cbggYD6a/DLc=; b=oCwNj9cppHWLrXu9Kj2hHr7dlx84CgZt0Y0bk23Pni+aNLLmvgNNr5lDKNkGBCC0jwKl CzDYVAS4wPi/GC7ufFRUV14TwGO766NJUJ8pQkNhvuGqvmPEnm7X9ofZzjiK3g0JZYGE XOBJ3fZVWC3k5UPCfq11v5d1LHR5V87yzZOBTUKGHZy34NNNXc8LLDpqcrbkwXhf47+d OwSNsQLgPcyMokt29WJpBQFlO1xNBnXl0OufMTTSkI54F3Q1E1ELlWy4MQgi+3hK9uAt PXj3zmMlQA20Gq2Wap+3h5O7IH2FOKHCOCCPU6eOdEfZqAWsOzzWa/+9wRRX5oeK8VFt MQ== Received: from rn-mailsvcp-mta-lapp02.rno.apple.com (rn-mailsvcp-mta-lapp02.rno.apple.com [10.225.203.150]) by ma1-aaemail-dr-lapp03.apple.com with ESMTP id 3bx4hkwpn9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Mon, 25 Oct 2021 20:11:16 -0700 Received: from rn-mailsvcp-mmp-lapp03.rno.apple.com (rn-mailsvcp-mmp-lapp03.rno.apple.com [17.179.253.16]) by rn-mailsvcp-mta-lapp02.rno.apple.com (Oracle Communications Messaging Server 8.1.0.12.20210903 64bit (built Sep 3 2021)) with ESMTPS id <0R1K007A6E6R8IH0@rn-mailsvcp-mta-lapp02.rno.apple.com>; Mon, 25 Oct 2021 20:11:15 -0700 (PDT) Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp03.rno.apple.com by rn-mailsvcp-mmp-lapp03.rno.apple.com (Oracle Communications Messaging Server 8.1.0.12.20210903 64bit (built Sep 3 2021)) id <0R1K00600E4RFQ00@rn-mailsvcp-mmp-lapp03.rno.apple.com>; Mon, 25 Oct 2021 20:11:15 -0700 (PDT) X-Va-A: X-Va-T-CD: 095fc62c348ed25dc0c10f0adec69295 X-Va-E-CD: cd8bf474a903d3c7bc0aa9da18c9a853 X-Va-R-CD: 540cfe524e2d48ca1445b34bbf6444c7 X-Va-CD: 0 X-Va-ID: c6f845f8-d97a-46c5-afa6-45717430e2ba X-V-A: X-V-T-CD: 095fc62c348ed25dc0c10f0adec69295 X-V-E-CD: cd8bf474a903d3c7bc0aa9da18c9a853 X-V-R-CD: 540cfe524e2d48ca1445b34bbf6444c7 X-V-CD: 0 X-V-ID: b886e011-fe58-4bae-9337-08199c0357b5 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.425, 18.0.790 definitions=2021-10-25_08:2021-10-25, 2021-10-25 signatures=0 Received: from [17.11.109.25] (unknown [17.11.109.25]) by rn-mailsvcp-mmp-lapp03.rno.apple.com (Oracle Communications Messaging Server 8.1.0.12.20210903 64bit (built Sep 3 2021)) with ESMTPSA id <0R1K00433E6K1M00@rn-mailsvcp-mmp-lapp03.rno.apple.com>; Mon, 25 Oct 2021 20:11:15 -0700 (PDT) Content-type: text/plain; charset=utf-8 MIME-version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.1\)) From: Stuart Cheshire In-reply-to: Date: Mon, 25 Oct 2021 20:11:07 -0700 Cc: "David P. Reed" , Cake List , =?utf-8?Q?Valdis_Kl=C4=93tnieks?= , Make-Wifi-fast , starlink@lists.bufferbloat.net, codel , Matt Mathis , cerowrt-devel , bloat , Steve Crocker , Vint Cerf , Neal Cardwell Content-transfer-encoding: quoted-printable Message-id: References: <1625188609.32718319@apps.rackspace.com> <989de0c1-e06c-cda9-ebe6-1f33df8a4c24@candelatech.com> <1625773080.94974089@apps.rackspace.com> <1625859083.09751240@apps.rackspace.com> <257851.1632110422@turing-police> <1632680642.869711321@apps.rackspace.com> To: Bob McMahon X-Mailer: Apple Mail (2.3608.120.23.2.1) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.425, 18.0.790 definitions=2021-10-25_08:2021-10-25, 2021-10-25 signatures=0 Subject: Re: [Make-wifi-fast] TCP_NOTSENT_LOWAT applied to e2e TCP msg latency X-BeenThere: make-wifi-fast@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Oct 2021 03:11:19 -0000 On 21 Oct 2021, at 17:51, Bob McMahon via Make-wifi-fast = wrote: > Hi All, >=20 > Sorry for the spam. I'm trying to support a meaningful TCP message = latency w/iperf 2 from the sender side w/o requiring e2e clock = synchronization. I thought I'd try to use the TCP_NOTSENT_LOWAT event to = help with this. It seems that this event goes off when the bytes are in = flight vs have reached the destination network stack. If that's the = case, then iperf 2 client (sender) may be able to produce the message = latency by adding the drain time (write start to TCP_NOTSENT_LOWAT) and = the sampled RTT. >=20 > Does this seem reasonable? I=E2=80=99m not 100% sure what you=E2=80=99re asking, but I will try to = help. When you set TCP_NOTSENT_LOWAT, the TCP implementation won=E2=80=99t = report your endpoint as writable (e.g., via kqueue or epoll) until less = than that threshold of data remains unsent. It won=E2=80=99t stop you = writing more bytes if you want to, up to the socket send buffer size, = but it won=E2=80=99t *ask* you for more data until the TCP_NOTSENT_LOWAT = threshold is reached. In other words, the TCP implementation attempts to = keep BDP bytes in flight + TCP_NOTSENT_LOWAT bytes buffered and ready to = go. The BDP of bytes in flight is necessary to fill the network pipe and = get good throughput. The TCP_NOTSENT_LOWAT of bytes buffered and ready = to go is provided to give the source software some advance notice that = the TCP implementation will soon be looking for more bytes to send, so = that the buffer doesn=E2=80=99t run dry, thereby lowering throughput. = (The old SO_SNDBUF option conflates both =E2=80=9Cbytes in flight=E2=80=9D= and =E2=80=9Cbytes buffered and ready to go=E2=80=9D into the same = number.) If you wait for the TCP_NOTSENT_LOWAT notification, write a chunk of n = bytes of data, and then wait for the next TCP_NOTSENT_LOWAT = notification, that will tell you roughly how long it took n bytes to = depart the machine. You won=E2=80=99t know why, though. The bytes could = depart the machine in response for acks indicating that the same number = of bytes have been accepted at the receiver. But the bytes can also = depart the machine because CWND is growing. Of course, both of those = things are usually happening at the same time. How to use TCP_NOTSENT_LOWAT is explained in this video: Later in the same video is a two-minute demo (time offset 42:00 to time = offset 44:00) showing a =E2=80=9Cbefore and after=E2=80=9D demo = illustrating the dramatic difference this makes for screen sharing = responsiveness. Stuart Cheshire=