From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.15.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 35B243CB4E; Wed, 27 Oct 2021 10:29:15 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1635344952; bh=S9v2XEB5kRZFzx+hD8JK4hcWbqqj6Ai9w8pCzh29Nfc=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=L+8SzuQxo47ohoPEpJtS1Gc36XCYCVUIaN5ZcLwiig657pKIyZNTPv6v0iqk7swMq WBd/CcRIvEW4LSApgoCbkIU/SFRp58bZH2HUIKbNEBOBQiT2cLJOmM4q9j9Gv6T++A mpELZdB1KRfNefp6SZMuXvDh+bFb0IfK46AXhiSg= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from [192.168.250.101] ([134.76.241.253]) by mail.gmx.net (mrgmx004 [212.227.17.190]) with ESMTPSA (Nemesis) id 1MgeoI-1n9bcq1Kkm-00hA1K; Wed, 27 Oct 2021 16:29:12 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.7\)) From: Sebastian Moeller In-Reply-To: Date: Wed, 27 Oct 2021 16:29:11 +0200 Cc: =?utf-8?Q?Bj=C3=B8rn_Ivar_Teigen?= , Cake List , Make-Wifi-fast , starlink@lists.bufferbloat.net, codel , cerowrt-devel , bloat Content-Transfer-Encoding: quoted-printable Message-Id: <77662A26-2CDC-4201-8148-6C0702702447@gmx.de> References: <1625188609.32718319@apps.rackspace.com> <989de0c1-e06c-cda9-ebe6-1f33df8a4c24@candelatech.com> <1625773080.94974089@apps.rackspace.com> <1625859083.09751240@apps.rackspace.com> <257851.1632110422@turing-police> <1632680642.869711321@apps.rackspace.com> To: Bob McMahon X-Mailer: Apple Mail (2.3608.120.23.2.7) X-Provags-ID: V03:K1:Qiv7NbnFNULUk5q49x8w3a1PYo11Io5AtSSXfN6LwzE8EkRCVks 7ZikZmjnrczeOKPBEaR8hvvHfeshQ/jDykQ67oNmEed3lBghPuUMDmpXAnbdEqEq1hirJjj SZOPuoYhVFGsYwnebIk+LovG/g74ZVpst1m1SGcIQ8DRI4MkHYzK5GcCe3W5GVENJg+NQtr oWwdmpTLpEPz2ayO3m+aQ== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:9eWLBeDTwvg=:XOIOn+4xMW00Si7GENNtEx mIiAvxa/Jro/flsHsaQc8B23pXdOUE6oOGv/hk8nGaiuaihAThSsGtqDdf+/qEDJc/EdNOvM9 UYiEDxvLFrc1dp5xS0q2TG0PG/M+hX80teB1JbYGMMPvFnElA1FXRHPGsC47XDhwHDyJ5Z2/9 LaQJjWOIt/BgO933Lwl67fzUWipWvSSM3M+bEOYQm7r/jTuwPeoKIG4ZxJjZ50SSPU/w8OTNj 3Ez4+PSqGYdSM79yCJELasS/t00fKqmLIlTv9uYXs70RdQmRszml5bO8c1kuf/QJzUHdHHfc3 GA3ncMn0kwMkTA60rfh61Oqwkm41MMooi+KpHSv8QijUnZuBuh5EWoUo2PYzQB4ZFndBy1wxf 31eDyzSN448N6JGOHGCPpPy8+qDwjkkut5Jt0nkZXvkVwQ97eXDsqKdQmCqDyBmU+bVtJn4aW BrmpWeQx5pDAMa1u3wgafs8kvoqhkMpqKudMElcJ7KJPk96iI+lqX7BFLPFCmmNM8mFIMyNWU IKtQ9kHQ8z2PlV1ZJhm3T9/FGckZYSf7YUAfv9F3CrdN8/C3GD3XIhkwDg4Yt4n/OILNqCQD4 WhgSGk/5hqXrQimnZ55GIPPPEcJZ9kGFQs+agX8DfDNQ4z2S0kgFS8bW/0aW5h6DMHOBAFgn6 nZi+gP0KpGegjbn54Lxye3dLJX/1kIgn/J1XQxERbexQaxQ+xv5w1QnjkOtVgynn8HnKOXHHF 61c8k09axuWJ0u9Lv7zh9jS0GsW9t6SkFVknKkIQOliQ+du7v7zq/eEBkKqf2ZmlIgHPyjRNp Xv8lR+PN3xZqPZAdShrbbSHPF6v72961DGQ8zwb6JeL4m+H1Q9467cJjtNzRPEvRWGF7DX4gc WxpdKb1owFp4uKjTaRSscrbc7J1p9m/qK1IXZxYhz+gHaElaWS6w1F0l71ljCWGpHPsZAnZpF Q/bFQTIoHhZlc51PpGWdr7os5sB5vOlM5mSvkBDeIzmCGGLf8jgX/Woc/+tL7H+tA7srnHUOF ZaAdloSascPB3Qdf3gSxtUOx0Xd4EmIr99up0GgduaYhFa6Sevxic8yC37ymWCivbaIn0QeqB t2DmrCHusbEyIY= Subject: Re: [Cerowrt-devel] [Make-wifi-fast] [Starlink] TCP_NOTSENT_LOWAT applied to e2e TCP msg latency X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Oct 2021 14:29:15 -0000 Hi Bob, OWD !=3D RTT/2 seems generically to be the rule on the internet not the = exception, even with perfectly symmetric access links. Routing between = AS often is asymmetric in it self (hot potato routing, where each AS = hands over packets destined to others as early as possible, means that = forward and backward path are often noticeably different; or rather they = are different but that is hard to notice unless one can get path = measurements like traceroutes from both directions). That last point is = what makes me believe that internet speedtests, should always also = include traceroutes from server to client and from client to server so = one at least has a rough idea where the packets are going, but I = digress... Regards Sebastian > On Oct 26, 2021, at 19:23, Bob McMahon via Make-wifi-fast = wrote: >=20 > Hi Bj=C3=B8rn, >=20 > I find, when possible, it's preferred to take telemetry data of actual = traffic (or reads and writes) vs a proxy. We had a case where TCP BE was = outperforming TCP w/VI because BE had the most engineering resources = assigned to it and engineers did a better job with BE. Using a proxy = protocol wouldn't have exercised the same logic paths (in this case it = was in the L2 driver) as TCP did. Hence, measuring actual TCP traffic = (or socket reads and socket writes) was needed to flush out the problem. = Note: I also find that network engineers tend to focus on the stack but = it's the e2e at the application level that impacts user experience. Send = side bloat can drive the OWD while the TCP stack's RTT may look fine. = For WiFi test & measurements, we've decided most testing should be using = TCP_NOSENT_LOWAT because it helps mitigate send side bloat which WiFi = engineering doesn't focus on per lack of ability to impact. >=20 > Also, I think OWD is under tested and two way based testing can give = incomplete and inaccurate information, particularly with respect to = things like an e2e transport's control loop. A most obvious example is = assuming 1/2 RTT is the same as OWD to/fro. For WiFi this assumption is = most always false. It also false for many residential internet = connections where OWD asymmetry is designed in. >=20 > Bob >=20 >=20 > On Tue, Oct 26, 2021 at 3:04 AM Bj=C3=B8rn Ivar Teigen = wrote: > Hi Bob, >=20 > My name is Bj=C3=B8rn Ivar Teigen and I'm working on modeling and = measuring WiFi MAC-layer protocol performance for my PhD. >=20 > Is it necessary to measure the latency using the TCP stream itself? I = had a similar problem in the past, and solved it by doing the latency = measurements using TWAMP running alongside the TCP traffic. The = requirement for this to work is that the TWAMP packets are placed in the = same queue(s) as the TCP traffic, and that the impact of measurement = traffic is small enough so as not to interfere too much with your TCP = results. > Just my two cents, hope it's helpful. >=20 > Bj=C3=B8rn >=20 > On Tue, 26 Oct 2021 at 06:32, Bob McMahon = wrote: > Thanks Stuart this is helpful. I'm measuring the time just before the = first write() (of potentially a burst of writes to achieve a burst size) = per a socket fd's select event occurring when TCP_NOT_SENT_LOWAT being = set to a small value, then sampling the RTT and CWND and providing = histograms for all three, all on that event. I'm not sure the = correctness of RTT and CWND at this sample point. This is a controlled = test over 802.11ax and OFDMA where the TCP acks per the WiFi clients are = being scheduled by the AP using 802.11ax trigger frames so the AP is = affecting the end/end BDP per scheduling the transmits and the acks. The = AP can grow the BDP or shrink it based on these scheduling decisions. = =46rom there we're trying to maximize network power (throughput/delay) = for elephant flows and just latency for mouse flows. (We also plan some = RF frequency stuff to per OFDMA) Anyway, the AP based scheduling along = with aggregation and OFDMA makes WiFi scheduling optimums non-obvious - = at least to me - and I'm trying to provide insights into how an AP is = affecting end/end performance. >=20 > The more direct approach for e2e TCP latency and network power has = been to measure first write() to final read() and compute the e2e delay. = This requires clock sync on the ends. (We're using ptp4l with GPS OCXO = atomic references for that but this is typically only available in some = labs.)=20 >=20 > Bob > =20 >=20 > On Mon, Oct 25, 2021 at 8:11 PM Stuart Cheshire = wrote: > On 21 Oct 2021, at 17:51, Bob McMahon via Make-wifi-fast = wrote: >=20 > > Hi All, > >=20 > > Sorry for the spam. I'm trying to support a meaningful TCP message = latency w/iperf 2 from the sender side w/o requiring e2e clock = synchronization. I thought I'd try to use the TCP_NOTSENT_LOWAT event to = help with this. It seems that this event goes off when the bytes are in = flight vs have reached the destination network stack. If that's the = case, then iperf 2 client (sender) may be able to produce the message = latency by adding the drain time (write start to TCP_NOTSENT_LOWAT) and = the sampled RTT. > >=20 > > Does this seem reasonable? >=20 > I=E2=80=99m not 100% sure what you=E2=80=99re asking, but I will try = to help. >=20 > When you set TCP_NOTSENT_LOWAT, the TCP implementation won=E2=80=99t = report your endpoint as writable (e.g., via kqueue or epoll) until less = than that threshold of data remains unsent. It won=E2=80=99t stop you = writing more bytes if you want to, up to the socket send buffer size, = but it won=E2=80=99t *ask* you for more data until the TCP_NOTSENT_LOWAT = threshold is reached. In other words, the TCP implementation attempts to = keep BDP bytes in flight + TCP_NOTSENT_LOWAT bytes buffered and ready to = go. The BDP of bytes in flight is necessary to fill the network pipe and = get good throughput. The TCP_NOTSENT_LOWAT of bytes buffered and ready = to go is provided to give the source software some advance notice that = the TCP implementation will soon be looking for more bytes to send, so = that the buffer doesn=E2=80=99t run dry, thereby lowering throughput. = (The old SO_SNDBUF option conflates both =E2=80=9Cbytes in flight=E2=80=9D= and =E2=80=9Cbytes buffered and ready to go=E2=80=9D into the same = number.) >=20 > If you wait for the TCP_NOTSENT_LOWAT notification, write a chunk of n = bytes of data, and then wait for the next TCP_NOTSENT_LOWAT = notification, that will tell you roughly how long it took n bytes to = depart the machine. You won=E2=80=99t know why, though. The bytes could = depart the machine in response for acks indicating that the same number = of bytes have been accepted at the receiver. But the bytes can also = depart the machine because CWND is growing. Of course, both of those = things are usually happening at the same time. >=20 > How to use TCP_NOTSENT_LOWAT is explained in this video: >=20 > >=20 > Later in the same video is a two-minute demo (time offset 42:00 to = time offset 44:00) showing a =E2=80=9Cbefore and after=E2=80=9D demo = illustrating the dramatic difference this makes for screen sharing = responsiveness. >=20 > >=20 > Stuart Cheshire >=20 > This electronic communication and the information and any files = transmitted with it, or attached to it, are confidential and are = intended solely for the use of the individual or entity to whom it is = addressed and may contain information that is confidential, legally = privileged, protected by privacy laws, or otherwise restricted from = disclosure to anyone else. If you are not the intended recipient or the = person responsible for delivering the e-mail to the intended recipient, = you are hereby notified that any use, copying, distributing, = dissemination, forwarding, printing, or copying of this e-mail is = strictly prohibited. If you received this e-mail in error, please return = the e-mail to the sender, delete it from your computer, and destroy any = printed copy of it._______________________________________________ > Starlink mailing list > Starlink@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/starlink >=20 >=20 > --=20 > Bj=C3=B8rn Ivar Teigen > Head of Research > +47 47335952 | bjorn@domos.no | www.domos.no > WiFi Slicing by Domos >=20 > This electronic communication and the information and any files = transmitted with it, or attached to it, are confidential and are = intended solely for the use of the individual or entity to whom it is = addressed and may contain information that is confidential, legally = privileged, protected by privacy laws, or otherwise restricted from = disclosure to anyone else. If you are not the intended recipient or the = person responsible for delivering the e-mail to the intended recipient, = you are hereby notified that any use, copying, distributing, = dissemination, forwarding, printing, or copying of this e-mail is = strictly prohibited. If you received this e-mail in error, please return = the e-mail to the sender, delete it from your computer, and destroy any = printed copy of it._______________________________________________ > Make-wifi-fast mailing list > Make-wifi-fast@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/make-wifi-fast