From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 46AE23B29E for ; Tue, 26 Oct 2021 13:23:40 -0400 (EDT) Received: by mail-ed1-x531.google.com with SMTP id s1so16608109edd.3 for ; Tue, 26 Oct 2021 10:23:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=zcM6M9fQyNtyobcPZyonOjNlTt2hcdCIixXBxrdmvOE=; b=RJdOAXeQCcKRuy20wrTuMXAEn69VfDL8kOotMIXUNWGfrQwOnaBZ3fDMoSGODatdJr TJO1KN4zWameBawD9x4wCWa8knEXsI7DS9qkqnPRKmMbDEiWLYWBMObrqvc4Xakh3mJC FRhm/WRth1IZrwUSdZUsxd+zrCPasMLxRGSJQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=zcM6M9fQyNtyobcPZyonOjNlTt2hcdCIixXBxrdmvOE=; b=NeKq9U6KmGm79kZA5V37fNu4Emw2i0KrX5EUqEBDk6r+INjlqKDyWeySW2S4teYp4U 02Gr/fNLg8Z4qvrn49owW4Gc4ylaGEQmXrsu88/DWSnOVapqXVxbI9lLcd+t6E3VIyxM L8ZxSIZsOv2TA72zPATz7/JTX7Q763WNmhhpnB33ys9o8XWs5sTCCbvP5cX6GgnE238w 39gdfhJQPJ1X7Doc5wU0RlawrQijr/yLj5w/84hs5q8efGWYG5xxr85shOAEf5W2QSNo UBy89F+g/ocae7+bazLP+w8fWpVBMWB5wLq8sZq1sM6QK5DTwkuyjkU3hb2VRbQ22jAS UjkQ== X-Gm-Message-State: AOAM530pADetOMbWPphx8EELwGqBeQL7ZMXh5K0HLEzR7dyI9CneG8ou nAaxQb/J87ITSqBgMHp/WS726gVHEPEOg25TK6oueJYWgofzTK4aOXX0FDfHX8HPqOGyKB4Z+u8 yvjEoaZGC1yjk3+ZoOTKr3syua0PNn0P1yUxLuJfe X-Google-Smtp-Source: ABdhPJxmJhw6TKH4tK8qX96wXWO2Mj9qPKYyfyrAUzaEYGt60pkez2omZmW4yahoHCdm2Lw36HwnmKfdkRCTfq9yOr8= X-Received: by 2002:a17:907:9714:: with SMTP id jg20mr12967539ejc.318.1635269019040; Tue, 26 Oct 2021 10:23:39 -0700 (PDT) MIME-Version: 1.0 References: <1625188609.32718319@apps.rackspace.com> <989de0c1-e06c-cda9-ebe6-1f33df8a4c24@candelatech.com> <1625773080.94974089@apps.rackspace.com> <1625859083.09751240@apps.rackspace.com> <257851.1632110422@turing-police> <1632680642.869711321@apps.rackspace.com> In-Reply-To: From: Bob McMahon Date: Tue, 26 Oct 2021 10:23:28 -0700 Message-ID: Subject: Re: [Starlink] [Make-wifi-fast] TCP_NOTSENT_LOWAT applied to e2e TCP msg latency To: =?UTF-8?Q?Bj=C3=B8rn_Ivar_Teigen?= Cc: Stuart Cheshire , starlink@lists.bufferbloat.net, =?UTF-8?Q?Valdis_Kl=C4=93tnieks?= , Make-Wifi-fast , "David P. Reed" , Cake List , codel , Matt Mathis , cerowrt-devel , bloat , Neal Cardwell Content-Type: multipart/alternative; boundary="000000000000fe418b05cf44bc39" X-List-Received-Date: Tue, 26 Oct 2021 17:23:40 -0000 --000000000000fe418b05cf44bc39 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Bj=C3=B8rn, I find, when possible, it's preferred to take telemetry data of actual traffic (or reads and writes) vs a proxy. We had a case where TCP BE was outperforming TCP w/VI because BE had the most engineering resources assigned to it and engineers did a better job with BE. Using a proxy protocol wouldn't have exercised the same logic paths (in this case it was in the L2 driver) as TCP did. Hence, measuring actual TCP traffic (or socket reads and socket writes) was needed to flush out the problem. Note: I also find that network engineers tend to focus on the stack but it's the e2e at the application level that impacts user experience. Send side bloat can drive the OWD while the TCP stack's RTT may look fine. For WiFi test & measurements, we've decided most testing should be using TCP_NOSENT_LOWAT because it helps mitigate send side bloat which WiFi engineering doesn't focus on per lack of ability to impact. Also, I think OWD is under tested and two way based testing can give incomplete and inaccurate information, particularly with respect to things like an e2e transport's control loop. A most obvious example is assuming 1/2 RTT is the same as OWD to/fro. For WiFi this assumption is most always false. It also false for many residential internet connections where OWD asymmetry is designed in. Bob On Tue, Oct 26, 2021 at 3:04 AM Bj=C3=B8rn Ivar Teigen wro= te: > Hi Bob, > > My name is Bj=C3=B8rn Ivar Teigen and I'm working on modeling and measuri= ng > WiFi MAC-layer protocol performance for my PhD. > > Is it necessary to measure the latency using the TCP stream itself? I had > a similar problem in the past, and solved it by doing the latency > measurements using TWAMP running alongside the TCP traffic. The requireme= nt > for this to work is that the TWAMP packets are placed in the same queue(s= ) > as the TCP traffic, and that the impact of measurement traffic is small > enough so as not to interfere too much with your TCP results. > Just my two cents, hope it's helpful. > > Bj=C3=B8rn > > On Tue, 26 Oct 2021 at 06:32, Bob McMahon > wrote: > >> Thanks Stuart this is helpful. I'm measuring the time just before the >> first write() (of potentially a burst of writes to achieve a burst size) >> per a socket fd's select event occurring when TCP_NOT_SENT_LOWAT being s= et >> to a small value, then sampling the RTT and CWND and providing histogram= s >> for all three, all on that event. I'm not sure the correctness of RTT an= d >> CWND at this sample point. This is a controlled test over 802.11ax and >> OFDMA where the TCP acks per the WiFi clients are being scheduled by the= AP >> using 802.11ax trigger frames so the AP is affecting the end/end BDP per >> scheduling the transmits and the acks. The AP can grow the BDP or shrink= it >> based on these scheduling decisions. From there we're trying to maximiz= e >> network power (throughput/delay) for elephant flows and just latency for >> mouse flows. (We also plan some RF frequency stuff to per OFDMA) Anyway, >> the AP based scheduling along with aggregation and OFDMA makes WiFi >> scheduling optimums non-obvious - at least to me - and I'm trying to >> provide insights into how an AP is affecting end/end performance. >> >> The more direct approach for e2e TCP latency and network power has been >> to measure first write() to final read() and compute the e2e delay. This >> requires clock sync on the ends. (We're using ptp4l with GPS OCXO >> atomic references for that but this is typically only available in some >> labs.) >> >> Bob >> >> >> On Mon, Oct 25, 2021 at 8:11 PM Stuart Cheshire >> wrote: >> >>> On 21 Oct 2021, at 17:51, Bob McMahon via Make-wifi-fast < >>> make-wifi-fast@lists.bufferbloat.net> wrote: >>> >>> > Hi All, >>> > >>> > Sorry for the spam. I'm trying to support a meaningful TCP message >>> latency w/iperf 2 from the sender side w/o requiring e2e clock >>> synchronization. I thought I'd try to use the TCP_NOTSENT_LOWAT event t= o >>> help with this. It seems that this event goes off when the bytes are in >>> flight vs have reached the destination network stack. If that's the cas= e, >>> then iperf 2 client (sender) may be able to produce the message latency= by >>> adding the drain time (write start to TCP_NOTSENT_LOWAT) and the sample= d >>> RTT. >>> > >>> > Does this seem reasonable? >>> >>> I=E2=80=99m not 100% sure what you=E2=80=99re asking, but I will try to= help. >>> >>> When you set TCP_NOTSENT_LOWAT, the TCP implementation won=E2=80=99t re= port your >>> endpoint as writable (e.g., via kqueue or epoll) until less than that >>> threshold of data remains unsent. It won=E2=80=99t stop you writing mor= e bytes if >>> you want to, up to the socket send buffer size, but it won=E2=80=99t *a= sk* you for >>> more data until the TCP_NOTSENT_LOWAT threshold is reached. In other wo= rds, >>> the TCP implementation attempts to keep BDP bytes in flight + >>> TCP_NOTSENT_LOWAT bytes buffered and ready to go. The BDP of bytes in >>> flight is necessary to fill the network pipe and get good throughput. T= he >>> TCP_NOTSENT_LOWAT of bytes buffered and ready to go is provided to give= the >>> source software some advance notice that the TCP implementation will so= on >>> be looking for more bytes to send, so that the buffer doesn=E2=80=99t r= un dry, >>> thereby lowering throughput. (The old SO_SNDBUF option conflates both >>> =E2=80=9Cbytes in flight=E2=80=9D and =E2=80=9Cbytes buffered and ready= to go=E2=80=9D into the same >>> number.) >>> >>> If you wait for the TCP_NOTSENT_LOWAT notification, write a chunk of n >>> bytes of data, and then wait for the next TCP_NOTSENT_LOWAT notificatio= n, >>> that will tell you roughly how long it took n bytes to depart the machi= ne. >>> You won=E2=80=99t know why, though. The bytes could depart the machine = in response >>> for acks indicating that the same number of bytes have been accepted at= the >>> receiver. But the bytes can also depart the machine because CWND is >>> growing. Of course, both of those things are usually happening at the s= ame >>> time. >>> >>> How to use TCP_NOTSENT_LOWAT is explained in this video: >>> >>> >>> >>> Later in the same video is a two-minute demo (time offset 42:00 to time >>> offset 44:00) showing a =E2=80=9Cbefore and after=E2=80=9D demo illustr= ating the dramatic >>> difference this makes for screen sharing responsiveness. >>> >>> >>> >>> Stuart Cheshire >> >> >> This electronic communication and the information and any files >> transmitted with it, or attached to it, are confidential and are intende= d >> solely for the use of the individual or entity to whom it is addressed a= nd >> may contain information that is confidential, legally privileged, protec= ted >> by privacy laws, or otherwise restricted from disclosure to anyone else.= If >> you are not the intended recipient or the person responsible for deliver= ing >> the e-mail to the intended recipient, you are hereby notified that any u= se, >> copying, distributing, dissemination, forwarding, printing, or copying o= f >> this e-mail is strictly prohibited. If you received this e-mail in error= , >> please return the e-mail to the sender, delete it from your computer, an= d >> destroy any printed copy of it. >> _______________________________________________ >> Starlink mailing list >> Starlink@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/starlink >> > > > -- > Bj=C3=B8rn Ivar Teigen > Head of Research > +47 47335952 | bjorn@domos.no | www.domos.no > WiFi Slicing by Domos > --=20 This electronic communication and the information and any files transmitted= =20 with it, or attached to it, are confidential and are intended solely for=20 the use of the individual or entity to whom it is addressed and may contain= =20 information that is confidential, legally privileged, protected by privacy= =20 laws, or otherwise restricted from disclosure to anyone else. If you are=20 not the intended recipient or the person responsible for delivering the=20 e-mail to the intended recipient, you are hereby notified that any use,=20 copying, distributing, dissemination, forwarding, printing, or copying of= =20 this e-mail is strictly prohibited. If you received this e-mail in error,= =20 please return the e-mail to the sender, delete it from your computer, and= =20 destroy any printed copy of it. --000000000000fe418b05cf44bc39 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Bj=C3=B8rn,

I find, when possible, it's pre= ferred to take telemetry data of actual traffic (or reads and writes) vs a = proxy. We had a case where TCP BE was outperforming=C2=A0TCP w/VI because B= E had the most engineering resources assigned to it and engineers did a bet= ter job with BE. Using a proxy protocol wouldn't have exercised the sam= e logic paths (in this case it was in the L2 driver) as TCP did. Hence, mea= suring actual TCP traffic (or socket reads and socket writes) was needed to= flush out the problem. Note: I also find that network engineers tend to fo= cus on the stack but it's the e2e at the application=C2=A0level that im= pacts user experience. Send side bloat can drive the OWD while the TCP stac= k's RTT may look fine. For WiFi test & measurements, we've deci= ded most testing=C2=A0should be using TCP_NOSENT_LOWAT because it helps mit= igate send side bloat which WiFi engineering doesn't focus on per lack = of ability to impact.

Also, I think OWD is under tested and two way = based testing can give incomplete and inaccurate information, particularly = with respect to things like an e2e transport's control loop.=C2=A0 A mo= st obvious example is assuming 1/2 RTT is the same as OWD to/fro. For WiFi = this assumption is most always false. It also false for many residential in= ternet connections where OWD asymmetry is designed in.

Bob


On Tue, Oct 26, 2021 at 3:04 AM Bj=C3=B8r= n Ivar Teigen <bjorn= @domos.no> wrote:
Hi Bob,

My name i= s Bj=C3=B8rn Ivar Teigen and I'm working on modeling and measuring WiFi= MAC-layer protocol performance for my PhD.

Is it = necessary to measure the latency using the TCP stream itself? I had a simil= ar problem in the past, and solved it by doing the latency measurements usi= ng TWAMP running alongside the TCP traffic. The requirement for this to wor= k is that the TWAMP packets are placed in the same queue(s) as the TCP traf= fic, and that the impact of measurement traffic is small enough so as not t= o interfere too much with your TCP results.
Just my two cents, ho= pe it's helpful.

Bj=C3=B8rn
On Tue, 2= 6 Oct 2021 at 06:32, Bob McMahon <bob.mcmahon@broadcom.com> wrote:
Thanks Stu= art this is helpful. I'm measuring=C2=A0the time just before the first = write() (of potentially a burst of writes to achieve a burst size) per a so= cket fd's select event occurring when TCP_NOT_SENT_LOWAT being set to a= small value, then sampling the RTT and CWND and providing histograms for a= ll three, all on that event. I'm not sure the correctness of RTT and CW= ND at this sample point. This is a controlled test over 802.11ax and OFDMA = where the TCP acks per the WiFi clients are being scheduled by the AP using= 802.11ax trigger frames so the AP is affecting the end/end BDP per schedul= ing the transmits and the acks. The AP can grow the BDP or shrink it based = on these scheduling decisions.=C2=A0 From there we're trying to maximiz= e network power (throughput/delay) for elephant flows and just latency for = mouse flows. (We also plan some RF frequency stuff to per OFDMA) Anyway, th= e AP based scheduling along with aggregation=C2=A0and OFDMA makes WiFi sche= duling optimums non-obvious - at least to me - and I'm trying to provid= e insights into how an AP is affecting end/end performance.

The more= direct approach for e2e TCP latency and network power has been to measure = first write() to final read() and compute the e2e delay. This requires cloc= k sync on the ends. (We're using ptp4l with GPS OCXO atomic=C2=A0refere= nces=C2=A0for that but this is typically only available in some labs.)=C2= =A0

Bob
=C2=A0

On Mon, Oct 25, 2021 at 8:11 PM Stuart C= heshire <cheshir= e@apple.com> wrote:
On 21 Oct 2021, at 17:51, Bob McMahon via Make-wifi-fast <make= -wifi-fast@lists.bufferbloat.net> wrote:

> Hi All,
>
> Sorry for the spam. I'm trying to support a meaningful TCP message= latency w/iperf 2 from the sender side w/o requiring e2e clock synchroniza= tion. I thought I'd try to use the TCP_NOTSENT_LOWAT event to help with= this. It seems that this event goes off when the bytes are in flight vs ha= ve reached the destination network stack. If that's the case, then iper= f 2 client (sender) may be able to produce the message latency by adding th= e drain time (write start to TCP_NOTSENT_LOWAT) and the sampled RTT.
>
> Does this seem reasonable?

I=E2=80=99m not 100% sure what you=E2=80=99re asking, but I will try to hel= p.

When you set TCP_NOTSENT_LOWAT, the TCP implementation won=E2=80=99t report= your endpoint as writable (e.g., via kqueue or epoll) until less than that= threshold of data remains unsent. It won=E2=80=99t stop you writing more b= ytes if you want to, up to the socket send buffer size, but it won=E2=80=99= t *ask* you for more data until the TCP_NOTSENT_LOWAT threshold is reached.= In other words, the TCP implementation attempts to keep BDP bytes in fligh= t + TCP_NOTSENT_LOWAT bytes buffered and ready to go. The BDP of bytes in f= light is necessary to fill the network pipe and get good throughput. The TC= P_NOTSENT_LOWAT of bytes buffered and ready to go is provided to give the s= ource software some advance notice that the TCP implementation will soon be= looking for more bytes to send, so that the buffer doesn=E2=80=99t run dry= , thereby lowering throughput. (The old SO_SNDBUF option conflates both =E2= =80=9Cbytes in flight=E2=80=9D and =E2=80=9Cbytes buffered and ready to go= =E2=80=9D into the same number.)

If you wait for the TCP_NOTSENT_LOWAT notification, write a chunk of n byte= s of data, and then wait for the next TCP_NOTSENT_LOWAT notification, that = will tell you roughly how long it took n bytes to depart the machine. You w= on=E2=80=99t know why, though. The bytes could depart the machine in respon= se for acks indicating that the same number of bytes have been accepted at = the receiver. But the bytes can also depart the machine because CWND is gro= wing. Of course, both of those things are usually happening at the same tim= e.

How to use TCP_NOTSENT_LOWAT is explained in this video:

<https://developer.apple.com/v= ideos/play/wwdc2015/719/?time=3D2199>

Later in the same video is a two-minute demo (time offset 42:00 to time off= set 44:00) showing a =E2=80=9Cbefore and after=E2=80=9D demo illustrating t= he dramatic difference this makes for screen sharing responsiveness.

<https://developer.apple.com/v= ideos/play/wwdc2015/719/?time=3D2520>

Stuart Cheshire

This ele= ctronic communication and the information and any files transmitted with it= , or attached to it, are confidential and are intended solely for the use o= f the individual or entity to whom it is addressed and may contain informat= ion that is confidential, legally privileged, protected by privacy laws, or= otherwise restricted from disclosure to anyone else. If you are not the in= tended recipient or the person responsible for delivering the e-mail to the= intended recipient, you are hereby notified that any use, copying, distrib= uting, dissemination, forwarding, printing, or copying of this e-mail is st= rictly prohibited. If you received this e-mail in error, please return the = e-mail to the sender, delete it from your computer, and destroy any printed= copy of it._______________________________________________ Starlink mailing list
Starlin= k@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink


--
Bj=C3=B8rn Ivar Teigen
Head of Research
+47 47335952 | bjorn@domos.no=C2=A0|=C2=A0www.domos.no
WiFi Slicing by Domos

This ele= ctronic communication and the information and any files transmitted with it= , or attached to it, are confidential and are intended solely for the use o= f the individual or entity to whom it is addressed and may contain informat= ion that is confidential, legally privileged, protected by privacy laws, or= otherwise restricted from disclosure to anyone else. If you are not the in= tended recipient or the person responsible for delivering the e-mail to the= intended recipient, you are hereby notified that any use, copying, distrib= uting, dissemination, forwarding, printing, or copying of this e-mail is st= rictly prohibited. If you received this e-mail in error, please return the = e-mail to the sender, delete it from your computer, and destroy any printed= copy of it. --000000000000fe418b05cf44bc39--