From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ej1-x62f.google.com (mail-ej1-x62f.google.com [IPv6:2a00:1450:4864:20::62f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 0D97C3B29D for ; Tue, 13 Sep 2022 15:25:34 -0400 (EDT) Received: by mail-ej1-x62f.google.com with SMTP id a26so1107586ejc.4 for ; Tue, 13 Sep 2022 12:25:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date; bh=e9Asoy5iI0Dy72o7Ur1gNa+MMATRPgO+FXuidTOCNro=; b=UIoK+6bVVcMaeeZa1Qo6hbe+BU8GEjJnQMj7CvksQ/3Dlg0E2RPySqjGPyQR2Nd4V8 ZL2ZYOH4+r1zb55XEf0QyBVB65exMepRE+ZcgoAQrinQibBxl5CEqDkiK5IFnBOekNfa /QbQqGpi9PSN9JUunxhpaZK2qHRyMO0KPoH5U= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date; bh=e9Asoy5iI0Dy72o7Ur1gNa+MMATRPgO+FXuidTOCNro=; b=6nqUYiwLWrwBupXOKUvKO+qrXccoMtS09Vci/zumkOP416mpxiHHQq2wHXpxA/lrAQ a8Kaghf0YEtPwP2iYnTz7w8/L4pTBel/cQr/0/LlMVr3bEnNBO82U0nKug8Fp6rxxVST NW+HeAEUd/93JKO4nKgo+26Bd1e/lLFuf54kNe6U9iEFdV9GKEB06FEaMQP6OTUY9B2n uPBogsxpDAzchidwCHQUI24jcKLjpWDcu6/WK6TSuv/enIn7Wb7PLbPROF8RsvDijiNL VWlrcfhXodN1XBa797o3y4rpQjzQLZ+BeX4BkjeVR6NqS/DxvqTV7SoFj6eUO1IzZrkf V0wg== X-Gm-Message-State: ACgBeo04A6k2WEZSu/T41Xr9WeBIjgSZwcIYbWsLgx9RujObwM+HTj3L V8CG0yGHV/jmO8/1SldFdmGmp7LwPj8ogLvFdE1vLdfwoe0gezON2rFhIIzj87y7DJYe/uYIpJe URPMKpH7ux2Jt5aiXMEsWH4Y33RY5updfM1rhvQY= X-Google-Smtp-Source: AA6agR59ZYXf+PxCOrD1DX6EuGs6m/N8Iz9xwPtQVHq5fLcyCpQgEiKPg8TJTUBzgOubb0lhCzsM769zmCZuGPnG3gY= X-Received: by 2002:a17:907:a051:b0:77a:e136:6ad2 with SMTP id gz17-20020a170907a05100b0077ae1366ad2mr12333939ejc.764.1663097132440; Tue, 13 Sep 2022 12:25:32 -0700 (PDT) MIME-Version: 1.0 References: <95b2dd55-9265-9976-5bd0-52f9a46dd118@candelatech.com> <93d66465-1a70-fef9-7cca-9cb58fe1cad5@candelatech.com> In-Reply-To: <93d66465-1a70-fef9-7cca-9cb58fe1cad5@candelatech.com> From: Bob McMahon Date: Tue, 13 Sep 2022 12:25:21 -0700 Message-ID: To: Ben Greear Cc: Dave Taht , Dave Taht via Starlink , Make-Wifi-fast Content-Type: multipart/alternative; boundary="000000000000ceac2005e893f9a7" Subject: Re: [Starlink] [Make-wifi-fast] RFC: Latency test case text and example report. X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Sep 2022 19:25:34 -0000 --000000000000ceac2005e893f9a7 Content-Type: text/plain; charset="UTF-8" We find iperf2 's --tcp-write-prefetch at some small value can be trusted and eliminates send side bloat. It sets TCP_NOTSENT_LOWAT. This probably should be used generally as send side bloat adds no value to e2e traffic. The iperf 2 bounce-back feature we find as useful too. It has both RTT sampling (read network level RTTs) and app level round trip times in the same sample report. You'll need version 2.1.8 for this. Openwrt doesn't have a maintainer for iperf 2 and supports a very old version. In general, this test seems contrived. Also, probably a good idea to add some variable phase shifters to the rig. Use a 5 branch tree to produce the distance matrices. Anyway, a lot can be done on the RF side beyond attenuators. Pass/fail should be statistically based. Latency should be analyzed at the tails of the distributions. Bob On Tue, Sep 13, 2022 at 12:09 PM Ben Greear via Make-wifi-fast < make-wifi-fast@lists.bufferbloat.net> wrote: > On 9/13/22 11:32 AM, Dave Taht wrote: > > On Tue, Sep 13, 2022 at 9:57 AM Ben Greear > wrote: > >> > >> On 9/13/22 9:12 AM, Dave Taht wrote: > >>> On Tue, Sep 13, 2022 at 8:58 AM Ben Greear > wrote: > >>>> > >>>> On 9/13/22 8:39 AM, Dave Taht wrote: > >>>>> hey, ben, I'm curious if this test made it into TR398? Is it possible > >>>>> to setup some of this or parts of TR398 to run over starlink? > >>>>> > >>>>> I'm also curious as to if any commercial ax APs were testing out > >>>>> better than when you tested about this time last year. I've just > gone > >>>>> through 9 months of pure hell getting openwrt's implementation of the > >>>>> mt76 and ath10k to multiplex a lot better, and making some forward > >>>>> progress again ( > >>>>> https://forum.openwrt.org/t/aql-and-the-ath10k-is-lovely/59002/830 ) > >>>>> and along the way ran into new problems with location scanning and > >>>>> apple's airdrop.... > >>>>> > >>>>> but I just got a batch of dismal results back from the ax210 and > >>>>> mt79... tell me that there's an AP shipping from someone that scales > a > >>>>> bit better? Lie if you must... > >>>> > >>>> An mtk7915 based AP that is running recent owrt did better than > others. > >>>> > >>>> > http://www.candelatech.com/examples/TR-398v2-2022-06-05-08-28-57-6.2.6-latency-virt-sta-new-atf-c/ > >>> > >>> I wanted to be happy, but... tcp... > >>> > >>> > http://www.candelatech.com/examples/TR-398v2-2022-06-05-08-28-57-6.2.6-latency-virt-sta-new-atf-c/chart-31.png > >>> > >>> what's the chipset driving these tests nowadays? > >> > >> That test was done with MTK virtual stations doing the station load > (and multi-gig Eth port > >> sending traffic towards the DUT in download direction). > > > > Openwrt driver or factory? > > I run my own kernel, but it would have been 5.17 plus a bunch of patches > from mtk tree that > owrt uses, plus my own hackings. > > > > > The last major patches for openwrt mt76 wifi landed aug 4, I think. > > There are a few more under test now that the OS is stable. > > > >> My assumption is that much of the TCP latency is very likely caused on > the > >> traffic generator itself, so that is why we measure udp latency for > pass/fail > >> metrics. > > > > I fear a great deal of it is real, on the path, in the DUT. However > > there is a lot in the local stack too. > > > > Here's some things to try. TCP small queues stops being effective (at > > this rate) at oh, 8-12 flows, > > and they start accruing in the stack and look like an RTT inflation. > > A big help is to set TCP_NONSENT_LOWAT to a low value (16k). > > > > sch_fq is actually worse than fq_codel on the driving host as it too > > accrues packets. > > > > Trying out tcp reno, and BBR on this workload might show a difference. > > I wish LEDBAT++ was available for linux... > > I have not much interest in trying to get the traffic generator to report > less TCP latency, > by tuning the traffic generator because whatever it reports, I do not > trust it to not be a > significant part of the over-all end-to-end latency. > > So, better question for me is how to get precise info on TCP latency > through the DUT for > a generic traffic generator. > > We put sequence numbers and time-stamps in our traffic generator payloads, > so we could > use wireshark/tshark captures on Eth and WiFi to detect latency from when > DUT would have received > the pkt on Eth port and transmitted it on WiFi. It would > be...interesting...to take a multi-million packet > capture of 32 stations doing 4 tcp streams each and try to make sense of > that. I don't think > I'd want to spend time on that now, but if you'd like a pair of packet > captures to try it yourself, I'd be happy > to generate them and make them available. > > Or, if you have other ideas for how to test DUT tcp latency under > load/scale without having to overly > trust the packet generator's latency reporting, please let me know. > > > > > > > ... going theoreticall ... > > > > There was some really great work on fractional windows that went into > > google's swift congestion control, this is an earlier paper on it: > > > > https://research.google/pubs/pub49448/ > > > > and a couple really great papers from google and others last week > > from: https://conferences.sigcomm.org/sigcomm/2022/program.html > > > > > >> > >> It would take some special logic, like sniffing eth port and air at > same time, > >> and matching packets by looking at the packet content closely to really > understand DUT TCP latency. > >> I'm not sure that is worth the effort. > > > > Heh. I of course, think it is, as TCP is the dominant protocol on the > > internet... anyway, > > to get a baseline comparison between tcp behaviors, you could just do > > a pure ethernet test, set it > > to what bandwidth you are getting out of this test via cake, and > > measure the tcp rtts that way. It would be nice to know what the test > > does without wifi in the way. > > With no backpressure, I think that test is useless, and with backpressure, > I'm > sure there will be lots of latency on the generator, so again back to > needing a way to > test DUT directly. > > > >> But, assuming we can properly measure slow-speed UDP latency through > DUT, do you still > >> think that it is possible that DUT is causing significantly different > latency to TCP > >> packets? > > > > Absolutely. It's the return path that's mostly at fault - every two > > tcp packets needs an ack, so > > even if you have working mu-mimo for 4 streams, that's 4 txops > > (minimum) that the clients are going to respond on. > > > > std Packet caps of this 32 station tcp test would be useful, and > > aircaps would show how effeciently the clients are responding. A lot > > of stations burn a whole txop on a single ack, then get the rest on > > another.... > > I'll get you some captures next time I have a chance to run this test. > > > No, thank you, for sharing. Can you point at some commercial AP we > > could test that does > > better than this on the tcp test? > > Not that I recall testing. > > Thanks, > Ben > > -- > Ben Greear > Candela Technologies Inc http://www.candelatech.com > > _______________________________________________ > Make-wifi-fast mailing list > Make-wifi-fast@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/make-wifi-fast -- This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it. --000000000000ceac2005e893f9a7 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
We find iperf2's --tcp-write-prefetch at some small value can be trusted= and eliminates send side bloat. It sets TCP_NOTSENT_LOWAT. This probably s= hould be used generally as send side bloat adds no value to e2e traffic.
The iperf 2 bounce-back feature we find as useful too. It has both RTT= sampling (read network level RTTs) and app level round trip times in the s= ame sample report. You'll need version 2.1.8 for this. Openwrt doesn= 9;t have a maintainer for iperf 2 and supports a very old version.

I= n general, this test seems contrived.=C2=A0 Also, probably a good idea to a= dd some variable phase shifters to the rig.=C2=A0 Use a 5 branch tree to pr= oduce the distance matrices.=C2=A0 Anyway, a lot can be done on the RF side= beyond attenuators.

Pass/fail should be statistically based. Latenc= y should be analyzed at the tails of the distributions.

Bob
On Tue, S= ep 13, 2022 at 12:09 PM Ben Greear via Make-wifi-fast <make-wifi-fast@lists.bufferbloat.net= > wrote:
= On 9/13/22 11:32 AM, Dave Taht wrote:
> On Tue, Sep 13, 2022 at 9:57 AM Ben Greear <greearb@candelatech.com> wrote= :
>>
>> On 9/13/22 9:12 AM, Dave Taht wrote:
>>> On Tue, Sep 13, 2022 at 8:58 AM Ben Greear <greearb@candelatech.com&g= t; wrote:
>>>>
>>>> On 9/13/22 8:39 AM, Dave Taht wrote:
>>>>> hey, ben, I'm curious if this test made it into TR= 398? Is it possible
>>>>> to setup some of this or parts of TR398 to run over st= arlink?
>>>>>
>>>>> I'm also curious as to if any commercial ax APs we= re testing out
>>>>> better than when you tested about this time last year.= =C2=A0 I've just gone
>>>>> through 9 months of pure hell getting openwrt's im= plementation of the
>>>>> mt76 and ath10k to multiplex a lot better, and making = some forward
>>>>> progress again (
>>>>> https://forum= .openwrt.org/t/aql-and-the-ath10k-is-lovely/59002/830 )
>>>>> and along the way ran into new problems with location = scanning and
>>>>> apple's airdrop....
>>>>>
>>>>> but I just got a batch of dismal results back from the= ax210 and
>>>>> mt79... tell me that there's an AP shipping from s= omeone that scales a
>>>>> bit better? Lie if you must...
>>>>
>>>> An mtk7915 based AP that is running recent owrt did better= than others.
>>>>
>>>> http://www.candelatech.com/examples/TR-398v2-2022-06-05-08-2= 8-57-6.2.6-latency-virt-sta-new-atf-c/
>>>
>>> I wanted to be happy, but... tcp...
>>>
>>> http://www.candelatech.com/examples/TR-398v2-2022-06= -05-08-28-57-6.2.6-latency-virt-sta-new-atf-c/chart-31.png
>>>
>>> what's the chipset driving these tests nowadays?
>>
>> That test was done with MTK virtual stations doing the station loa= d (and multi-gig Eth port
>> sending traffic towards the DUT in download direction).
>
> Openwrt driver or factory?

I run my own kernel, but it would have been 5.17 plus a bunch of patches fr= om mtk tree that
owrt uses, plus my own hackings.

>
> The last major patches for openwrt mt76 wifi landed aug 4, I think. > There are a few more under test now that the OS is stable.
>
>> My assumption is that much of the TCP latency is very likely cause= d on the
>> traffic generator itself, so that is why we measure udp latency fo= r pass/fail
>> metrics.
>
> I fear a great deal of it is real, on the path, in the DUT. However > there is a lot in the local stack too.
>
> Here's some things to try. TCP small queues stops being effective = (at
> this rate) at oh, 8-12 flows,
> and they start accruing in the stack and look like an RTT inflation. > A big help is to set TCP_NONSENT_LOWAT to a low value (16k).
>
> sch_fq is actually worse than fq_codel on the driving host as it too > accrues packets.
>
> Trying out tcp reno, and BBR on this workload might show a difference.=
> I wish LEDBAT++ was available for linux...

I have not much interest in trying to get the traffic generator to report l= ess TCP latency,
by tuning the traffic generator because whatever it reports, I do not trust= it to not be a
significant part of the over-all end-to-end latency.

So, better question for me is how to get precise info on TCP latency throug= h the DUT for
a generic traffic generator.

We put sequence numbers and time-stamps in our traffic generator payloads, = so we could
use wireshark/tshark captures on Eth and WiFi to detect latency from when D= UT would have received
the pkt on Eth port and transmitted it on WiFi.=C2=A0 It would be...interes= ting...to take a multi-million packet
capture of 32 stations doing 4 tcp streams each and try to make sense of th= at.=C2=A0 I don't think
I'd want to spend time on that now, but if you'd like a pair of pac= ket captures to try it yourself, I'd be happy
to generate them and make them available.

Or, if you have other ideas for how to test DUT tcp latency under load/scal= e without having to overly
trust the packet generator's latency reporting, please let me know.

>
>
> ... going theoreticall ...
>
> There was some really great work on fractional windows that went into<= br> > google's swift congestion control, this is an earlier paper on it:=
>
> https://research.google/pubs/pub49448/
>
> and a couple really great papers from google and others last week
> from: https://conferences.sigcomm.org/= sigcomm/2022/program.html
>
>
>>
>> It would take some special logic, like sniffing eth port and air a= t same time,
>> and matching packets by looking at the packet content closely to r= eally understand DUT TCP latency.
>> I'm not sure that is worth the effort.
>
> Heh. I of course, think it is, as TCP is the dominant protocol on the<= br> > internet... anyway,
> to get a baseline comparison between tcp behaviors, you could just do<= br> > a pure ethernet test, set it
> to what bandwidth you are getting out of this test via cake, and
> measure the tcp rtts that way. It would be nice to know what the test<= br> > does without wifi in the way.

With no backpressure, I think that test is useless, and with backpressure, = I'm
sure there will be lots of latency on the generator, so again back to needi= ng a way to
test DUT directly.


>> But, assuming we can properly measure slow-speed UDP latency throu= gh DUT, do you still
>> think that it is possible that DUT is causing significantly differ= ent latency to TCP
>> packets?
>
> Absolutely. It's the return path that's mostly at fault - ever= y two
> tcp packets needs an ack, so
> even if you have working mu-mimo for 4 streams, that's 4 txops
> (minimum) that the clients are going to respond on.
>
> std Packet caps of this 32 station tcp test would be useful, and
> aircaps would show how effeciently the clients are responding. A lot > of stations burn a whole txop on a single ack, then get the rest on > another....

I'll get you some captures next time I have a chance to run this test.<= br>
> No, thank you, for sharing. Can you point at some commercial AP we
> could test that does
> better than this on the tcp test?

Not that I recall testing.

Thanks,
Ben

--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc=C2=A0 http://www.candelatech.com

_______________________________________________
Make-wifi-fast mailing list
M= ake-wifi-fast@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/make-wif= i-fast

This ele= ctronic communication and the information and any files transmitted with it= , or attached to it, are confidential and are intended solely for the use o= f the individual or entity to whom it is addressed and may contain informat= ion that is confidential, legally privileged, protected by privacy laws, or= otherwise restricted from disclosure to anyone else. If you are not the in= tended recipient or the person responsible for delivering the e-mail to the= intended recipient, you are hereby notified that any use, copying, distrib= uting, dissemination, forwarding, printing, or copying of this e-mail is st= rictly prohibited. If you received this e-mail in error, please return the = e-mail to the sender, delete it from your computer, and destroy any printed= copy of it. --000000000000ceac2005e893f9a7--