Re: [Starlink] RFC: Latency test case text and example report.

[Dave Taht <dave.taht@gmail.com>]

On Tue, Sep 13, 2022 at 8:58 AM Ben Greear <greearb@candelatech.com> wrote:
>
> On 9/13/22 8:39 AM, Dave Taht wrote:
> > hey, ben, I'm curious if this test made it into TR398? Is it possible
> > to setup some of this or parts of TR398 to run over starlink?
> >
> > I'm also curious as to if any commercial ax APs were testing out
> > better than when you tested about this time last year.  I've just gone
> > through 9 months of pure hell getting openwrt's implementation of the
> > mt76 and ath10k to multiplex a lot better, and making some forward
> > progress again (
> > https://forum.openwrt.org/t/aql-and-the-ath10k-is-lovely/59002/830 )
> > and along the way ran into new problems with location scanning and
> > apple's airdrop....
> >
> > but I just got a batch of dismal results back from the ax210 and
> > mt79... tell me that there's an AP shipping from someone that scales a
> > bit better? Lie if you must...
>
> An mtk7915 based AP that is running recent owrt did better than others.
>
> http://www.candelatech.com/examples/TR-398v2-2022-06-05-08-28-57-6.2.6-latency-virt-sta-new-atf-c/

I wanted to be happy, but... tcp...

http://www.candelatech.com/examples/TR-398v2-2022-06-05-08-28-57-6.2.6-latency-virt-sta-new-atf-c/chart-31.png

what's the chipset driving these tests nowadays?

> The test was at least tentatively accepted into tr398v3, but I don't think anyone other than ourselves has implemented
> or tested it.  I think the pass/fail will need to be adjusted to make it easier to pass.  Some APs were showing
> multiple seconds of latency, so maybe a few hundred MS is really OK.
>
> The test should be able to run over WAN if desired, though it would take a bit
> of extra setup to place an upstream LANforge endpoint on a cloud VM.
>
> If someone@spacex wants to run this test, please contact me off list and we can help
> make it happen.
>
> Thanks,
> Ben
>
> >
> > On Sun, Sep 26, 2021 at 2:59 PM Ben Greear <greearb@candelatech.com> wrote:
> >>
> >> I have been working on a latency test that I hope can be included in the TR398 issue 3
> >> document.  It is based somewhat on Toke's paper on buffer bloat and latency testing,
> >> with a notable change that I'm doing this on 32 stations in part of the test.
> >>
> >> I implemented this test case, and an example run against an enterprise grade AX AP
> >> is here.  There could still be bugs in my implementation, but I think it is at least
> >> close to correct:
> >>
> >> http://www.candelatech.com/examples/tr398v3-latency-report.pdf
> >>
> >> TLDR:  Runs OK with single station, but sees 1+second one-way latency with 32 stations and high load, and UDP often
> >>     is not able to see any throughput at all, I guess due to too many packets being lost
> >>     or something.  I hope to run against some cutting-edge OpenWRT APs soon.
> >>
> >> One note on TCP Latency:  This is time to transmit a 64k chunk of data over TCP, not a single
> >> frame.
> >>
> >> My testbed used 32 Intel ax210 radios as stations in this test.
> >>
> >> I am interested in feedback from this list if anyone has opinions.
> >>
> >> Here is text of the test case:
> >>
> >> The Latency test intends to verify latency under low, high, and maximum AP traffic load, with
> >> 1 and 32 stations. Traffic load is 4 bi-directional TCP streams for each station, plus a
> >> low speed UDP connection to probe latency.
> >>
> >> Test Procedure
> >>
> >> DUT should be configured for 20Mhz on 2.4Ghz and 80Mhz on 5Ghz and stations should use
> >> two spatial streams.
> >>
> >> 1: For each combination of:  2.4Ghz N, 5Ghz AC, 2.4Ghz AX, 5Ghz AX:
> >>
> >> 2: Configure attenuators to emulate 2-meter distance between stations and AP.
> >>
> >> 3: Create 32 stations and allow one to associate with the DUT.  The other 31 are admin-down.
> >>
> >> 4: Create AP to Station (download) TCP stream, and run for 120 seconds, recoard
> >>      throughput as 'maximum_load'.  Stop this connection.
> >>
> >> 5: Calculate offered_load as 1% of maximum_load.
> >>
> >> 6: Create 4 TCP streams on each active station, each configured for Upload and Download rate of
> >>      offered_load / (4 * active_station_count * 2).
> >>
> >> 6: Create 1 UDP stream on each active station, configured for 56kbps traffic Upload and 56kbps traffic Download.
> >>
> >> 7: Start all TCP and UDP connections.  Wait 30 seconds to let traffic settle.
> >>
> >> 8: Every 10 seconds for 120 seconds, record one-way download latency over the last 10 seconds for each UDP connection.  Depending on test
> >>      equipment features, this may mean you need to start/stop the UDP every 10 seconds or clear the UDP connection
> >>      counters.
> >>
> >> 9: Calculate offered_load as 70% of maximum_load, and repeat steps 6 - 9 inclusive.
> >>
> >> 10: Calculate offered_load as 125% of maximum_load, and repeat steps 6 - 9 inclusive.
> >>
> >> 11: Allow the other 31 stations to associate, and repeat steps 5 - 11 inclusive with all 32 stations active.
> >>
> >>
> >> Pass/Fail Criteria
> >>
> >> 1: For each test configuration running at 1% of maximum load:  Average of all UDP latency samples must be less than 10ms.
> >> 2: For each test configuration running at 1% of maximum load:  Maximum of all UDP latency samples must be less than 20ms.
> >> 3: For each test configuration running at 70% of maximum load:  Average of all UDP latency samples must be less than 20ms.
> >> 4: For each test configuration running at 70% of maximum load:  Maximum of all UDP latency samples must be less than 40ms.
> >> 5: For each test configuration running at 125% of maximum load:  Average of all UDP latency samples must be less than 50ms.
> >> 6: For each test configuration running at 125% of maximum load:  Maximum of all UDP latency samples must be less than 100ms.
> >> 7: For each test configuration: Each UDP connection upload throughput must be at least 1/2 of requested UDP speed for final 10-second test interval.
> >> 8: For each test configuration: Each UDP connection download throughput must be at least 1/2 of requested UDP speed for final 10-second test interval.
> >>
> >>
> >> --
> >> Ben Greear <greearb@candelatech.com>
> >> Candela Technologies Inc  http://www.candelatech.com
> >> _______________________________________________
> >> Starlink mailing list
> >> Starlink@lists.bufferbloat.net
> >> https://lists.bufferbloat.net/listinfo/starlink
> >
> >
> >
>
>
> --
> Ben Greear <greearb@candelatech.com>
> Candela Technologies Inc  http://www.candelatech.com
>


-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC