[Starlink] some router hardware we use, stats, and test suites

[Dave Taht]

One thing that’s not well understood about running the internet at higher rates is that above 40Mbit, most of the bufferbloat shifts to the wifi. eero’s prior to wifi 6 products had a good implementation of fq_codel throughout, in particular. gfiber still does. 1000s of other products shipping today have this in the 802.11ac and prior wifi. 

For the wifi device drivers that support fq_codel (ath9k, ath10k, mt76, iwl(intel)), you can poll
the “aqm” statistics for drops and rfc3168 ecn marks by looking at the “aqm” files in, essentially, 

find /sys/kernel/debug/iee*/phy*/ -name aqm

Given all the interest in pretty plots around here, I’d hope we could
start showing those stats to users also. Being partially blind, I find the color scheme y’all use so far as almost impossible to read, but please carry on….

Also useful (on ath9k) are the xmit and rc_stats files. retransmits and AC queue usage are especially interesting. Not a lot of people deeply understand wifi rate control, the best paper on “minstrel” I know of is here: http://blog.cerowrt.org/post/minstrel/

Our reference routers are the venerable wndr3800 and wndr3700 series from netgear. We have several that have been up for 7+ years minus some power failures, and they are entirely open source top to bottom. You can still get ‘em off of ebay!

We use ubuntu linux and openwrt as our main development platforms. We really need to add more pi4s to our mix. 

To drive multi-user wifi tests, we also tend to use old discarded laptops off of ebay (the lenovo R61i has a wifi card that only does 802.11a and thus is a good, simple test of 20Mbit wifi performance), a bunch of random wifi products from apple, google, and others, the pcengines apu2 with various wifi cards, several different qotom products,  evenroute v4 pro (unreleased as yet), the edgerouter X, the archer c7v2 and the turris omnia. 

The omnia is interesting because it has a powerful cpu and few offloads, as well as an implementation of unbound rather than dnsmasq and a routing protocol. 

After coping with qualcomm for too many years[1], most of our still out of tree fq_codel improvements like ack-filtering have shifted over to the mediatek mt76 products. Mediatek may be #3, but they try harder. An example off the shelf mt76 802.11x router you can reflash with the current openwrt release candidate is here:

https://www.amazon.com/gp/product/B08LMQLG7X/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1

We hope to slam this and a turris omnia into one of our starlink testbeds on the 25th. Hope is also several more of our volunteers will have sqm and cake up and running so we can resume p2p multi-user/multi-client ipv6 testing on a variety of fronts.

There are some really heavy duty tests left to run out of the cake, fq_codel, and other test suites.

these are linked to off the relevant cake and fq_codel papers on https://bufferbloat-and-beyond.net/

as for more recent work (there have been 4 years of development since we published that book and folded all the apis into the linux kernel), repeating the enormous test series' of the “L4S” vs “SCE” battle over high fidelity lossless congestion signaling, on starlinks network, is on my mind, but we are running low on resources to set all that up. 

Ironically I am on neither side of that debate, as I think RFC3168 is
“good enough”, and easiest to deploy more fully. I mildly favor the SCE side as it’s fully backward compatible with RFC3168 which is what’s in fq-codel, cake, and fq-pie, and after 7 years of deployment rather hard to call back, but I do expect surprising and interesting results from testing a real world 40ms rtt LEO network with the test suites we have whenever we can get around to it. 

https://github.com/heistp/l4s-tests

IMHO: A little lossless congestion control is nice, but too much, not so much.


[1] A few months back we had a pretty good meeting with one of QCA’s subcontractors where we showed them their current driver design for the ath11k was going wrong.