On Thu, Sep 1, 2022 at 12:24 PM Luis A. Cornejo wrote: > > Dave, > > Did you leave your ingress at 0? I could never find an operating point that was suitable. > How about ack filtering and overhead on egress? Do you mind sharing your tc qdisc commands? ack-filter works better than ack-filter-aggressive, though I switch back and forth. > Have you run the auto rate script? I used to - but my own usage is principally bound by annoyance at upload speeds, so I reverted to just using ack-filtering and a 6Mbit rate on uploads for the sqm-scripts. See attached. For new subscribers to this list, the genesis of it all was the idea of doing a cerowrt II project targetted at making all of openwrt easily capable of interoperating with starlink's products, and in at least a few ways, superior. Despite pitching this in multiple directions, no funding arrived, and Starlink stopped communicating with us at all... https://docs.google.com/document/d/1rVGC-iNq2NZ0jk4f3IAiVUHz2S9O6P-F3vVZU2yBYtw/edit#heading=h.qev8j7cst4lc and instead we've been poking at various subprojects in loose formation, off of that list. Notably, the cake-autorate work hit 1.0 with some decent solutions for LTE/5G that I hope more are using, some details on that here: https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379 Given how the starlink network is (d)evolving, and my continued, fervent hope they are upgrading their dishys and downlinks to manage congestion better, I went back to paid work and polishing up the openwrt 22.03 release. (I think we fixed the last major wifi bug a week ago). LibreQos is experiencing a small explosion of popularity, and I hope more small ISPs are leaping on that. When we started developing cake in 2015, XDP, ebpf didn't exist, and htb was too locky to scale much past a gbit, we recently got LibreQos past 10gbits and 5000 60/6 mbit users on really cheap hardware. Next stop, 20Gbit! https://github.com/rchac/LibreQoS I picked up a pretty cool FPGA the other day, as the stop after that is 100Gbits. 6ns per packet is difficult. A couple days worth of dishy cake stats: tc -s qdisc show dev wlp3s0 qdisc cake 818c: root refcnt 2 bandwidth 6Mbit diffserv3 triple-isolate nonat nowash ack-filter split-gso rtt 100.0ms raw overhead 0 Sent 1090433207 bytes 3498702 pkt (dropped 240608, overlimits 2592116 requeues 0) backlog 0b 0p requeues 0 memory used: 68Kb of 4Mb capacity estimate: 6Mbit min/max network layer size: 42 / 1514 min/max overhead-adjusted size: 42 / 1514 average network hdr offset: 14 Bulk Best Effort Voice thresh 375Kbit 6Mbit 1500Kbit target 48.4ms 5.0ms 12.1ms interval 143.4ms 100.0ms 107.1ms pk_delay 0us 3.2ms 384us av_delay 0us 415us 15us sp_delay 0us 3us 2us backlog 0b 0b 0b pkts 0 3732804 6506 bytes 0 1105842362 295688 way_inds 0 254569 0 way_miss 0 32630 84 way_cols 0 0 0 drops 0 46 0 marks 0 160 0 ack_drop 0 240562 0 sp_flows 0 2 0 bk_flows 0 1 0 un_flows 0 0 0 max_len 0 13446 329 quantum 300 300 300 For contrast here's two days worth of stats for a 60Mbit customer of this wisp. Overall we see about a 1% drop rate... qdisc cake c538: parent 7:1218 bandwidth unlimited diffserv4 triple-isolate nona t nowash ack-filter split-gso rtt 100ms raw overhead 0 Sent 1068282731 bytes 6570868 pkt (dropped 187845, overlimits 0 requeues 0) backlog 0b 0p requeues 0 memory used: 292608b of 15140Kb capacity estimate: 0bit min/max network layer size: 60 / 1494 min/max overhead-adjusted size: 60 / 1494 average network hdr offset: 14 Bulk Best Effort Video Voice thresh 0bit 0bit 0bit 0bit target 5ms 5ms 5ms 5ms interval 100ms 100ms 100ms 100ms pk_delay 1.08ms 196us 130us 44us av_delay 26us 4us 3us 2us sp_delay 0us 0us 1us 0us backlog 0b 0b 0b 0b pkts 258 6753105 320 5030 bytes 15480 1081835162 24846 860791 way_inds 0 499764 0 0 way_miss 90 63092 229 309 way_cols 0 0 0 0 drops 0 1062 0 2 marks 0 32 0 0 ack_drop 0 186781 0 0 sp_flows 0 1 0 1 bk_flows 0 1 0 0 un_flows 0 0 0 0 max_len 60 1494 1422 590 quantum 1514 1514 1514 1514 > > -Luis > > On Wed, Aug 31, 2022 at 10:22 AM Dave Taht via Starlink wrote: >> >> I have just been leaving cake at 6Mbit on the upload and that fq and >> control make it much >> more tolerable for my mix of videoconferencing and uploads. That said, >> I hadn't looked >> at the native performance in a while which is markedly better than it >> was a few months ago, at least for this quick test run. >> >> If anyone can make sense of these... for the first 1/3 of the trace >> throughput is low and RTTs are *NICE*, >> the second looks like a sat switch - still nice... then another sat >> switch where I get full upload throughput >> and the latency grows significantly. >> >> If anyone is into packet captures: >> >> http://www.taht.net/~d/starlink-1-auto.cap # starlink through their wifi >> http://www.taht.net/~d/starlink-1-cake6.cap # starlink through their >> wifi, with cake bandwidth 6mbit on mine >> >> Graphs produced with xplot. >> _______________________________________________ >> Starlink mailing list >> Starlink@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/starlink -- FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/ Dave Täht CEO, TekLibre, LLC