Re: [Starlink] starlink upload behavior

[Dave Taht <dave.taht@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 6958 bytes --]

On Thu, Sep 1, 2022 at 12:24 PM Luis A. Cornejo
<luis.a.cornejo@gmail.com> wrote:
>
> Dave,
>
> Did you leave your ingress at 0?

I could never find an operating point that was suitable.

> How about ack filtering and overhead on egress? Do you mind sharing your tc qdisc commands?

ack-filter works better than ack-filter-aggressive, though I switch
back and forth.

> Have you run the auto rate script?

I used to - but my own usage is principally bound by annoyance at
upload speeds, so I reverted to just using ack-filtering and a 6Mbit
rate on uploads for the sqm-scripts. See attached.

For new subscribers to this list, the genesis of it all was the idea
of doing a cerowrt II project
targetted at making all of openwrt easily capable of interoperating
with starlink's products, and
in at least a few ways, superior. Despite pitching this in multiple
directions, no funding arrived, and Starlink stopped communicating
with us at all...

https://docs.google.com/document/d/1rVGC-iNq2NZ0jk4f3IAiVUHz2S9O6P-F3vVZU2yBYtw/edit#heading=h.qev8j7cst4lc

and instead we've been poking at various subprojects in loose
formation, off of that list. Notably, the cake-autorate work hit 1.0
with some decent solutions for LTE/5G that I hope more are using, some
details on that here:
https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379

Given how the starlink network is (d)evolving, and my continued,
fervent hope they are upgrading their dishys and downlinks to manage
congestion better, I went back to paid work and polishing up the
openwrt 22.03 release. (I think we fixed the last major wifi bug a
week ago).

LibreQos is experiencing a small explosion of popularity, and I hope
more small ISPs are leaping on that. When we started developing cake
in 2015, XDP, ebpf didn't exist, and htb was too locky to scale much
past a gbit, we recently got LibreQos past 10gbits and 5000 60/6 mbit
users on really cheap hardware. Next stop, 20Gbit!

https://github.com/rchac/LibreQoS

I picked up a pretty cool FPGA the other day, as the stop after that
is 100Gbits. 6ns per packet is difficult.

A couple days worth of dishy cake stats:

tc -s qdisc show dev wlp3s0
qdisc cake 818c: root refcnt 2 bandwidth 6Mbit diffserv3
triple-isolate nonat nowash ack-filter split-gso rtt 100.0ms raw
overhead 0
 Sent 1090433207 bytes 3498702 pkt (dropped 240608, overlimits 2592116
requeues 0)
 backlog 0b 0p requeues 0
 memory used: 68Kb of 4Mb
 capacity estimate: 6Mbit
 min/max network layer size:           42 /    1514
 min/max overhead-adjusted size:       42 /    1514
 average network hdr offset:           14

                   Bulk  Best Effort        Voice
  thresh        375Kbit        6Mbit     1500Kbit
  target         48.4ms        5.0ms       12.1ms
  interval      143.4ms      100.0ms      107.1ms
  pk_delay          0us        3.2ms        384us
  av_delay          0us        415us         15us
  sp_delay          0us          3us          2us
  backlog            0b           0b           0b
  pkts                0      3732804         6506
  bytes               0   1105842362       295688
  way_inds            0       254569            0
  way_miss            0        32630           84
  way_cols            0            0            0
  drops               0           46            0
  marks               0          160            0
  ack_drop            0       240562            0
  sp_flows            0            2            0
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len             0        13446          329
  quantum           300          300          300

For contrast here's two days worth of stats for a 60Mbit customer of
this wisp. Overall we see about a 1% drop rate...

qdisc cake c538: parent 7:1218 bandwidth unlimited diffserv4 triple-isolate nona
t nowash ack-filter split-gso rtt 100ms raw overhead 0
 Sent 1068282731 bytes 6570868 pkt (dropped 187845, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 292608b of 15140Kb
 capacity estimate: 0bit
 min/max network layer size:           60 /    1494
 min/max overhead-adjusted size:       60 /    1494
 average network hdr offset:           14

                   Bulk  Best Effort        Video        Voice
  thresh           0bit         0bit         0bit         0bit
  target            5ms          5ms          5ms          5ms
  interval        100ms        100ms        100ms        100ms
  pk_delay       1.08ms        196us        130us         44us
  av_delay         26us          4us          3us          2us
  sp_delay          0us          0us          1us          0us
  backlog            0b           0b           0b           0b
  pkts              258      6753105          320         5030
  bytes           15480   1081835162        24846       860791
  way_inds            0       499764            0            0
  way_miss           90        63092          229          309
  way_cols            0            0            0            0
  drops               0         1062            0            2
  marks               0           32            0            0
  ack_drop            0       186781            0            0
  sp_flows            0            1            0            1
  bk_flows            0            1            0            0
  un_flows            0            0            0            0
  max_len            60         1494         1422          590
  quantum          1514         1514         1514         1514



>
> -Luis
>
> On Wed, Aug 31, 2022 at 10:22 AM Dave Taht via Starlink <starlink@lists.bufferbloat.net> wrote:
>>
>> I have just been leaving cake at 6Mbit on the upload and that fq and
>> control make it much
>> more tolerable for my mix of videoconferencing and uploads. That said,
>> I hadn't looked
>> at the native performance in a while which is markedly better than it
>> was a few months ago, at least for this quick test run.
>>
>> If anyone can make sense of these... for the first 1/3 of the trace
>> throughput is low and RTTs are *NICE*,
>> the second looks like a sat switch - still nice... then another sat
>> switch where I get full upload throughput
>> and the latency grows significantly.
>>
>> If anyone is into packet captures:
>>
>> http://www.taht.net/~d/starlink-1-auto.cap # starlink through their wifi
>> http://www.taht.net/~d/starlink-1-cake6.cap # starlink through their
>> wifi, with cake bandwidth 6mbit on mine
>>
>> Graphs produced with xplot.
>> _______________________________________________
>> Starlink mailing list
>> Starlink@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/starlink



-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

[-- Attachment #2: wlp3s0.iface.conf --]
[-- Type: application/octet-stream, Size: 999 bytes --]

# Default SQM config; the variables defined here will be applied to all
# interfaces. To override values for a particular interface, copy this file to
# <dev>.iface.conf (e.g., "eth0.iface.conf" for eth0).
#
# When using ifupdown, the interface config file needs to exist for sqm-scripts
# to be activated on that interface. However, these defaults are still applied,
# so the interface config can be empty.

# Uplink and Downlink values are in kbps
UPLINK=6000
DOWNLINK=0
#DOWNLINK=40000

# SQM recipe to use. For more information, see /usr/lib/sqm/*.help
SCRIPT=piece_of_cake.qos

# Optional/advanced config

ENABLED=1
QDISC=cake

#LLAM=tc_stab
#LINKLAYER=none
#OVERHEAD=0
#STAB_MTU=2047
#STAB_TSIZE=512
#STAB_MPU=0

#ILIMIT=
#ELIMIT=
#ITARGET=
#ETARGET=

# ECN ingress resp. egress. Values are ECN or NOECN.
IECN=ECN
EECN=ECN

# Extra qdisc options ingress resp. egress
#IQDISC_OPTS=""
EQDISC_OPTS="ack-filter-aggressive"

# CoDel target
#TARGET=5ms

#ZERO_DSCP_INGRESS=1
#IGNORE_DSCP_INGRESS=1