From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 13D8F3B29E for ; Thu, 1 Sep 2022 15:50:14 -0400 (EDT) Received: by mail-wr1-x434.google.com with SMTP id s7so8804176wro.2 for ; Thu, 01 Sep 2022 12:50:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date; bh=q+9zFGV4kEjTD569Tin6/KSMFNoS/3tlBtlztiVIHpw=; b=k/bxEqmbVcOwwOxoFLzq/TTQBZGuz2dksRRdGSPlGi1XIbyR8tdYIOkSOf54MUxIzk 1QRViggpCmyyc/sG/cD2lrPiCA+tSV8TIahSuOde3KiUXYQD5UEDPbbdYhS3EYnde0U0 j3kz6iPljs3+A58MJg5p6p1rYwwWRJ68uLrWBa+eGu/zdXiw+TESfJtUt+0ZM52R65XB LJFrD7Shk/e5H8wdf+QWvk6NtslFJRqJYUWtxKddbarrDjof2rO/nwxeW8ptKMuuNBpI 799VCcOGZAMIscLKRAuDrrsik3tIZTIoxcQo3vozGJg1pIPKQY4HdFWUufQY178t+TMT naSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date; bh=q+9zFGV4kEjTD569Tin6/KSMFNoS/3tlBtlztiVIHpw=; b=vOydW+hqUMeu6TwshbjG87r/FtSIo9Psdnp9XE4z4f12+hD3fwy0dx25Swx9bxi5QD C4MK77WA0sfDKvZXx7OrtvNO5fp5GrlP157gF2++/9qvIHzkTz8C7b9YaOCQnYoKFILX nKy/ZnaklhuCNQedo6qGoXWyyFRsTYZj5GCSKJOUGszuj+kVCWD/TVm08lDn667JCmcK 03B7Jvr8hVvnLIvP6ji8VmErsJJzFmr5kw4MHqilvU6YbOHHbM52EMu8uxcq9daTRkUw W3Ku3LIzgcof+cH+XhjJ4ZRtq8ixLan4YIhBP1Sba8A7u5QDXmR/2jCmCEom4MZRUzQU fzpA== X-Gm-Message-State: ACgBeo3mkU7TSGgSw1bMKKqWbGUiYA5mpvrbhaw+GM/2OH8Dej7z1Ye5 TSUdDKJ8Y2gt4AH90+OaLzhG6vBqGVlgB7o5jo8xP8VYko4= X-Google-Smtp-Source: AA6agR725HdnnAeEUHLdcqSt1c5jyX6RBeYy8SSZC2l0+vUbaMMoaI4MGrFlKRiggJcPZBL5CqVLKv/cVfiI5ELhknM= X-Received: by 2002:adf:f58f:0:b0:226:d216:bcfd with SMTP id f15-20020adff58f000000b00226d216bcfdmr14798326wro.5.1662061813591; Thu, 01 Sep 2022 12:50:13 -0700 (PDT) MIME-Version: 1.0 References: <1661878433.14064713@apps.rackspace.com> In-Reply-To: From: Dave Taht Date: Thu, 1 Sep 2022 12:50:01 -0700 Message-ID: To: "Luis A. Cornejo" Cc: Dave Taht via Starlink Content-Type: multipart/mixed; boundary="000000000000feae0c05e7a2eb63" Subject: Re: [Starlink] starlink upload behavior X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Sep 2022 19:50:15 -0000 --000000000000feae0c05e7a2eb63 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Sep 1, 2022 at 12:24 PM Luis A. Cornejo wrote: > > Dave, > > Did you leave your ingress at 0? I could never find an operating point that was suitable. > How about ack filtering and overhead on egress? Do you mind sharing your = tc qdisc commands? ack-filter works better than ack-filter-aggressive, though I switch back and forth. > Have you run the auto rate script? I used to - but my own usage is principally bound by annoyance at upload speeds, so I reverted to just using ack-filtering and a 6Mbit rate on uploads for the sqm-scripts. See attached. For new subscribers to this list, the genesis of it all was the idea of doing a cerowrt II project targetted at making all of openwrt easily capable of interoperating with starlink's products, and in at least a few ways, superior. Despite pitching this in multiple directions, no funding arrived, and Starlink stopped communicating with us at all... https://docs.google.com/document/d/1rVGC-iNq2NZ0jk4f3IAiVUHz2S9O6P-F3vVZU2y= BYtw/edit#heading=3Dh.qev8j7cst4lc and instead we've been poking at various subprojects in loose formation, off of that list. Notably, the cake-autorate work hit 1.0 with some decent solutions for LTE/5G that I hope more are using, some details on that here: https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379 Given how the starlink network is (d)evolving, and my continued, fervent hope they are upgrading their dishys and downlinks to manage congestion better, I went back to paid work and polishing up the openwrt 22.03 release. (I think we fixed the last major wifi bug a week ago). LibreQos is experiencing a small explosion of popularity, and I hope more small ISPs are leaping on that. When we started developing cake in 2015, XDP, ebpf didn't exist, and htb was too locky to scale much past a gbit, we recently got LibreQos past 10gbits and 5000 60/6 mbit users on really cheap hardware. Next stop, 20Gbit! https://github.com/rchac/LibreQoS I picked up a pretty cool FPGA the other day, as the stop after that is 100Gbits. 6ns per packet is difficult. A couple days worth of dishy cake stats: tc -s qdisc show dev wlp3s0 qdisc cake 818c: root refcnt 2 bandwidth 6Mbit diffserv3 triple-isolate nonat nowash ack-filter split-gso rtt 100.0ms raw overhead 0 Sent 1090433207 bytes 3498702 pkt (dropped 240608, overlimits 2592116 requeues 0) backlog 0b 0p requeues 0 memory used: 68Kb of 4Mb capacity estimate: 6Mbit min/max network layer size: 42 / 1514 min/max overhead-adjusted size: 42 / 1514 average network hdr offset: 14 Bulk Best Effort Voice thresh 375Kbit 6Mbit 1500Kbit target 48.4ms 5.0ms 12.1ms interval 143.4ms 100.0ms 107.1ms pk_delay 0us 3.2ms 384us av_delay 0us 415us 15us sp_delay 0us 3us 2us backlog 0b 0b 0b pkts 0 3732804 6506 bytes 0 1105842362 295688 way_inds 0 254569 0 way_miss 0 32630 84 way_cols 0 0 0 drops 0 46 0 marks 0 160 0 ack_drop 0 240562 0 sp_flows 0 2 0 bk_flows 0 1 0 un_flows 0 0 0 max_len 0 13446 329 quantum 300 300 300 For contrast here's two days worth of stats for a 60Mbit customer of this wisp. Overall we see about a 1% drop rate... qdisc cake c538: parent 7:1218 bandwidth unlimited diffserv4 triple-isolate= nona t nowash ack-filter split-gso rtt 100ms raw overhead 0 Sent 1068282731 bytes 6570868 pkt (dropped 187845, overlimits 0 requeues 0= ) backlog 0b 0p requeues 0 memory used: 292608b of 15140Kb capacity estimate: 0bit min/max network layer size: 60 / 1494 min/max overhead-adjusted size: 60 / 1494 average network hdr offset: 14 Bulk Best Effort Video Voice thresh 0bit 0bit 0bit 0bit target 5ms 5ms 5ms 5ms interval 100ms 100ms 100ms 100ms pk_delay 1.08ms 196us 130us 44us av_delay 26us 4us 3us 2us sp_delay 0us 0us 1us 0us backlog 0b 0b 0b 0b pkts 258 6753105 320 5030 bytes 15480 1081835162 24846 860791 way_inds 0 499764 0 0 way_miss 90 63092 229 309 way_cols 0 0 0 0 drops 0 1062 0 2 marks 0 32 0 0 ack_drop 0 186781 0 0 sp_flows 0 1 0 1 bk_flows 0 1 0 0 un_flows 0 0 0 0 max_len 60 1494 1422 590 quantum 1514 1514 1514 1514 > > -Luis > > On Wed, Aug 31, 2022 at 10:22 AM Dave Taht via Starlink wrote: >> >> I have just been leaving cake at 6Mbit on the upload and that fq and >> control make it much >> more tolerable for my mix of videoconferencing and uploads. That said, >> I hadn't looked >> at the native performance in a while which is markedly better than it >> was a few months ago, at least for this quick test run. >> >> If anyone can make sense of these... for the first 1/3 of the trace >> throughput is low and RTTs are *NICE*, >> the second looks like a sat switch - still nice... then another sat >> switch where I get full upload throughput >> and the latency grows significantly. >> >> If anyone is into packet captures: >> >> http://www.taht.net/~d/starlink-1-auto.cap # starlink through their wifi >> http://www.taht.net/~d/starlink-1-cake6.cap # starlink through their >> wifi, with cake bandwidth 6mbit on mine >> >> Graphs produced with xplot. >> _______________________________________________ >> Starlink mailing list >> Starlink@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/starlink --=20 FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_code= l/ Dave T=C3=A4ht CEO, TekLibre, LLC --000000000000feae0c05e7a2eb63 Content-Type: application/octet-stream; name="wlp3s0.iface.conf" Content-Disposition: attachment; filename="wlp3s0.iface.conf" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_l7jg1rc10 IyBEZWZhdWx0IFNRTSBjb25maWc7IHRoZSB2YXJpYWJsZXMgZGVmaW5lZCBoZXJlIHdpbGwgYmUg YXBwbGllZCB0byBhbGwKIyBpbnRlcmZhY2VzLiBUbyBvdmVycmlkZSB2YWx1ZXMgZm9yIGEgcGFy dGljdWxhciBpbnRlcmZhY2UsIGNvcHkgdGhpcyBmaWxlIHRvCiMgPGRldj4uaWZhY2UuY29uZiAo ZS5nLiwgImV0aDAuaWZhY2UuY29uZiIgZm9yIGV0aDApLgojCiMgV2hlbiB1c2luZyBpZnVwZG93 biwgdGhlIGludGVyZmFjZSBjb25maWcgZmlsZSBuZWVkcyB0byBleGlzdCBmb3Igc3FtLXNjcmlw dHMKIyB0byBiZSBhY3RpdmF0ZWQgb24gdGhhdCBpbnRlcmZhY2UuIEhvd2V2ZXIsIHRoZXNlIGRl ZmF1bHRzIGFyZSBzdGlsbCBhcHBsaWVkLAojIHNvIHRoZSBpbnRlcmZhY2UgY29uZmlnIGNhbiBi ZSBlbXB0eS4KCiMgVXBsaW5rIGFuZCBEb3dubGluayB2YWx1ZXMgYXJlIGluIGticHMKVVBMSU5L PTYwMDAKRE9XTkxJTks9MAojRE9XTkxJTks9NDAwMDAKCiMgU1FNIHJlY2lwZSB0byB1c2UuIEZv ciBtb3JlIGluZm9ybWF0aW9uLCBzZWUgL3Vzci9saWIvc3FtLyouaGVscApTQ1JJUFQ9cGllY2Vf b2ZfY2FrZS5xb3MKCiMgT3B0aW9uYWwvYWR2YW5jZWQgY29uZmlnCgpFTkFCTEVEPTEKUURJU0M9 Y2FrZQoKI0xMQU09dGNfc3RhYgojTElOS0xBWUVSPW5vbmUKI09WRVJIRUFEPTAKI1NUQUJfTVRV PTIwNDcKI1NUQUJfVFNJWkU9NTEyCiNTVEFCX01QVT0wCgojSUxJTUlUPQojRUxJTUlUPQojSVRB UkdFVD0KI0VUQVJHRVQ9CgojIEVDTiBpbmdyZXNzIHJlc3AuIGVncmVzcy4gVmFsdWVzIGFyZSBF Q04gb3IgTk9FQ04uCklFQ049RUNOCkVFQ049RUNOCgojIEV4dHJhIHFkaXNjIG9wdGlvbnMgaW5n cmVzcyByZXNwLiBlZ3Jlc3MKI0lRRElTQ19PUFRTPSIiCkVRRElTQ19PUFRTPSJhY2stZmlsdGVy LWFnZ3Jlc3NpdmUiCgojIENvRGVsIHRhcmdldAojVEFSR0VUPTVtcwoKI1pFUk9fRFNDUF9JTkdS RVNTPTEKI0lHTk9SRV9EU0NQX0lOR1JFU1M9MQoK --000000000000feae0c05e7a2eb63--