From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 4851E3B2A4 for ; Tue, 22 Jan 2019 03:50:51 -0500 (EST) Received: by mail-wr1-x436.google.com with SMTP id r10so26271906wrs.10 for ; Tue, 22 Jan 2019 00:50:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heistp.net; s=google; h=from:mime-version:subject:message-id:date:to; bh=dLkeVvgdiAHyfdlKNapBB2C65a0PudxWi+8VAWJExo8=; b=JSYbabS51T16hRfbrVzRTEOksxzrc32/GdNHD4pYnVYa+rHQovQIcyR/NS0ytHfLR3 TpWv81d94btckzl3NUayfnYMDw3uzk80/tFrPas9bqdMgQ9E6hpI85Vg4iPiO8etM7U1 rkjK+rkiXjpozvAON0E/e/bkVqZOlVGCNNNVme15zfo82uiRjQA5c6g7MA7pZ/xfAWkV qnAay96wCp4lYLl3BDpAhA8bNLsejeL7YvDPs8T+XrhpF+3TIiwYo/frinsDOwgegMI+ 0iPBVTO8ZqDgalWzKzSHClsELz1edEZXwp0h9Ho3uZOMvQW8Uz2PBP73EglrBOzS6YHc WL9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:mime-version:subject:message-id:date:to; bh=dLkeVvgdiAHyfdlKNapBB2C65a0PudxWi+8VAWJExo8=; b=j+3hVUZsPV8FHRltvUDjB+aj/FMmfI65T5cmYZojFvbzX4qGPE7NB8TfD1fVvF1Cnb rrhaHRMAVUazWf6Jjm7vLGvsu06sLDKMLYE/ictwSBpjBIedflJyMDmR9yuv4XiUmvh0 lKAgmowfIJDDftgFt6lRz43V+BVkiVJNW43bLx0ZWntUoQ/Xeta/FlWR+bWGvZHYRDsA ehrj3VbTTlyOq1/rtKLZU7kHsNYBwEjWNEQMzG0yZrGvLrBGBOvJkS4fQwNku21w5Yhf WN4S19xkx5Dm+acJCcvlYEvHd5Wtg96LLRQFsOedjGDoH8+6qT5GsMSH+UqHdO2YWsea xKLg== X-Gm-Message-State: AJcUukeIMIB0chIlg1otdlBa24yiWkznbSdnGv1Qxhs4CyKYLW3oPg85 n93IaZ44ewHMZyncq+yqFeSnLsd6biE= X-Google-Smtp-Source: ALg8bN7eMROGrQZl7BNKtbaoOfbizZFtPT55htMoPBIZ5HQOYyjWSCDXZyMHyG4iX/Csr0eZC/AT8g== X-Received: by 2002:adf:e383:: with SMTP id e3mr31277814wrm.31.1548147050033; Tue, 22 Jan 2019 00:50:50 -0800 (PST) Received: from tron.luk.heistp.net (h-1169.lbcfree.net. [185.193.85.130]) by smtp.gmail.com with ESMTPSA id v4sm43973296wme.6.2019.01.22.00.50.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 22 Jan 2019 00:50:49 -0800 (PST) From: Pete Heist Content-Type: multipart/alternative; boundary="Apple-Mail=_1E0C2E6F-4F14-4CAF-A2BA-0CD23F3925D6" Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Message-Id: <9ECF8538-357C-448F-BE27-40A21A5CC048@heistp.net> Date: Tue, 22 Jan 2019 09:50:48 +0100 To: Make-Wifi-fast X-Mailer: Apple Mail (2.3445.9.1) Subject: [Make-wifi-fast] Ubiquiti fixed framing experience X-BeenThere: make-wifi-fast@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jan 2019 08:50:51 -0000 --Apple-Mail=_1E0C2E6F-4F14-4CAF-A2BA-0CD23F3925D6 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 We=E2=80=99ve been using some new PrismStation 5ACs at my ISP that offer = 5 =E2=80=9CTDD Framing=E2=80=9D options: "Flexible (legacy)=E2=80=9D - traditional airMAX, probably =E2=80=9CFlexible (NEW)=E2=80=9D - newfangled airMAX =E2=80=9CFixed (5 ms)=E2=80=9D - fixed 5 ms framing =E2=80=9CFixed (8 ms)=E2=80=9D - fixed 8 ms framing =E2=80=9CFixed (10 ms)=E2=80=9D - fixed 10 ms framing Since installation last summer we=E2=80=99d been running with =E2=80=9CFix= ed (5 ms)=E2=80=9D with a 33/66 upload/download ratio, because Ubiquiti = touts the scalability of the fixed framing modes. There=E2=80=99s also a = GPS sync option that allows APs to sync with fixed framing, allowing = channel re-use. Sounds nice. But even fixed 5ms frames mean minimum RTTs to the AP of 10ms, and on = higher load, noise, etc, RTTs that can look roughly quantized to 15ms, = 20ms and up. Whereas I=E2=80=99d see 1.3ms mean RTT to the old Rocket = M5, I=E2=80=99d see mean 13ms to the new PrismStation 5AC. ISP members were reporting speedtest.net results = of only up to about 5Mbit during evening load times and 20Mbit in the = middle of the night. But tests to fast.com and = dslreports.com/speedtest , which both = use 8 or more flows by default, could show 40-50Mbit at the very same = time that speedtest.net was showing 20Mbit. = Sounds like a TCP window scaling problem. Meanwhile, bi-directional = throughput tests straight to the AP with Ubiquiti=E2=80=99s web UI speed = test looked great (180 Mbit!!) One morning I decided to try =E2=80=9CFlexible (NEW)=E2=80=9D framing. = Right away, ping to the AP dropped to mean 6.5ms with far less = variation, and late night speedtest.net results = jumped to 80Mbit, limited mostly by our backhaul uplink. During the day, = users reported =E2=80=9Cmy speed over doubled=E2=80=9D and I could feel = the latency improvement. SmokePing to the AP agreed: https://www.heistp.net/downloads/ff_smokeping.png = (on which 3am = morning did we switch to flex framing?) IRTT to an Internet host showed large drops in send delay and send IPDV = (less so on the receive path): https://www.heistp.net/downloads/ff_send_delay.png = https://www.heistp.net/downloads/ff_send_ipdv.png = Then I found out we=E2=80=99re not the only ones: = https://community.ubnt.com/t5/airMAX-AC/New-Flexible-framing-and-huge-impr= ovements-in-TCP-latency/td-p/2385132 = Maybe fixed framing is useful in urban environments with hundreds of = clients, but with two clients on our sector it was rather disastrous and = led to a few months of bad Internet until we figured it out. 10ms chunks = of discarded time can really mean a lot. I wish we could try OpenWRT here with airtime fairness for comparison, = but AirControl is too engrained as an administrative tool, so it=E2=80=99s= tough to switch. And the AP is on top of a tower, on top of a castle on = a hill, so it doesn=E2=80=99t get worked on often. Anyway that=E2=80=99s my story, it=E2=80=99s 2019 and it seems that = vendors are still not thinking about what latency means, even when it = can start to affect TCP... Pete --Apple-Mail=_1E0C2E6F-4F14-4CAF-A2BA-0CD23F3925D6 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 We=E2=80=99ve been = using some new PrismStation 5ACs at my ISP that offer 5 =E2=80=9CTDD = Framing=E2=80=9D options:

"Flexible (legacy)=E2=80=9D - traditional airMAX, = probably
=E2=80=9CFlexible (NEW)=E2=80=9D - = newfangled airMAX
=E2=80=9CFixed (5 ms)=E2=80=9D - = fixed 5 ms framing
=E2=80=9CFixed (8 ms)=E2=80=9D - = fixed 8 ms framing
=E2=80=9CFixed (10 ms)=E2=80=9D = - fixed 10 ms framing

Since installation last summer we=E2=80=99d been running with = =E2=80=9CFixed (5 ms)=E2=80=9D with a 33/66 upload/download ratio, = because Ubiquiti touts the scalability of the fixed framing modes. = There=E2=80=99s also a GPS sync option that allows APs to sync with = fixed framing, allowing channel re-use. Sounds nice.

But even fixed 5ms = frames mean minimum RTTs to the AP of 10ms, and on higher load, noise, = etc, RTTs that can look roughly quantized to 15ms, 20ms and up. Whereas = I=E2=80=99d see 1.3ms mean RTT to the old Rocket M5, I=E2=80=99d see = mean 13ms to the new PrismStation 5AC.

ISP members were reporting speedtest.net results = of only up to about 5Mbit during evening load times and 20Mbit in the = middle of the night. But tests to fast.com and dslreports.com/speedtest, which both use 8 or more flows = by default, could show 40-50Mbit at the very same time that speedtest.net was = showing 20Mbit. Sounds like a TCP window scaling problem. Meanwhile, = bi-directional throughput tests straight to the AP with Ubiquiti=E2=80=99s= web UI speed test looked great (180 Mbit!!)

One morning I decided to try = =E2=80=9CFlexible (NEW)=E2=80=9D framing. Right away, ping to the AP = dropped to mean 6.5ms with far less variation, and late night speedtest.net results = jumped to 80Mbit, limited mostly by our backhaul uplink. During the day, = users reported =E2=80=9Cmy speed over doubled=E2=80=9D and I could feel = the latency improvement. SmokePing to the AP agreed:

https://www.heistp.net/downloads/ff_smokeping.png (on = which 3am morning did we switch to flex framing?)
IRTT to an Internet host showed large = drops in send delay and send IPDV (less so on the receive = path):

https://www.heistp.net/downloads/ff_send_delay.png

https://www.heistp.net/downloads/ff_send_ipdv.png

Then I found out = we=E2=80=99re not the only ones:

https://community.ubnt.com/t5/airMAX-AC/New-Flexible-framing-an= d-huge-improvements-in-TCP-latency/td-p/2385132

Maybe fixed framing is = useful in urban environments with hundreds of clients, but with two = clients on our sector it was rather disastrous and led to a few months = of bad Internet until we figured it out. 10ms chunks of discarded time = can really mean a lot.

I wish we could try OpenWRT here with airtime fairness for = comparison, but AirControl is too engrained as an administrative tool, = so it=E2=80=99s tough to switch. And the AP is on top of a tower, on top = of a castle on a hill, so it doesn=E2=80=99t get worked on = often.

Anyway = that=E2=80=99s my story, it=E2=80=99s 2019 and it seems that vendors are = still not thinking about what latency means, even when it can start to = affect TCP...

Pete

= --Apple-Mail=_1E0C2E6F-4F14-4CAF-A2BA-0CD23F3925D6--