From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from sender4-op-o12.zoho.com (sender4-op-o12.zoho.com [136.143.188.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 57B303B29E for ; Fri, 4 Nov 2022 16:50:37 -0400 (EDT) ARC-Seal: i=1; a=rsa-sha256; t=1667595026; cv=none; d=zohomail.com; s=zohoarc; b=BIfF+OZUAmuO2z9Nd7fqAkEK6ACW4WrwUAK9jCQq8GFJNgtYsavKP3ELVvVDJ7RoKP/eeYrWkQW2OSshBUpQ/jwA83cgyKty65iKqEmuyeUWYg/HnFwfTN4dXUK134cNpEMz3q4kwOrqT9h32IrpjcfwWL2wKArTcTp20gQ6NlA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1667595026; h=Content-Type:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=hTJXm3Db+WQjbyrtglWVDIgePYJqLwfWTyO2aIR+JRY=; b=YDOjz6CXJkoHTldYZA4MUR6QtJ9ZsIX/Dl3Tj0H9TNiGvdilc4lTVvPA5gSvKVFjIu/UxqwpBHER3yu4q6oM9jfgfs/PN66c2bwdHT2ZwiLHk8y5i5ufuFEHVSxY4AGLSnLRcUeB1ju5u4lqmpHxvzys1pyNQ79+bJAGaPGUC+o= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=jonathanfoulkes.com; spf=pass smtp.mailfrom=jf@jonathanfoulkes.com; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1667595026; s=zoho; d=jonathanfoulkes.com; i=jf@jonathanfoulkes.com; h=From:From:Message-Id:Message-Id:Content-Type:Mime-Version:Subject:Subject:Date:Date:In-Reply-To:Cc:Cc:To:To:References:Reply-To; bh=hTJXm3Db+WQjbyrtglWVDIgePYJqLwfWTyO2aIR+JRY=; b=LRhk+b9CwILi5+evJjqHKblBki7PDvL3MwU3C3TGNYZ36GivCSG8JKH/rQBCHMYB AfbBR2SEYqX7N2vL3bUbiyFQtV87OEa3QA1ZpH1Wpm85ai3krqhzkcrHEolIg3GJS48 Y94QSQ+MQyOd9A5mhFOnbKfer/6FXlvzww+9bdEU= Received: from smtpclient.apple (h38.249.20.98.dynamic.ip.windstream.net [98.20.249.38]) by mx.zohomail.com with SMTPS id 166759502546541.0647967601584; Fri, 4 Nov 2022 13:50:25 -0700 (PDT) From: "jf@jonathanfoulkes.com" Message-Id: <18106767-7105-44EB-8E9B-C431D69544DD@jonathanfoulkes.com> Content-Type: multipart/alternative; boundary="Apple-Mail=_0CA51065-E806-4ACF-82EC-E500012D6863" Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.200.110.1.12\)) Date: Fri, 4 Nov 2022 16:50:13 -0400 In-Reply-To: <49CC4E18-2165-4027-BA1D-B3C998BBF54D@jonathanfoulkes.com> Cc: Rpm To: Christoph Paasch References: <49CC4E18-2165-4027-BA1D-B3C998BBF54D@jonathanfoulkes.com> X-Mailer: Apple Mail (2.3731.200.110.1.12) X-ZohoMailClient: External Subject: Re: [Rpm] Changes to RPM calculation for MacOS Ventura? X-BeenThere: rpm@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: revolutions per minute - a new metric for measuring responsiveness List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Nov 2022 20:50:37 -0000 --Apple-Mail=_0CA51065-E806-4ACF-82EC-E500012D6863 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 As I mentioned, here are the results of running with ECN enabled vs = disabled. First, we retest with ECN enabled at the router, the SQM instance and = fully enabled on the MacOS (see a prior post with the settings). This is a DOCSIS 3.0 line at 300Mbps down and 25Mbps Up, router is an = x86 i5-4200 running OpenWRT 22.03 with layer_cake QoS. =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D Uplink capacity: 20.971 Mbps (Accuracy: High) Downlink capacity: 267.643 Mbps (Accuracy: High) Uplink Responsiveness: Medium (550 RPM) (Accuracy: High) Downlink Responsiveness: Medium (900 RPM) (Accuracy: High) Idle Latency: 21.417 milli-seconds (Accuracy: High) Interface: en6 Uplink bytes transferred: 24.359 MB Downlink bytes transferred: 298.730 MB Uplink Flow count: 12 Downlink Flow count: 12 Start: 11/4/22, 3:41:30 PM End: 11/4/22, 3:41:49 PM OS Version: Version 13.0 (Build 22A380) Then I disabled ECN support on the router, the sqm instance, and the = MacOS, then re-ran a couple of times, and this is a representative = result: =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D Uplink capacity: 20.961 Mbps (Accuracy: High) Downlink capacity: 255.146 Mbps (Accuracy: High) Uplink Responsiveness: Medium (475 RPM) (Accuracy: High) Downlink Responsiveness: Medium (591 RPM) (Accuracy: High) Idle Latency: 23.583 milli-seconds (Accuracy: High) Interface: en6 Uplink bytes transferred: 24.593 MB Downlink bytes transferred: 278.096 MB Uplink Flow count: 12 Downlink Flow count: 12 Start: 11/4/22, 4:11:59 PM End: 11/4/22, 4:12:18 PM OS Version: Version 13.0 (Build 22A380) The absence of ECN does have an impact, most significantly on download, = where it drops from 900 to 591 RPM My takeaway is that ECN does help with responsiveness overall. What I=E2=80=99m still puzzled by here is the bloat and why changes to = QoS settings do not make a difference. I have questions about how the lag within a bulk flow is a factor in the = calculation, but I=E2=80=99ll first go read the latest version of the = document. In case I missed it, is there a discussion thread someone can point me = to where this change is mentioned? Thanks, Jonathan > On Nov 3, 2022, at 6:09 PM, jf--- via Rpm = wrote: >=20 > Hi Christoph, >=20 > Thanks for the reply, it clarifies why the metric would be different = but it then leads to questions regarding how / where bufferbloat is = occurring on the links creating the load. I noted the tc stats show a = peak delay of 81ms on download and 139ms on upload, so there is indeed = some queue build-up in the router. >=20 > So even testing cutting QoS to 50% of the upstream capacity and still = getting no better than medium RPM rates. >=20 > Here is the test run on that unit set for roughly half ( 80 / 10 ) of = the upstream line ( 180 / 24 ), this one from a 12.6 OS >=20 > =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D = =20 > Upload capacity: 8.679 Mbps > Download capacity: 75.213 Mbps > Upload flows: 20 > Download flows: 12 > Upload Responsiveness: High (2659 RPM) > Download Responsiveness: High (2587 RPM) > Base RTT: 14 > Start: 11/3/22, 4:05:01 PM > End: 11/3/22, 4:05:26 PM > OS Version: Version 12.6 (Build 21G115) >=20 > And this one from Ventura (13) >=20 > =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D > Uplink capacity: 9.328 Mbps (Accuracy: High) > Downlink capacity: 76.555 Mbps (Accuracy: High) > Uplink Responsiveness: Low (143 RPM) (Accuracy: High) > Downlink Responsiveness: Medium (380 RPM) (Accuracy: High) > Idle Latency: 29.000 milli-seconds (Accuracy: High) > Interface: en6 > Uplink bytes transferred: 16.734 MB > Downlink bytes transferred: 85.637 MB > Uplink Flow count: 20 > Downlink Flow count: 12 > Start: 11/3/22, 4:03:33 PM > End: 11/3/22, 4:03:58 PM > OS Version: Version 13.0 (Build 22A380) >=20 > Does all-out use of ECN cause a penalty? >=20 > On download, we recorded 9504 marks, but only 38 drops. So the flows = should have been well managed with all the early feedback to senders.=20 >=20 > Do the metrics look for drops and thus this low drop rate seems like = there is bloat given the amount of traffic in flight? >=20 > The device under test is an MT7621 running stock OpenWRT 22.03.2 with = SQM installed and using layer-cake. But we see similar metrics on an = i5-4200 x86 box with Intel NICs. So it=E2=80=99s not horsepower related. > I just retested on the x86, with the same ballpark results. >=20 > I=E2=80=99ll re-test tomorrow with all the ECN features on the Mac and = the router disabled to see what that does to the metrics. >=20 > Thanks, >=20 > Jonathan >=20 >=20 >=20 >> On Nov 1, 2022, at 5:52 PM, Christoph Paasch = wrote: >>=20 >> Hello Jonathan, >>=20 >>> On Oct 28, 2022, at 2:45 PM, jf--- via Rpm = wrote: >>>=20 >>> Hopefully, Christoph can provide some details on the changes from = the prior networkQuality test, as we=E2=80=99re seeing some pretty large = changes in results for the latest RPM tests. >>>=20 >>> Where before we=E2=80=99d see results in the >1,500 RPM (and = multiple >2,000 RPM results) for a DOCSIS 3.1 line with QoS enabled (180 = down/35 up), it now returns peak download RPM of ~600 and ~800 for = upload. >>>=20 >>> latest results: >>>=20 >>> =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D >>> Uplink capacity: 25.480 Mbps (Accuracy: High) >>> Downlink capacity: 137.768 Mbps (Accuracy: High) >>> Uplink Responsiveness: Medium (385 RPM) (Accuracy: High) >>> Downlink Responsiveness: Medium (376 RPM) (Accuracy: High) >>> Idle Latency: 43.875 milli-seconds (Accuracy: High) >>> Interface: en8 >>> Uplink bytes transferred: 35.015 MB >>> Downlink bytes transferred: 154.649 MB >>> Uplink Flow count: 16 >>> Downlink Flow count: 12 >>> Start: 10/28/22, 5:12:30 PM >>> End: 10/28/22, 5:12:54 PM >>> OS Version: Version 13.0 (Build 22A380) >>>=20 >>> Latencies (as monitored via PingPlotter) stay absolutely steady = during these tests, >>>=20 >>> So unless my ISP coincidentally started having major service issues, = I=E2=80=99m scratching my head as to why. >>>=20 >>> For contrast, the Ookla result is as follows: = https://www.speedtest.net/result/13865976456 with 15ms down, 18ms up = loaded latencies. >>=20 >> In Ventura, we started adding the latency on the load-generating = connections to the final RPM-calulcation as well. The formula being used = is now exactly what is in the v01 IETF draft. >>=20 >> Very likely the bottleneck in your network does FQ, and so latency on = separate connections is very low, while your load-generating connections = are still bufferbloated. >>=20 >>=20 >> Ookla measures latency only on separate connections, thus will also = be heavily impacted by FQ. >>=20 >>=20 >> Does that clarify it? >>=20 >>=20 >> Cheers, >> Christoph=20 >>=20 >>=20 >>>=20 >>> Further machine details: MacBook Pro 16=E2=80=9D (2019) using a = USB-C to Ethernet adapter. >>> I run with full ECN enabled: >>> sudo sysctl -w net.inet.tcp.disable_tcp_heuristics=3D1=20 >>>=20 >>> sudo sysctl -w net.inet.tcp.ecn_initiate_out=3D1 >>>=20 >>> sudo sysctl -w net.inet.tcp.ecn_negotiate_in=3D1 >>>=20 >>> and also with instant ack replies: >>>=20 >>> sysctl net.inet.tcp.delayed_ack >>> net.inet.tcp.delayed_ack: 0 >>>=20 >>> I did try with delayed_ack=3D1, and the results were about the same. >>>=20 >>> Thanks in advance, >>>=20 >>> Jonathan Foulkes >>>=20 >>> _______________________________________________ >>> Rpm mailing list >>> Rpm@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/rpm >>=20 >=20 > _______________________________________________ > Rpm mailing list > Rpm@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/rpm --Apple-Mail=_0CA51065-E806-4ACF-82EC-E500012D6863 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 As I = mentioned, here are the results of running with ECN enabled vs = disabled.

First, we retest with ECN enabled at the = router, the SQM instance and fully enabled on the MacOS (see a prior = post with the settings).

This is a DOCSIS 3.0 = line at 300Mbps down and 25Mbps Up, router is an x86 i5-4200 running = OpenWRT 22.03 with layer_cake QoS.

=3D=3D=3D= =3D SUMMARY =3D=3D=3D=3D
Uplink capacity: 20.971 Mbps = (Accuracy: High)
Downlink capacity: 267.643 Mbps (Accuracy: = High)
Uplink Responsiveness: Medium (550 RPM) (Accuracy: = High)
Downlink Responsiveness: Medium (900 RPM) (Accuracy: = High)
Idle Latency: 21.417 milli-seconds (Accuracy: = High)
Interface: en6
Uplink bytes transferred: = 24.359 MB
Downlink bytes transferred: 298.730 = MB
Uplink Flow count: 12
Downlink Flow count: = 12
Start: 11/4/22, 3:41:30 PM
End: 11/4/22, 3:41:49 = PM
OS Version: Version 13.0 (Build = 22A380)

Then I disabled ECN support on the = router, the sqm instance, and the MacOS, then re-ran a couple of times, = and this is a representative result:

=3D=3D=3D= =3D SUMMARY =3D=3D=3D=3D
Uplink capacity: 20.961 Mbps = (Accuracy: High)
Downlink capacity: 255.146 Mbps (Accuracy: = High)
Uplink Responsiveness: Medium (475 RPM) (Accuracy: = High)
Downlink Responsiveness: Medium (591 RPM) (Accuracy: = High)
Idle Latency: 23.583 milli-seconds (Accuracy: = High)
Interface: en6
Uplink bytes transferred: = 24.593 MB
Downlink bytes transferred: 278.096 = MB
Uplink Flow count: 12
Downlink Flow count: = 12
Start: 11/4/22, 4:11:59 PM
End: 11/4/22, 4:12:18 = PM
OS Version: Version 13.0 (Build = 22A380)

The absence of ECN does have an = impact, most significantly on download, where it drops from 900 to 591 = RPM

My takeaway is that ECN does help with = responsiveness overall.

What I=E2=80=99m still = puzzled by here is the bloat and why changes to QoS settings do not make = a difference.

I have questions about how the = lag within a bulk flow is a factor in the calculation, but I=E2=80=99ll = first go read the latest version of the document.
In case I = missed it, is there a discussion thread someone can point me to where = this change is = mentioned?

Thanks,

Jonat= han


On Nov = 3, 2022, at 6:09 PM, jf--- via Rpm <rpm@lists.bufferbloat.net> = wrote:

Hi Christoph,

Thanks = for the reply, it clarifies why the metric would be different but it = then leads to questions regarding how / where bufferbloat is occurring = on the links creating the load. I noted the tc stats show a peak delay = of 81ms on download and 139ms on upload, so there is indeed some queue = build-up in the router.

So even testing cutting = QoS to 50% of the upstream capacity and still getting no better than = medium RPM rates.

Here is the test run on that = unit set for roughly half ( 80 / 10 ) of the upstream line ( 180 / 24 ), = this one from a 12.6 OS

=3D=3D=3D=3D = SUMMARY =3D=3D=3D=3D               =                     =                     =                     =               
Upload = capacity: 8.679 Mbps
Download capacity: 75.213 = Mbps
Upload flows: 20
Download flows: = 12
Upload Responsiveness: High (2659 RPM)
Download = Responsiveness: High (2587 RPM)
Base RTT: 14
Start: = 11/3/22, 4:05:01 PM
End: 11/3/22, 4:05:26 PM
OS = Version: Version 12.6 (Build 21G115)

And this = one from Ventura (13)

=3D=3D=3D=3D SUMMARY = =3D=3D=3D=3D
Uplink capacity: 9.328 Mbps (Accuracy: = High)
Downlink capacity: 76.555 Mbps (Accuracy: = High)
Uplink Responsiveness: Low (143 RPM) (Accuracy: = High)
Downlink Responsiveness: Medium (380 RPM) (Accuracy: = High)
Idle Latency: 29.000 milli-seconds (Accuracy: = High)
Interface: en6
Uplink bytes transferred: = 16.734 MB
Downlink bytes transferred: 85.637 = MB
Uplink Flow count: 20
Downlink Flow count: = 12
Start: 11/3/22, 4:03:33 PM
End: 11/3/22, 4:03:58 = PM
OS Version: Version 13.0 (Build = 22A380)

Does all-out use of ECN cause a = penalty?

On download, we recorded 9504 marks, = but only 38 drops. So the flows should have been well managed with all = the early feedback to senders. 

Do the = metrics look for drops and thus this low drop rate seems like there is = bloat given the amount of traffic in = flight?

The device under test is an MT7621 = running stock OpenWRT 22.03.2 with SQM installed and using layer-cake. = But we see similar metrics on an i5-4200 x86 box with Intel NICs. So = it=E2=80=99s not horsepower related.
I just retested on the = x86, with the same ballpark results.

I=E2=80=99ll= re-test tomorrow with all the ECN features on the Mac and the router = disabled to see what that does to the = metrics.

Thanks,

Jonatha= n



On Nov 1, 2022, at 5:52 PM, Christoph Paasch = <cpaasch@apple.com> wrote:

Hello Jonathan,

On Oct 28, 2022, at 2:45 PM, jf--- via Rpm = <rpm@lists.bufferbloat.net> wrote:

Hopefully, Christoph can provide some details on the = changes from the prior networkQuality test, as we=E2=80=99re seeing some = pretty large changes in results for the latest RPM = tests.

Where before we=E2=80=99d see results in the = >1,500 RPM (and multiple >2,000 RPM results) for a DOCSIS 3.1 line = with QoS enabled (180 down/35 up), it now returns peak download RPM of = ~600 and ~800 for upload.

latest = results:

=3D=3D=3D=3D SUMMARY = =3D=3D=3D=3D
Uplink capacity: 25.480 Mbps (Accuracy: = High)
Downlink capacity: 137.768 Mbps (Accuracy: = High)
Uplink Responsiveness: Medium (385 RPM) (Accuracy: = High)
Downlink Responsiveness: Medium (376 RPM) (Accuracy: = High)
Idle Latency: 43.875 milli-seconds (Accuracy: = High)
Interface: en8
Uplink bytes transferred: = 35.015 MB
Downlink bytes transferred: 154.649 = MB
Uplink Flow count: 16
Downlink Flow count: = 12
Start: 10/28/22, 5:12:30 PM
End: 10/28/22, = 5:12:54 PM
OS Version: Version 13.0 (Build = 22A380)

Latencies (as monitored via = PingPlotter) stay absolutely steady during these = tests,

So unless my ISP coincidentally started = having major service issues, I=E2=80=99m scratching my head as to = why.

For contrast, the Ookla result is as = follows: https://www.speedtes= t.net/result/13865976456  with 15ms down, 18ms up loaded = latencies.

In Ventura, = we started adding the latency on the load-generating connections to the = final RPM-calulcation as well. The formula being used is now exactly = what is in the v01 IETF draft.

Very likely the = bottleneck in your network does FQ, and so latency on separate = connections is very low, while your load-generating connections are = still bufferbloated.


Ookla = measures latency only on separate connections, thus will also be heavily = impacted by FQ.


Does that = clarify = it?


Cheers,
Christoph&nb= sp;



Further machine = details: MacBook Pro 16=E2=80=9D (2019) using a USB-C to Ethernet = adapter.
I run with full ECN enabled:

sudo sysctl -w = net.inet.tcp.disable_tcp_heuristics=3D1 

sudo sysctl -w net.inet.tcp.ecn_initiate_out=3D1

sudo sysctl -w = net.inet.tcp.ecn_negotiate_in=3D1

a= nd also with instant ack replies:

sysctl = net.inet.tcp.delayed_ack
net.inet.tcp.delayed_ack: = 0

I did try with delayed_ack=3D1, and the = results were about the same.

Thanks in = advance,

Jonathan = Foulkes

________________________________________= _______
Rpm mailing = list
Rpm@lists.bufferbloat.net
https://lists.bufferbloat.net/listinf= o/rpm


<= /div>
_______________________________________________
Rpm = mailing = list
Rpm@lists.bufferbloat.net
https://lists.bufferbloat.net/listinf= o/rpm

= --Apple-Mail=_0CA51065-E806-4ACF-82EC-E500012D6863--