From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ma1-aaemail-dr-lapp01.apple.com (ma1-aaemail-dr-lapp01.apple.com [17.171.2.60]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id A85F43B29E for ; Thu, 10 Nov 2022 15:28:26 -0500 (EST) Received: from pps.filterd (ma1-aaemail-dr-lapp01.apple.com [127.0.0.1]) by ma1-aaemail-dr-lapp01.apple.com (8.16.0.42/8.16.0.42) with SMTP id 2AAKFscZ040949; Thu, 10 Nov 2022 12:28:25 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=CoSPg1ASrmQ/oh4tGpYP10fPCGI6TJmBKHqYTKfNRcU=; b=W7nnnqXm8/JFb5m5y6i2h17SrvpHrjkE3wbH9h3L+dMRgpVJOsOOcbu1MqnTGXVUFSPe U/w6uX++u8M0sCgi8Pd3xXSpzHmacyf2fRCiNxhWch3qqTxfXixnSx0utAfPnHDX+uVb KrTotBM9uYlztey3bS1voBzssGSXo48fUlFHQRP5SZSMRqxZ09SSHlWFoo1mcCJY1SQG rDMBzWzct6jQ8p/4CT4wJXUUXAtTmo351QwLZC/3IrCWipDWxGaI65Hf3uyshdPKutv7 LGnhy+T/a8hZb6/TfAovnugcfc2i8aA5ttOZfGUxar+m/ueOhhrAQ/5n0iSr1sVntv37 7A== Received: from rn-mailsvcp-mta-lapp02.rno.apple.com (rn-mailsvcp-mta-lapp02.rno.apple.com [10.225.203.150]) by ma1-aaemail-dr-lapp01.apple.com with ESMTP id 3knpg39nug-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Thu, 10 Nov 2022 12:28:25 -0800 Received: from rn-mailsvcp-mmp-lapp04.rno.apple.com (rn-mailsvcp-mmp-lapp04.rno.apple.com [17.179.253.17]) by rn-mailsvcp-mta-lapp02.rno.apple.com (Oracle Communications Messaging Server 8.1.0.20.20220923 64bit (built Sep 23 2022)) with ESMTPS id <0RL500PX7FJCMSH0@rn-mailsvcp-mta-lapp02.rno.apple.com>; Thu, 10 Nov 2022 12:28:24 -0800 (PST) Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp04.rno.apple.com by rn-mailsvcp-mmp-lapp04.rno.apple.com (Oracle Communications Messaging Server 8.1.0.20.20220923 64bit (built Sep 23 2022)) id <0RL500P00FIHGB00@rn-mailsvcp-mmp-lapp04.rno.apple.com>; Thu, 10 Nov 2022 12:28:24 -0800 (PST) X-Va-A: X-Va-T-CD: 87fb32bc225d5809fa663d8e3ce5e235 X-Va-E-CD: e6522529a1c3a53151bafdb6b02f4ea2 X-Va-R-CD: c39ddcbb2776b6dc4205ccce0a2c0401 X-Va-CD: 0 X-Va-ID: 61bbee5a-05bc-41c8-9504-a6b057317396 X-V-A: X-V-T-CD: 87fb32bc225d5809fa663d8e3ce5e235 X-V-E-CD: e6522529a1c3a53151bafdb6b02f4ea2 X-V-R-CD: c39ddcbb2776b6dc4205ccce0a2c0401 X-V-CD: 0 X-V-ID: 044e238e-0e1c-4cab-8ec1-84d62fd12281 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.545, 18.0.895 definitions=2022-11-10_12:2022-11-09, 2022-11-10 signatures=0 Received: from smtpclient.apple ([17.11.121.197]) by rn-mailsvcp-mmp-lapp04.rno.apple.com (Oracle Communications Messaging Server 8.1.0.20.20220923 64bit (built Sep 23 2022)) with ESMTPSA id <0RL500N6JFJC4X00@rn-mailsvcp-mmp-lapp04.rno.apple.com>; Thu, 10 Nov 2022 12:28:24 -0800 (PST) From: Christoph Paasch Message-id: <50EF2D37-0C16-498A-9395-47610B12EEB2@apple.com> Content-type: multipart/alternative; boundary="Apple-Mail=_7389F3F6-6569-4F3B-95F3-93919942BA21" MIME-version: 1.0 (Mac OS X Mail 16.0 \(3731.200.92\)) Date: Thu, 10 Nov 2022 12:28:13 -0800 In-reply-to: <49CC4E18-2165-4027-BA1D-B3C998BBF54D@jonathanfoulkes.com> Cc: Rpm To: "jf@jonathanfoulkes.com" References: <49CC4E18-2165-4027-BA1D-B3C998BBF54D@jonathanfoulkes.com> X-Mailer: Apple Mail (2.3731.200.92) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.545, 18.0.895 definitions=2022-11-10_12:2022-11-09, 2022-11-10 signatures=0 Subject: Re: [Rpm] Changes to RPM calculation for MacOS Ventura? X-BeenThere: rpm@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: revolutions per minute - a new metric for measuring responsiveness List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Nov 2022 20:28:26 -0000 --Apple-Mail=_7389F3F6-6569-4F3B-95F3-93919942BA21 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hello Jonathan, > On Nov 3, 2022, at 3:09 PM, jf@jonathanfoulkes.com wrote: >=20 > Hi Christoph, >=20 > Thanks for the reply, it clarifies why the metric would be different = but it then leads to questions regarding how / where bufferbloat is = occurring on the links creating the load. I noted the tc stats show a = peak delay of 81ms on download and 139ms on upload, so there is indeed = some queue build-up in the router. Yes, there are multiple sources of bufferbloat. First, it happens on the = bottleneck link. But then, it also happens in the sender=E2=80=99s = TCP-stack (thus, the importance of TCP_NOTSENT_LOWAT). Add flow-queuing = in the mix it gets even more tricky :-) But, in the end, we want low latency not only on separate connections, = but also on the connections that are carrying the high-throughput = traffic. These days with H2/H3, connections are aggressively reused for = transmitting the data, thus potentially mixing a bulk-transfer with a = short latency-sensitive transfer on the same connection.=20 >=20 > So even testing cutting QoS to 50% of the upstream capacity and still = getting no better than medium RPM rates. >=20 > Here is the test run on that unit set for roughly half ( 80 / 10 ) of = the upstream line ( 180 / 24 ), this one from a 12.6 OS >=20 > =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D = =20 > Upload capacity: 8.679 Mbps > Download capacity: 75.213 Mbps > Upload flows: 20 > Download flows: 12 > Upload Responsiveness: High (2659 RPM) > Download Responsiveness: High (2587 RPM) > Base RTT: 14 > Start: 11/3/22, 4:05:01 PM > End: 11/3/22, 4:05:26 PM > OS Version: Version 12.6 (Build 21G115) >=20 > And this one from Ventura (13) >=20 > =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D > Uplink capacity: 9.328 Mbps (Accuracy: High) > Downlink capacity: 76.555 Mbps (Accuracy: High) > Uplink Responsiveness: Low (143 RPM) (Accuracy: High) > Downlink Responsiveness: Medium (380 RPM) (Accuracy: High) > Idle Latency: 29.000 milli-seconds (Accuracy: High) > Interface: en6 > Uplink bytes transferred: 16.734 MB > Downlink bytes transferred: 85.637 MB > Uplink Flow count: 20 > Downlink Flow count: 12 > Start: 11/3/22, 4:03:33 PM > End: 11/3/22, 4:03:58 PM > OS Version: Version 13.0 (Build 22A380) >=20 > Does all-out use of ECN cause a penalty? >=20 > On download, we recorded 9504 marks, but only 38 drops. So the flows = should have been well managed with all the early feedback to senders.=20 >=20 > Do the metrics look for drops and thus this low drop rate seems like = there is bloat given the amount of traffic in flight? No, we don=E2=80=99t look for drops. Christoph >=20 > The device under test is an MT7621 running stock OpenWRT 22.03.2 with = SQM installed and using layer-cake. But we see similar metrics on an = i5-4200 x86 box with Intel NICs. So it=E2=80=99s not horsepower related. > I just retested on the x86, with the same ballpark results. >=20 > I=E2=80=99ll re-test tomorrow with all the ECN features on the Mac and = the router disabled to see what that does to the metrics. >=20 > Thanks, >=20 > Jonathan >=20 >=20 >=20 >> On Nov 1, 2022, at 5:52 PM, Christoph Paasch = wrote: >>=20 >> Hello Jonathan, >>=20 >>> On Oct 28, 2022, at 2:45 PM, jf--- via Rpm = wrote: >>>=20 >>> Hopefully, Christoph can provide some details on the changes from = the prior networkQuality test, as we=E2=80=99re seeing some pretty large = changes in results for the latest RPM tests. >>>=20 >>> Where before we=E2=80=99d see results in the >1,500 RPM (and = multiple >2,000 RPM results) for a DOCSIS 3.1 line with QoS enabled (180 = down/35 up), it now returns peak download RPM of ~600 and ~800 for = upload. >>>=20 >>> latest results: >>>=20 >>> =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D >>> Uplink capacity: 25.480 Mbps (Accuracy: High) >>> Downlink capacity: 137.768 Mbps (Accuracy: High) >>> Uplink Responsiveness: Medium (385 RPM) (Accuracy: High) >>> Downlink Responsiveness: Medium (376 RPM) (Accuracy: High) >>> Idle Latency: 43.875 milli-seconds (Accuracy: High) >>> Interface: en8 >>> Uplink bytes transferred: 35.015 MB >>> Downlink bytes transferred: 154.649 MB >>> Uplink Flow count: 16 >>> Downlink Flow count: 12 >>> Start: 10/28/22, 5:12:30 PM >>> End: 10/28/22, 5:12:54 PM >>> OS Version: Version 13.0 (Build 22A380) >>>=20 >>> Latencies (as monitored via PingPlotter) stay absolutely steady = during these tests, >>>=20 >>> So unless my ISP coincidentally started having major service issues, = I=E2=80=99m scratching my head as to why. >>>=20 >>> For contrast, the Ookla result is as follows: = https://www.speedtest.net/result/13865976456 with 15ms down, 18ms up = loaded latencies. >>=20 >> In Ventura, we started adding the latency on the load-generating = connections to the final RPM-calulcation as well. The formula being used = is now exactly what is in the v01 IETF draft. >>=20 >> Very likely the bottleneck in your network does FQ, and so latency on = separate connections is very low, while your load-generating connections = are still bufferbloated. >>=20 >>=20 >> Ookla measures latency only on separate connections, thus will also = be heavily impacted by FQ. >>=20 >>=20 >> Does that clarify it? >>=20 >>=20 >> Cheers, >> Christoph=20 >>=20 >>=20 >>>=20 >>> Further machine details: MacBook Pro 16=E2=80=9D (2019) using a = USB-C to Ethernet adapter. >>> I run with full ECN enabled: >>> sudo sysctl -w net.inet.tcp.disable_tcp_heuristics=3D1=20 >>>=20 >>> sudo sysctl -w net.inet.tcp.ecn_initiate_out=3D1 >>>=20 >>> sudo sysctl -w net.inet.tcp.ecn_negotiate_in=3D1 >>>=20 >>> and also with instant ack replies: >>>=20 >>> sysctl net.inet.tcp.delayed_ack >>> net.inet.tcp.delayed_ack: 0 >>>=20 >>> I did try with delayed_ack=3D1, and the results were about the same. >>>=20 >>> Thanks in advance, >>>=20 >>> Jonathan Foulkes >>>=20 >>> _______________________________________________ >>> Rpm mailing list >>> Rpm@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/rpm >>=20 >=20 --Apple-Mail=_7389F3F6-6569-4F3B-95F3-93919942BA21 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hello = Jonathan,

On Nov 3, 2022, at = 3:09 PM, jf@jonathanfoulkes.com wrote:

Hi = Christoph,

Thanks for the reply, it clarifies why the = metric would be different but it then leads to questions regarding how / = where bufferbloat is occurring on the links creating the load. I noted = the tc stats show a peak delay of 81ms on download and 139ms on upload, = so there is indeed some queue build-up in the = router.

Yes, there are = multiple sources of bufferbloat. First, it happens on the bottleneck = link. But then, it also happens in the sender=E2=80=99s TCP-stack (thus, = the importance of TCP_NOTSENT_LOWAT). Add flow-queuing in the mix it = gets even more tricky :-)

But, in the end, we = want low latency not only on separate connections, but also on the = connections that are carrying the high-throughput traffic. These days = with H2/H3, connections are aggressively reused for transmitting the = data, thus potentially mixing a bulk-transfer with a short = latency-sensitive transfer on the same = connection. 


So even testing = cutting QoS to 50% of the upstream capacity and still getting no better = than medium RPM rates.

Here is the test run on = that unit set for roughly half ( 80 / 10 ) of the upstream line ( 180 / = 24 ), this one from a 12.6 OS

=3D=3D=3D=3D = SUMMARY =3D=3D=3D=3D               =                     =                     =                     =               
Upload = capacity: 8.679 Mbps
Download capacity: 75.213 = Mbps
Upload flows: 20
Download flows: = 12
Upload Responsiveness: High (2659 RPM)
Download = Responsiveness: High (2587 RPM)
Base RTT: 14
Start: = 11/3/22, 4:05:01 PM
End: 11/3/22, 4:05:26 PM
OS = Version: Version 12.6 (Build 21G115)

And this = one from Ventura (13)

=3D=3D=3D=3D SUMMARY = =3D=3D=3D=3D
Uplink capacity: 9.328 Mbps (Accuracy: = High)
Downlink capacity: 76.555 Mbps (Accuracy: = High)
Uplink Responsiveness: Low (143 RPM) (Accuracy: = High)
Downlink Responsiveness: Medium (380 RPM) (Accuracy: = High)
Idle Latency: 29.000 milli-seconds (Accuracy: = High)
Interface: en6
Uplink bytes transferred: = 16.734 MB
Downlink bytes transferred: 85.637 = MB
Uplink Flow count: 20
Downlink Flow count: = 12
Start: 11/3/22, 4:03:33 PM
End: 11/3/22, 4:03:58 = PM
OS Version: Version 13.0 (Build = 22A380)

Does all-out use of ECN cause a = penalty?

On download, we recorded 9504 marks, = but only 38 drops. So the flows should have been well managed with all = the early feedback to senders. 

Do the = metrics look for drops and thus this low drop rate seems like there is = bloat given the amount of traffic in = flight?

No, we = don=E2=80=99t look for = drops.


Christoph


The device under test is an = MT7621 running stock OpenWRT 22.03.2 with SQM installed and using = layer-cake. But we see similar metrics on an i5-4200 x86 box with Intel = NICs. So it=E2=80=99s not horsepower related.
I just retested = on the x86, with the same ballpark = results.

I=E2=80=99ll re-test tomorrow with all = the ECN features on the Mac and the router disabled to see what that = does to the = metrics.

Thanks,

Jonatha= n



On Nov 1, 2022, at 5:52 PM, Christoph Paasch = <cpaasch@apple.com> wrote:

Hello Jonathan,

On Oct 28, 2022, at 2:45 PM, jf--- via Rpm = <rpm@lists.bufferbloat.net> wrote:

Hopefully, Christoph can provide some details on the = changes from the prior networkQuality test, as we=E2=80=99re seeing some = pretty large changes in results for the latest RPM = tests.

Where before we=E2=80=99d see results in the = >1,500 RPM (and multiple >2,000 RPM results) for a DOCSIS 3.1 line = with QoS enabled (180 down/35 up), it now returns peak download RPM of = ~600 and ~800 for upload.

latest = results:

=3D=3D=3D=3D SUMMARY = =3D=3D=3D=3D
Uplink capacity: 25.480 Mbps (Accuracy: = High)
Downlink capacity: 137.768 Mbps (Accuracy: = High)
Uplink Responsiveness: Medium (385 RPM) (Accuracy: = High)
Downlink Responsiveness: Medium (376 RPM) (Accuracy: = High)
Idle Latency: 43.875 milli-seconds (Accuracy: = High)
Interface: en8
Uplink bytes transferred: = 35.015 MB
Downlink bytes transferred: 154.649 = MB
Uplink Flow count: 16
Downlink Flow count: = 12
Start: 10/28/22, 5:12:30 PM
End: 10/28/22, = 5:12:54 PM
OS Version: Version 13.0 (Build = 22A380)

Latencies (as monitored via = PingPlotter) stay absolutely steady during these = tests,

So unless my ISP coincidentally started = having major service issues, I=E2=80=99m scratching my head as to = why.

For contrast, the Ookla result is as = follows: https://www.speedtes= t.net/result/13865976456  with 15ms down, 18ms up loaded = latencies.

In Ventura, = we started adding the latency on the load-generating connections to the = final RPM-calulcation as well. The formula being used is now exactly = what is in the v01 IETF draft.

Very likely the = bottleneck in your network does FQ, and so latency on separate = connections is very low, while your load-generating connections are = still bufferbloated.


Ookla = measures latency only on separate connections, thus will also be heavily = impacted by FQ.


Does that = clarify = it?


Cheers,
Christoph&nb= sp;



Further machine = details: MacBook Pro 16=E2=80=9D (2019) using a USB-C to Ethernet = adapter.
I run with full ECN enabled:

sudo sysctl -w = net.inet.tcp.disable_tcp_heuristics=3D1 

sudo sysctl -w net.inet.tcp.ecn_initiate_out=3D1

sudo sysctl -w = net.inet.tcp.ecn_negotiate_in=3D1

a= nd also with instant ack replies:

sysctl = net.inet.tcp.delayed_ack
net.inet.tcp.delayed_ack: = 0

I did try with delayed_ack=3D1, and the = results were about the same.

Thanks in = advance,

Jonathan = Foulkes

________________________________________= _______
Rpm mailing = list
Rpm@lists.bufferbloat.net
https://lists.bufferbloat.net/listinf= o/rpm


<= /div>

= --Apple-Mail=_7389F3F6-6569-4F3B-95F3-93919942BA21--