From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from rn-mailsvcp-ppex-lapp34.apple.com (rn-mailsvcp-ppex-lapp34.rno.apple.com [17.179.253.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 21B023B29E for ; Thu, 10 Nov 2022 15:41:45 -0500 (EST) Received: from pps.filterd (rn-mailsvcp-ppex-lapp34.rno.apple.com [127.0.0.1]) by rn-mailsvcp-ppex-lapp34.rno.apple.com (8.16.1.2/8.16.1.2) with SMTP id 2AAKb2ej027626; Thu, 10 Nov 2022 12:41:43 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=HVk2cEGDS3pscb3mXL/f3PsaOavhxV+q170mEQBPI9Y=; b=uaB+ZHrpEQ9jYzIyXkRvMUVRkA7PfcJecFKWW+Hh1a7FZKcC2O8TEczbLnBBA+XoG8bN um3RG+fdbJ3mZZLXJa/max8SsA/2omO2Lq5eXts5d8ccAGMVE+SfxyS8BDAq3oGH1+KI 05sx7us3yn3PFHVAJQeODlSARzPJY1iodYSx83ogKpxGYO4RnIalzLl5OynxV+sE6KaH JzDfGMWSdywwJKOEV4XyHEaXw/yoq5MUtj3LdNYdehlgc1iIuXvBP7vG+1vP1v8fPega s3RjSmQTdQaDwQfD5qke5OMKm119hmAJLXnHSp4P5YJqy4V6f+fw2p3JurDCbUBErSZ2 Fg== Received: from rn-mailsvcp-mta-lapp01.rno.apple.com (rn-mailsvcp-mta-lapp01.rno.apple.com [10.225.203.149]) by rn-mailsvcp-ppex-lapp34.rno.apple.com with ESMTP id 3krjsx9xxn-7 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Thu, 10 Nov 2022 12:41:43 -0800 Received: from rn-mailsvcp-mmp-lapp04.rno.apple.com (rn-mailsvcp-mmp-lapp04.rno.apple.com [17.179.253.17]) by rn-mailsvcp-mta-lapp01.rno.apple.com (Oracle Communications Messaging Server 8.1.0.20.20220923 64bit (built Sep 23 2022)) with ESMTPS id <0RL500E34G5IBTF0@rn-mailsvcp-mta-lapp01.rno.apple.com>; Thu, 10 Nov 2022 12:41:42 -0800 (PST) Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp04.rno.apple.com by rn-mailsvcp-mmp-lapp04.rno.apple.com (Oracle Communications Messaging Server 8.1.0.20.20220923 64bit (built Sep 23 2022)) id <0RL500J00FZ17Y00@rn-mailsvcp-mmp-lapp04.rno.apple.com>; Thu, 10 Nov 2022 12:41:42 -0800 (PST) X-Va-A: X-Va-T-CD: 5eb40c22202d18605434c505fbef57d8 X-Va-E-CD: e6522529a1c3a53151bafdb6b02f4ea2 X-Va-R-CD: c39ddcbb2776b6dc4205ccce0a2c0401 X-Va-CD: 0 X-Va-ID: 47cd781a-9d7c-4b1a-93da-ec7c8a38e62c X-V-A: X-V-T-CD: 5eb40c22202d18605434c505fbef57d8 X-V-E-CD: e6522529a1c3a53151bafdb6b02f4ea2 X-V-R-CD: c39ddcbb2776b6dc4205ccce0a2c0401 X-V-CD: 0 X-V-ID: 95078c8b-cca6-4b20-8094-9064fca7739a X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.545, 18.0.895 definitions=2022-11-10_12:2022-11-09, 2022-11-10 signatures=0 Received: from smtpclient.apple ([17.11.121.197]) by rn-mailsvcp-mmp-lapp04.rno.apple.com (Oracle Communications Messaging Server 8.1.0.20.20220923 64bit (built Sep 23 2022)) with ESMTPSA id <0RL500N6IG5I4X20@rn-mailsvcp-mmp-lapp04.rno.apple.com>; Thu, 10 Nov 2022 12:41:42 -0800 (PST) From: Christoph Paasch Message-id: <034C022C-49CD-4922-9C65-1CFB63348033@apple.com> Content-type: multipart/alternative; boundary="Apple-Mail=_2706DF7D-B4EE-44E4-994E-579899F9FA94" MIME-version: 1.0 (Mac OS X Mail 16.0 \(3731.200.92\)) Date: Thu, 10 Nov 2022 12:41:32 -0800 In-reply-to: <18106767-7105-44EB-8E9B-C431D69544DD@jonathanfoulkes.com> Cc: Rpm To: jf@jonathanfoulkes.com References: <49CC4E18-2165-4027-BA1D-B3C998BBF54D@jonathanfoulkes.com> <18106767-7105-44EB-8E9B-C431D69544DD@jonathanfoulkes.com> X-Mailer: Apple Mail (2.3731.200.92) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.545, 18.0.895 definitions=2022-11-10_12:2022-11-09, 2022-11-10 signatures=0 Subject: Re: [Rpm] Changes to RPM calculation for MacOS Ventura? X-BeenThere: rpm@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: revolutions per minute - a new metric for measuring responsiveness List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Nov 2022 20:41:45 -0000 --Apple-Mail=_2706DF7D-B4EE-44E4-994E-579899F9FA94 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Nov 4, 2022, at 1:50 PM, jf@jonathanfoulkes.com wrote: >=20 > As I mentioned, here are the results of running with ECN enabled vs = disabled. >=20 > First, we retest with ECN enabled at the router, the SQM instance and = fully enabled on the MacOS (see a prior post with the settings). >=20 > This is a DOCSIS 3.0 line at 300Mbps down and 25Mbps Up, router is an = x86 i5-4200 running OpenWRT 22.03 with layer_cake QoS. >=20 > =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D > Uplink capacity: 20.971 Mbps (Accuracy: High) > Downlink capacity: 267.643 Mbps (Accuracy: High) > Uplink Responsiveness: Medium (550 RPM) (Accuracy: High) > Downlink Responsiveness: Medium (900 RPM) (Accuracy: High) > Idle Latency: 21.417 milli-seconds (Accuracy: High) > Interface: en6 > Uplink bytes transferred: 24.359 MB > Downlink bytes transferred: 298.730 MB > Uplink Flow count: 12 > Downlink Flow count: 12 > Start: 11/4/22, 3:41:30 PM > End: 11/4/22, 3:41:49 PM > OS Version: Version 13.0 (Build 22A380) >=20 > Then I disabled ECN support on the router, the sqm instance, and the = MacOS, then re-ran a couple of times, and this is a representative = result: >=20 > =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D > Uplink capacity: 20.961 Mbps (Accuracy: High) > Downlink capacity: 255.146 Mbps (Accuracy: High) > Uplink Responsiveness: Medium (475 RPM) (Accuracy: High) > Downlink Responsiveness: Medium (591 RPM) (Accuracy: High) > Idle Latency: 23.583 milli-seconds (Accuracy: High) > Interface: en6 > Uplink bytes transferred: 24.593 MB > Downlink bytes transferred: 278.096 MB > Uplink Flow count: 12 > Downlink Flow count: 12 > Start: 11/4/22, 4:11:59 PM > End: 11/4/22, 4:12:18 PM > OS Version: Version 13.0 (Build 22A380) >=20 > The absence of ECN does have an impact, most significantly on = download, where it drops from 900 to 591 RPM That=E2=80=99s interesting and good to know that the tool can measure = the positive impact of ECN! > My takeaway is that ECN does help with responsiveness overall. >=20 > What I=E2=80=99m still puzzled by here is the bloat and why changes to = QoS settings do not make a difference. That is interesting indeed. What configurations are you testing? At = this level, pcaps may be helpful. Christoph >=20 > I have questions about how the lag within a bulk flow is a factor in = the calculation, but I=E2=80=99ll first go read the latest version of = the document. > In case I missed it, is there a discussion thread someone can point me = to where this change is mentioned? The draft itself has had latency on bulk-flows since always. We just = hadn=E2=80=99t backed this into the implementation, due to a bad = server-side responsiveness which we have addressed now. >=20 > Thanks, >=20 > Jonathan >=20 >=20 >> On Nov 3, 2022, at 6:09 PM, jf--- via Rpm = wrote: >>=20 >> Hi Christoph, >>=20 >> Thanks for the reply, it clarifies why the metric would be different = but it then leads to questions regarding how / where bufferbloat is = occurring on the links creating the load. I noted the tc stats show a = peak delay of 81ms on download and 139ms on upload, so there is indeed = some queue build-up in the router. >>=20 >> So even testing cutting QoS to 50% of the upstream capacity and still = getting no better than medium RPM rates. >>=20 >> Here is the test run on that unit set for roughly half ( 80 / 10 ) of = the upstream line ( 180 / 24 ), this one from a 12.6 OS >>=20 >> =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D = =20 >> Upload capacity: 8.679 Mbps >> Download capacity: 75.213 Mbps >> Upload flows: 20 >> Download flows: 12 >> Upload Responsiveness: High (2659 RPM) >> Download Responsiveness: High (2587 RPM) >> Base RTT: 14 >> Start: 11/3/22, 4:05:01 PM >> End: 11/3/22, 4:05:26 PM >> OS Version: Version 12.6 (Build 21G115) >>=20 >> And this one from Ventura (13) >>=20 >> =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D >> Uplink capacity: 9.328 Mbps (Accuracy: High) >> Downlink capacity: 76.555 Mbps (Accuracy: High) >> Uplink Responsiveness: Low (143 RPM) (Accuracy: High) >> Downlink Responsiveness: Medium (380 RPM) (Accuracy: High) >> Idle Latency: 29.000 milli-seconds (Accuracy: High) >> Interface: en6 >> Uplink bytes transferred: 16.734 MB >> Downlink bytes transferred: 85.637 MB >> Uplink Flow count: 20 >> Downlink Flow count: 12 >> Start: 11/3/22, 4:03:33 PM >> End: 11/3/22, 4:03:58 PM >> OS Version: Version 13.0 (Build 22A380) >>=20 >> Does all-out use of ECN cause a penalty? >>=20 >> On download, we recorded 9504 marks, but only 38 drops. So the flows = should have been well managed with all the early feedback to senders.=20 >>=20 >> Do the metrics look for drops and thus this low drop rate seems like = there is bloat given the amount of traffic in flight? >>=20 >> The device under test is an MT7621 running stock OpenWRT 22.03.2 with = SQM installed and using layer-cake. But we see similar metrics on an = i5-4200 x86 box with Intel NICs. So it=E2=80=99s not horsepower related. >> I just retested on the x86, with the same ballpark results. >>=20 >> I=E2=80=99ll re-test tomorrow with all the ECN features on the Mac = and the router disabled to see what that does to the metrics. >>=20 >> Thanks, >>=20 >> Jonathan >>=20 >>=20 >>=20 >>> On Nov 1, 2022, at 5:52 PM, Christoph Paasch = wrote: >>>=20 >>> Hello Jonathan, >>>=20 >>>> On Oct 28, 2022, at 2:45 PM, jf--- via Rpm = wrote: >>>>=20 >>>> Hopefully, Christoph can provide some details on the changes from = the prior networkQuality test, as we=E2=80=99re seeing some pretty large = changes in results for the latest RPM tests. >>>>=20 >>>> Where before we=E2=80=99d see results in the >1,500 RPM (and = multiple >2,000 RPM results) for a DOCSIS 3.1 line with QoS enabled (180 = down/35 up), it now returns peak download RPM of ~600 and ~800 for = upload. >>>>=20 >>>> latest results: >>>>=20 >>>> =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D >>>> Uplink capacity: 25.480 Mbps (Accuracy: High) >>>> Downlink capacity: 137.768 Mbps (Accuracy: High) >>>> Uplink Responsiveness: Medium (385 RPM) (Accuracy: High) >>>> Downlink Responsiveness: Medium (376 RPM) (Accuracy: High) >>>> Idle Latency: 43.875 milli-seconds (Accuracy: High) >>>> Interface: en8 >>>> Uplink bytes transferred: 35.015 MB >>>> Downlink bytes transferred: 154.649 MB >>>> Uplink Flow count: 16 >>>> Downlink Flow count: 12 >>>> Start: 10/28/22, 5:12:30 PM >>>> End: 10/28/22, 5:12:54 PM >>>> OS Version: Version 13.0 (Build 22A380) >>>>=20 >>>> Latencies (as monitored via PingPlotter) stay absolutely steady = during these tests, >>>>=20 >>>> So unless my ISP coincidentally started having major service = issues, I=E2=80=99m scratching my head as to why. >>>>=20 >>>> For contrast, the Ookla result is as follows: = https://www.speedtest.net/result/13865976456 with 15ms down, 18ms up = loaded latencies. >>>=20 >>> In Ventura, we started adding the latency on the load-generating = connections to the final RPM-calulcation as well. The formula being used = is now exactly what is in the v01 IETF draft. >>>=20 >>> Very likely the bottleneck in your network does FQ, and so latency = on separate connections is very low, while your load-generating = connections are still bufferbloated. >>>=20 >>>=20 >>> Ookla measures latency only on separate connections, thus will also = be heavily impacted by FQ. >>>=20 >>>=20 >>> Does that clarify it? >>>=20 >>>=20 >>> Cheers, >>> Christoph=20 >>>=20 >>>=20 >>>>=20 >>>> Further machine details: MacBook Pro 16=E2=80=9D (2019) using a = USB-C to Ethernet adapter. >>>> I run with full ECN enabled: >>>> sudo sysctl -w net.inet.tcp.disable_tcp_heuristics=3D1=20 >>>>=20 >>>> sudo sysctl -w net.inet.tcp.ecn_initiate_out=3D1 >>>>=20 >>>> sudo sysctl -w net.inet.tcp.ecn_negotiate_in=3D1 >>>>=20 >>>> and also with instant ack replies: >>>>=20 >>>> sysctl net.inet.tcp.delayed_ack >>>> net.inet.tcp.delayed_ack: 0 >>>>=20 >>>> I did try with delayed_ack=3D1, and the results were about the = same. >>>>=20 >>>> Thanks in advance, >>>>=20 >>>> Jonathan Foulkes >>>>=20 >>>> _______________________________________________ >>>> Rpm mailing list >>>> Rpm@lists.bufferbloat.net >>>> https://lists.bufferbloat.net/listinfo/rpm >>>=20 >>=20 >> _______________________________________________ >> Rpm mailing list >> Rpm@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/rpm >=20 --Apple-Mail=_2706DF7D-B4EE-44E4-994E-579899F9FA94 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On Nov = 4, 2022, at 1:50 PM, jf@jonathanfoulkes.com wrote:

As = I mentioned, here are the results of running with ECN enabled vs = disabled.

First, we retest with ECN enabled at the = router, the SQM instance and fully enabled on the MacOS (see a prior = post with the settings).

This is a DOCSIS 3.0 = line at 300Mbps down and 25Mbps Up, router is an x86 i5-4200 running = OpenWRT 22.03 with layer_cake QoS.

=3D=3D=3D= =3D SUMMARY =3D=3D=3D=3D
Uplink capacity: 20.971 Mbps = (Accuracy: High)
Downlink capacity: 267.643 Mbps (Accuracy: = High)
Uplink Responsiveness: Medium (550 RPM) (Accuracy: = High)
Downlink Responsiveness: Medium (900 RPM) (Accuracy: = High)
Idle Latency: 21.417 milli-seconds (Accuracy: = High)
Interface: en6
Uplink bytes transferred: = 24.359 MB
Downlink bytes transferred: 298.730 = MB
Uplink Flow count: 12
Downlink Flow count: = 12
Start: 11/4/22, 3:41:30 PM
End: 11/4/22, 3:41:49 = PM
OS Version: Version 13.0 (Build = 22A380)

Then I disabled ECN support on the = router, the sqm instance, and the MacOS, then re-ran a couple of times, = and this is a representative result:

=3D=3D=3D= =3D SUMMARY =3D=3D=3D=3D
Uplink capacity: 20.961 Mbps = (Accuracy: High)
Downlink capacity: 255.146 Mbps (Accuracy: = High)
Uplink Responsiveness: Medium (475 RPM) (Accuracy: = High)
Downlink Responsiveness: Medium (591 RPM) (Accuracy: = High)
Idle Latency: 23.583 milli-seconds (Accuracy: = High)
Interface: en6
Uplink bytes transferred: = 24.593 MB
Downlink bytes transferred: 278.096 = MB
Uplink Flow count: 12
Downlink Flow count: = 12
Start: 11/4/22, 4:11:59 PM
End: 11/4/22, 4:12:18 = PM
OS Version: Version 13.0 (Build = 22A380)

The absence of ECN does have an = impact, most significantly on download, where it drops from 900 to 591 = RPM

That=E2=80=99s= interesting and good to know that the tool can measure the positive = impact of ECN!

My takeaway is that ECN does = help with responsiveness overall.

What I=E2=80=99= m still puzzled by here is the bloat and why changes to QoS settings do = not make a = difference.

 = That is interesting indeed. What configurations are you testing? At this = level, pcaps may be = helpful.


Christoph

<= /div>

I have questions about how = the lag within a bulk flow is a factor in the calculation, but I=E2=80=99l= l first go read the latest version of the document.
In case I = missed it, is there a discussion thread someone can point me to where = this change is = mentioned?

The = draft itself has had latency on bulk-flows since always. We just = hadn=E2=80=99t backed this into the implementation, due to a bad = server-side responsiveness which we have addressed = now.


Thanks,

<= div>Jonathan


On Nov 3, 2022, at 6:09 PM, jf--- via Rpm = <rpm@lists.bufferbloat.net> wrote:

Hi = Christoph,

Thanks for the reply, it clarifies why the = metric would be different but it then leads to questions regarding how / = where bufferbloat is occurring on the links creating the load. I noted = the tc stats show a peak delay of 81ms on download and 139ms on upload, = so there is indeed some queue build-up in the = router.

So even testing cutting QoS to 50% of = the upstream capacity and still getting no better than medium RPM = rates.

Here is the test run on that unit set = for roughly half ( 80 / 10 ) of the upstream line ( 180 / 24 ), this one = from a 12.6 OS

=3D=3D=3D=3D SUMMARY =3D=3D=3D= =3D                   =                     =                     =                     =           
Upload capacity: = 8.679 Mbps
Download capacity: 75.213 Mbps
Upload = flows: 20
Download flows: 12
Upload Responsiveness: = High (2659 RPM)
Download Responsiveness: High (2587 = RPM)
Base RTT: 14
Start: 11/3/22, 4:05:01 = PM
End: 11/3/22, 4:05:26 PM
OS Version: Version 12.6 = (Build 21G115)

And this one from Ventura = (13)

=3D=3D=3D=3D SUMMARY = =3D=3D=3D=3D
Uplink capacity: 9.328 Mbps (Accuracy: = High)
Downlink capacity: 76.555 Mbps (Accuracy: = High)
Uplink Responsiveness: Low (143 RPM) (Accuracy: = High)
Downlink Responsiveness: Medium (380 RPM) (Accuracy: = High)
Idle Latency: 29.000 milli-seconds (Accuracy: = High)
Interface: en6
Uplink bytes transferred: = 16.734 MB
Downlink bytes transferred: 85.637 = MB
Uplink Flow count: 20
Downlink Flow count: = 12
Start: 11/3/22, 4:03:33 PM
End: 11/3/22, 4:03:58 = PM
OS Version: Version 13.0 (Build = 22A380)

Does all-out use of ECN cause a = penalty?

On download, we recorded 9504 marks, = but only 38 drops. So the flows should have been well managed with all = the early feedback to senders. 

Do the = metrics look for drops and thus this low drop rate seems like there is = bloat given the amount of traffic in = flight?

The device under test is an MT7621 = running stock OpenWRT 22.03.2 with SQM installed and using layer-cake. = But we see similar metrics on an i5-4200 x86 box with Intel NICs. So = it=E2=80=99s not horsepower related.
I just retested on the = x86, with the same ballpark results.

I=E2=80=99ll= re-test tomorrow with all the ECN features on the Mac and the router = disabled to see what that does to the = metrics.

Thanks,

Jonatha= n



On Nov 1, 2022, at 5:52 PM, Christoph Paasch = <cpaasch@apple.com> wrote:

Hello Jonathan,

On Oct 28, 2022, at 2:45 PM, jf--- via Rpm = <rpm@lists.bufferbloat.net> wrote:

Hopefully, Christoph can provide some details on the = changes from the prior networkQuality test, as we=E2=80=99re seeing some = pretty large changes in results for the latest RPM = tests.

Where before we=E2=80=99d see results in the = >1,500 RPM (and multiple >2,000 RPM results) for a DOCSIS 3.1 line = with QoS enabled (180 down/35 up), it now returns peak download RPM of = ~600 and ~800 for upload.

latest = results:

=3D=3D=3D=3D SUMMARY = =3D=3D=3D=3D
Uplink capacity: 25.480 Mbps (Accuracy: = High)
Downlink capacity: 137.768 Mbps (Accuracy: = High)
Uplink Responsiveness: Medium (385 RPM) (Accuracy: = High)
Downlink Responsiveness: Medium (376 RPM) (Accuracy: = High)
Idle Latency: 43.875 milli-seconds (Accuracy: = High)
Interface: en8
Uplink bytes transferred: = 35.015 MB
Downlink bytes transferred: 154.649 = MB
Uplink Flow count: 16
Downlink Flow count: = 12
Start: 10/28/22, 5:12:30 PM
End: 10/28/22, = 5:12:54 PM
OS Version: Version 13.0 (Build = 22A380)

Latencies (as monitored via = PingPlotter) stay absolutely steady during these = tests,

So unless my ISP coincidentally started = having major service issues, I=E2=80=99m scratching my head as to = why.

For contrast, the Ookla result is as = follows: https://www.speedtes= t.net/result/13865976456  with 15ms down, 18ms up loaded = latencies.

In Ventura, = we started adding the latency on the load-generating connections to the = final RPM-calulcation as well. The formula being used is now exactly = what is in the v01 IETF draft.

Very likely the = bottleneck in your network does FQ, and so latency on separate = connections is very low, while your load-generating connections are = still bufferbloated.


Ookla = measures latency only on separate connections, thus will also be heavily = impacted by FQ.


Does that = clarify = it?


Cheers,
Christoph&nb= sp;



Further machine = details: MacBook Pro 16=E2=80=9D (2019) using a USB-C to Ethernet = adapter.
I run with full ECN enabled:

sudo sysctl -w = net.inet.tcp.disable_tcp_heuristics=3D1 

sudo sysctl -w net.inet.tcp.ecn_initiate_out=3D1

sudo sysctl -w = net.inet.tcp.ecn_negotiate_in=3D1

a= nd also with instant ack replies:

sysctl = net.inet.tcp.delayed_ack
net.inet.tcp.delayed_ack: = 0

I did try with delayed_ack=3D1, and the = results were about the same.

Thanks in = advance,

Jonathan = Foulkes

________________________________________= _______
Rpm mailing = list
Rpm@lists.bufferbloat.net
https://lists.bufferbloat.net/listinf= o/rpm


<= /div>
_______________________________________________
Rpm = mailing = list
Rpm@lists.bufferbloat.net
https://lists.bufferbloat.net/listinf= o/rpm


= --Apple-Mail=_2706DF7D-B4EE-44E4-994E-579899F9FA94--