From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id C1B263B2A4; Sun, 3 Sep 2023 14:54:23 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de; s=s31663417; t=1693767262; x=1694372062; i=moeller0@gmx.de; bh=6ANoYGyRWnWO+rsxgbpZbq44LWVq+7LS1p8FYTBmS8s=; h=X-UI-Sender-Class:From:Subject:Date:To; b=XM4T5RaVTX6l89e/uDWbkeefHSnVUgBMq/Fil/UOMO/jGlVAK7dph6J8EESUmPkEDBGI8M9 KrblMrP3iqUrq3tP3AzSF0CW0gat0uoaRNDpa+x82ZbnaJ6J3xTO+MpbnYsGQIsU3a9h/ejlT 2elXBoHEQpHwBU8ZTMUEvpSVdXDFWdHVi+ri0u+fv1faCQUZP6bCUGaB+gNqiiENz96KMD88M rmQC84dH99/b36HLp7i5qRw3M3XIF7ENEOYP/SnhWn4+X7C/U99aG3BezTrzKITwXkKe82jWt XQ/siOlpCcfxq1JBYHNaEnpBEOJ5UZSZaqIpFm/7jBFGlVgCcpkw== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Received: from smtpclient.apple ([95.112.161.66]) by mail.gmx.net (mrgmx105 [212.227.17.168]) with ESMTPSA (Nemesis) id 1N4hvb-1pcpGj1WlD-011g26; Sun, 03 Sep 2023 20:54:22 +0200 From: Sebastian Moeller Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.4\)) Message-Id: <758B6681-14B5-473E-9706-45B7E2927505@gmx.de> Date: Sun, 3 Sep 2023 20:54:21 +0200 To: bloat , ECN-Sane X-Mailer: Apple Mail (2.3696.120.41.1.4) X-Provags-ID: V03:K1:9EReixxP7mReP0mEIJIaMqanZ0O41nB/BCLBivjXT84MYpm0bCq q1jGfmB1lBsZoTzcVqGijgknSAGLDKom7LIyExA5i+mrGS+wbBSHGoso7iVoxFG/TC2/DNO e0cXfAK86Q3OCRS+63H9I0Er+hO0i9Fpn9t71kI7h+sLUc5C8RDJE/OipRKnqYwbJcsgUQx 12OFSVPFKKiOyt/8dnO6g== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:CUJSAbMZoJs=;rbZ0wjBAkFxRdmKsCRDhXDbBbkj HZGV13FAGtVGafOnFjDzubRg9FuDmhPkeENYYtoqWIcKeWx8dGhw5GRFFNYQGDOj3kMSide/5 BbrgGKiNk+TisN525ZAW23HZi/rKRaUhfa5rkGFmQS6M+H66RSXS2xStP84ViRW4XmCuvgST8 NZ03Q07Dim18tBZ3aDaStkuQtbGcYELCKpDwwdjMlu1a5DMN3DBYadM4bx5LHFRdcBdKDhIC4 vfMQy7OG8UcqfY0LC8RSsUiCPVTN90hpwTtpTG2zfoIhwT7fkl+rRKiIegnCCgwy0LUkZBbYl bii7jGq+cwvtgbG6RQ/q4GjdgmqshYaY4+jWFVVqmHXIzCsWRKcuDRupt5SYQmMry02lTImj+ xU85kQVRNNHeXnlobP+FXK7Cz+BY3pFH1X50gRKP6D1p26tiHMR8CFWjFnCgS6KJ+QHbg/COc yIxi5YYUpQAj7RidVeVJ9wZeX8ZAAVEZA1mHLwGFoNOb3SVzWQ3/57MOYLo/M5kJ2SOPyyeOe IvI1whMe1yEWJ/0QYv9tZZPcPvD+g9skqBfo3FDlC8kIKGhoxpwv2P+NvAqp3m0zEeE9XhdlN 43ufmiRnJCGTK3F2e76VzoVOErjRmHS/VkFMRwKD/l+7C4VlSvwmlI8oWIpPoS9HSrr7oBp+z qQ/2dPDf7vAzfRom9SQ1nDE4xJV/wOiSOTZ6BaMNF40EUFJpDuOL/OOgN1v+u1ggDCKp6z6fK mjoDR9jOebIUOTlaKP+uBvB/DtOOO+yxdD+4cTq2Xp1DyzZKfLZ8uPzDCpvK82l89gJPyDQmV CaaM6cYEonn0a2Kw0pGgKV2stVDOOJL04QbaxcZrIQCxbYjW/My2qn7vef4TUf6aAWolElPaH 1XUwvJcUs20lZ1bYNrYvZDn/7hJs80oy4lC1TrELmd21RmSvxNXWOBkcuGrPPOWZI24PLZ6l4 LVOvwQ== Subject: [Ecn-sane] The curious case of "cursed-ECN" steam downloads X-BeenThere: ecn-sane@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of explicit congestion notification's impact on the Internet List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Sep 2023 18:54:24 -0000 Dear ECN experts, I want to report some oddities I encounter with downloading data on = steam's network. (My kids started playing steam games recently, so my = network started to see steam loads). Some games apparently need routine = updates every few weeks in the multi GB range which on the one hand = seems quite excessive to me, but on the other hand it presents a nice = way to look closer at how modern CDN-backed downloads operate. All of = this is flowing trough my OpenWrt-based router using cake = traffic-shaper/scheduler/AQM combos in both directions. (Cake by default = uses rfc3168 ECN signaling, for packets presenting either ECT(0) or = ECT(1), but concurrently employs a BLUE to increase pure drop = probability to eventually also reign in unresponsive flows). So far I have seen two things worth noting: A) excessive CE marking (this happened with a multi-GB download served = from cloudflare CDN nodes) on the range of 70% of packets marked; = Jonathan gave a reasonable explanation that this might be BBRv1 in = action. Side-note: am I the only one slightly miffed that BBR apparently = ignored to implement either an appropriate CE response or to make sure = BBR using TCPs refrain from negotiating ECN? Unfortunately I did not = take packet captures of this event (only tc -s qdisc snapshots before = and after). This I posted earlier on the list already. B) Excessive ECT(1) marking (this happened with a multi-GB download) Here is cctrace output for the respective flow (cctrace was developed as = part of SCE and the reported SCE events are equivalent with ECT(1)) 55827-443: = = =20 Up: SCE=3D0, CE=3D0, ECE=3D79538, CWR=3D0, NS=3D0, total=3D104222 = = =20 Down: SCE=3D200260, CE=3D0, ECE=3D0, CWR=3D2691, NS=3D0, total=3D200429= =20 Here is what wireshark reported for the same TCP flow { "Address A": "2a01:c22:8c6c:8700:b84b:c89e:6424:8c74", "Address B": "2a01:bc80:7:100::9b85:f813", "Bits/s A =E2=86=92 B": "289152", "Bits/s B =E2=86=92 A": "9205750", "Bytes": "305873523", "Bytes A =E2=86=92 B": "9314897", "Bytes B =E2=86=92 A": "296558626", "Duration": "257.715992", "Packets": "304651", "Packets A =E2=86=92 B": "104222", "Packets B =E2=86=92 A": "200429", "Percent Filtered": "0", "Port A": "55827", "Port B": "443", "Rel Start": "314.512067", "Stream ID": "259", "Total Packets": "0" }, So all in all roughly 5% of the total download volume, but essentially = all ECT(1). Here I took a packet capture (albeit upstream of my ingress shaper so I = see no CE markings or dropped packets), but I failed to take the tc -s = qdisc snapshots to get easy access to number of CE marks and drops = (these would not be per flow anyway). 123-1234567:CAKE-autorate user$ sudo mtr -ezb6w -c 100 = 2a01:bc80:7:100::9b85:f813 Password: Start: 2023-09-03T19:53:45+0200 HOST: 123-1234567.local = Loss% Snt Last Avg Best = Wrst StDev 1. AS6805 = dynamic-2a01-0c23-9012-7900-0000-0000-0000-0001.c23.pool.telefonica.de = (2a01:c23:9012:7900::1) 1.0% 100 1.1 1.2 0.8 2.7 0.3 2. AS6805 2a02:3001::11e = 0.0% 100 15.9 46.2 12.9 = 149.6 26.4 3. AS6805 2a02:3001::1b7 = 56.0% 100 13.7 12.7 11.4 = 13.9 0.6 4. AS6805 2a02:3040:0:10::1c = 46.0% 100 12.2 12.8 11.7 = 15.1 0.6 5. AS??? ??? = 100.0 100 0.0 0.0 0.0 = 0.0 0.0 6. AS??? ??? = 100.0 100 0.0 0.0 0.0 = 0.0 0.0 7. AS??? amsix-v6.valve.net (2001:7f8:1::a503:2590:1) = 0.0% 100 28.6 31.7 26.0 = 116.4 18.1 8. AS32590 2a01:bc80:7:ffff::9b85:f8fb = 0.0% 100 28.7 29.0 27.5 = 32.6 0.8 9. AS32590 2a01:bc80:7:100::9b85:f813 = 0.0% 100 26.9 26.5 24.7 = 31.8 1.0 At an average 9.2 Mbps, I saw one CWR for every 79538/2691 =3D 29.5 = ECEs. With ~9Mbps this would be around 1000*(29.5*1534*8)/(9.2 * 1000^2) = =3D 39.4 ms time between sending the first ECE before seeing a CWR. This = seems within range of the 24.7ms unloaded RTT to that IPv6. (Note when I = first reported this I mtr's against the wrong server IPv6 and reported = ~10ms RTT, but looking at the actual TCP stream and mtr-ing that today I = got the above.) The CWR response to the ECEs indicates to me, that the flow was = responsive to ECN signaling (how well I can not say, as I did not = capture CE marks nor packet drops) and the network stayed responsive = during the download, so to me this looks like ECT(1) was used for a = genuine ECN-enabled and CE-responsive flow. Would be interesting if others could check whether they see ECT(1) in = actual use on their homelinks? Initially I thought this might be my ISP = doing some mismarking, but it would be quite lucky to re-mark a genuine = ECT(0) flow to TOS 0x01 and hence ECT(1) so that ECN signaling would = still work, while leaving the other ~8 flows from the same remote = address at ECT(0) (also with working ECN). Just in case my ISP is AS6805 = Telef=C3=B3nica Germany GmbH & Co. OHG.=20 For quick monitoring on my OpenWrt router, I use the following tcpdump = invocation just to see what is happening (pppoe-wan is my wan = interface's name): tcpdump -i pppoe-wan -v -n '(ip6 and (ip6[0:2] & 0x30) >> 4 =3D=3D 1)' = or '(ip and (ip[1] & 0x3) =3D=3D 1)' # ECT(1) (to exercise/test this I use (under macos):=20 ping -z 0x01 -c 10 one.one.one.one or ping6 -z 0x01 -c 10 one.one.one.one) and to see ECN activity in TCP I use: tcpdump -i pppoe-wan -v -n '(tcp[tcpflags] & (tcp-ece|tcp-cwr) !=3D 0)' = or '((ip6[6] =3D 6) and (ip6[53] & 0xC0 !=3D 0))' # TCP ECN flags, ECN = in action Regards Sebastian