* [Ecn-sane] The curious case of "cursed-ECN" steam downloads
@ 2023-09-03 18:54 Sebastian Moeller
2023-09-03 22:36 ` Jonathan Morton
0 siblings, 1 reply; 3+ messages in thread
From: Sebastian Moeller @ 2023-09-03 18:54 UTC (permalink / raw)
To: bloat, ECN-Sane
Dear ECN experts,
I want to report some oddities I encounter with downloading data on steam's network. (My kids started playing steam games recently, so my network started to see steam loads). Some games apparently need routine updates every few weeks in the multi GB range which on the one hand seems quite excessive to me, but on the other hand it presents a nice way to look closer at how modern CDN-backed downloads operate. All of this is flowing trough my OpenWrt-based router using cake traffic-shaper/scheduler/AQM combos in both directions. (Cake by default uses rfc3168 ECN signaling, for packets presenting either ECT(0) or ECT(1), but concurrently employs a BLUE to increase pure drop probability to eventually also reign in unresponsive flows).
So far I have seen two things worth noting:
A) excessive CE marking (this happened with a multi-GB download served from cloudflare CDN nodes) on the range of 70% of packets marked; Jonathan gave a reasonable explanation that this might be BBRv1 in action. Side-note: am I the only one slightly miffed that BBR apparently ignored to implement either an appropriate CE response or to make sure BBR using TCPs refrain from negotiating ECN? Unfortunately I did not take packet captures of this event (only tc -s qdisc snapshots before and after). This I posted earlier on the list already.
B) Excessive ECT(1) marking (this happened with a multi-GB download)
Here is cctrace output for the respective flow (cctrace was developed as part of SCE and the reported SCE events are equivalent with ECT(1))
55827-443:
Up: SCE=0, CE=0, ECE=79538, CWR=0, NS=0, total=104222
Down: SCE=200260, CE=0, ECE=0, CWR=2691, NS=0, total=200429
Here is what wireshark reported for the same TCP flow
{
"Address A": "2a01:c22:8c6c:8700:b84b:c89e:6424:8c74",
"Address B": "2a01:bc80:7:100::9b85:f813",
"Bits/s A → B": "289152",
"Bits/s B → A": "9205750",
"Bytes": "305873523",
"Bytes A → B": "9314897",
"Bytes B → A": "296558626",
"Duration": "257.715992",
"Packets": "304651",
"Packets A → B": "104222",
"Packets B → A": "200429",
"Percent Filtered": "0",
"Port A": "55827",
"Port B": "443",
"Rel Start": "314.512067",
"Stream ID": "259",
"Total Packets": "0"
},
So all in all roughly 5% of the total download volume, but essentially all ECT(1).
Here I took a packet capture (albeit upstream of my ingress shaper so I see no CE markings or dropped packets), but I failed to take the tc -s qdisc snapshots to get easy access to number of CE marks and drops (these would not be per flow anyway).
123-1234567:CAKE-autorate user$ sudo mtr -ezb6w -c 100 2a01:bc80:7:100::9b85:f813
Password:
Start: 2023-09-03T19:53:45+0200
HOST: 123-1234567.local Loss% Snt Last Avg Best Wrst StDev
1. AS6805 dynamic-2a01-0c23-9012-7900-0000-0000-0000-0001.c23.pool.telefonica.de (2a01:c23:9012:7900::1) 1.0% 100 1.1 1.2 0.8 2.7 0.3
2. AS6805 2a02:3001::11e 0.0% 100 15.9 46.2 12.9 149.6 26.4
3. AS6805 2a02:3001::1b7 56.0% 100 13.7 12.7 11.4 13.9 0.6
4. AS6805 2a02:3040:0:10::1c 46.0% 100 12.2 12.8 11.7 15.1 0.6
5. AS??? ??? 100.0 100 0.0 0.0 0.0 0.0 0.0
6. AS??? ??? 100.0 100 0.0 0.0 0.0 0.0 0.0
7. AS??? amsix-v6.valve.net (2001:7f8:1::a503:2590:1) 0.0% 100 28.6 31.7 26.0 116.4 18.1
8. AS32590 2a01:bc80:7:ffff::9b85:f8fb 0.0% 100 28.7 29.0 27.5 32.6 0.8
9. AS32590 2a01:bc80:7:100::9b85:f813 0.0% 100 26.9 26.5 24.7 31.8 1.0
At an average 9.2 Mbps, I saw one CWR for every 79538/2691 = 29.5 ECEs. With ~9Mbps this would be around 1000*(29.5*1534*8)/(9.2 * 1000^2) = 39.4 ms time between sending the first ECE before seeing a CWR. This seems within range of the 24.7ms unloaded RTT to that IPv6. (Note when I first reported this I mtr's against the wrong server IPv6 and reported ~10ms RTT, but looking at the actual TCP stream and mtr-ing that today I got the above.)
The CWR response to the ECEs indicates to me, that the flow was responsive to ECN signaling (how well I can not say, as I did not capture CE marks nor packet drops) and the network stayed responsive during the download, so to me this looks like ECT(1) was used for a genuine ECN-enabled and CE-responsive flow.
Would be interesting if others could check whether they see ECT(1) in actual use on their homelinks? Initially I thought this might be my ISP doing some mismarking, but it would be quite lucky to re-mark a genuine ECT(0) flow to TOS 0x01 and hence ECT(1) so that ECN signaling would still work, while leaving the other ~8 flows from the same remote address at ECT(0) (also with working ECN). Just in case my ISP is AS6805 Telefónica Germany GmbH & Co. OHG.
For quick monitoring on my OpenWrt router, I use the following tcpdump invocation just to see what is happening (pppoe-wan is my wan interface's name):
tcpdump -i pppoe-wan -v -n '(ip6 and (ip6[0:2] & 0x30) >> 4 == 1)' or '(ip and (ip[1] & 0x3) == 1)' # ECT(1)
(to exercise/test this I use (under macos):
ping -z 0x01 -c 10 one.one.one.one
or
ping6 -z 0x01 -c 10 one.one.one.one)
and to see ECN activity in TCP I use:
tcpdump -i pppoe-wan -v -n '(tcp[tcpflags] & (tcp-ece|tcp-cwr) != 0)' or '((ip6[6] = 6) and (ip6[53] & 0xC0 != 0))' # TCP ECN flags, ECN in action
Regards
Sebastian
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [Ecn-sane] The curious case of "cursed-ECN" steam downloads
2023-09-03 18:54 [Ecn-sane] The curious case of "cursed-ECN" steam downloads Sebastian Moeller
@ 2023-09-03 22:36 ` Jonathan Morton
2023-09-04 7:14 ` Sebastian Moeller
0 siblings, 1 reply; 3+ messages in thread
From: Jonathan Morton @ 2023-09-03 22:36 UTC (permalink / raw)
To: Sebastian Moeller; +Cc: bloat, ECN-Sane
> On 3 Sep, 2023, at 9:54 pm, Sebastian Moeller via Ecn-sane <ecn-sane@lists.bufferbloat.net> wrote:
>
> B) Excessive ECT(1) marking (this happened with a multi-GB download)
This *could* be a badly configured middlebox attempting to apply a DSCP, but clobbering the entire TOS byte instead of the (left justified) DSCP field. Apparently Comcast just found a whole raft of these in their own network as part of rolling out L4S support. Funny how they didn't notice them previously.
- Jonathan Morton
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [Ecn-sane] The curious case of "cursed-ECN" steam downloads
2023-09-03 22:36 ` Jonathan Morton
@ 2023-09-04 7:14 ` Sebastian Moeller
0 siblings, 0 replies; 3+ messages in thread
From: Sebastian Moeller @ 2023-09-04 7:14 UTC (permalink / raw)
To: Jonathan Morton; +Cc: bloat, ECN-Sane
Hi Jonathan
On 4 September 2023 00:36:03 CEST, Jonathan Morton <chromatix99@gmail.com> wrote:
>> On 3 Sep, 2023, at 9:54 pm, Sebastian Moeller via Ecn-sane <ecn-sane@lists.bufferbloat.net> wrote:
>>
>> B) Excessive ECT(1) marking (this happened with a multi-GB download)
>
>This *could* be a badly configured middlebox attempting to apply a DSCP, but clobbering the entire TOS byte instead of the (left justified) DSCP field.
[SM] Mmmh, would that likely be DSCP decimal 1, aka LE? Or someone mapping e.g. a 3bit VLAN priority PCP directly onto the TOS field? What puzzles me is that from the same IPv6 I got one more data flow, 182 MB using ECT(0) and ECN... odd...
>Apparently Comcast just found a whole raft of these in their own network as part of rolling out L4S support. Funny how they didn't notice them previously.
[SM] In retrospect re-dedicating bits out of an already used header field might have been a bit optimistic....
Regards
Sebastian
>
> - Jonathan Morton
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2023-09-04 7:14 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-09-03 18:54 [Ecn-sane] The curious case of "cursed-ECN" steam downloads Sebastian Moeller
2023-09-03 22:36 ` Jonathan Morton
2023-09-04 7:14 ` Sebastian Moeller
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox