<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:monospace;font-size:small"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Nov 12, 2019 at 2:02 PM Toke Høiland-Jørgensen <<a href="mailto:toke@toke.dk">toke@toke.dk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Mikael Abrahamsson <<a href="mailto:swmike@swm.pp.se" target="_blank">swmike@swm.pp.se</a>> writes:<br>
<br>
> On Tue, 12 Nov 2019, Toke Høiland-Jørgensen wrote:<br>
><br>
>> I'm not on the nanog list, but feel free to cross-post; would be good to <br>
>> actually get to the bottom of this issue! Marek and I already had an <br>
>> off-list back-and-forth after that original thread, and we couldn't find <br>
>> anything wrong on the Cloudflare side. And the RSTs have a higher TTL <br>
>> than the actual traffic, indicating an in-path problem...<br>
><br>
> tcptraceroute supports setting/clearing ECN bits (-E), would be very <br>
> interesting to see difference between those tcptraceroutes?<br>
<br>
No difference. But the RST is not being sent as a response to the SYN;<br>
it is sent in response to the first data packet...<br>
<br>
... and now that I'm re-testing, things were working for a little while,<br>
but now the bug is back. I got an intermittent successful connection<br>
with the same TTL that I was previously getting the RST from. And now<br>
I'm back to getting RSTed.<br>
<br>
So I guess there's some kind of multipath issue here; ECMP path,<br>
multiple routing upstreams, or a broken load balancer? Any other ideas?<br></blockquote><div><br></div><div><br></div><div><div class="gmail_default" style="font-family:monospace;font-size:small">It makes me think of some usage of anycast TCP on the cloudflare side.</div><div class="gmail_default" style="font-family:monospace;font-size:small">What service is this Toke?</div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
-Toke<br>
<br>
<br>
tcpdump output:<br>
<br>
With ECN, and failing. Notice TTL 59 for the SYNACK, but TTL 61 for the<br>
RST:<br>
<br>
00:00:00.000000 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 21817, offset 0, flags [DF], proto TCP (6), length 60)<br>
85.204.121.218.33376 > 1.1.1.1.80: Flags [SEW], cksum 0x5284 (correct), seq 1677914250, win 64240, options [mss 1460,sackOK,TS val 438384324 ecr 0,nop,wscale 7], length 0<br>
00:00:00.006962 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto TCP (6), length 52)<br>
1.1.1.1.80 > 85.204.121.218.33376: Flags [S.E], cksum 0x4e79 (correct), seq 1887212753, ack 1677914251, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 10], length 0<br>
00:00:00.000614 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 63, id 21818, offset 0, flags [DF], proto TCP (6), length 40)<br>
85.204.121.218.33376 > 1.1.1.1.80: Flags [.], cksum 0xffa8 (correct), seq 1, ack 1, win 502, length 0<br>
00:00:00.000255 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0800), length 125: (tos 0x2,ECT(0), ttl 63, id 21819, offset 0, flags [DF], proto TCP (6), length 111)<br>
85.204.121.218.33376 > 1.1.1.1.80: Flags [P.], cksum 0x05e5 (correct), seq 1:72, ack 1, win 502, length 71: HTTP, length: 71<br>
GET / HTTP/1.1<br>
Host: 1.1.1.1<br>
User-Agent: curl/7.66.0<br>
Accept: */*<br>
<br>
00:00:00.001714 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0x0800), length 60: (tos 0x2,ECT(0), ttl 61, id 0, offset 0, flags [DF], proto TCP (6), length 40)<br>
1.1.1.1.80 > 85.204.121.218.33376: Flags [R], cksum 0x5639 (correct), seq 1887212754, win 0, length 0<br>
<br>
<br>
Without ECN; succeeding, with TTL 59:<br>
<br>
00:00:00.000000 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 54830, offset 0, flags [DF], proto TCP (6), length 60)<br>
85.204.121.218.33362 > 1.1.1.1.80: Flags [S], cksum 0x5430 (correct), seq 922398600, win 64240, options [mss 1460,sackOK,TS val 438346737 ecr 0,nop,wscale 7], length 0<br>
00:00:00.006895 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto TCP (6), length 52)<br>
1.1.1.1.80 > 85.204.121.218.33362: Flags [S.], cksum 0xbdf8 (correct), seq 1251654028, ack 922398601, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 10], length 0<br>
00:00:00.000570 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 63, id 54831, offset 0, flags [DF], proto TCP (6), length 40)<br>
85.204.121.218.33362 > 1.1.1.1.80: Flags [.], cksum 0x6ee8 (correct), seq 1, ack 1, win 502, length 0<br>
00:00:00.000261 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0800), length 125: (tos 0x0, ttl 63, id 54832, offset 0, flags [DF], proto TCP (6), length 111)<br>
85.204.121.218.33362 > 1.1.1.1.80: Flags [P.], cksum 0x7524 (correct), seq 1:72, ack 1, win 502, length 71: HTTP, length: 71<br>
GET / HTTP/1.1<br>
Host: 1.1.1.1<br>
User-Agent: curl/7.66.0<br>
Accept: */*<br>
<br>
00:00:00.006955 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 59, id 46658, offset 0, flags [DF], proto TCP (6), length 40)<br>
1.1.1.1.80 > 85.204.121.218.33362: Flags [.], cksum 0x707a (correct), seq 1, ack 72, win 29, length 0<br>
00:00:00.004938 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0x0800), length 609: (tos 0x0, ttl 59, id 46659, offset 0, flags [DF], proto TCP (6), length 595)<br>
1.1.1.1.80 > 85.204.121.218.33362: Flags [P.], cksum 0x13dc (correct), seq 1:556, ack 72, win 29, length 555: HTTP, length: 555<br>
HTTP/1.1 301 Moved Permanently<br>
Date: Fri, 20 Sep 2019 09:33:56 GMT<br>
Content-Type: text/html<br>
Transfer-Encoding: chunked<br>
Connection: keep-alive<br>
Location: <a href="https://1.1.1.1/" rel="noreferrer" target="_blank">https://1.1.1.1/</a><br>
Served-In-Seconds: 0.000<br>
CF-Cache-Status: HIT<br>
Age: 3920<br>
Expires: Fri, 20 Sep 2019 13:33:56 GMT<br>
Cache-Control: public, max-age=14400<br>
Server: cloudflare<br>
CF-RAY: 5192ccfbeeefd47b-HAM<br>
<br>
ba<br>
<html><br>
<head><title>301 Moved Permanently</title></head><br>
<body bgcolor="white"><br>
<center><h1>301 Moved Permanently</h1></center><br>
<hr><center>cloudflare-lb</center><br>
</body><br>
</html><br>
<br>
00:00:00.000002 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 59, id 46660, offset 0, flags [DF], proto TCP (6), length 45)<br>
1.1.1.1.80 > 85.204.121.218.33362: Flags [P.], cksum 0x2a28 (correct), seq 556:561, ack 72, win 29, length 5: HTTP<br>
00:00:00.000549 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 63, id 54833, offset 0, flags [DF], proto TCP (6), length 40)<br>
85.204.121.218.33362 > 1.1.1.1.80: Flags [.], cksum 0x6c77 (correct), seq 72, ack 556, win 501, length 0<br>
00:00:00.000266 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 63, id 54834, offset 0, flags [DF], proto TCP (6), length 40)<br>
85.204.121.218.33362 > 1.1.1.1.80: Flags [.], cksum 0x6c72 (correct), seq 72, ack 561, win 501, length 0<br>
00:00:00.000217 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 63, id 54835, offset 0, flags [DF], proto TCP (6), length 40)<br>
85.204.121.218.33362 > 1.1.1.1.80: Flags [F.], cksum 0x6c71 (correct), seq 72, ack 561, win 501, length 0<br>
00:00:00.007287 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 59, id 46661, offset 0, flags [DF], proto TCP (6), length 40)<br>
1.1.1.1.80 > 85.204.121.218.33362: Flags [F.], cksum 0x6e48 (correct), seq 561, ack 73, win 29, length 0<br>
00:00:00.000504 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 63, id 54836, offset 0, flags [DF], proto TCP (6), length 40)<br>
85.204.121.218.33362 > 1.1.1.1.80: Flags [.], cksum 0x6c70 (correct), seq 73, ack 562, win 501, length 0<br>
00:00:05.170886 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 11852, offset 0, flags [DF], proto TCP (6), length 60)<br>
<br>
<br>
And that one time it worked, with TTL 61:<br>
<br>
13:47:54.908967 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 53207, offset 0, flags [DF], proto TCP (6), length 60)<br>
85.204.121.218.48924 > 1.1.1.1.80: Flags [SEW], cksum 0xa5de (correct), seq 3526272449, win 64240, options [mss 1460,sackOK,TS val 513441489 ecr 0,nop,wscale 7], length 0<br>
13:47:54.910220 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 61, id 0, offset 0, flags [DF], proto TCP (6), length 52)<br>
1.1.1.1.80 > 85.204.121.218.48924: Flags [S.E], cksum 0x17dd (correct), seq 633452041, ack 3526272450, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 10], length 0<br>
13:47:54.910747 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 63, id 53208, offset 0, flags [DF], proto TCP (6), length 40)<br>
85.204.121.218.48924 > 1.1.1.1.80: Flags [.], cksum 0xc90c (correct), seq 1, ack 1, win 502, length 0<br>
13:47:54.910990 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0800), length 125: (tos 0x2,ECT(0), ttl 63, id 53209, offset 0, flags [DF], proto TCP (6), length 111)<br>
85.204.121.218.48924 > 1.1.1.1.80: Flags [P.], cksum 0xcf48 (correct), seq 1:72, ack 1, win 502, length 71: HTTP, length: 71<br>
GET / HTTP/1.1<br>
Host: 1.1.1.1<br>
User-Agent: curl/7.66.0<br>
Accept: */*<br>
<br>
13:47:55.119451 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0800), length 125: (tos 0x0, ttl 63, id 53210, offset 0, flags [DF], proto TCP (6), length 111)<br>
85.204.121.218.48924 > 1.1.1.1.80: Flags [P.], cksum 0xcf48 (correct), seq 1:72, ack 1, win 502, length 71: HTTP, length: 71<br>
GET / HTTP/1.1<br>
Host: 1.1.1.1<br>
User-Agent: curl/7.66.0<br>
Accept: */*<br>
<br>
13:47:55.120638 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 61, id 41447, offset 0, flags [DF], proto TCP (6), length 40)<br>
1.1.1.1.80 > 85.204.121.218.48924: Flags [.], cksum 0xca9e (correct), seq 1, ack 72, win 29, length 0<br>
13:47:55.130264 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0x0800), length 609: (tos 0x2,ECT(0), ttl 61, id 41448, offset 0, flags [DF], proto TCP (6), length 595)<br>
1.1.1.1.80 > 85.204.121.218.48924: Flags [P.], cksum 0xde5e (correct), seq 1:556, ack 72, win 29, length 555: HTTP, length: 555<br>
HTTP/1.1 301 Moved Permanently<br>
Date: Tue, 12 Nov 2019 12:47:55 GMT<br>
Content-Type: text/html<br>
Transfer-Encoding: chunked<br>
Connection: keep-alive<br>
Location: <a href="https://1.1.1.1/" rel="noreferrer" target="_blank">https://1.1.1.1/</a><br>
Served-In-Seconds: 0.000<br>
CF-Cache-Status: HIT<br>
Age: 2976<br>
Expires: Tue, 12 Nov 2019 16:47:55 GMT<br>
Cache-Control: public, max-age=14400<br>
Server: cloudflare<br>
CF-RAY: 53489e018ad8d885-CPH<br>
<br>
ba<br>
<html><br>
<head><title>301 Moved Permanently</title></head><br>
<body bgcolor="white"><br>
<center><h1>301 Moved Permanently</h1></center><br>
<hr><center>cloudflare-lb</center><br>
</body><br>
</html><br>
<br>
13:47:55.130265 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0x0800), length 60: (tos 0x2,ECT(0), ttl 61, id 41449, offset 0, flags [DF], proto TCP (6), length 45)<br>
1.1.1.1.80 > 85.204.121.218.48924: Flags [P.], cksum 0x844c (correct), seq 556:561, ack 72, win 29, length 5: HTTP<br>
13:47:55.130777 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 63, id 53211, offset 0, flags [DF], proto TCP (6), length 40)<br>
85.204.121.218.48924 > 1.1.1.1.80: Flags [.], cksum 0xc69b (correct), seq 72, ack 556, win 501, length 0<br>
13:47:55.131097 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 63, id 53212, offset 0, flags [DF], proto TCP (6), length 40)<br>
85.204.121.218.48924 > 1.1.1.1.80: Flags [.], cksum 0xc696 (correct), seq 72, ack 561, win 501, length 0<br>
13:47:55.131491 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 63, id 53213, offset 0, flags [DF], proto TCP (6), length 40)<br>
85.204.121.218.48924 > 1.1.1.1.80: Flags [F.], cksum 0xc695 (correct), seq 72, ack 561, win 501, length 0<br>
13:47:55.132804 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 61, id 41450, offset 0, flags [DF], proto TCP (6), length 40)<br>
1.1.1.1.80 > 85.204.121.218.48924: Flags [F.], cksum 0xc86c (correct), seq 561, ack 73, win 29, length 0<br>
13:47:55.133281 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 63, id 53214, offset 0, flags [DF], proto TCP (6), length 40)<br>
85.204.121.218.48924 > 1.1.1.1.80: Flags [.], cksum 0xc694 (correct), seq 73, ack 562, win 501, length 0<br>
<br>
_______________________________________________<br>
Ecn-sane mailing list<br>
<a href="mailto:Ecn-sane@lists.bufferbloat.net" target="_blank">Ecn-sane@lists.bufferbloat.net</a><br>
<a href="https://lists.bufferbloat.net/listinfo/ecn-sane" rel="noreferrer" target="_blank">https://lists.bufferbloat.net/listinfo/ecn-sane</a><br>
</blockquote></div></div>