From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.toke.dk (mail.toke.dk [IPv6:2a00:7660:6da:2001::664]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 99DBC3B29D for ; Tue, 12 Nov 2019 08:02:25 -0500 (EST) From: Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=toke.dk; s=20161023; t=1573563743; bh=dRZVPU6ez7G8vxGdUFeMJ2lSqka6hmWCFvxyhAoFsI8=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=ny9WNZCPTO6HhYKlY+OXHpocJugxfr4kngT05CLgyYfJlVbqVzcg7bUYsdL7YH9kV t/o9YDaFNzZnCIWgAHVMBuR64POPCkST/S1It3H9XFKhbZW1LNKv+PvP01WNXdEB4p D+tle4oPSn2MvquvTWWgLzLd9J/RVLDmyT1FsAEviQJbSLuv+qw65jjSMjTIg21+dU 5CIXQWCmx8v06m/f15Xz0hjUFLhVbySkszdeuVSVT71lpDyVj0q7cYzF4vRkMc/tZ7 OtYGYghTpSYcR7qsKqEd6xjenxSBamtTaXrAVDhHBZwgln1D7SrkBcW5UunikKGIpx FddX2kBr/d/sg== To: Mikael Abrahamsson Cc: Rich Brown , ecn-sane@lists.bufferbloat.net In-Reply-To: References: <1CF67BBC-B528-4667-97AE-760DCE027466@gmail.com> <87y2wlp9p4.fsf@toke.dk> Date: Tue, 12 Nov 2019 14:02:23 +0100 X-Clacks-Overhead: GNU Terry Pratchett Message-ID: <87v9rpp7s0.fsf@toke.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [Ecn-sane] Meanwhile, over on NANOG... X-BeenThere: ecn-sane@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of explicit congestion notification's impact on the Internet List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2019 13:02:25 -0000 Mikael Abrahamsson writes: > On Tue, 12 Nov 2019, Toke H=C3=B8iland-J=C3=B8rgensen wrote: > >> I'm not on the nanog list, but feel free to cross-post; would be good to= =20 >> actually get to the bottom of this issue! Marek and I already had an=20 >> off-list back-and-forth after that original thread, and we couldn't find= =20 >> anything wrong on the Cloudflare side. And the RSTs have a higher TTL=20 >> than the actual traffic, indicating an in-path problem... > > tcptraceroute supports setting/clearing ECN bits (-E), would be very=20 > interesting to see difference between those tcptraceroutes? No difference. But the RST is not being sent as a response to the SYN; it is sent in response to the first data packet... ... and now that I'm re-testing, things were working for a little while, but now the bug is back. I got an intermittent successful connection with the same TTL that I was previously getting the RST from. And now I'm back to getting RSTed. So I guess there's some kind of multipath issue here; ECMP path, multiple routing upstreams, or a broken load balancer? Any other ideas? -Toke tcpdump output: With ECN, and failing. Notice TTL 59 for the SYNACK, but TTL 61 for the RST: 00:00:00.000000 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0= 800), length 74: (tos 0x0, ttl 63, id 21817, offset 0, flags [DF], proto TC= P (6), length 60) 85.204.121.218.33376 > 1.1.1.1.80: Flags [SEW], cksum 0x5284 (correct),= seq 1677914250, win 64240, options [mss 1460,sackOK,TS val 438384324 ecr 0= ,nop,wscale 7], length 0 00:00:00.006962 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0x0= 800), length 66: (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto TCP (6= ), length 52) 1.1.1.1.80 > 85.204.121.218.33376: Flags [S.E], cksum 0x4e79 (correct),= seq 1887212753, ack 1677914251, win 29200, options [mss 1460,nop,nop,sackO= K,nop,wscale 10], length 0 00:00:00.000614 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0= 800), length 54: (tos 0x0, ttl 63, id 21818, offset 0, flags [DF], proto TC= P (6), length 40) 85.204.121.218.33376 > 1.1.1.1.80: Flags [.], cksum 0xffa8 (correct), s= eq 1, ack 1, win 502, length 0 00:00:00.000255 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0= 800), length 125: (tos 0x2,ECT(0), ttl 63, id 21819, offset 0, flags [DF], = proto TCP (6), length 111) 85.204.121.218.33376 > 1.1.1.1.80: Flags [P.], cksum 0x05e5 (correct), = seq 1:72, ack 1, win 502, length 71: HTTP, length: 71 GET / HTTP/1.1 Host: 1.1.1.1 User-Agent: curl/7.66.0 Accept: */* 00:00:00.001714 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0x0= 800), length 60: (tos 0x2,ECT(0), ttl 61, id 0, offset 0, flags [DF], proto= TCP (6), length 40) 1.1.1.1.80 > 85.204.121.218.33376: Flags [R], cksum 0x5639 (correct), s= eq 1887212754, win 0, length 0 Without ECN; succeeding, with TTL 59: 00:00:00.000000 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0= 800), length 74: (tos 0x0, ttl 63, id 54830, offset 0, flags [DF], proto TC= P (6), length 60) 85.204.121.218.33362 > 1.1.1.1.80: Flags [S], cksum 0x5430 (correct), s= eq 922398600, win 64240, options [mss 1460,sackOK,TS val 438346737 ecr 0,no= p,wscale 7], length 0 00:00:00.006895 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0x0= 800), length 66: (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto TCP (6= ), length 52) 1.1.1.1.80 > 85.204.121.218.33362: Flags [S.], cksum 0xbdf8 (correct), = seq 1251654028, ack 922398601, win 29200, options [mss 1460,nop,nop,sackOK,= nop,wscale 10], length 0 00:00:00.000570 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0= 800), length 54: (tos 0x0, ttl 63, id 54831, offset 0, flags [DF], proto TC= P (6), length 40) 85.204.121.218.33362 > 1.1.1.1.80: Flags [.], cksum 0x6ee8 (correct), s= eq 1, ack 1, win 502, length 0 00:00:00.000261 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0= 800), length 125: (tos 0x0, ttl 63, id 54832, offset 0, flags [DF], proto T= CP (6), length 111) 85.204.121.218.33362 > 1.1.1.1.80: Flags [P.], cksum 0x7524 (correct), = seq 1:72, ack 1, win 502, length 71: HTTP, length: 71 GET / HTTP/1.1 Host: 1.1.1.1 User-Agent: curl/7.66.0 Accept: */* 00:00:00.006955 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0x0= 800), length 60: (tos 0x0, ttl 59, id 46658, offset 0, flags [DF], proto TC= P (6), length 40) 1.1.1.1.80 > 85.204.121.218.33362: Flags [.], cksum 0x707a (correct), s= eq 1, ack 72, win 29, length 0 00:00:00.004938 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0x0= 800), length 609: (tos 0x0, ttl 59, id 46659, offset 0, flags [DF], proto T= CP (6), length 595) 1.1.1.1.80 > 85.204.121.218.33362: Flags [P.], cksum 0x13dc (correct), = seq 1:556, ack 72, win 29, length 555: HTTP, length: 555 HTTP/1.1 301 Moved Permanently Date: Fri, 20 Sep 2019 09:33:56 GMT Content-Type: text/html Transfer-Encoding: chunked Connection: keep-alive Location: https://1.1.1.1/ Served-In-Seconds: 0.000 CF-Cache-Status: HIT Age: 3920 Expires: Fri, 20 Sep 2019 13:33:56 GMT Cache-Control: public, max-age=3D14400 Server: cloudflare CF-RAY: 5192ccfbeeefd47b-HAM ba 301 Moved Permanently

301 Moved Permanently

cloudflare-lb
00:00:00.000002 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0x0= 800), length 60: (tos 0x0, ttl 59, id 46660, offset 0, flags [DF], proto TC= P (6), length 45) 1.1.1.1.80 > 85.204.121.218.33362: Flags [P.], cksum 0x2a28 (correct), = seq 556:561, ack 72, win 29, length 5: HTTP 00:00:00.000549 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0= 800), length 54: (tos 0x0, ttl 63, id 54833, offset 0, flags [DF], proto TC= P (6), length 40) 85.204.121.218.33362 > 1.1.1.1.80: Flags [.], cksum 0x6c77 (correct), s= eq 72, ack 556, win 501, length 0 00:00:00.000266 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0= 800), length 54: (tos 0x0, ttl 63, id 54834, offset 0, flags [DF], proto TC= P (6), length 40) 85.204.121.218.33362 > 1.1.1.1.80: Flags [.], cksum 0x6c72 (correct), s= eq 72, ack 561, win 501, length 0 00:00:00.000217 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0= 800), length 54: (tos 0x0, ttl 63, id 54835, offset 0, flags [DF], proto TC= P (6), length 40) 85.204.121.218.33362 > 1.1.1.1.80: Flags [F.], cksum 0x6c71 (correct), = seq 72, ack 561, win 501, length 0 00:00:00.007287 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0x0= 800), length 60: (tos 0x0, ttl 59, id 46661, offset 0, flags [DF], proto TC= P (6), length 40) 1.1.1.1.80 > 85.204.121.218.33362: Flags [F.], cksum 0x6e48 (correct), = seq 561, ack 73, win 29, length 0 00:00:00.000504 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0= 800), length 54: (tos 0x0, ttl 63, id 54836, offset 0, flags [DF], proto TC= P (6), length 40) 85.204.121.218.33362 > 1.1.1.1.80: Flags [.], cksum 0x6c70 (correct), s= eq 73, ack 562, win 501, length 0 00:00:05.170886 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x0= 800), length 74: (tos 0x0, ttl 64, id 11852, offset 0, flags [DF], proto TC= P (6), length 60) And that one time it worked, with TTL 61: 13:47:54.908967 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x08= 00), length 74: (tos 0x0, ttl 63, id 53207, offset 0, flags [DF], proto TCP= (6), length 60) 85.204.121.218.48924 > 1.1.1.1.80: Flags [SEW], cksum 0xa5de (correct),= seq 3526272449, win 64240, options [mss 1460,sackOK,TS val 513441489 ecr 0= ,nop,wscale 7], length 0 13:47:54.910220 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0x08= 00), length 66: (tos 0x0, ttl 61, id 0, offset 0, flags [DF], proto TCP (6)= , length 52) 1.1.1.1.80 > 85.204.121.218.48924: Flags [S.E], cksum 0x17dd (correct),= seq 633452041, ack 3526272450, win 29200, options [mss 1460,nop,nop,sackOK= ,nop,wscale 10], length 0 13:47:54.910747 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x08= 00), length 54: (tos 0x0, ttl 63, id 53208, offset 0, flags [DF], proto TCP= (6), length 40) 85.204.121.218.48924 > 1.1.1.1.80: Flags [.], cksum 0xc90c (correct), s= eq 1, ack 1, win 502, length 0 13:47:54.910990 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x08= 00), length 125: (tos 0x2,ECT(0), ttl 63, id 53209, offset 0, flags [DF], p= roto TCP (6), length 111) 85.204.121.218.48924 > 1.1.1.1.80: Flags [P.], cksum 0xcf48 (correct), = seq 1:72, ack 1, win 502, length 71: HTTP, length: 71 GET / HTTP/1.1 Host: 1.1.1.1 User-Agent: curl/7.66.0 Accept: */* =09 13:47:55.119451 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x08= 00), length 125: (tos 0x0, ttl 63, id 53210, offset 0, flags [DF], proto TC= P (6), length 111) 85.204.121.218.48924 > 1.1.1.1.80: Flags [P.], cksum 0xcf48 (correct), = seq 1:72, ack 1, win 502, length 71: HTTP, length: 71 GET / HTTP/1.1 Host: 1.1.1.1 User-Agent: curl/7.66.0 Accept: */* =09 13:47:55.120638 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0x08= 00), length 60: (tos 0x0, ttl 61, id 41447, offset 0, flags [DF], proto TCP= (6), length 40) 1.1.1.1.80 > 85.204.121.218.48924: Flags [.], cksum 0xca9e (correct), s= eq 1, ack 72, win 29, length 0 13:47:55.130264 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0x08= 00), length 609: (tos 0x2,ECT(0), ttl 61, id 41448, offset 0, flags [DF], p= roto TCP (6), length 595) 1.1.1.1.80 > 85.204.121.218.48924: Flags [P.], cksum 0xde5e (correct), = seq 1:556, ack 72, win 29, length 555: HTTP, length: 555 HTTP/1.1 301 Moved Permanently Date: Tue, 12 Nov 2019 12:47:55 GMT Content-Type: text/html Transfer-Encoding: chunked Connection: keep-alive Location: https://1.1.1.1/ Served-In-Seconds: 0.000 CF-Cache-Status: HIT Age: 2976 Expires: Tue, 12 Nov 2019 16:47:55 GMT Cache-Control: public, max-age=3D14400 Server: cloudflare CF-RAY: 53489e018ad8d885-CPH =09 ba 301 Moved Permanently

301 Moved Permanently

cloudflare-lb
=09 13:47:55.130265 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0x08= 00), length 60: (tos 0x2,ECT(0), ttl 61, id 41449, offset 0, flags [DF], pr= oto TCP (6), length 45) 1.1.1.1.80 > 85.204.121.218.48924: Flags [P.], cksum 0x844c (correct), = seq 556:561, ack 72, win 29, length 5: HTTP 13:47:55.130777 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x08= 00), length 54: (tos 0x0, ttl 63, id 53211, offset 0, flags [DF], proto TCP= (6), length 40) 85.204.121.218.48924 > 1.1.1.1.80: Flags [.], cksum 0xc69b (correct), s= eq 72, ack 556, win 501, length 0 13:47:55.131097 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x08= 00), length 54: (tos 0x0, ttl 63, id 53212, offset 0, flags [DF], proto TCP= (6), length 40) 85.204.121.218.48924 > 1.1.1.1.80: Flags [.], cksum 0xc696 (correct), s= eq 72, ack 561, win 501, length 0 13:47:55.131491 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x08= 00), length 54: (tos 0x0, ttl 63, id 53213, offset 0, flags [DF], proto TCP= (6), length 40) 85.204.121.218.48924 > 1.1.1.1.80: Flags [F.], cksum 0xc695 (correct), = seq 72, ack 561, win 501, length 0 13:47:55.132804 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0x08= 00), length 60: (tos 0x0, ttl 61, id 41450, offset 0, flags [DF], proto TCP= (6), length 40) 1.1.1.1.80 > 85.204.121.218.48924: Flags [F.], cksum 0xc86c (correct), = seq 561, ack 73, win 29, length 0 13:47:55.133281 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0x08= 00), length 54: (tos 0x0, ttl 63, id 53214, offset 0, flags [DF], proto TCP= (6), length 40) 85.204.121.218.48924 > 1.1.1.1.80: Flags [.], cksum 0xc694 (correct), s= eq 73, ack 562, win 501, length 0