From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.toke.dk (mail.toke.dk [52.28.52.200]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id AAF183B29D for ; Tue, 12 Nov 2019 17:01:45 -0500 (EST) From: Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=toke.dk; s=20161023; t=1573596103; bh=ugng/Egpl2M+jafiX+6/clPvcRowineIsfoaNah0830=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=mVZtMCk2jPyra9viqt5asIJBWJkzBHcGHZf+DXN1kcOPE6IWYfEbXam0LQgb70vdF UMeN5BHPDDIaSrhCCXbcEPFo7cq5cyTS/Z4wq+yD8Xur6lcm4TUG+f7XtK3vrfQp5E n6vaoN2fuScSPPPvTAYk3X2Giz2PMfk322cERq/mRiuovpfPvAndN3TECOAZg8q88a nDnXZxM4eqlWxRLg8FQ5bVHMPyP7byjcRXoaw0Vz9U8On2CEBOkXJTVGgB+FvdT8yI qdBEYivDH5s3Je3cc3rc9xPM3EeqTYEcG2ODr+qmv/sXlDqXAYRBh199jXK1q9hddE jacThWKc1mKfw== To: Luca Muscariello Cc: Mikael Abrahamsson , Rich Brown , ECN-Sane In-Reply-To: <87pnhxp3gy.fsf@toke.dk> References: <1CF67BBC-B528-4667-97AE-760DCE027466@gmail.com> <87y2wlp9p4.fsf@toke.dk> <87v9rpp7s0.fsf@toke.dk> <87pnhxp3gy.fsf@toke.dk> Date: Tue, 12 Nov 2019 23:01:42 +0100 X-Clacks-Overhead: GNU Terry Pratchett Message-ID: <878sokpxdl.fsf@toke.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [Ecn-sane] Meanwhile, over on NANOG... X-BeenThere: ecn-sane@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of explicit congestion notification's impact on the Internet List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2019 22:01:46 -0000 Toke H=C3=B8iland-J=C3=B8rgensen writes: > Luca Muscariello writes: > >> On Tue, Nov 12, 2019 at 2:02 PM Toke H=C3=B8iland-J=C3=B8rgensen wrote: >> >>> Mikael Abrahamsson writes: >>> >>> > On Tue, 12 Nov 2019, Toke H=C3=B8iland-J=C3=B8rgensen wrote: >>> > >>> >> I'm not on the nanog list, but feel free to cross-post; would be good >>> to >>> >> actually get to the bottom of this issue! Marek and I already had an >>> >> off-list back-and-forth after that original thread, and we couldn't >>> find >>> >> anything wrong on the Cloudflare side. And the RSTs have a higher TTL >>> >> than the actual traffic, indicating an in-path problem... >>> > >>> > tcptraceroute supports setting/clearing ECN bits (-E), would be very >>> > interesting to see difference between those tcptraceroutes? >>> >>> No difference. But the RST is not being sent as a response to the SYN; >>> it is sent in response to the first data packet... >>> >>> ... and now that I'm re-testing, things were working for a little while, >>> but now the bug is back. I got an intermittent successful connection >>> with the same TTL that I was previously getting the RST from. And now >>> I'm back to getting RSTed. >>> >>> So I guess there's some kind of multipath issue here; ECMP path, >>> multiple routing upstreams, or a broken load balancer? Any other ideas? >>> >> >> >> It makes me think of some usage of anycast TCP on the cloudflare side. >> What service is this Toke? > > Yeah, I did also think about anycast when I said "multiple routing > upstreams". For testing I've just been doing 'curl 1.1.1.1'. But > Cloudflare-hosted sites in general seem to have this problem; for > instance, 'curl -4 bufferbloat.net' also fails (but IPv6 is fine). Right, so I've played around with tcptraceroute a bit more, and looked at some more packet dumps, and I think I'm starting to form a theory: I get two different traceroutes; this was from running two traceroutes right after one another: $ sudo tcptraceroute 1.1.1.1 Selected device eth0, address 10.42.3.130, port 42177 for outgoing packets Tracing the path to 1.1.1.1 on TCP port 80 (http), 30 hops max 1 10.42.3.1 0.318 ms 0.325 ms 0.321 ms 2 albertslund-edge1-lo.net.gigabit.dk (185.24.171.254) 1.337 ms 5.390 m= s 3.194 ms 3 customer-185-24-168-46.ip4.gigabit.dk (185.24.168.46) 1.319 ms 1.120 = ms 1.256 ms 4 te0-1-1-5.rcr21.cph01.atlas.cogentco.com (149.6.137.49) 1.533 ms 1.61= 2 ms 1.392 ms 5 be2306.ccr42.ham01.atlas.cogentco.com (130.117.3.237) 6.787 ms 6.822 = ms 6.721 ms 6 149.6.142.130 7.000 ms 6.939 ms 6.948 ms 7 one.one.one.one (1.1.1.1) [open] 6.957 ms 6.967 ms 6.893 ms =20 $ sudo tcptraceroute 1.1.1.1 Selected device eth0, address 10.42.3.130, port 38681 for outgoing packets Tracing the path to 1.1.1.1 on TCP port 80 (http), 30 hops max 1 10.42.3.1 0.290 ms 0.287 ms 0.292 ms 2 albertslund-edge1-lo.net.gigabit.dk (185.24.171.254) 1.857 ms 5.382 m= s 18.654 ms 3 customer-185-24-168-38.ip4.gigabit.dk (185.24.168.38) 1.249 ms 1.121 = ms 1.521 ms 4 10ge1-2.core1.cph1.he.net (216.66.83.101) 1.375 ms 2.495 ms 1.440 ms 5 dix.as13335.net (192.38.7.70) 2.093 ms 1.895 ms 1.790 ms 6 one.one.one.one (1.1.1.1) [open] 1.783 ms 1.861 ms 1.817 ms Notice how one is one hop longer than the other. So definitely something to do with anycast; maybe ECMP over both paths since it's changing pretty often? Now, what I was seeing with the ECN errors was that the SYN-ACK would have a different TTL than the first data packet. So what I'm thinking is that maybe there's an ECMP hash that hashes on the wrong parts of the TCP header, and so considers the SYN packet with the ECN bit set to be part of a different flow than the subsequent packets. The result being that the flow is split between two anycasted endpoints, causing the RST. Does this sound completely out in the weeds? Has anyone else run into an ECMP device that did something similar? -Toke