From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <4bone@gndrsh.dnsmgr.net> Received: from gndrsh.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 8B1983B29E for ; Wed, 13 Nov 2019 10:25:34 -0500 (EST) Received: from gndrsh.dnsmgr.net (localhost [127.0.0.1]) by gndrsh.dnsmgr.net (8.13.3/8.13.3) with ESMTP id xADFPRLA044390; Wed, 13 Nov 2019 07:25:27 -0800 (PST) (envelope-from 4bone@gndrsh.dnsmgr.net) Received: (from 4bone@localhost) by gndrsh.dnsmgr.net (8.13.3/8.13.3/Submit) id xADFPQ61044389; Wed, 13 Nov 2019 07:25:26 -0800 (PST) (envelope-from 4bone) From: "Rodney W. Grimes" <4bone@gndrsh.dnsmgr.net> Message-Id: <201911131525.xADFPQ61044389@gndrsh.dnsmgr.net> In-Reply-To: <875zjooy46.fsf@toke.dk> To: "Toke H?iland-J?rgensen" Date: Wed, 13 Nov 2019 07:25:26 -0800 (PST) CC: "Rodney W. Grimes" <4bone@gndrsh.dnsmgr.net>, Luca Muscariello , Rich Brown , ECN-Sane X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII Subject: Re: [Ecn-sane] Meanwhile, over on NANOG... X-BeenThere: ecn-sane@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of explicit congestion notification's impact on the Internet List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Nov 2019 15:25:34 -0000 > "Rodney W. Grimes" <4bone@gndrsh.dnsmgr.net> writes: > > >> Toke H?iland-J?rgensen writes: > >> > >> > Luca Muscariello writes: > >> > > >> >> On Tue, Nov 12, 2019 at 2:02 PM Toke H?iland-J?rgensen wrote: > >> >> > >> >>> Mikael Abrahamsson writes: > >> >>> > >> >>> > On Tue, 12 Nov 2019, Toke H?iland-J?rgensen wrote: > >> >>> > > >> >>> >> I'm not on the nanog list, but feel free to cross-post; would be good > >> >>> to > >> >>> >> actually get to the bottom of this issue! Marek and I already had an > >> >>> >> off-list back-and-forth after that original thread, and we couldn't > >> >>> find > >> >>> >> anything wrong on the Cloudflare side. And the RSTs have a higher TTL > >> >>> >> than the actual traffic, indicating an in-path problem... > >> >>> > > >> >>> > tcptraceroute supports setting/clearing ECN bits (-E), would be very > >> >>> > interesting to see difference between those tcptraceroutes? > >> >>> > >> >>> No difference. But the RST is not being sent as a response to the SYN; > >> >>> it is sent in response to the first data packet... > >> >>> > >> >>> ... and now that I'm re-testing, things were working for a little while, > >> >>> but now the bug is back. I got an intermittent successful connection > >> >>> with the same TTL that I was previously getting the RST from. And now > >> >>> I'm back to getting RSTed. > >> >>> > >> >>> So I guess there's some kind of multipath issue here; ECMP path, > >> >>> multiple routing upstreams, or a broken load balancer? Any other ideas? > >> >>> > >> >> > >> >> > >> >> It makes me think of some usage of anycast TCP on the cloudflare side. > >> >> What service is this Toke? > >> > > >> > Yeah, I did also think about anycast when I said "multiple routing > >> > upstreams". For testing I've just been doing 'curl 1.1.1.1'. But > >> > Cloudflare-hosted sites in general seem to have this problem; for > >> > instance, 'curl -4 bufferbloat.net' also fails (but IPv6 is fine). > >> > >> Right, so I've played around with tcptraceroute a bit more, and looked > >> at some more packet dumps, and I think I'm starting to form a theory: > >> > >> I get two different traceroutes; this was from running two traceroutes > >> right after one another: > >> > >> $ sudo tcptraceroute 1.1.1.1 > >> Selected device eth0, address 10.42.3.130, port 42177 for outgoing packets > >> Tracing the path to 1.1.1.1 on TCP port 80 (http), 30 hops max > >> 1 10.42.3.1 0.318 ms 0.325 ms 0.321 ms > >> 2 albertslund-edge1-lo.net.gigabit.dk (185.24.171.254) 1.337 ms 5.390 ms 3.194 ms > >> 3 customer-185-24-168-46.ip4.gigabit.dk (185.24.168.46) 1.319 ms 1.120 ms 1.256 ms > >> 4 te0-1-1-5.rcr21.cph01.atlas.cogentco.com (149.6.137.49) 1.533 ms 1.612 ms 1.392 ms > >> 5 be2306.ccr42.ham01.atlas.cogentco.com (130.117.3.237) 6.787 ms 6.822 ms 6.721 ms > >> 6 149.6.142.130 7.000 ms 6.939 ms 6.948 ms > >> 7 one.one.one.one (1.1.1.1) [open] 6.957 ms 6.967 ms 6.893 ms > >> > >> $ sudo tcptraceroute 1.1.1.1 > >> Selected device eth0, address 10.42.3.130, port 38681 for outgoing packets > >> Tracing the path to 1.1.1.1 on TCP port 80 (http), 30 hops max > >> 1 10.42.3.1 0.290 ms 0.287 ms 0.292 ms > >> 2 albertslund-edge1-lo.net.gigabit.dk (185.24.171.254) 1.857 ms 5.382 ms 18.654 ms > >> 3 customer-185-24-168-38.ip4.gigabit.dk (185.24.168.38) 1.249 ms 1.121 ms 1.521 ms > >> 4 10ge1-2.core1.cph1.he.net (216.66.83.101) 1.375 ms 2.495 ms 1.440 ms > >> 5 dix.as13335.net (192.38.7.70) 2.093 ms 1.895 ms 1.790 ms > >> 6 one.one.one.one (1.1.1.1) [open] 1.783 ms 1.861 ms 1.817 ms > >> > >> > >> Notice how one is one hop longer than the other. > > > > Worse than just longer, it appears as if the exit hop from gigabit.dk > > goes to 2 different providers (hop 4 above). If these are packets towards > > an anycast address that is going to cause exactly what you see. ECMP > > accross multiple AS's towards anycast is.. umm.. very fragile and your > > seeing one of the problems with anycast. > > > > It is very unlikely that he.net and cogentco.com end up at the same > > 1.1.1.1 box. > > Yeah, did notice it was two different upstreams :) > > >> So definitely something > >> to do with anycast; maybe ECMP over both paths since it's changing > >> pretty often? > > > > And the multipath is set to round robin perhaps? > > Not round-robin. That it was changing simply at random turns out to be > my mistake; by default tcptraceroute will pick a new source port each > time. If I fix the source port I get the same path each time, so it > looks like it's hashing on headers. > > Going back to regular UDP-based trace route I finally found what looks > to be the smoking gun: > > $ traceroute 1.1.1.1 -q 1 --sport=10000 -t 1 > traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets > 1 _gateway (10.42.3.1) 0.304 ms > 2 albertslund-edge1-lo.net.gigabit.dk (185.24.171.254) 3.935 ms > 3 customer-185-24-168-46.ip4.gigabit.dk (185.24.168.46) 1.005 ms > 4 te0-1-1-5.rcr21.cph01.atlas.cogentco.com (149.6.137.49) 1.361 ms > 5 netnod-ix-cph-blue-9000.cloudflare.com (212.237.192.246) 1.250 ms > 6 one.one.one.one (1.1.1.1) 1.380 ms > > $ traceroute 1.1.1.1 -q 1 --sport=10000 -t 2 > traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets > 1 _gateway (10.42.3.1) 0.236 ms > 2 albertslund-edge1-lo.net.gigabit.dk (185.24.171.254) 53.833 ms > 3 customer-185-24-168-38.ip4.gigabit.dk (185.24.168.38) 1.195 ms > 4 10ge1-2.core1.cph1.he.net (216.66.83.101) 1.979 ms > 5 be2306.ccr42.ham01.atlas.cogentco.com (130.117.3.237) 6.851 ms > 6 149.6.142.130 (149.6.142.130) 13.081 ms > 7 one.one.one.one (1.1.1.1) 1.842 ms > > -t is the TOS value; so those two happen to correspond to ECT(1) and > ECT(0); and as you can see they go two different paths. Which would be > consistent with the SYN going one way and the data packets going > another. Perhaps Old enough that maybe they are treating that as TOS byte? Looks like you have nailed it though, someone has a broken hash. > -Toke -- Rod Grimes rgrimes@freebsd.org