<div><div><div dir="auto">TCP anycast fails in this case and I would not blame the load balancer for that.</div></div><div dir="auto">Some people will have a different opinion on that.</div><div dir="auto"><br></div><div dir="auto">The current Internet just does not support well these use cases. </div><div dir="auto"><br></div></div><div dir="auto">At the same time this DNS service is supposed to be used in a different way. So we may even blame the user? Toke in this case ?</div><div dir="auto"><br></div><div dir="auto">DNS anycast works as long as it uses UDP.</div><div dir="auto">The IP address returned by the resolver should be unicast and TCP should run over unicast addresses.</div><div dir="auto"><br></div><div dir="auto">Toke, Looks like you are doing an HTTP GET directly toward an anycast address. This is where things are supposed to break and they break.</div><div dir="auto"><br></div><div dir="auto">If you traceroute over unicast addresses you should see the load balancer providing stickiness.</div><div dir="auto"><br></div><div dir="auto"><br></div><div><div dir="auto"><br></div><div>On Wed 13 Nov 2019 at 01:04, Rodney W. Grimes <<a href="mailto:4bone@gndrsh.dnsmgr.net" target="_blank">4bone@gndrsh.dnsmgr.net</a>> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">> Toke H?iland-J?rgensen <<a href="mailto:toke@toke.dk" target="_blank">toke@toke.dk</a>> writes:<br>
> <br>
> > Luca Muscariello <<a href="mailto:muscariello@ieee.org" target="_blank">muscariello@ieee.org</a>> writes:<br>
> ><br>
> >> On Tue, Nov 12, 2019 at 2:02 PM Toke H?iland-J?rgensen <<a href="mailto:toke@toke.dk" target="_blank">toke@toke.dk</a>> wrote:<br>
> >><br>
> >>> Mikael Abrahamsson <<a href="mailto:swmike@swm.pp.se" target="_blank">swmike@swm.pp.se</a>> writes:<br>
> >>><br>
> >>> > On Tue, 12 Nov 2019, Toke H?iland-J?rgensen wrote:<br>
> >>> ><br>
> >>> >> I'm not on the nanog list, but feel free to cross-post; would be good<br>
> >>> to<br>
> >>> >> actually get to the bottom of this issue! Marek and I already had an<br>
> >>> >> off-list back-and-forth after that original thread, and we couldn't<br>
> >>> find<br>
> >>> >> anything wrong on the Cloudflare side. And the RSTs have a higher TTL<br>
> >>> >> than the actual traffic, indicating an in-path problem...<br>
> >>> ><br>
> >>> > tcptraceroute supports setting/clearing ECN bits (-E), would be very<br>
> >>> > interesting to see difference between those tcptraceroutes?<br>
> >>><br>
> >>> No difference. But the RST is not being sent as a response to the SYN;<br>
> >>> it is sent in response to the first data packet...<br>
> >>><br>
> >>> ... and now that I'm re-testing, things were working for a little while,<br>
> >>> but now the bug is back. I got an intermittent successful connection<br>
> >>> with the same TTL that I was previously getting the RST from. And now<br>
> >>> I'm back to getting RSTed.<br>
> >>><br>
> >>> So I guess there's some kind of multipath issue here; ECMP path,<br>
> >>> multiple routing upstreams, or a broken load balancer? Any other ideas?<br>
> >>><br>
> >><br>
> >><br>
> >> It makes me think of some usage of anycast TCP on the cloudflare side.<br>
> >> What service is this Toke?<br>
> ><br>
> > Yeah, I did also think about anycast when I said "multiple routing<br>
> > upstreams". For testing I've just been doing 'curl 1.1.1.1'. But<br>
> > Cloudflare-hosted sites in general seem to have this problem; for<br>
> > instance, 'curl -4 <a href="http://bufferbloat.net" rel="noreferrer" target="_blank">bufferbloat.net</a>' also fails (but IPv6 is fine).<br>
> <br>
> Right, so I've played around with tcptraceroute a bit more, and looked<br>
> at some more packet dumps, and I think I'm starting to form a theory:<br>
> <br>
> I get two different traceroutes; this was from running two traceroutes<br>
> right after one another:<br>
> <br>
> $ sudo tcptraceroute 1.1.1.1<br>
> Selected device eth0, address 10.42.3.130, port 42177 for outgoing packets<br>
> Tracing the path to 1.1.1.1 on TCP port 80 (http), 30 hops max<br>
> 1 10.42.3.1 0.318 ms 0.325 ms 0.321 ms<br>
> 2 <a href="http://albertslund-edge1-lo.net.gigabit.dk" rel="noreferrer" target="_blank">albertslund-edge1-lo.net.gigabit.dk</a> (185.24.171.254) 1.337 ms 5.390 ms 3.194 ms<br>
> 3 <a href="http://customer-185-24-168-46.ip4.gigabit.dk" rel="noreferrer" target="_blank">customer-185-24-168-46.ip4.gigabit.dk</a> (185.24.168.46) 1.319 ms 1.120 ms 1.256 ms<br>
> 4 <a href="http://te0-1-1-5.rcr21.cph01.atlas.cogentco.com" rel="noreferrer" target="_blank">te0-1-1-5.rcr21.cph01.atlas.cogentco.com</a> (149.6.137.49) 1.533 ms 1.612 ms 1.392 ms<br>
> 5 <a href="http://be2306.ccr42.ham01.atlas.cogentco.com" rel="noreferrer" target="_blank">be2306.ccr42.ham01.atlas.cogentco.com</a> (130.117.3.237) 6.787 ms 6.822 ms 6.721 ms<br>
> 6 149.6.142.130 7.000 ms 6.939 ms 6.948 ms<br>
> 7 one.one.one.one (1.1.1.1) [open] 6.957 ms 6.967 ms 6.893 ms<br>
> <br>
> $ sudo tcptraceroute 1.1.1.1<br>
> Selected device eth0, address 10.42.3.130, port 38681 for outgoing packets<br>
> Tracing the path to 1.1.1.1 on TCP port 80 (http), 30 hops max<br>
> 1 10.42.3.1 0.290 ms 0.287 ms 0.292 ms<br>
> 2 <a href="http://albertslund-edge1-lo.net.gigabit.dk" rel="noreferrer" target="_blank">albertslund-edge1-lo.net.gigabit.dk</a> (185.24.171.254) 1.857 ms 5.382 ms 18.654 ms<br>
> 3 <a href="http://customer-185-24-168-38.ip4.gigabit.dk" rel="noreferrer" target="_blank">customer-185-24-168-38.ip4.gigabit.dk</a> (185.24.168.38) 1.249 ms 1.121 ms 1.521 ms<br>
> 4 <a href="http://10ge1-2.core1.cph1.he.net" rel="noreferrer" target="_blank">10ge1-2.core1.cph1.he.net</a> (216.66.83.101) 1.375 ms 2.495 ms 1.440 ms<br>
> 5 <a href="http://dix.as13335.net" rel="noreferrer" target="_blank">dix.as13335.net</a> (192.38.7.70) 2.093 ms 1.895 ms 1.790 ms<br>
> 6 one.one.one.one (1.1.1.1) [open] 1.783 ms 1.861 ms 1.817 ms<br>
> <br>
> <br>
> Notice how one is one hop longer than the other.<br>
<br>
Worse than just longer, it appears as if the exit hop from <a href="http://gigabit.dk" rel="noreferrer" target="_blank">gigabit.dk</a><br>
goes to 2 different providers (hop 4 above). If these are packets towards<br>
an anycast address that is going to cause exactly what you see. ECMP<br>
accross multiple AS's towards anycast is.. umm.. very fragile and your<br>
seeing one of the problems with anycast.<br>
<br>
It is very unlikely that <a href="http://he.net" rel="noreferrer" target="_blank">he.net</a> and <a href="http://cogentco.com" rel="noreferrer" target="_blank">cogentco.com</a> end up at the same<br>
1.1.1.1 box.<br>
<br>
> So definitely something<br>
> to do with anycast; maybe ECMP over both paths since it's changing<br>
> pretty often?<br>
<br>
And the multipath is set to round robin perhaps?<br>
<br>
> Now, what I was seeing with the ECN errors was that the SYN-ACK would<br>
> have a different TTL than the first data packet. So what I'm thinking is<br>
> that maybe there's an ECMP hash that hashes on the wrong parts of the<br>
> TCP header, and so considers the SYN packet with the ECN bit set to be<br>
> part of a different flow than the subsequent packets. The result being<br>
> that the flow is split between two anycasted endpoints, causing the RST.<br>
> <br>
> Does this sound completely out in the weeds?<br>
Nope, your spot on, other than this is a ECMP issue, not an ECN issue.<br>
> Has anyone else run into an<br>
> ECMP device that did something similar?<br>
<br>
Yes. When round robin path selection is in use.<br>
<br>
> -Toke<br>
<br>
-- <br>
Rod Grimes <a href="mailto:rgrimes@freebsd.org" target="_blank">rgrimes@freebsd.org</a><br>
</blockquote></div></div>
</div>