[Ecn-sane] Meanwhile, over on NANOG...

[Rodney W. Grimes]

> Toke H?iland-J?rgensen <toke at toke.dk> writes:
> 
> > Luca Muscariello <muscariello at ieee.org> writes:
> >
> >> On Tue, Nov 12, 2019 at 2:02 PM Toke H?iland-J?rgensen <toke at toke.dk> wrote:
> >>
> >>> Mikael Abrahamsson <swmike at swm.pp.se> writes:
> >>>
> >>> > On Tue, 12 Nov 2019, Toke H?iland-J?rgensen wrote:
> >>> >
> >>> >> I'm not on the nanog list, but feel free to cross-post; would be good
> >>> to
> >>> >> actually get to the bottom of this issue! Marek and I already had an
> >>> >> off-list back-and-forth after that original thread, and we couldn't
> >>> find
> >>> >> anything wrong on the Cloudflare side. And the RSTs have a higher TTL
> >>> >> than the actual traffic, indicating an in-path problem...
> >>> >
> >>> > tcptraceroute supports setting/clearing ECN bits (-E), would be very
> >>> > interesting to see difference between those tcptraceroutes?
> >>>
> >>> No difference. But the RST is not being sent as a response to the SYN;
> >>> it is sent in response to the first data packet...
> >>>
> >>> ... and now that I'm re-testing, things were working for a little while,
> >>> but now the bug is back. I got an intermittent successful connection
> >>> with the same TTL that I was previously getting the RST from. And now
> >>> I'm back to getting RSTed.
> >>>
> >>> So I guess there's some kind of multipath issue here; ECMP path,
> >>> multiple routing upstreams, or a broken load balancer? Any other ideas?
> >>>
> >>
> >>
> >> It makes me think of some usage of anycast TCP on the cloudflare side.
> >> What service is this Toke?
> >
> > Yeah, I did also think about anycast when I said "multiple routing
> > upstreams". For testing I've just been doing 'curl 1.1.1.1'. But
> > Cloudflare-hosted sites in general seem to have this problem; for
> > instance, 'curl -4 bufferbloat.net' also fails (but IPv6 is fine).
> 
> Right, so I've played around with tcptraceroute a bit more, and looked
> at some more packet dumps, and I think I'm starting to form a theory:
> 
> I get two different traceroutes; this was from running two traceroutes
> right after one another:
> 
> $ sudo tcptraceroute 1.1.1.1
> Selected device eth0, address 10.42.3.130, port 42177 for outgoing packets
> Tracing the path to 1.1.1.1 on TCP port 80 (http), 30 hops max
>  1  10.42.3.1  0.318 ms  0.325 ms  0.321 ms
>  2  albertslund-edge1-lo.net.gigabit.dk (185.24.171.254)  1.337 ms  5.390 ms  3.194 ms
>  3  customer-185-24-168-46.ip4.gigabit.dk (185.24.168.46)  1.319 ms  1.120 ms  1.256 ms
>  4  te0-1-1-5.rcr21.cph01.atlas.cogentco.com (149.6.137.49)  1.533 ms  1.612 ms  1.392 ms
>  5  be2306.ccr42.ham01.atlas.cogentco.com (130.117.3.237)  6.787 ms  6.822 ms  6.721 ms
>  6  149.6.142.130  7.000 ms  6.939 ms  6.948 ms
>  7  one.one.one.one (1.1.1.1) [open]  6.957 ms  6.967 ms  6.893 ms
>  
> $ sudo tcptraceroute 1.1.1.1
> Selected device eth0, address 10.42.3.130, port 38681 for outgoing packets
> Tracing the path to 1.1.1.1 on TCP port 80 (http), 30 hops max
>  1  10.42.3.1  0.290 ms  0.287 ms  0.292 ms
>  2  albertslund-edge1-lo.net.gigabit.dk (185.24.171.254)  1.857 ms  5.382 ms  18.654 ms
>  3  customer-185-24-168-38.ip4.gigabit.dk (185.24.168.38)  1.249 ms  1.121 ms  1.521 ms
>  4  10ge1-2.core1.cph1.he.net (216.66.83.101)  1.375 ms  2.495 ms  1.440 ms
>  5  dix.as13335.net (192.38.7.70)  2.093 ms  1.895 ms  1.790 ms
>  6  one.one.one.one (1.1.1.1) [open]  1.783 ms  1.861 ms  1.817 ms
> 
> 
> Notice how one is one hop longer than the other.

Worse than just longer, it appears as if the exit hop from gigabit.dk
goes to 2 different providers (hop 4 above).  If these are packets towards
an anycast address that is going to cause exactly what you see.  ECMP
accross multiple AS's towards anycast is.. umm.. very fragile and your
seeing one of the problems with anycast.

It is very unlikely that he.net and cogentco.com end up at the same
1.1.1.1 box.

> So definitely something
> to do with anycast; maybe ECMP over both paths since it's changing
> pretty often?

And the multipath is set to round robin perhaps?

> Now, what I was seeing with the ECN errors was that the SYN-ACK would
> have a different TTL than the first data packet. So what I'm thinking is
> that maybe there's an ECMP hash that hashes on the wrong parts of the
> TCP header, and so considers the SYN packet with the ECN bit set to be
> part of a different flow than the subsequent packets. The result being
> that the flow is split between two anycasted endpoints, causing the RST.
> 
> Does this sound completely out in the weeds?
Nope, your spot on, other than this is a ECMP issue, not an ECN issue.
> Has anyone else run into an
> ECMP device that did something similar?

Yes.  When round robin path selection is in use.

> -Toke

-- 
Rod Grimes                                                 rgrimes at freebsd.org