[Ecn-sane] Meanwhile, over on NANOG...

[Luca Muscariello]

TCP anycast fails in this case and I would not blame the load balancer for
that.
Some people will have a different opinion on that.

The current Internet just does not support well these use cases.

At the same time this DNS service is supposed to be used in a different
way. So we may even blame the user? Toke in this case ?

DNS anycast works as long as it uses UDP.
The IP address returned by the resolver should be unicast and TCP should
run over unicast addresses.

Toke,  Looks like you are doing an HTTP GET directly toward an anycast
address. This is where things are supposed to break and they break.

If you traceroute over unicast addresses you should see the load balancer
providing stickiness.



On Wed 13 Nov 2019 at 01:04, Rodney W. Grimes <4bone at gndrsh.dnsmgr.net>
wrote:

> > Toke H?iland-J?rgensen <toke at toke.dk> writes:
> >
> > > Luca Muscariello <muscariello at ieee.org> writes:
> > >
> > >> On Tue, Nov 12, 2019 at 2:02 PM Toke H?iland-J?rgensen <toke at toke.dk>
> wrote:
> > >>
> > >>> Mikael Abrahamsson <swmike at swm.pp.se> writes:
> > >>>
> > >>> > On Tue, 12 Nov 2019, Toke H?iland-J?rgensen wrote:
> > >>> >
> > >>> >> I'm not on the nanog list, but feel free to cross-post; would be
> good
> > >>> to
> > >>> >> actually get to the bottom of this issue! Marek and I already had
> an
> > >>> >> off-list back-and-forth after that original thread, and we
> couldn't
> > >>> find
> > >>> >> anything wrong on the Cloudflare side. And the RSTs have a higher
> TTL
> > >>> >> than the actual traffic, indicating an in-path problem...
> > >>> >
> > >>> > tcptraceroute supports setting/clearing ECN bits (-E), would be
> very
> > >>> > interesting to see difference between those tcptraceroutes?
> > >>>
> > >>> No difference. But the RST is not being sent as a response to the
> SYN;
> > >>> it is sent in response to the first data packet...
> > >>>
> > >>> ... and now that I'm re-testing, things were working for a little
> while,
> > >>> but now the bug is back. I got an intermittent successful connection
> > >>> with the same TTL that I was previously getting the RST from. And now
> > >>> I'm back to getting RSTed.
> > >>>
> > >>> So I guess there's some kind of multipath issue here; ECMP path,
> > >>> multiple routing upstreams, or a broken load balancer? Any other
> ideas?
> > >>>
> > >>
> > >>
> > >> It makes me think of some usage of anycast TCP on the cloudflare side.
> > >> What service is this Toke?
> > >
> > > Yeah, I did also think about anycast when I said "multiple routing
> > > upstreams". For testing I've just been doing 'curl 1.1.1.1'. But
> > > Cloudflare-hosted sites in general seem to have this problem; for
> > > instance, 'curl -4 bufferbloat.net' also fails (but IPv6 is fine).
> >
> > Right, so I've played around with tcptraceroute a bit more, and looked
> > at some more packet dumps, and I think I'm starting to form a theory:
> >
> > I get two different traceroutes; this was from running two traceroutes
> > right after one another:
> >
> > $ sudo tcptraceroute 1.1.1.1
> > Selected device eth0, address 10.42.3.130, port 42177 for outgoing
> packets
> > Tracing the path to 1.1.1.1 on TCP port 80 (http), 30 hops max
> >  1  10.42.3.1  0.318 ms  0.325 ms  0.321 ms
> >  2  albertslund-edge1-lo.net.gigabit.dk (185.24.171.254)  1.337 ms
> 5.390 ms  3.194 ms
> >  3  customer-185-24-168-46.ip4.gigabit.dk (185.24.168.46)  1.319 ms
> 1.120 ms  1.256 ms
> >  4  te0-1-1-5.rcr21.cph01.atlas.cogentco.com (149.6.137.49)  1.533 ms
> 1.612 ms  1.392 ms
> >  5  be2306.ccr42.ham01.atlas.cogentco.com (130.117.3.237)  6.787 ms
> 6.822 ms  6.721 ms
> >  6  149.6.142.130  7.000 ms  6.939 ms  6.948 ms
> >  7  one.one.one.one (1.1.1.1) [open]  6.957 ms  6.967 ms  6.893 ms
> >
> > $ sudo tcptraceroute 1.1.1.1
> > Selected device eth0, address 10.42.3.130, port 38681 for outgoing
> packets
> > Tracing the path to 1.1.1.1 on TCP port 80 (http), 30 hops max
> >  1  10.42.3.1  0.290 ms  0.287 ms  0.292 ms
> >  2  albertslund-edge1-lo.net.gigabit.dk (185.24.171.254)  1.857 ms
> 5.382 ms  18.654 ms
> >  3  customer-185-24-168-38.ip4.gigabit.dk (185.24.168.38)  1.249 ms
> 1.121 ms  1.521 ms
> >  4  10ge1-2.core1.cph1.he.net (216.66.83.101)  1.375 ms  2.495 ms
> 1.440 ms
> >  5  dix.as13335.net (192.38.7.70)  2.093 ms  1.895 ms  1.790 ms
> >  6  one.one.one.one (1.1.1.1) [open]  1.783 ms  1.861 ms  1.817 ms
> >
> >
> > Notice how one is one hop longer than the other.
>
> Worse than just longer, it appears as if the exit hop from gigabit.dk
> goes to 2 different providers (hop 4 above).  If these are packets towards
> an anycast address that is going to cause exactly what you see.  ECMP
> accross multiple AS's towards anycast is.. umm.. very fragile and your
> seeing one of the problems with anycast.
>
> It is very unlikely that he.net and cogentco.com end up at the same
> 1.1.1.1 box.
>
> > So definitely something
> > to do with anycast; maybe ECMP over both paths since it's changing
> > pretty often?
>
> And the multipath is set to round robin perhaps?
>
> > Now, what I was seeing with the ECN errors was that the SYN-ACK would
> > have a different TTL than the first data packet. So what I'm thinking is
> > that maybe there's an ECMP hash that hashes on the wrong parts of the
> > TCP header, and so considers the SYN packet with the ECN bit set to be
> > part of a different flow than the subsequent packets. The result being
> > that the flow is split between two anycasted endpoints, causing the RST.
> >
> > Does this sound completely out in the weeds?
> Nope, your spot on, other than this is a ECMP issue, not an ECN issue.
> > Has anyone else run into an
> > ECMP device that did something similar?
>
> Yes.  When round robin path selection is in use.
>
> > -Toke
>
> --
> Rod Grimes
> rgrimes at freebsd.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/ecn-sane/attachments/20191113/0161059b/attachment-0001.html>