From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 00DCA3B29D for ; Wed, 13 Nov 2019 03:05:50 -0500 (EST) Received: by mail-wm1-x32c.google.com with SMTP id t26so897139wmi.4 for ; Wed, 13 Nov 2019 00:05:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ieee.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Ul5w3kOgEg9N0peHTdskT+Zrdtlx4vqTr1uyRwZuRcg=; b=Ku2F5YYclWsmYQ5GPlw9IGyd617mOSqtDRC76T402/KXx6ay+n9iBea23Wgh2RwBNO pQhasN2iaRf6gRnbtEQfm/jxAkyqsZPAIb9uxLb2Ioi9sZ7Qk1cB5Mfjd6CqjslOfHol c8lIai1ZxitRV4mW/w7MdbIancYC/VrbcGBpU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Ul5w3kOgEg9N0peHTdskT+Zrdtlx4vqTr1uyRwZuRcg=; b=PpHaILAjX1Dm0ozMSOFl4u6/GEAszeYko6ZH9HItrLsFR9ZOj//HNQAnPPjDPsRiQB y4DeOl3ymhtEKIFxhvz4mD1m+wE+7UmXq0oeYcgQWIENkoGpWHUyZrlLeQTlqcd/uMOk sHSriYDtGf/cD7D2hkExyZjSC8XUNDgedhXX2ZwS9mfCN/3LpVFzWwy0yFPV70bB0gwl zIuiPgmF1oVnRl2lHmjj02ofAxA6m1DzdlQkl203060qO27spCsJJjFNBFuiIHZ4O4+Z EEzHR/wO5XTrJmsckpFFfvzwfXS1vCojB5urGrr3+5ANviY/H4y06KBMx0B/FpkzG/mr Dr4g== X-Gm-Message-State: APjAAAVoIe8fCM8cqxQDd4mP9lubWaDZjskbH2gFxqZkejLLF4kmDz8a B4Sd4M7lgp3jJGVuNUWc6pDeyI7rKn/LKcujms+UNg== X-Google-Smtp-Source: APXvYqyDUbV+sW2LPooasY4bQ+raHrvvH8TYxv3WrU4288pBLaZq8tUcGmgKFJe20fAe76wHJOt/F+udvbNGXkqwWFc= X-Received: by 2002:a1c:1f14:: with SMTP id f20mr1331346wmf.147.1573632349998; Wed, 13 Nov 2019 00:05:49 -0800 (PST) MIME-Version: 1.0 References: <878sokpxdl.fsf@toke.dk> <201911130004.xAD04Vx6041534@gndrsh.dnsmgr.net> In-Reply-To: <201911130004.xAD04Vx6041534@gndrsh.dnsmgr.net> From: Luca Muscariello Date: Wed, 13 Nov 2019 09:05:38 +0100 Message-ID: To: "Rodney W. Grimes" <4bone@gndrsh.dnsmgr.net> Cc: ECN-Sane , Rich Brown , "Toke H?iland-J?rgensen" Content-Type: multipart/alternative; boundary="0000000000003abea9059735d505" Subject: Re: [Ecn-sane] Meanwhile, over on NANOG... X-BeenThere: ecn-sane@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of explicit congestion notification's impact on the Internet List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Nov 2019 08:05:51 -0000 --0000000000003abea9059735d505 Content-Type: text/plain; charset="UTF-8" TCP anycast fails in this case and I would not blame the load balancer for that. Some people will have a different opinion on that. The current Internet just does not support well these use cases. At the same time this DNS service is supposed to be used in a different way. So we may even blame the user? Toke in this case ? DNS anycast works as long as it uses UDP. The IP address returned by the resolver should be unicast and TCP should run over unicast addresses. Toke, Looks like you are doing an HTTP GET directly toward an anycast address. This is where things are supposed to break and they break. If you traceroute over unicast addresses you should see the load balancer providing stickiness. On Wed 13 Nov 2019 at 01:04, Rodney W. Grimes <4bone@gndrsh.dnsmgr.net> wrote: > > Toke H?iland-J?rgensen writes: > > > > > Luca Muscariello writes: > > > > > >> On Tue, Nov 12, 2019 at 2:02 PM Toke H?iland-J?rgensen > wrote: > > >> > > >>> Mikael Abrahamsson writes: > > >>> > > >>> > On Tue, 12 Nov 2019, Toke H?iland-J?rgensen wrote: > > >>> > > > >>> >> I'm not on the nanog list, but feel free to cross-post; would be > good > > >>> to > > >>> >> actually get to the bottom of this issue! Marek and I already had > an > > >>> >> off-list back-and-forth after that original thread, and we > couldn't > > >>> find > > >>> >> anything wrong on the Cloudflare side. And the RSTs have a higher > TTL > > >>> >> than the actual traffic, indicating an in-path problem... > > >>> > > > >>> > tcptraceroute supports setting/clearing ECN bits (-E), would be > very > > >>> > interesting to see difference between those tcptraceroutes? > > >>> > > >>> No difference. But the RST is not being sent as a response to the > SYN; > > >>> it is sent in response to the first data packet... > > >>> > > >>> ... and now that I'm re-testing, things were working for a little > while, > > >>> but now the bug is back. I got an intermittent successful connection > > >>> with the same TTL that I was previously getting the RST from. And now > > >>> I'm back to getting RSTed. > > >>> > > >>> So I guess there's some kind of multipath issue here; ECMP path, > > >>> multiple routing upstreams, or a broken load balancer? Any other > ideas? > > >>> > > >> > > >> > > >> It makes me think of some usage of anycast TCP on the cloudflare side. > > >> What service is this Toke? > > > > > > Yeah, I did also think about anycast when I said "multiple routing > > > upstreams". For testing I've just been doing 'curl 1.1.1.1'. But > > > Cloudflare-hosted sites in general seem to have this problem; for > > > instance, 'curl -4 bufferbloat.net' also fails (but IPv6 is fine). > > > > Right, so I've played around with tcptraceroute a bit more, and looked > > at some more packet dumps, and I think I'm starting to form a theory: > > > > I get two different traceroutes; this was from running two traceroutes > > right after one another: > > > > $ sudo tcptraceroute 1.1.1.1 > > Selected device eth0, address 10.42.3.130, port 42177 for outgoing > packets > > Tracing the path to 1.1.1.1 on TCP port 80 (http), 30 hops max > > 1 10.42.3.1 0.318 ms 0.325 ms 0.321 ms > > 2 albertslund-edge1-lo.net.gigabit.dk (185.24.171.254) 1.337 ms > 5.390 ms 3.194 ms > > 3 customer-185-24-168-46.ip4.gigabit.dk (185.24.168.46) 1.319 ms > 1.120 ms 1.256 ms > > 4 te0-1-1-5.rcr21.cph01.atlas.cogentco.com (149.6.137.49) 1.533 ms > 1.612 ms 1.392 ms > > 5 be2306.ccr42.ham01.atlas.cogentco.com (130.117.3.237) 6.787 ms > 6.822 ms 6.721 ms > > 6 149.6.142.130 7.000 ms 6.939 ms 6.948 ms > > 7 one.one.one.one (1.1.1.1) [open] 6.957 ms 6.967 ms 6.893 ms > > > > $ sudo tcptraceroute 1.1.1.1 > > Selected device eth0, address 10.42.3.130, port 38681 for outgoing > packets > > Tracing the path to 1.1.1.1 on TCP port 80 (http), 30 hops max > > 1 10.42.3.1 0.290 ms 0.287 ms 0.292 ms > > 2 albertslund-edge1-lo.net.gigabit.dk (185.24.171.254) 1.857 ms > 5.382 ms 18.654 ms > > 3 customer-185-24-168-38.ip4.gigabit.dk (185.24.168.38) 1.249 ms > 1.121 ms 1.521 ms > > 4 10ge1-2.core1.cph1.he.net (216.66.83.101) 1.375 ms 2.495 ms > 1.440 ms > > 5 dix.as13335.net (192.38.7.70) 2.093 ms 1.895 ms 1.790 ms > > 6 one.one.one.one (1.1.1.1) [open] 1.783 ms 1.861 ms 1.817 ms > > > > > > Notice how one is one hop longer than the other. > > Worse than just longer, it appears as if the exit hop from gigabit.dk > goes to 2 different providers (hop 4 above). If these are packets towards > an anycast address that is going to cause exactly what you see. ECMP > accross multiple AS's towards anycast is.. umm.. very fragile and your > seeing one of the problems with anycast. > > It is very unlikely that he.net and cogentco.com end up at the same > 1.1.1.1 box. > > > So definitely something > > to do with anycast; maybe ECMP over both paths since it's changing > > pretty often? > > And the multipath is set to round robin perhaps? > > > Now, what I was seeing with the ECN errors was that the SYN-ACK would > > have a different TTL than the first data packet. So what I'm thinking is > > that maybe there's an ECMP hash that hashes on the wrong parts of the > > TCP header, and so considers the SYN packet with the ECN bit set to be > > part of a different flow than the subsequent packets. The result being > > that the flow is split between two anycasted endpoints, causing the RST. > > > > Does this sound completely out in the weeds? > Nope, your spot on, other than this is a ECMP issue, not an ECN issue. > > Has anyone else run into an > > ECMP device that did something similar? > > Yes. When round robin path selection is in use. > > > -Toke > > -- > Rod Grimes > rgrimes@freebsd.org > --0000000000003abea9059735d505 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
TCP anycast fails in this case and I would not = blame the load balancer for that.
Some people = will have a different opinion on that.

The current Internet just does not support well these use ca= ses.=C2=A0

At the = same time this DNS service is supposed to be used in a different way. So we= may even blame the user? Toke in this case ?

DNS anycast works as long as it uses UDP.
The IP address returned by the resolver should be unicast and T= CP should run over unicast addresses.

Toke, =C2=A0Looks like you are doing an HTTP GET directly tow= ard an anycast address. This is where things are supposed to break and they= break.

If you tracerout= e over unicast addresses you should see the load balancer providing stickin= ess.



On Wed 13 Nov 2019 at 01:04, Rodney W. Grimes = <4bone@gndr= sh.dnsmgr.net> wrote:
> Toke H?iland-J?rgensen <toke@toke.dk> writes:
>
> > Luca Muscariello <muscariello@ieee.org> writes:
> >
> >> On Tue, Nov 12, 2019 at 2:02 PM Toke H?iland-J?rgensen <toke@toke.dk> wrote:=
> >>
> >>> Mikael Abrahamsson <swmike@swm.pp.se> writes:
> >>>
> >>> > On Tue, 12 Nov 2019, Toke H?iland-J?rgensen wrote: > >>> >
> >>> >> I'm not on the nanog list, but feel free to = cross-post; would be good
> >>> to
> >>> >> actually get to the bottom of this issue! Marek = and I already had an
> >>> >> off-list back-and-forth after that original thre= ad, and we couldn't
> >>> find
> >>> >> anything wrong on the Cloudflare side. And the R= STs have a higher TTL
> >>> >> than the actual traffic, indicating an in-path p= roblem...
> >>> >
> >>> > tcptraceroute supports setting/clearing ECN bits (-E= ), would be very
> >>> > interesting to see difference between those tcptrace= routes?
> >>>
> >>> No difference. But the RST is not being sent as a respons= e to the SYN;
> >>> it is sent in response to the first data packet...
> >>>
> >>> ... and now that I'm re-testing, things were working = for a little while,
> >>> but now the bug is back. I got an intermittent successful= connection
> >>> with the same TTL that I was previously getting the RST f= rom. And now
> >>> I'm back to getting RSTed.
> >>>
> >>> So I guess there's some kind of multipath issue here;= ECMP path,
> >>> multiple routing upstreams, or a broken load balancer? An= y other ideas?
> >>>
> >>
> >>
> >> It makes me think of some usage of anycast TCP on the cloudfl= are side.
> >> What service is this Toke?
> >
> > Yeah, I did also think about anycast when I said "multiple r= outing
> > upstreams". For testing I've just been doing 'curl 1= .1.1.1'. But
> > Cloudflare-hosted sites in general seem to have this problem; for=
> > instance, 'curl -4 bufferbloat.net' also fails (but IPv6= is fine).
>
> Right, so I've played around with tcptraceroute a bit more, and lo= oked
> at some more packet dumps, and I think I'm starting to form a theo= ry:
>
> I get two different traceroutes; this was from running two traceroutes=
> right after one another:
>
> $ sudo tcptraceroute 1.1.1.1
> Selected device eth0, address 10.42.3.130, port 42177 for outgoing pac= kets
> Tracing the path to 1.1.1.1 on TCP port 80 (http), 30 hops max
>=C2=A0 1=C2=A0 10.42.3.1=C2=A0 0.318 ms=C2=A0 0.325 ms=C2=A0 0.321 ms >=C2=A0 2=C2=A0 albertslund-edge1-lo.net.gigabit.dk= (185.24.171.254)=C2=A0 1.337 ms=C2=A0 5.390 ms=C2=A0 3.194 ms
>=C2=A0 3=C2=A0 customer-185-24-168-46.ip4.gigabit.dk= (185.24.168.46)=C2=A0 1.319 ms=C2=A0 1.120 ms=C2=A0 1.256 ms
>=C2=A0 4=C2=A0 te0-1-1-5.rcr21.cph01.atlas.cogent= co.com (149.6.137.49)=C2=A0 1.533 ms=C2=A0 1.612 ms=C2=A0 1.392 ms
>=C2=A0 5=C2=A0 be2306.ccr42.ham01.atlas.cogentco.com= (130.117.3.237)=C2=A0 6.787 ms=C2=A0 6.822 ms=C2=A0 6.721 ms
>=C2=A0 6=C2=A0 149.6.142.130=C2=A0 7.000 ms=C2=A0 6.939 ms=C2=A0 6.948 = ms
>=C2=A0 7=C2=A0 one.one.one.one (1.1.1.1) [open]=C2=A0 6.957 ms=C2=A0 6.= 967 ms=C2=A0 6.893 ms
>=C2=A0
> $ sudo tcptraceroute 1.1.1.1
> Selected device eth0, address 10.42.3.130, port 38681 for outgoing pac= kets
> Tracing the path to 1.1.1.1 on TCP port 80 (http), 30 hops max
>=C2=A0 1=C2=A0 10.42.3.1=C2=A0 0.290 ms=C2=A0 0.287 ms=C2=A0 0.292 ms >=C2=A0 2=C2=A0 albertslund-edge1-lo.net.gigabit.dk= (185.24.171.254)=C2=A0 1.857 ms=C2=A0 5.382 ms=C2=A0 18.654 ms
>=C2=A0 3=C2=A0 customer-185-24-168-38.ip4.gigabit.dk= (185.24.168.38)=C2=A0 1.249 ms=C2=A0 1.121 ms=C2=A0 1.521 ms
>=C2=A0 4=C2=A0 10ge1-2.core1.cph1.he.net (216.66.83.101)=C2= =A0 1.375 ms=C2=A0 2.495 ms=C2=A0 1.440 ms
>=C2=A0 5=C2=A0 dix.as13335.net (192.38.7.70)=C2=A0 2.093 ms=C2=A0 1.89= 5 ms=C2=A0 1.790 ms
>=C2=A0 6=C2=A0 one.one.one.one (1.1.1.1) [open]=C2=A0 1.783 ms=C2=A0 1.= 861 ms=C2=A0 1.817 ms
>
>
> Notice how one is one hop longer than the other.

Worse than just longer, it appears as if the exit hop from gigabit.dk
goes to 2 different providers (hop 4 above).=C2=A0 If these are packets tow= ards
an anycast address that is going to cause exactly what you see.=C2=A0 ECMP<= br> accross multiple AS's towards anycast is.. umm.. very fragile and your<= br> seeing one of the problems with anycast.

It is very unlikely that he.net and cogentco.com end up at the same
1.1.1.1 box.

> So definitely something
> to do with anycast; maybe ECMP over both paths since it's changing=
> pretty often?

And the multipath is set to round robin perhaps?

> Now, what I was seeing with the ECN errors was that the SYN-ACK would<= br> > have a different TTL than the first data packet. So what I'm think= ing is
> that maybe there's an ECMP hash that hashes on the wrong parts of = the
> TCP header, and so considers the SYN packet with the ECN bit set to be=
> part of a different flow than the subsequent packets. The result being=
> that the flow is split between two anycasted endpoints, causing the RS= T.
>
> Does this sound completely out in the weeds?
Nope, your spot on, other than this is a ECMP issue, not an ECN issue.
> Has anyone else run into an
> ECMP device that did something similar?

Yes.=C2=A0 When round robin path selection is in use.

> -Toke

--
Rod Grimes=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rgrimes@freebsd.org
--0000000000003abea9059735d505--