From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.toke.dk (mail.toke.dk [IPv6:2a00:7660:6da:2001::664]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id E19AD3B29D for ; Tue, 12 Nov 2019 09:35:28 -0500 (EST) From: Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=toke.dk; s=20161023; t=1573569326; bh=ExOWIdT5QUnS9HicbHk5MLkLyQV1UMZISc2YEuofw8Q=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=urJubA+52CTRGx+xOV2fXZE1ifsZLXJ06HiX1/+IXXeNBETqhwImLGlO3I2Ws4ffq pYUAJ4xq8/cMvRe4Y2F4SLcM6ByRKmYtlD5DupUBBT5O/kAUCqsMZU8vVoAKEKL7RK DV5TGpm2PLPK79sTaHaNFW7syvt6rrEudrOD6HCUKr20XNKSyTRCGd+t09bL42q389 BGr3z/fLoydPFnqaEB2D4lJtt+kpvTeb0/LHBQoFP0d+5EjSL/JSx4FoXDsmTu2T+3 69d3rppxe8gCukNFiru+pUSkt/XoCoum1Q2bf/eNGOSpA2PsX9aQQoDu1UaHEesM7s N/PyFj8gQ2Kxg== To: Luca Muscariello Cc: Mikael Abrahamsson , Rich Brown , ECN-Sane In-Reply-To: References: <1CF67BBC-B528-4667-97AE-760DCE027466@gmail.com> <87y2wlp9p4.fsf@toke.dk> <87v9rpp7s0.fsf@toke.dk> Date: Tue, 12 Nov 2019 15:35:25 +0100 X-Clacks-Overhead: GNU Terry Pratchett Message-ID: <87pnhxp3gy.fsf@toke.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [Ecn-sane] Meanwhile, over on NANOG... X-BeenThere: ecn-sane@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of explicit congestion notification's impact on the Internet List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2019 14:35:29 -0000 Luca Muscariello writes: > On Tue, Nov 12, 2019 at 2:02 PM Toke H=C3=B8iland-J=C3=B8rgensen wrote: > >> Mikael Abrahamsson writes: >> >> > On Tue, 12 Nov 2019, Toke H=C3=B8iland-J=C3=B8rgensen wrote: >> > >> >> I'm not on the nanog list, but feel free to cross-post; would be good >> to >> >> actually get to the bottom of this issue! Marek and I already had an >> >> off-list back-and-forth after that original thread, and we couldn't >> find >> >> anything wrong on the Cloudflare side. And the RSTs have a higher TTL >> >> than the actual traffic, indicating an in-path problem... >> > >> > tcptraceroute supports setting/clearing ECN bits (-E), would be very >> > interesting to see difference between those tcptraceroutes? >> >> No difference. But the RST is not being sent as a response to the SYN; >> it is sent in response to the first data packet... >> >> ... and now that I'm re-testing, things were working for a little while, >> but now the bug is back. I got an intermittent successful connection >> with the same TTL that I was previously getting the RST from. And now >> I'm back to getting RSTed. >> >> So I guess there's some kind of multipath issue here; ECMP path, >> multiple routing upstreams, or a broken load balancer? Any other ideas? >> > > > It makes me think of some usage of anycast TCP on the cloudflare side. > What service is this Toke? Yeah, I did also think about anycast when I said "multiple routing upstreams". For testing I've just been doing 'curl 1.1.1.1'. But Cloudflare-hosted sites in general seem to have this problem; for instance, 'curl -4 bufferbloat.net' also fails (but IPv6 is fine). -Toke