From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 66AB83B29D for ; Tue, 12 Nov 2019 08:55:11 -0500 (EST) Received: by mail-wm1-x331.google.com with SMTP id l1so3275136wme.2 for ; Tue, 12 Nov 2019 05:55:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ieee.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=HTld3Gqfa245xwy/qU7IsVWOI22QPbdcbyoWBRsiomo=; b=hQk8e4ZOKbNvZh32Ly0MzUnYjVET/OTiUv23Wt9mYiV8pNoDJCAUSgwzQ4r65IlaM4 gVJj7v1aBAZqgtOENjaEfgHmlBlWpASt64pv/CZUQnUA3L36YuJrGA7u/bdZTGvrvHV+ TlXB0SrzxH7ThK+KFUphsFV3XgFGRtrjgRC7s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=HTld3Gqfa245xwy/qU7IsVWOI22QPbdcbyoWBRsiomo=; b=C494xEeHgYeUParyKuLku20XeNtG/66Bv2nPxhNT042IzAcjaJALetal8c2hAxgWE5 UnIFMgvDkomlzc4OYzJ22egsV5T1OUD5wiQ7nOv2V6O9+d9PjCCwjEsWobxm7qQuHfsB +fla+HN7b2JzNjde4jnFuJ+V6ak6HU0zxQfuDbZgsv+lrTsKQcV5wDf0658+xKLNL4nz cQ6JnAotN2odrn/R3TAKoghdxhK18o4FiGnoyS6LvYkVDbfOXsK9T9jArkK3G7orn9RR vA7x0yvQbsvlB43HNu+khtBXByL/gBjTmrszUxaJgGTwNnz+O3UIDXZ6yQQVU9wuGUM+ OCIg== X-Gm-Message-State: APjAAAX7z0FIbJvbHN/fE/2CXefTVa7DakNVmyWS5YzGs5ZENznfZGNO gM2XD14j7ZQf2wD0QOw44r5rsdfJKHdYuOzVjk7VAQ== X-Google-Smtp-Source: APXvYqwWMH1yht4pePGlMi+uNdVHVFZBpEk7QzpOtRKOekuhfBoFU2fCZhFyB1R3VK4dSP3U40W/T/FlbLbZFfTRB4c= X-Received: by 2002:a1c:1f14:: with SMTP id f20mr3696183wmf.147.1573566910404; Tue, 12 Nov 2019 05:55:10 -0800 (PST) MIME-Version: 1.0 References: <1CF67BBC-B528-4667-97AE-760DCE027466@gmail.com> <87y2wlp9p4.fsf@toke.dk> <87v9rpp7s0.fsf@toke.dk> In-Reply-To: <87v9rpp7s0.fsf@toke.dk> From: Luca Muscariello Date: Tue, 12 Nov 2019 14:54:59 +0100 Message-ID: To: =?UTF-8?B?VG9rZSBIw7hpbGFuZC1Kw7hyZ2Vuc2Vu?= Cc: Mikael Abrahamsson , Rich Brown , ECN-Sane Content-Type: multipart/alternative; boundary="000000000000b9c3b50597269862" Subject: Re: [Ecn-sane] Meanwhile, over on NANOG... X-BeenThere: ecn-sane@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of explicit congestion notification's impact on the Internet List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2019 13:55:11 -0000 --000000000000b9c3b50597269862 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Nov 12, 2019 at 2:02 PM Toke H=C3=B8iland-J=C3=B8rgensen wrote: > Mikael Abrahamsson writes: > > > On Tue, 12 Nov 2019, Toke H=C3=B8iland-J=C3=B8rgensen wrote: > > > >> I'm not on the nanog list, but feel free to cross-post; would be good > to > >> actually get to the bottom of this issue! Marek and I already had an > >> off-list back-and-forth after that original thread, and we couldn't > find > >> anything wrong on the Cloudflare side. And the RSTs have a higher TTL > >> than the actual traffic, indicating an in-path problem... > > > > tcptraceroute supports setting/clearing ECN bits (-E), would be very > > interesting to see difference between those tcptraceroutes? > > No difference. But the RST is not being sent as a response to the SYN; > it is sent in response to the first data packet... > > ... and now that I'm re-testing, things were working for a little while, > but now the bug is back. I got an intermittent successful connection > with the same TTL that I was previously getting the RST from. And now > I'm back to getting RSTed. > > So I guess there's some kind of multipath issue here; ECMP path, > multiple routing upstreams, or a broken load balancer? Any other ideas? > It makes me think of some usage of anycast TCP on the cloudflare side. What service is this Toke? > > -Toke > > > tcpdump output: > > With ECN, and failing. Notice TTL 59 for the SYNACK, but TTL 61 for the > RST: > > 00:00:00.000000 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 > (0x0800), length 74: (tos 0x0, ttl 63, id 21817, offset 0, flags [DF], > proto TCP (6), length 60) > 85.204.121.218.33376 > 1.1.1.1.80: Flags [SEW], cksum 0x5284 > (correct), seq 1677914250, win 64240, options [mss 1460,sackOK,TS val > 438384324 ecr 0,nop,wscale 7], length 0 > 00:00:00.006962 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 > (0x0800), length 66: (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto > TCP (6), length 52) > 1.1.1.1.80 > 85.204.121.218.33376: Flags [S.E], cksum 0x4e79 > (correct), seq 1887212753, ack 1677914251, win 29200, options [mss > 1460,nop,nop,sackOK,nop,wscale 10], length 0 > 00:00:00.000614 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 > (0x0800), length 54: (tos 0x0, ttl 63, id 21818, offset 0, flags [DF], > proto TCP (6), length 40) > 85.204.121.218.33376 > 1.1.1.1.80: Flags [.], cksum 0xffa8 (correct), > seq 1, ack 1, win 502, length 0 > 00:00:00.000255 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 > (0x0800), length 125: (tos 0x2,ECT(0), ttl 63, id 21819, offset 0, flags > [DF], proto TCP (6), length 111) > 85.204.121.218.33376 > 1.1.1.1.80: Flags [P.], cksum 0x05e5 (correct)= , > seq 1:72, ack 1, win 502, length 71: HTTP, length: 71 > GET / HTTP/1.1 > Host: 1.1.1.1 > User-Agent: curl/7.66.0 > Accept: */* > > 00:00:00.001714 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 > (0x0800), length 60: (tos 0x2,ECT(0), ttl 61, id 0, offset 0, flags [DF], > proto TCP (6), length 40) > 1.1.1.1.80 > 85.204.121.218.33376: Flags [R], cksum 0x5639 (correct), > seq 1887212754, win 0, length 0 > > > Without ECN; succeeding, with TTL 59: > > 00:00:00.000000 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 > (0x0800), length 74: (tos 0x0, ttl 63, id 54830, offset 0, flags [DF], > proto TCP (6), length 60) > 85.204.121.218.33362 > 1.1.1.1.80: Flags [S], cksum 0x5430 (correct), > seq 922398600, win 64240, options [mss 1460,sackOK,TS val 438346737 ecr > 0,nop,wscale 7], length 0 > 00:00:00.006895 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 > (0x0800), length 66: (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto > TCP (6), length 52) > 1.1.1.1.80 > 85.204.121.218.33362: Flags [S.], cksum 0xbdf8 (correct)= , > seq 1251654028, ack 922398601, win 29200, options [mss > 1460,nop,nop,sackOK,nop,wscale 10], length 0 > 00:00:00.000570 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 > (0x0800), length 54: (tos 0x0, ttl 63, id 54831, offset 0, flags [DF], > proto TCP (6), length 40) > 85.204.121.218.33362 > 1.1.1.1.80: Flags [.], cksum 0x6ee8 (correct), > seq 1, ack 1, win 502, length 0 > 00:00:00.000261 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 > (0x0800), length 125: (tos 0x0, ttl 63, id 54832, offset 0, flags [DF], > proto TCP (6), length 111) > 85.204.121.218.33362 > 1.1.1.1.80: Flags [P.], cksum 0x7524 (correct)= , > seq 1:72, ack 1, win 502, length 71: HTTP, length: 71 > GET / HTTP/1.1 > Host: 1.1.1.1 > User-Agent: curl/7.66.0 > Accept: */* > > 00:00:00.006955 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 > (0x0800), length 60: (tos 0x0, ttl 59, id 46658, offset 0, flags [DF], > proto TCP (6), length 40) > 1.1.1.1.80 > 85.204.121.218.33362: Flags [.], cksum 0x707a (correct), > seq 1, ack 72, win 29, length 0 > 00:00:00.004938 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 > (0x0800), length 609: (tos 0x0, ttl 59, id 46659, offset 0, flags [DF], > proto TCP (6), length 595) > 1.1.1.1.80 > 85.204.121.218.33362: Flags [P.], cksum 0x13dc (correct)= , > seq 1:556, ack 72, win 29, length 555: HTTP, length: 555 > HTTP/1.1 301 Moved Permanently > Date: Fri, 20 Sep 2019 09:33:56 GMT > Content-Type: text/html > Transfer-Encoding: chunked > Connection: keep-alive > Location: https://1.1.1.1/ > Served-In-Seconds: 0.000 > CF-Cache-Status: HIT > Age: 3920 > Expires: Fri, 20 Sep 2019 13:33:56 GMT > Cache-Control: public, max-age=3D14400 > Server: cloudflare > CF-RAY: 5192ccfbeeefd47b-HAM > > ba > > 301 Moved Permanently > >

301 Moved Permanently

>
cloudflare-lb
> > > > 00:00:00.000002 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 > (0x0800), length 60: (tos 0x0, ttl 59, id 46660, offset 0, flags [DF], > proto TCP (6), length 45) > 1.1.1.1.80 > 85.204.121.218.33362: Flags [P.], cksum 0x2a28 (correct)= , > seq 556:561, ack 72, win 29, length 5: HTTP > 00:00:00.000549 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 > (0x0800), length 54: (tos 0x0, ttl 63, id 54833, offset 0, flags [DF], > proto TCP (6), length 40) > 85.204.121.218.33362 > 1.1.1.1.80: Flags [.], cksum 0x6c77 (correct), > seq 72, ack 556, win 501, length 0 > 00:00:00.000266 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 > (0x0800), length 54: (tos 0x0, ttl 63, id 54834, offset 0, flags [DF], > proto TCP (6), length 40) > 85.204.121.218.33362 > 1.1.1.1.80: Flags [.], cksum 0x6c72 (correct), > seq 72, ack 561, win 501, length 0 > 00:00:00.000217 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 > (0x0800), length 54: (tos 0x0, ttl 63, id 54835, offset 0, flags [DF], > proto TCP (6), length 40) > 85.204.121.218.33362 > 1.1.1.1.80: Flags [F.], cksum 0x6c71 (correct)= , > seq 72, ack 561, win 501, length 0 > 00:00:00.007287 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 > (0x0800), length 60: (tos 0x0, ttl 59, id 46661, offset 0, flags [DF], > proto TCP (6), length 40) > 1.1.1.1.80 > 85.204.121.218.33362: Flags [F.], cksum 0x6e48 (correct)= , > seq 561, ack 73, win 29, length 0 > 00:00:00.000504 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 > (0x0800), length 54: (tos 0x0, ttl 63, id 54836, offset 0, flags [DF], > proto TCP (6), length 40) > 85.204.121.218.33362 > 1.1.1.1.80: Flags [.], cksum 0x6c70 (correct), > seq 73, ack 562, win 501, length 0 > 00:00:05.170886 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 > (0x0800), length 74: (tos 0x0, ttl 64, id 11852, offset 0, flags [DF], > proto TCP (6), length 60) > > > And that one time it worked, with TTL 61: > > 13:47:54.908967 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 > (0x0800), length 74: (tos 0x0, ttl 63, id 53207, offset 0, flags [DF], > proto TCP (6), length 60) > 85.204.121.218.48924 > 1.1.1.1.80: Flags [SEW], cksum 0xa5de > (correct), seq 3526272449, win 64240, options [mss 1460,sackOK,TS val > 513441489 ecr 0,nop,wscale 7], length 0 > 13:47:54.910220 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 > (0x0800), length 66: (tos 0x0, ttl 61, id 0, offset 0, flags [DF], proto > TCP (6), length 52) > 1.1.1.1.80 > 85.204.121.218.48924: Flags [S.E], cksum 0x17dd > (correct), seq 633452041, ack 3526272450, win 29200, options [mss > 1460,nop,nop,sackOK,nop,wscale 10], length 0 > 13:47:54.910747 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 > (0x0800), length 54: (tos 0x0, ttl 63, id 53208, offset 0, flags [DF], > proto TCP (6), length 40) > 85.204.121.218.48924 > 1.1.1.1.80: Flags [.], cksum 0xc90c (correct), > seq 1, ack 1, win 502, length 0 > 13:47:54.910990 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 > (0x0800), length 125: (tos 0x2,ECT(0), ttl 63, id 53209, offset 0, flags > [DF], proto TCP (6), length 111) > 85.204.121.218.48924 > 1.1.1.1.80: Flags [P.], cksum 0xcf48 (correct)= , > seq 1:72, ack 1, win 502, length 71: HTTP, length: 71 > GET / HTTP/1.1 > Host: 1.1.1.1 > User-Agent: curl/7.66.0 > Accept: */* > > 13:47:55.119451 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 > (0x0800), length 125: (tos 0x0, ttl 63, id 53210, offset 0, flags [DF], > proto TCP (6), length 111) > 85.204.121.218.48924 > 1.1.1.1.80: Flags [P.], cksum 0xcf48 (correct)= , > seq 1:72, ack 1, win 502, length 71: HTTP, length: 71 > GET / HTTP/1.1 > Host: 1.1.1.1 > User-Agent: curl/7.66.0 > Accept: */* > > 13:47:55.120638 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 > (0x0800), length 60: (tos 0x0, ttl 61, id 41447, offset 0, flags [DF], > proto TCP (6), length 40) > 1.1.1.1.80 > 85.204.121.218.48924: Flags [.], cksum 0xca9e (correct), > seq 1, ack 72, win 29, length 0 > 13:47:55.130264 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 > (0x0800), length 609: (tos 0x2,ECT(0), ttl 61, id 41448, offset 0, flags > [DF], proto TCP (6), length 595) > 1.1.1.1.80 > 85.204.121.218.48924: Flags [P.], cksum 0xde5e (correct)= , > seq 1:556, ack 72, win 29, length 555: HTTP, length: 555 > HTTP/1.1 301 Moved Permanently > Date: Tue, 12 Nov 2019 12:47:55 GMT > Content-Type: text/html > Transfer-Encoding: chunked > Connection: keep-alive > Location: https://1.1.1.1/ > Served-In-Seconds: 0.000 > CF-Cache-Status: HIT > Age: 2976 > Expires: Tue, 12 Nov 2019 16:47:55 GMT > Cache-Control: public, max-age=3D14400 > Server: cloudflare > CF-RAY: 53489e018ad8d885-CPH > > ba > > 301 Moved Permanently > >

301 Moved Permanently

>
cloudflare-lb
> > > > 13:47:55.130265 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 > (0x0800), length 60: (tos 0x2,ECT(0), ttl 61, id 41449, offset 0, flags > [DF], proto TCP (6), length 45) > 1.1.1.1.80 > 85.204.121.218.48924: Flags [P.], cksum 0x844c (correct)= , > seq 556:561, ack 72, win 29, length 5: HTTP > 13:47:55.130777 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 > (0x0800), length 54: (tos 0x0, ttl 63, id 53211, offset 0, flags [DF], > proto TCP (6), length 40) > 85.204.121.218.48924 > 1.1.1.1.80: Flags [.], cksum 0xc69b (correct), > seq 72, ack 556, win 501, length 0 > 13:47:55.131097 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 > (0x0800), length 54: (tos 0x0, ttl 63, id 53212, offset 0, flags [DF], > proto TCP (6), length 40) > 85.204.121.218.48924 > 1.1.1.1.80: Flags [.], cksum 0xc696 (correct), > seq 72, ack 561, win 501, length 0 > 13:47:55.131491 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 > (0x0800), length 54: (tos 0x0, ttl 63, id 53213, offset 0, flags [DF], > proto TCP (6), length 40) > 85.204.121.218.48924 > 1.1.1.1.80: Flags [F.], cksum 0xc695 (correct)= , > seq 72, ack 561, win 501, length 0 > 13:47:55.132804 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 > (0x0800), length 60: (tos 0x0, ttl 61, id 41450, offset 0, flags [DF], > proto TCP (6), length 40) > 1.1.1.1.80 > 85.204.121.218.48924: Flags [F.], cksum 0xc86c (correct)= , > seq 561, ack 73, win 29, length 0 > 13:47:55.133281 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 > (0x0800), length 54: (tos 0x0, ttl 63, id 53214, offset 0, flags [DF], > proto TCP (6), length 40) > 85.204.121.218.48924 > 1.1.1.1.80: Flags [.], cksum 0xc694 (correct), > seq 73, ack 562, win 501, length 0 > > _______________________________________________ > Ecn-sane mailing list > Ecn-sane@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/ecn-sane > --000000000000b9c3b50597269862 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Tue, Nov 12, 2019 at 2:02 = PM Toke H=C3=B8iland-J=C3=B8rgensen <tok= e@toke.dk> wrote:
Mikael Abrahamsson <swmike@swm.pp.se> writes:

> On Tue, 12 Nov 2019, Toke H=C3=B8iland-J=C3=B8rgensen wrote:
>
>> I'm not on the nanog list, but feel free to cross-post; would = be good to
>> actually get to the bottom of this issue! Marek and I already had = an
>> off-list back-and-forth after that original thread, and we couldn&= #39;t find
>> anything wrong on the Cloudflare side. And the RSTs have a higher = TTL
>> than the actual traffic, indicating an in-path problem...
>
> tcptraceroute supports setting/clearing ECN bits (-E), would be very <= br> > interesting to see difference between those tcptraceroutes?

No difference. But the RST is not being sent as a response to the SYN;
it is sent in response to the first data packet...

... and now that I'm re-testing, things were working for a little while= ,
but now the bug is back. I got an intermittent successful connection
with the same TTL that I was previously getting the RST from. And now
I'm back to getting RSTed.

So I guess there's some kind of multipath issue here; ECMP path,
multiple routing upstreams, or a broken load balancer? Any other ideas?
=


It makes me think of some= usage of anycast TCP on the cloudflare side.
What service is this T= oke?


=C2=A0

-Toke


tcpdump output:

With ECN, and failing. Notice TTL 59 for the SYNACK, but TTL 61 for the
RST:

=C2=A000:00:00.000000 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype I= Pv4 (0x0800), length 74: (tos 0x0, ttl 63, id 21817, offset 0, flags [DF], = proto TCP (6), length 60)
=C2=A0 =C2=A0 85.204.121.218.33376 > 1.1.1.1.80: Flags [SEW], cksum 0x52= 84 (correct), seq 1677914250, win 64240, options [mss 1460,sackOK,TS val 43= 8384324 ecr 0,nop,wscale 7], length 0
=C2=A000:00:00.006962 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype I= Pv4 (0x0800), length 66: (tos 0x0, ttl 59, id 0, offset 0, flags [DF], prot= o TCP (6), length 52)
=C2=A0 =C2=A0 1.1.1.1.80 > 85.204.121.218.33376: Flags [S.E], cksum 0x4e= 79 (correct), seq 1887212753, ack 1677914251, win 29200, options [mss 1460,= nop,nop,sackOK,nop,wscale 10], length 0
=C2=A000:00:00.000614 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype I= Pv4 (0x0800), length 54: (tos 0x0, ttl 63, id 21818, offset 0, flags [DF], = proto TCP (6), length 40)
=C2=A0 =C2=A0 85.204.121.218.33376 > 1.1.1.1.80: Flags [.], cksum 0xffa8= (correct), seq 1, ack 1, win 502, length 0
=C2=A000:00:00.000255 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype I= Pv4 (0x0800), length 125: (tos 0x2,ECT(0), ttl 63, id 21819, offset 0, flag= s [DF], proto TCP (6), length 111)
=C2=A0 =C2=A0 85.204.121.218.33376 > 1.1.1.1.80: Flags [P.], cksum 0x05e= 5 (correct), seq 1:72, ack 1, win 502, length 71: HTTP, length: 71
=C2=A0 =C2=A0 =C2=A0 =C2=A0 GET / HTTP/1.1
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Host: 1.1.1.1
=C2=A0 =C2=A0 =C2=A0 =C2=A0 User-Agent: curl/7.66.0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Accept: */*

=C2=A000:00:00.001714 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype I= Pv4 (0x0800), length 60: (tos 0x2,ECT(0), ttl 61, id 0, offset 0, flags [DF= ], proto TCP (6), length 40)
=C2=A0 =C2=A0 1.1.1.1.80 > 85.204.121.218.33376: Flags [R], cksum 0x5639= (correct), seq 1887212754, win 0, length 0


Without ECN; succeeding, with TTL 59:

=C2=A000:00:00.000000 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype I= Pv4 (0x0800), length 74: (tos 0x0, ttl 63, id 54830, offset 0, flags [DF], = proto TCP (6), length 60)
=C2=A0 =C2=A0 85.204.121.218.33362 > 1.1.1.1.80: Flags [S], cksum 0x5430= (correct), seq 922398600, win 64240, options [mss 1460,sackOK,TS val 43834= 6737 ecr 0,nop,wscale 7], length 0
=C2=A000:00:00.006895 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype I= Pv4 (0x0800), length 66: (tos 0x0, ttl 59, id 0, offset 0, flags [DF], prot= o TCP (6), length 52)
=C2=A0 =C2=A0 1.1.1.1.80 > 85.204.121.218.33362: Flags [S.], cksum 0xbdf= 8 (correct), seq 1251654028, ack 922398601, win 29200, options [mss 1460,no= p,nop,sackOK,nop,wscale 10], length 0
=C2=A000:00:00.000570 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype I= Pv4 (0x0800), length 54: (tos 0x0, ttl 63, id 54831, offset 0, flags [DF], = proto TCP (6), length 40)
=C2=A0 =C2=A0 85.204.121.218.33362 > 1.1.1.1.80: Flags [.], cksum 0x6ee8= (correct), seq 1, ack 1, win 502, length 0
=C2=A000:00:00.000261 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype I= Pv4 (0x0800), length 125: (tos 0x0, ttl 63, id 54832, offset 0, flags [DF],= proto TCP (6), length 111)
=C2=A0 =C2=A0 85.204.121.218.33362 > 1.1.1.1.80: Flags [P.], cksum 0x752= 4 (correct), seq 1:72, ack 1, win 502, length 71: HTTP, length: 71
=C2=A0 =C2=A0 =C2=A0 =C2=A0 GET / HTTP/1.1
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Host: 1.1.1.1
=C2=A0 =C2=A0 =C2=A0 =C2=A0 User-Agent: curl/7.66.0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Accept: */*

=C2=A000:00:00.006955 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype I= Pv4 (0x0800), length 60: (tos 0x0, ttl 59, id 46658, offset 0, flags [DF], = proto TCP (6), length 40)
=C2=A0 =C2=A0 1.1.1.1.80 > 85.204.121.218.33362: Flags [.], cksum 0x707a= (correct), seq 1, ack 72, win 29, length 0
=C2=A000:00:00.004938 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype I= Pv4 (0x0800), length 609: (tos 0x0, ttl 59, id 46659, offset 0, flags [DF],= proto TCP (6), length 595)
=C2=A0 =C2=A0 1.1.1.1.80 > 85.204.121.218.33362: Flags [P.], cksum 0x13d= c (correct), seq 1:556, ack 72, win 29, length 555: HTTP, length: 555
=C2=A0 =C2=A0 =C2=A0 =C2=A0 HTTP/1.1 301 Moved Permanently
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Date: Fri, 20 Sep 2019 09:33:56 GMT
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Content-Type: text/html
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Transfer-Encoding: chunked
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Connection: keep-alive
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Location: https://1.1.1.1/
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Served-In-Seconds: 0.000
=C2=A0 =C2=A0 =C2=A0 =C2=A0 CF-Cache-Status: HIT
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Age: 3920
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Expires: Fri, 20 Sep 2019 13:33:56 GMT
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Cache-Control: public, max-age=3D14400
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Server: cloudflare
=C2=A0 =C2=A0 =C2=A0 =C2=A0 CF-RAY: 5192ccfbeeefd47b-HAM

=C2=A0 =C2=A0 =C2=A0 =C2=A0 ba
=C2=A0 =C2=A0 =C2=A0 =C2=A0 <html>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 <head><title>301 Moved Permanently&= lt;/title></head>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 <body bgcolor=3D"white">
=C2=A0 =C2=A0 =C2=A0 =C2=A0 <center><h1>301 Moved Permanently&l= t;/h1></center>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 <hr><center>cloudflare-lb</cente= r>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 </body>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 </html>

=C2=A000:00:00.000002 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype I= Pv4 (0x0800), length 60: (tos 0x0, ttl 59, id 46660, offset 0, flags [DF], = proto TCP (6), length 45)
=C2=A0 =C2=A0 1.1.1.1.80 > 85.204.121.218.33362: Flags [P.], cksum 0x2a2= 8 (correct), seq 556:561, ack 72, win 29, length 5: HTTP
=C2=A000:00:00.000549 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype I= Pv4 (0x0800), length 54: (tos 0x0, ttl 63, id 54833, offset 0, flags [DF], = proto TCP (6), length 40)
=C2=A0 =C2=A0 85.204.121.218.33362 > 1.1.1.1.80: Flags [.], cksum 0x6c77= (correct), seq 72, ack 556, win 501, length 0
=C2=A000:00:00.000266 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype I= Pv4 (0x0800), length 54: (tos 0x0, ttl 63, id 54834, offset 0, flags [DF], = proto TCP (6), length 40)
=C2=A0 =C2=A0 85.204.121.218.33362 > 1.1.1.1.80: Flags [.], cksum 0x6c72= (correct), seq 72, ack 561, win 501, length 0
=C2=A000:00:00.000217 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype I= Pv4 (0x0800), length 54: (tos 0x0, ttl 63, id 54835, offset 0, flags [DF], = proto TCP (6), length 40)
=C2=A0 =C2=A0 85.204.121.218.33362 > 1.1.1.1.80: Flags [F.], cksum 0x6c7= 1 (correct), seq 72, ack 561, win 501, length 0
=C2=A000:00:00.007287 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype I= Pv4 (0x0800), length 60: (tos 0x0, ttl 59, id 46661, offset 0, flags [DF], = proto TCP (6), length 40)
=C2=A0 =C2=A0 1.1.1.1.80 > 85.204.121.218.33362: Flags [F.], cksum 0x6e4= 8 (correct), seq 561, ack 73, win 29, length 0
=C2=A000:00:00.000504 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype I= Pv4 (0x0800), length 54: (tos 0x0, ttl 63, id 54836, offset 0, flags [DF], = proto TCP (6), length 40)
=C2=A0 =C2=A0 85.204.121.218.33362 > 1.1.1.1.80: Flags [.], cksum 0x6c70= (correct), seq 73, ack 562, win 501, length 0
=C2=A000:00:05.170886 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype I= Pv4 (0x0800), length 74: (tos 0x0, ttl 64, id 11852, offset 0, flags [DF], = proto TCP (6), length 60)


And that one time it worked, with TTL 61:

13:47:54.908967 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0= x0800), length 74: (tos 0x0, ttl 63, id 53207, offset 0, flags [DF], proto = TCP (6), length 60)
=C2=A0 =C2=A0 85.204.121.218.48924 > 1.1.1.1.80: Flags [SEW], cksum 0xa5= de (correct), seq 3526272449, win 64240, options [mss 1460,sackOK,TS val 51= 3441489 ecr 0,nop,wscale 7], length 0
13:47:54.910220 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0= x0800), length 66: (tos 0x0, ttl 61, id 0, offset 0, flags [DF], proto TCP = (6), length 52)
=C2=A0 =C2=A0 1.1.1.1.80 > 85.204.121.218.48924: Flags [S.E], cksum 0x17= dd (correct), seq 633452041, ack 3526272450, win 29200, options [mss 1460,n= op,nop,sackOK,nop,wscale 10], length 0
13:47:54.910747 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0= x0800), length 54: (tos 0x0, ttl 63, id 53208, offset 0, flags [DF], proto = TCP (6), length 40)
=C2=A0 =C2=A0 85.204.121.218.48924 > 1.1.1.1.80: Flags [.], cksum 0xc90c= (correct), seq 1, ack 1, win 502, length 0
13:47:54.910990 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0= x0800), length 125: (tos 0x2,ECT(0), ttl 63, id 53209, offset 0, flags [DF]= , proto TCP (6), length 111)
=C2=A0 =C2=A0 85.204.121.218.48924 > 1.1.1.1.80: Flags [P.], cksum 0xcf4= 8 (correct), seq 1:72, ack 1, win 502, length 71: HTTP, length: 71
=C2=A0 =C2=A0 =C2=A0 =C2=A0 GET / HTTP/1.1
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Host: 1.1.1.1
=C2=A0 =C2=A0 =C2=A0 =C2=A0 User-Agent: curl/7.66.0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Accept: */*

13:47:55.119451 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0= x0800), length 125: (tos 0x0, ttl 63, id 53210, offset 0, flags [DF], proto= TCP (6), length 111)
=C2=A0 =C2=A0 85.204.121.218.48924 > 1.1.1.1.80: Flags [P.], cksum 0xcf4= 8 (correct), seq 1:72, ack 1, win 502, length 71: HTTP, length: 71
=C2=A0 =C2=A0 =C2=A0 =C2=A0 GET / HTTP/1.1
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Host: 1.1.1.1
=C2=A0 =C2=A0 =C2=A0 =C2=A0 User-Agent: curl/7.66.0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Accept: */*

13:47:55.120638 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0= x0800), length 60: (tos 0x0, ttl 61, id 41447, offset 0, flags [DF], proto = TCP (6), length 40)
=C2=A0 =C2=A0 1.1.1.1.80 > 85.204.121.218.48924: Flags [.], cksum 0xca9e= (correct), seq 1, ack 72, win 29, length 0
13:47:55.130264 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0= x0800), length 609: (tos 0x2,ECT(0), ttl 61, id 41448, offset 0, flags [DF]= , proto TCP (6), length 595)
=C2=A0 =C2=A0 1.1.1.1.80 > 85.204.121.218.48924: Flags [P.], cksum 0xde5= e (correct), seq 1:556, ack 72, win 29, length 555: HTTP, length: 555
=C2=A0 =C2=A0 =C2=A0 =C2=A0 HTTP/1.1 301 Moved Permanently
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Date: Tue, 12 Nov 2019 12:47:55 GMT
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Content-Type: text/html
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Transfer-Encoding: chunked
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Connection: keep-alive
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Location: https://1.1.1.1/
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Served-In-Seconds: 0.000
=C2=A0 =C2=A0 =C2=A0 =C2=A0 CF-Cache-Status: HIT
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Age: 2976
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Expires: Tue, 12 Nov 2019 16:47:55 GMT
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Cache-Control: public, max-age=3D14400
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Server: cloudflare
=C2=A0 =C2=A0 =C2=A0 =C2=A0 CF-RAY: 53489e018ad8d885-CPH

=C2=A0 =C2=A0 =C2=A0 =C2=A0 ba
=C2=A0 =C2=A0 =C2=A0 =C2=A0 <html>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 <head><title>301 Moved Permanently&= lt;/title></head>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 <body bgcolor=3D"white">
=C2=A0 =C2=A0 =C2=A0 =C2=A0 <center><h1>301 Moved Permanently&l= t;/h1></center>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 <hr><center>cloudflare-lb</cente= r>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 </body>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 </html>

13:47:55.130265 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0= x0800), length 60: (tos 0x2,ECT(0), ttl 61, id 41449, offset 0, flags [DF],= proto TCP (6), length 45)
=C2=A0 =C2=A0 1.1.1.1.80 > 85.204.121.218.48924: Flags [P.], cksum 0x844= c (correct), seq 556:561, ack 72, win 29, length 5: HTTP
13:47:55.130777 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0= x0800), length 54: (tos 0x0, ttl 63, id 53211, offset 0, flags [DF], proto = TCP (6), length 40)
=C2=A0 =C2=A0 85.204.121.218.48924 > 1.1.1.1.80: Flags [.], cksum 0xc69b= (correct), seq 72, ack 556, win 501, length 0
13:47:55.131097 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0= x0800), length 54: (tos 0x0, ttl 63, id 53212, offset 0, flags [DF], proto = TCP (6), length 40)
=C2=A0 =C2=A0 85.204.121.218.48924 > 1.1.1.1.80: Flags [.], cksum 0xc696= (correct), seq 72, ack 561, win 501, length 0
13:47:55.131491 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0= x0800), length 54: (tos 0x0, ttl 63, id 53213, offset 0, flags [DF], proto = TCP (6), length 40)
=C2=A0 =C2=A0 85.204.121.218.48924 > 1.1.1.1.80: Flags [F.], cksum 0xc69= 5 (correct), seq 72, ack 561, win 501, length 0
13:47:55.132804 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4 (0= x0800), length 60: (tos 0x0, ttl 61, id 41450, offset 0, flags [DF], proto = TCP (6), length 40)
=C2=A0 =C2=A0 1.1.1.1.80 > 85.204.121.218.48924: Flags [F.], cksum 0xc86= c (correct), seq 561, ack 73, win 29, length 0
13:47:55.133281 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4 (0= x0800), length 54: (tos 0x0, ttl 63, id 53214, offset 0, flags [DF], proto = TCP (6), length 40)
=C2=A0 =C2=A0 85.204.121.218.48924 > 1.1.1.1.80: Flags [.], cksum 0xc694= (correct), seq 73, ack 562, win 501, length 0

_______________________________________________
Ecn-sane mailing list
Ecn-san= e@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/ecn-sane
--000000000000b9c3b50597269862--