From: Luca Muscariello <muscariello@ieee.org>
To: "Toke Høiland-Jørgensen" <toke@toke.dk>
Cc: Mikael Abrahamsson <swmike@swm.pp.se>,
Rich Brown <richb.hanover@gmail.com>,
ECN-Sane <ecn-sane@lists.bufferbloat.net>
Subject: Re: [Ecn-sane] Meanwhile, over on NANOG...
Date: Tue, 12 Nov 2019 14:54:59 +0100 [thread overview]
Message-ID: <CAH8sseTU+p7tv3iLzs8Ddmnzwv0GT__1ehmDqKDkME0pf9hAtQ@mail.gmail.com> (raw)
In-Reply-To: <87v9rpp7s0.fsf@toke.dk>
[-- Attachment #1: Type: text/plain, Size: 13244 bytes --]
On Tue, Nov 12, 2019 at 2:02 PM Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> Mikael Abrahamsson <swmike@swm.pp.se> writes:
>
> > On Tue, 12 Nov 2019, Toke Høiland-Jørgensen wrote:
> >
> >> I'm not on the nanog list, but feel free to cross-post; would be good
> to
> >> actually get to the bottom of this issue! Marek and I already had an
> >> off-list back-and-forth after that original thread, and we couldn't
> find
> >> anything wrong on the Cloudflare side. And the RSTs have a higher TTL
> >> than the actual traffic, indicating an in-path problem...
> >
> > tcptraceroute supports setting/clearing ECN bits (-E), would be very
> > interesting to see difference between those tcptraceroutes?
>
> No difference. But the RST is not being sent as a response to the SYN;
> it is sent in response to the first data packet...
>
> ... and now that I'm re-testing, things were working for a little while,
> but now the bug is back. I got an intermittent successful connection
> with the same TTL that I was previously getting the RST from. And now
> I'm back to getting RSTed.
>
> So I guess there's some kind of multipath issue here; ECMP path,
> multiple routing upstreams, or a broken load balancer? Any other ideas?
>
It makes me think of some usage of anycast TCP on the cloudflare side.
What service is this Toke?
>
> -Toke
>
>
> tcpdump output:
>
> With ECN, and failing. Notice TTL 59 for the SYNACK, but TTL 61 for the
> RST:
>
> 00:00:00.000000 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4
> (0x0800), length 74: (tos 0x0, ttl 63, id 21817, offset 0, flags [DF],
> proto TCP (6), length 60)
> 85.204.121.218.33376 > 1.1.1.1.80: Flags [SEW], cksum 0x5284
> (correct), seq 1677914250, win 64240, options [mss 1460,sackOK,TS val
> 438384324 ecr 0,nop,wscale 7], length 0
> 00:00:00.006962 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4
> (0x0800), length 66: (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto
> TCP (6), length 52)
> 1.1.1.1.80 > 85.204.121.218.33376: Flags [S.E], cksum 0x4e79
> (correct), seq 1887212753, ack 1677914251, win 29200, options [mss
> 1460,nop,nop,sackOK,nop,wscale 10], length 0
> 00:00:00.000614 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4
> (0x0800), length 54: (tos 0x0, ttl 63, id 21818, offset 0, flags [DF],
> proto TCP (6), length 40)
> 85.204.121.218.33376 > 1.1.1.1.80: Flags [.], cksum 0xffa8 (correct),
> seq 1, ack 1, win 502, length 0
> 00:00:00.000255 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4
> (0x0800), length 125: (tos 0x2,ECT(0), ttl 63, id 21819, offset 0, flags
> [DF], proto TCP (6), length 111)
> 85.204.121.218.33376 > 1.1.1.1.80: Flags [P.], cksum 0x05e5 (correct),
> seq 1:72, ack 1, win 502, length 71: HTTP, length: 71
> GET / HTTP/1.1
> Host: 1.1.1.1
> User-Agent: curl/7.66.0
> Accept: */*
>
> 00:00:00.001714 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4
> (0x0800), length 60: (tos 0x2,ECT(0), ttl 61, id 0, offset 0, flags [DF],
> proto TCP (6), length 40)
> 1.1.1.1.80 > 85.204.121.218.33376: Flags [R], cksum 0x5639 (correct),
> seq 1887212754, win 0, length 0
>
>
> Without ECN; succeeding, with TTL 59:
>
> 00:00:00.000000 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4
> (0x0800), length 74: (tos 0x0, ttl 63, id 54830, offset 0, flags [DF],
> proto TCP (6), length 60)
> 85.204.121.218.33362 > 1.1.1.1.80: Flags [S], cksum 0x5430 (correct),
> seq 922398600, win 64240, options [mss 1460,sackOK,TS val 438346737 ecr
> 0,nop,wscale 7], length 0
> 00:00:00.006895 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4
> (0x0800), length 66: (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto
> TCP (6), length 52)
> 1.1.1.1.80 > 85.204.121.218.33362: Flags [S.], cksum 0xbdf8 (correct),
> seq 1251654028, ack 922398601, win 29200, options [mss
> 1460,nop,nop,sackOK,nop,wscale 10], length 0
> 00:00:00.000570 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4
> (0x0800), length 54: (tos 0x0, ttl 63, id 54831, offset 0, flags [DF],
> proto TCP (6), length 40)
> 85.204.121.218.33362 > 1.1.1.1.80: Flags [.], cksum 0x6ee8 (correct),
> seq 1, ack 1, win 502, length 0
> 00:00:00.000261 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4
> (0x0800), length 125: (tos 0x0, ttl 63, id 54832, offset 0, flags [DF],
> proto TCP (6), length 111)
> 85.204.121.218.33362 > 1.1.1.1.80: Flags [P.], cksum 0x7524 (correct),
> seq 1:72, ack 1, win 502, length 71: HTTP, length: 71
> GET / HTTP/1.1
> Host: 1.1.1.1
> User-Agent: curl/7.66.0
> Accept: */*
>
> 00:00:00.006955 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4
> (0x0800), length 60: (tos 0x0, ttl 59, id 46658, offset 0, flags [DF],
> proto TCP (6), length 40)
> 1.1.1.1.80 > 85.204.121.218.33362: Flags [.], cksum 0x707a (correct),
> seq 1, ack 72, win 29, length 0
> 00:00:00.004938 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4
> (0x0800), length 609: (tos 0x0, ttl 59, id 46659, offset 0, flags [DF],
> proto TCP (6), length 595)
> 1.1.1.1.80 > 85.204.121.218.33362: Flags [P.], cksum 0x13dc (correct),
> seq 1:556, ack 72, win 29, length 555: HTTP, length: 555
> HTTP/1.1 301 Moved Permanently
> Date: Fri, 20 Sep 2019 09:33:56 GMT
> Content-Type: text/html
> Transfer-Encoding: chunked
> Connection: keep-alive
> Location: https://1.1.1.1/
> Served-In-Seconds: 0.000
> CF-Cache-Status: HIT
> Age: 3920
> Expires: Fri, 20 Sep 2019 13:33:56 GMT
> Cache-Control: public, max-age=14400
> Server: cloudflare
> CF-RAY: 5192ccfbeeefd47b-HAM
>
> ba
> <html>
> <head><title>301 Moved Permanently</title></head>
> <body bgcolor="white">
> <center><h1>301 Moved Permanently</h1></center>
> <hr><center>cloudflare-lb</center>
> </body>
> </html>
>
> 00:00:00.000002 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4
> (0x0800), length 60: (tos 0x0, ttl 59, id 46660, offset 0, flags [DF],
> proto TCP (6), length 45)
> 1.1.1.1.80 > 85.204.121.218.33362: Flags [P.], cksum 0x2a28 (correct),
> seq 556:561, ack 72, win 29, length 5: HTTP
> 00:00:00.000549 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4
> (0x0800), length 54: (tos 0x0, ttl 63, id 54833, offset 0, flags [DF],
> proto TCP (6), length 40)
> 85.204.121.218.33362 > 1.1.1.1.80: Flags [.], cksum 0x6c77 (correct),
> seq 72, ack 556, win 501, length 0
> 00:00:00.000266 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4
> (0x0800), length 54: (tos 0x0, ttl 63, id 54834, offset 0, flags [DF],
> proto TCP (6), length 40)
> 85.204.121.218.33362 > 1.1.1.1.80: Flags [.], cksum 0x6c72 (correct),
> seq 72, ack 561, win 501, length 0
> 00:00:00.000217 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4
> (0x0800), length 54: (tos 0x0, ttl 63, id 54835, offset 0, flags [DF],
> proto TCP (6), length 40)
> 85.204.121.218.33362 > 1.1.1.1.80: Flags [F.], cksum 0x6c71 (correct),
> seq 72, ack 561, win 501, length 0
> 00:00:00.007287 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4
> (0x0800), length 60: (tos 0x0, ttl 59, id 46661, offset 0, flags [DF],
> proto TCP (6), length 40)
> 1.1.1.1.80 > 85.204.121.218.33362: Flags [F.], cksum 0x6e48 (correct),
> seq 561, ack 73, win 29, length 0
> 00:00:00.000504 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4
> (0x0800), length 54: (tos 0x0, ttl 63, id 54836, offset 0, flags [DF],
> proto TCP (6), length 40)
> 85.204.121.218.33362 > 1.1.1.1.80: Flags [.], cksum 0x6c70 (correct),
> seq 73, ack 562, win 501, length 0
> 00:00:05.170886 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4
> (0x0800), length 74: (tos 0x0, ttl 64, id 11852, offset 0, flags [DF],
> proto TCP (6), length 60)
>
>
> And that one time it worked, with TTL 61:
>
> 13:47:54.908967 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4
> (0x0800), length 74: (tos 0x0, ttl 63, id 53207, offset 0, flags [DF],
> proto TCP (6), length 60)
> 85.204.121.218.48924 > 1.1.1.1.80: Flags [SEW], cksum 0xa5de
> (correct), seq 3526272449, win 64240, options [mss 1460,sackOK,TS val
> 513441489 ecr 0,nop,wscale 7], length 0
> 13:47:54.910220 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4
> (0x0800), length 66: (tos 0x0, ttl 61, id 0, offset 0, flags [DF], proto
> TCP (6), length 52)
> 1.1.1.1.80 > 85.204.121.218.48924: Flags [S.E], cksum 0x17dd
> (correct), seq 633452041, ack 3526272450, win 29200, options [mss
> 1460,nop,nop,sackOK,nop,wscale 10], length 0
> 13:47:54.910747 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4
> (0x0800), length 54: (tos 0x0, ttl 63, id 53208, offset 0, flags [DF],
> proto TCP (6), length 40)
> 85.204.121.218.48924 > 1.1.1.1.80: Flags [.], cksum 0xc90c (correct),
> seq 1, ack 1, win 502, length 0
> 13:47:54.910990 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4
> (0x0800), length 125: (tos 0x2,ECT(0), ttl 63, id 53209, offset 0, flags
> [DF], proto TCP (6), length 111)
> 85.204.121.218.48924 > 1.1.1.1.80: Flags [P.], cksum 0xcf48 (correct),
> seq 1:72, ack 1, win 502, length 71: HTTP, length: 71
> GET / HTTP/1.1
> Host: 1.1.1.1
> User-Agent: curl/7.66.0
> Accept: */*
>
> 13:47:55.119451 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4
> (0x0800), length 125: (tos 0x0, ttl 63, id 53210, offset 0, flags [DF],
> proto TCP (6), length 111)
> 85.204.121.218.48924 > 1.1.1.1.80: Flags [P.], cksum 0xcf48 (correct),
> seq 1:72, ack 1, win 502, length 71: HTTP, length: 71
> GET / HTTP/1.1
> Host: 1.1.1.1
> User-Agent: curl/7.66.0
> Accept: */*
>
> 13:47:55.120638 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4
> (0x0800), length 60: (tos 0x0, ttl 61, id 41447, offset 0, flags [DF],
> proto TCP (6), length 40)
> 1.1.1.1.80 > 85.204.121.218.48924: Flags [.], cksum 0xca9e (correct),
> seq 1, ack 72, win 29, length 0
> 13:47:55.130264 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4
> (0x0800), length 609: (tos 0x2,ECT(0), ttl 61, id 41448, offset 0, flags
> [DF], proto TCP (6), length 595)
> 1.1.1.1.80 > 85.204.121.218.48924: Flags [P.], cksum 0xde5e (correct),
> seq 1:556, ack 72, win 29, length 555: HTTP, length: 555
> HTTP/1.1 301 Moved Permanently
> Date: Tue, 12 Nov 2019 12:47:55 GMT
> Content-Type: text/html
> Transfer-Encoding: chunked
> Connection: keep-alive
> Location: https://1.1.1.1/
> Served-In-Seconds: 0.000
> CF-Cache-Status: HIT
> Age: 2976
> Expires: Tue, 12 Nov 2019 16:47:55 GMT
> Cache-Control: public, max-age=14400
> Server: cloudflare
> CF-RAY: 53489e018ad8d885-CPH
>
> ba
> <html>
> <head><title>301 Moved Permanently</title></head>
> <body bgcolor="white">
> <center><h1>301 Moved Permanently</h1></center>
> <hr><center>cloudflare-lb</center>
> </body>
> </html>
>
> 13:47:55.130265 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4
> (0x0800), length 60: (tos 0x2,ECT(0), ttl 61, id 41449, offset 0, flags
> [DF], proto TCP (6), length 45)
> 1.1.1.1.80 > 85.204.121.218.48924: Flags [P.], cksum 0x844c (correct),
> seq 556:561, ack 72, win 29, length 5: HTTP
> 13:47:55.130777 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4
> (0x0800), length 54: (tos 0x0, ttl 63, id 53211, offset 0, flags [DF],
> proto TCP (6), length 40)
> 85.204.121.218.48924 > 1.1.1.1.80: Flags [.], cksum 0xc69b (correct),
> seq 72, ack 556, win 501, length 0
> 13:47:55.131097 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4
> (0x0800), length 54: (tos 0x0, ttl 63, id 53212, offset 0, flags [DF],
> proto TCP (6), length 40)
> 85.204.121.218.48924 > 1.1.1.1.80: Flags [.], cksum 0xc696 (correct),
> seq 72, ack 561, win 501, length 0
> 13:47:55.131491 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4
> (0x0800), length 54: (tos 0x0, ttl 63, id 53213, offset 0, flags [DF],
> proto TCP (6), length 40)
> 85.204.121.218.48924 > 1.1.1.1.80: Flags [F.], cksum 0xc695 (correct),
> seq 72, ack 561, win 501, length 0
> 13:47:55.132804 cc:1a:fa:e2:bb:20 > d8:58:d7:00:1d:2c, ethertype IPv4
> (0x0800), length 60: (tos 0x0, ttl 61, id 41450, offset 0, flags [DF],
> proto TCP (6), length 40)
> 1.1.1.1.80 > 85.204.121.218.48924: Flags [F.], cksum 0xc86c (correct),
> seq 561, ack 73, win 29, length 0
> 13:47:55.133281 d8:58:d7:00:1d:2c > cc:1a:fa:e2:bb:20, ethertype IPv4
> (0x0800), length 54: (tos 0x0, ttl 63, id 53214, offset 0, flags [DF],
> proto TCP (6), length 40)
> 85.204.121.218.48924 > 1.1.1.1.80: Flags [.], cksum 0xc694 (correct),
> seq 73, ack 562, win 501, length 0
>
> _______________________________________________
> Ecn-sane mailing list
> Ecn-sane@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/ecn-sane
>
[-- Attachment #2: Type: text/html, Size: 15145 bytes --]
next prev parent reply other threads:[~2019-11-12 13:55 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-11-12 12:07 Rich Brown
2019-11-12 12:20 ` Toke Høiland-Jørgensen
2019-11-12 12:25 ` Mikael Abrahamsson
2019-11-12 13:02 ` Toke Høiland-Jørgensen
2019-11-12 13:54 ` Luca Muscariello [this message]
2019-11-12 14:35 ` Toke Høiland-Jørgensen
2019-11-12 22:01 ` Toke Høiland-Jørgensen
2019-11-13 0:04 ` Rodney W. Grimes
2019-11-13 8:05 ` Luca Muscariello
2019-11-13 10:45 ` Toke Høiland-Jørgensen
2019-11-13 15:36 ` Rodney W. Grimes
2019-11-13 10:43 ` Toke Høiland-Jørgensen
2019-11-13 15:25 ` Rodney W. Grimes
2019-11-13 15:35 ` Toke Høiland-Jørgensen
2019-11-13 15:36 ` Luca Muscariello
2019-11-13 15:42 ` Rodney W. Grimes
2019-11-13 15:52 ` Toke Høiland-Jørgensen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://lists.bufferbloat.net/postorius/lists/ecn-sane.lists.bufferbloat.net/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAH8sseTU+p7tv3iLzs8Ddmnzwv0GT__1ehmDqKDkME0pf9hAtQ@mail.gmail.com \
--to=muscariello@ieee.org \
--cc=ecn-sane@lists.bufferbloat.net \
--cc=richb.hanover@gmail.com \
--cc=swmike@swm.pp.se \
--cc=toke@toke.dk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox