[Ecn-sane] cautionary tcp tale

[Ecn-sane] cautionary tcp tale

[Dave Taht]

https://www.snellman.net/blog/archive/2017-07-20-s3-mystery/


-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Ecn-sane] cautionary tcp tale

[Jonathan Morton]

> On 13 Aug, 2019, at 12:30 am, Dave Taht <dave.taht@gmail.com> wrote:
> 
> https://www.snellman.net/blog/archive/2017-07-20-s3-mystery/

TL;DR summary:

Buggy checksum recalculation in a cable modem caused minor packet loss (through rejection at the receiver).  In the absence of TCP Timestamps, the retransmissions of these lost packets were identical and triggered the same bug.  Result, connections to certain particular servers which had the unusual property of disabling TCP Timestamps would quickly stall.

I think a tool could be made to watch a sample of received traffic for this pattern: incorrect checksums where the correct checksum is the same each time (though different per deployment).  How much network equipment exhibits this bug?

 - Jonathan Morton

Re: [Ecn-sane] cautionary tcp tale

[David P. Reed]

This is the stupidity of NAT, which encouraged modifying TCP checksums, which are supposed to be not modified in the network at all.

The whole point of an "end-to-end" checksum is that the middle shouldn't touch it!

Now I get why NAT became popular, though at the time many of us pretty much vomited all over the idea as completely bogus.

I do worry that now there is TCP checksum offloading in NICs, that middleboxes are feeling freer just to throw away and recalculate checksums as they pass through, and not bother to forward non-TCP, nonUDP, nonICMP IP packets at all.

This would be the kind of thing that Cisco, for example, might just do, since they have long thought that they owned the Internet design as a corporate entity.


On Monday, August 12, 2019 7:37pm, "Jonathan Morton" <chromatix99@gmail.com> said:

>> On 13 Aug, 2019, at 12:30 am, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> https://www.snellman.net/blog/archive/2017-07-20-s3-mystery/
> 
> TL;DR summary:
> 
> Buggy checksum recalculation in a cable modem caused minor packet loss (through
> rejection at the receiver).  In the absence of TCP Timestamps, the retransmissions
> of these lost packets were identical and triggered the same bug.  Result,
> connections to certain particular servers which had the unusual property of
> disabling TCP Timestamps would quickly stall.
> 
> I think a tool could be made to watch a sample of received traffic for this
> pattern: incorrect checksums where the correct checksum is the same each time
> (though different per deployment).  How much network equipment exhibits this bug?
> 
>  - Jonathan Morton
> _______________________________________________
> Ecn-sane mailing list
> Ecn-sane@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/ecn-sane
> 

Re: [Ecn-sane] cautionary tcp tale

[Rodney W. Grimes]

> This is the stupidity of NAT, which encouraged modifying TCP checksums, which are supposed to be not modified in the network at all.

Yes, I never liked that this had to be done.

> 
> The whole point of an "end-to-end" checksum is that the middle shouldn't touch it!

:-)

> Now I get why NAT became popular, though at the time many of us pretty much vomited all over the idea as completely bogus.

Still vomiting for some of us that have to deal with it on a continuous basis.

> I do worry that now there is TCP checksum offloading in NICs, that middleboxes are feeling freer just to throw away and recalculate checksums as they pass through, and not bother to forward non-TCP, nonUDP, nonICMP IP packets at all.

Scarry thought, do these nic cards have ECC on the buffers?   Is the datapath through the chip protected by ECC?

> This would be the kind of thing that Cisco, for example, might just do, since they have long thought that they owned the Internet design as a corporate entity.

Well thankful a few competitors have shown up to level the game, a bit anyway.

> On Monday, August 12, 2019 7:37pm, "Jonathan Morton" <chromatix99@gmail.com> said:
> 
> >> On 13 Aug, 2019, at 12:30 am, Dave Taht <dave.taht@gmail.com> wrote:
> >>
> >> https://www.snellman.net/blog/archive/2017-07-20-s3-mystery/
> > 
> > TL;DR summary:
> > 
> > Buggy checksum recalculation in a cable modem caused minor packet loss (through
> > rejection at the receiver).  In the absence of TCP Timestamps, the retransmissions
> > of these lost packets were identical and triggered the same bug.  Result,
> > connections to certain particular servers which had the unusual property of
> > disabling TCP Timestamps would quickly stall.
> > 
> > I think a tool could be made to watch a sample of received traffic for this
> > pattern: incorrect checksums where the correct checksum is the same each time
> > (though different per deployment).  How much network equipment exhibits this bug?
> > 
> >  - Jonathan Morton
> > _______________________________________________
> > Ecn-sane mailing list
> > Ecn-sane@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/ecn-sane
> > 
> 
> 
> _______________________________________________
> Ecn-sane mailing list
> Ecn-sane@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/ecn-sane
> 
> 

-- 
Rod Grimes                                                 rgrimes@freebsd.org

Re: [Ecn-sane] cautionary tcp tale

[Jonathan Morton]

> On 13 Aug, 2019, at 4:01 am, Rodney W. Grimes <4bone@gndrsh.dnsmgr.net> wrote:
> 
> Still vomiting for some of us that have to deal with it on a continuous basis.

I personally have to deal with triple or possibly quadruple NAT (depending on what dark voodoo my ISP does) on a continuous basis.  Two layers of it are inside a single physical box, and neither of them can be turned off.  Welcome to the wonderful world of LTE.

 - Jonathan Morton

Re: [Ecn-sane] cautionary tcp tale

[Michael Richardson]

Thanks.
Also a good story as to why middle boxes should stay away from mangling
packets without an audit trail.

Re: [Ecn-sane] cautionary tcp tale

[David P. Reed]

I'm of a mind to suggest an RFC specifically reiterating that non-endpoints MUST never modify the content part of any IP datagrams, ever, with the exception of the TCP and UDP extended routing header (the ports), and that solely for the purpose of implementing NAT as defined in the NAT standard.

I think vint Cerf and I would be happy to be co-authors, maybe along with Dave Clark, Noel Chiappa, and a crew of original Internet Protocol designers.

I had thought this was a well-understood invariant, core to the design of the entire Internet.

Part of the reason, but certainly not all of it, was that we all intended that the content within the IP datagram contents would be treated as sacrosanct, as if encrypted by a key unknown to the network.

We could not require end-to-end encryption because of ITAR rules at the time. But it is absolutely clear that NOTHING in the network transport system was expected to attempt to understand or to modify those bits until they reached the destination, unchanged.

It wasn't just a "good idea", it was a design requirement.

On Tuesday, August 13, 2019 12:39pm, "Michael Richardson" <mcr@sandelman.ca> said:

> Thanks.
> Also a good story as to why middle boxes should stay away from mangling
> packets without an audit trail.
> 
> 
> 
> _______________________________________________
> Ecn-sane mailing list
> Ecn-sane@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/ecn-sane
> 

Re: [Ecn-sane] cautionary tcp tale

[Rodney W. Grimes]

> I'm of a mind to suggest an RFC specifically reiterating that non-endpoints MUST never modify the content part of any IP datagrams, ever, with the exception of the TCP and UDP extended routing header (the ports), and that solely for the purpose of implementing NAT as defined in the NAT standard.

Now I think you really mean to say "modify the content part of any IP datagram PAYLOAD, ever" I am in agreement, the IP header itself is going to get modified a lot.

> 
> I think vint Cerf and I would be happy to be co-authors, maybe along with Dave Clark, Noel Chiappa, and a crew of original Internet Protocol designers.
> 
> I had thought this was a well-understood invariant, core to the design of the entire Internet.

People forget history, reasons, etc, I am not even sure that it is well documented that IP payload should not be modified, though it may be well known information in some cicles, I would say that circle is of diminishing size.

> 
> Part of the reason, but certainly not all of it, was that we all intended that the content within the IP datagram contents would be treated as sacrosanct, as if encrypted by a key unknown to the network.

Isnt it interesting that they are actually proposing that now to protect the IP payload from the malicious crap that is going on, your proposal would make a rule, the encryption solution would silently enforce that rule without question.

> We could not require end-to-end encryption because of ITAR rules at the time. But it is absolutely clear that NOTHING in the network transport system was expected to attempt to understand or to modify those bits until they reached the destination, unchanged.
> 
> It wasn't just a "good idea", it was a design requirement.

Perhaps a poorly documented one?  Can you site any RFC verbage that addresses this?

I would support any effort to codify this in a I-D.

> On Tuesday, August 13, 2019 12:39pm, "Michael Richardson" <mcr@sandelman.ca> said:
> 
> > Thanks.
> > Also a good story as to why middle boxes should stay away from mangling
> > packets without an audit trail.
> > 
> 

-- 
Rod Grimes                                                 rgrimes@freebsd.org

Re: [Ecn-sane] cautionary tcp tale

[David P. Reed]

I'm pretty sure it was stated in Cerf and Kahn's paper "A Transmission Control Protocol" as published in IEEE Proceedings. I know it was in the Transmission Control Protocol Working Group email and paper documents, though I don't have a personal copy. Is it in an RFC? Probably. It's important to remember that RFC's were literally Requests for Comment in the 1970's. They weren't the entire record.

But this kind of gradual rot does creep into systems design communities. THat's why I'm wondering whether the rationale and decisions ought to be restated.

On Tuesday, August 13, 2019 5:21pm, "Rodney W. Grimes" <4bone@gndrsh.dnsmgr.net> said:

>> I'm of a mind to suggest an RFC specifically reiterating that non-endpoints MUST
>> never modify the content part of any IP datagrams, ever, with the exception of
>> the TCP and UDP extended routing header (the ports), and that solely for the
>> purpose of implementing NAT as defined in the NAT standard.
> 
> Now I think you really mean to say "modify the content part of any IP datagram
> PAYLOAD, ever" I am in agreement, the IP header itself is going to get modified a
> lot.
> 
>>
>> I think vint Cerf and I would be happy to be co-authors, maybe along with Dave
>> Clark, Noel Chiappa, and a crew of original Internet Protocol designers.
>>
>> I had thought this was a well-understood invariant, core to the design of the
>> entire Internet.
> 
> People forget history, reasons, etc, I am not even sure that it is well documented
> that IP payload should not be modified, though it may be well known information in
> some cicles, I would say that circle is of diminishing size.
> 
>>
>> Part of the reason, but certainly not all of it, was that we all intended that
>> the content within the IP datagram contents would be treated as sacrosanct, as if
>> encrypted by a key unknown to the network.
> 
> Isnt it interesting that they are actually proposing that now to protect the IP
> payload from the malicious crap that is going on, your proposal would make a rule,
> the encryption solution would silently enforce that rule without question.
> 
>> We could not require end-to-end encryption because of ITAR rules at the time. But
>> it is absolutely clear that NOTHING in the network transport system was expected
>> to attempt to understand or to modify those bits until they reached the
>> destination, unchanged.
>>
>> It wasn't just a "good idea", it was a design requirement.
> 
> Perhaps a poorly documented one?  Can you site any RFC verbage that addresses
> this?
> 
> I would support any effort to codify this in a I-D.
> 
>> On Tuesday, August 13, 2019 12:39pm, "Michael Richardson" <mcr@sandelman.ca>
>> said:
>>
>> > Thanks.
>> > Also a good story as to why middle boxes should stay away from mangling
>> > packets without an audit trail.
>> >
>>
> 
> --
> Rod Grimes                                                 rgrimes@freebsd.org
> 

Re: [Ecn-sane] cautionary tcp tale

[Rodney W. Grimes]

> I'm pretty sure it was stated in Cerf and Kahn's paper "A Transmission Control Protocol" as published in IEEE Proceedings. I know it was in the Transmission Control Protocol Working Group email and paper documents, though I don't have a personal copy. Is it in an RFC? Probably. It's important to remember that RFC's were literally Requests for Comment in the 1970's. They weren't the entire record.
> 
> But this kind of gradual rot does creep into systems design communities. THat's why I'm wondering whether the rationale and decisions ought to be restated.

Without clear and concise documentation on the "engineering process" that arrvies at conclusion there is little to pass on to the next generation such that this "folke lore" does not become forgotten.

It is probably also mentioned in Stevens and Comer, which I am glad to see are still top hit search results.

But I doubt many new network developers read such books any more, if anything a good oral histroy should be recorded before we loose those that know the actual history and reasoning in that decision process, not that I am saying it is the necassarily still the best decisions, but it is important to know so that we do not nievly undo sound engineering.

Gorry Fairhurst has started a wonderful document on ECN:  Guidelines for Internet Congestion Control at Endpoints, https://tools.ietf.org/html/draft-fairhurst-tsvwg-cc.  The SCE developers are in process of preparing a review of it, which Pete Heist has done a wonderful job so far with.  This document collects in one place almost all of the references in RFC's to CC's.

A similiar I-D could be started about fundemental IP and transport layer paradigms, and incase your not aware rfc793bis is in process and could use more eyes, imho.  https://tools.ietf.org/html/draft-ietf-tcpm-rfc793bis

> On Tuesday, August 13, 2019 5:21pm, "Rodney W. Grimes" <4bone@gndrsh.dnsmgr.net> said:
> 
> >> I'm of a mind to suggest an RFC specifically reiterating that non-endpoints MUST
> >> never modify the content part of any IP datagrams, ever, with the exception of
> >> the TCP and UDP extended routing header (the ports), and that solely for the
> >> purpose of implementing NAT as defined in the NAT standard.
> > 
> > Now I think you really mean to say "modify the content part of any IP datagram
> > PAYLOAD, ever" I am in agreement, the IP header itself is going to get modified a
> > lot.
> > 
> >>
> >> I think vint Cerf and I would be happy to be co-authors, maybe along with Dave
> >> Clark, Noel Chiappa, and a crew of original Internet Protocol designers.
> >>
> >> I had thought this was a well-understood invariant, core to the design of the
> >> entire Internet.
> > 
> > People forget history, reasons, etc, I am not even sure that it is well documented
> > that IP payload should not be modified, though it may be well known information in
> > some cicles, I would say that circle is of diminishing size.
> > 
> >>
> >> Part of the reason, but certainly not all of it, was that we all intended that
> >> the content within the IP datagram contents would be treated as sacrosanct, as if
> >> encrypted by a key unknown to the network.
> > 
> > Isnt it interesting that they are actually proposing that now to protect the IP
> > payload from the malicious crap that is going on, your proposal would make a rule,
> > the encryption solution would silently enforce that rule without question.
> > 
> >> We could not require end-to-end encryption because of ITAR rules at the time. But
> >> it is absolutely clear that NOTHING in the network transport system was expected
> >> to attempt to understand or to modify those bits until they reached the
> >> destination, unchanged.
> >>
> >> It wasn't just a "good idea", it was a design requirement.
> > 
> > Perhaps a poorly documented one?  Can you site any RFC verbage that addresses
> > this?
> > 
> > I would support any effort to codify this in a I-D.
> > 
> >> On Tuesday, August 13, 2019 12:39pm, "Michael Richardson" <mcr@sandelman.ca>
> >> said:
> >>
> >> > Thanks.
> >> > Also a good story as to why middle boxes should stay away from mangling
> >> > packets without an audit trail.
> > --
> > Rod Grimes                                                 rgrimes@freebsd.org
-- 
Rod Grimes                                                 rgrimes@freebsd.org