[Bloat] mosh, ecn, and diffserv marking

[Bloat] mosh, ecn, and diffserv marking

[Dave Taht]

mosh list: sorry for broadening this discussion to my own list.

bloat list:

This is a discussion of the innovative new partial ssh replacement, mosh,
which is documented here:

http://mosh.mit.edu/

This particular protocol has made my (often crashy) wifi lab,
intercontinental connections, and life so much more manageable and
productive of late that I encourage as many folk as possible to play
with it, and port it to various pieces of equipment...

and we're playing with diffserv/ecn in this new context...

On Wed, Jun 13, 2012 at 3:27 PM, Keith Winstein <keithw@mit.edu> wrote:
> Hi Dave,
>
> We have been testing it (mosh set to AF42 + ECT) at MIT for a few weeks with no reported difficulty
> getting packets through. (I didn't put this in the 1.2.2 release though
> because we haven't really tested it enough, and I'm skitting about sending
> ECT when we just ignore ECN -- it seems like cheating.)
>
> I have found:
>
> (a) OS X refuses to let users set the DiffServ codepoint & ECT and returns
> an error. I think OS X may only permit the original IP TOS types. I haven't
> tested whether the DiffServ codepoint or ECT works separately.

I have asked some other mac folk I know to look into this.

> It works fine
> on Linux and I assume FreeBSD (haven't heard any reports to the contrary).

My principal concern with enabling ECN in mosh was sites that would
filter out udp packets marked such entirely, so that they would not
reach their destination. Thus far, in my limited testing (and yours),
that seems not to be the case.

If it were, I would recomend a backoff step for mosh's packets -
reverting to not having ECN marking after a lossy period, and
attempting to renable after recovery. This might still be a good
idea...

I have found multiple firewalls that don't allow mosh packets back at
all, but that has nothing to do with diffserv or ECN marking.
>
> (b) MIT's routers seem to strip the DiffServ codepoint after one hop.

The whole thing? Yeesh.

> ===
>
> Separate question: If we're setting ECT, what do you recommend we _do_ when
> we get ECN?

> My best idea is that we just alter the timestamp_reply we send
> back, which would be a clean way of getting the other side to gradually slow
> down its transmissions (because it would affect the SRTT, which governs our
> "frame rate").

Sounds promising to me! I would hope to get some of the more radical
thinkers on my list to make suggestions, also...

>
> But I'm a little concerned about the security implications here -- I don't
> want a bad guy to be able to sniff one packet from us and then replay it a
> million times with ECN set to cause a DoS.

The current implementations of codel and fq_codel have this flaw in
the general case (although ecn defaults to off in codel and on in
fq_codel), any misbehaving stream with ECT(2) set will bypass the
default fq_codel drop strategy by merely setting ECT(3), so it's of
much larger concern generally.

(I do have to note that it would take many ECN marked streams to
actually cause a problem with fq_codel, as it will balance all streams
as best as possible)

as it is very early days of codel deployment I feel we have time to
address this issue with various strategies. The simplest would be to
drop ecn marked packets when the codel target is being exceeded by
(say) 2x.

>So my thinking is we would only
> respond to ECN if it's a _new_ packet (with sequence number greater than any
> seen before). We use the same filter to decide whether to update our timing
> statistics so it seems somewhat natural.

Sounds good to me. Because your frame rate is so low in the first place
I have actually only seen codel mark a mosh packet twice in 10s of GB
of bg transfers!

However for future mosh-like applications with higher transfer rates a
satisfactory answer needs to be derived.

One possibility is to actually back off even more than what a packet
drop would normally do. This is an option I'm exploring with uTP as I
write.


>
> Have you seen anybody discuss the appropriate use of ECN in a
> security-sensitive transport protocol? TCP does not seem worried about this
> DoS.

Oh brave new world that has such protocols in it!

Not yet. cc'ing the bloat list, too.

>
> ===
>
> Third question for you: I had to take out the "out-of-order delivery"
> warning because we found that MIT's 802.11n network aggressively reordered
> packets, apparently because of the "block acks" feature in the protocol. In
> a full-throttle TCP download, something like 30-50% of all packets will be
> "out of order"! This seems like it will hurt TCP performance (since TCP will
> send an ACK immediately on any out-of-order packet). I have also heard a few
> reports from people outside MIT.
>
> RFC 3366 ("Advice to link designers on link Automatic Repeat reQuest")
> basically says that if a link layer is going to reorder packets, it should
> not cause a huge number of out-of-order deliveries to IP (and should put
> packets back in order itself if necessary and possible without a large
> delay).
>
> Have you heard anything about this and whether it's common in major 802.11n
> deployments or was discussed by the IETF community? It seems like it may be
> really hurting TCP performance.

I am painfully aware of issues like these in the wireless-n deployment
and should probably talk to them in a separate message series than this,
and perhaps on different lists entirely, too.

See also scary stuff like the infinite retry bugs present in millions
of deployed wireless-n network drivers...

http://www.bufferbloat.net/issues/390
http://www.bufferbloat.net/issues/216

A few more major bits of data regarding dense wireless networks also
went by my eyeballs recently, I'll have to find the latest and repost
it
(summary: in dense 802.11 networks over 70% of packets are not actual
data but frame and beacon related)

Wifi has a major, major, major set of interrelated issues that is
going to take a major effort to resolve.

>
> Cheers,
> Keith
>
>
> On Wed, 13 Jun 2012, Dave Taht wrote:
>
>> I am curious if you were able to measure/see
>> any differences in network latency (particularly over wifi) by the
>> switch to AF42 + ECN, or
>> had any difficulties getting packets through?
>>
>> I saw all sorts of interesting behavior on a recent test across the
>> country as to stripping bits (some ended up with ECN set, but nothing
>> else, others ended up with CS1 set, some preserved all the bits), but
>> lacking a means to compare A/B
>>
>> all I can say is that it feels better on saturated best effort wifi
>> networks.
>>
>>
>> On Wed, Jun 13, 2012 at 12:31 PM, Keith Winstein <keithw@mit.edu> wrote:
>>>
>>> Hello Mosh users and developers,
>>>
>>> mosh 1.2.2 has been released.
>>>
>>> The source code is at:
>>> https://github.com/downloads/keithw/mosh/mosh-1.2.2.tar.gz
>>>
>>> This is a minor maintenance release that:
>>>
>>> * Removes the warning on out-of-order packets (this was firing too
>>> often on some Wi-Fi networks)
>>>
>>> * Adds an experimental "--predict=experimental" mode to predict
>>> immediately, even if recent predictions were incorrect
>>>
>>> ===
>>>
>>> mosh 1.2.2 is backwards-compatible with mosh clients back to 0.96 and
>>> mosh servers back to 1.0.9. Please let us know of any problems
>>> (https://github.com/keithw/mosh/issues).
>>>
>>> Best regards from the Mosh team,
>>> Keith
>>> _______________________________________________
>>> mosh-devel mailing list
>>> mosh-devel@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/mosh-devel
>>
>>
>>
>>
>> --
>> Dave Täht
>> SKYPE: davetaht
>> http://ronsravings.blogspot.com/



-- 
Dave Täht
SKYPE: davetaht
http://ronsravings.blogspot.com/

Re: [Bloat] mosh, ecn, and diffserv marking

[Michael Welzl]

Hi,

On 6/14/12 5:02 AM, Dave Taht wrote:
> This is a discussion of the innovative new partial ssh replacement, mosh,
> which is documented here:
>
> http://mosh.mit.edu/
Looks cool to me!


> My principal concern with enabling ECN in mosh was sites that would
> filter out udp packets marked such entirely, so that they would not
> reach their destination. Thus far, in my limited testing (and yours),
> that seems not to be the case.
>
> If it were, I would recomend a backoff step for mosh's packets -
> reverting to not having ECN marking after a lossy period, and
> attempting to renable after recovery. This might still be a good
> idea...

I have this recurring thought that people should find ways to *try* ECN 
and back-off from it after a while instead of disabling it altogether. 
Maybe there is a way to sneak this functionality into TCP somehow... 
anyway, it's not a part of the current spec.

What's interesting here is that this sounds as if you're trying to come 
up with this functionality for a UDP-based application, in user space. I 
like that, I think this is the right approach for ECN *in general*, not 
just for Mosh!


>> But I'm a little concerned about the security implications here -- I don't
>> want a bad guy to be able to sniff one packet from us and then replay it a
>> million times with ECN set to cause a DoS.
(snip)
> Have you seen anybody discuss the appropriate use of ECN in a
> security-sensitive transport protocol? TCP does not seem worried about this
> DoS.
> Oh brave new world that has such protocols in it!
Perhaps what it would help to try to understand *why* this is of no 
concern in TCP?

First of all, TCP doesn't do ECN on retransmits, i.e. it is only applied 
to new packets.
Second, TCP reacts to loss or ECN only once per RTT.

With TCP, the described replay attack would require the use of correct 
sequence numbers, and take effect only once per RTT.

However, I think that the key issue is the following:

IF you're able to construct a million correct looking packets and inject 
it into the TCP stream, ECN is the least of your problems (you could 
just as well inject RST, FIN, you name it). Yes, people have been 
looking into these issues too, long debates in TCPM, see 
http://datatracker.ietf.org/wg/tcpm/ for some documents.

One ECN-specific concern that was addressed is that it's often in the 
interest of the receiver, but not the sender, to lie about ECN and 
simply cheat (reflect "nonono, no congestion at all" back to the 
sender). This is addressed by RFC3540, which is experimental and not 
really used.

Out of that box, take what you like, I'd say!   :-)

I think that it would actually be nice to have an RFC that gives 
recommendations on how ECN should be used by UDP based applications.
Strange that this isn't a part of RFC 5405? Maybe an update of this one 
would be appropriate.

Cheers,
Michael

Re: [Bloat] mosh, ecn, and diffserv marking

[Eric Dumazet]

On Thu, 2012-06-14 at 09:22 +0200, Michael Welzl wrote:

> One ECN-specific concern that was addressed is that it's often in the 
> interest of the receiver, but not the sender, to lie about ECN and 
> simply cheat (reflect "nonono, no congestion at all" back to the 
> sender). This is addressed by RFC3540, which is experimental and not 
> really used.

Yes, apparently :(

Re: [Bloat] mosh, ecn, and diffserv marking

[Michael Welzl]

On 6/14/12 10:54 AM, Eric Dumazet wrote:
> On Thu, 2012-06-14 at 09:22 +0200, Michael Welzl wrote:
>
>> One ECN-specific concern that was addressed is that it's often in the
>> interest of the receiver, but not the sender, to lie about ECN and
>> simply cheat (reflect "nonono, no congestion at all" back to the
>> sender). This is addressed by RFC3540, which is experimental and not
>> really used.
> Yes, apparently :(
>
I tried to restrain myself from self-promoting, but you asked for it   :-)
http://heim.ifi.uio.no/~michawe/research/projects/spurious/index.html

There is code!  - and there are possible additional benefits from the nonce.

I didn't push for that in the IETF because there is a conflict with 
conex (I think? I lost track of all their plans for bits)  for using the 
bit-combination required by the nonce. And conex is a bigger, better 
plan than my "Nonce++" suggestion.

Cheers,
Michael