[Ecn-sane] rfc3168 sec 6.1.2

[Ecn-sane] rfc3168 sec 6.1.2

[Dave Taht]

It would explain a lot if this was not actually implemented in Linux.
I'm afraid to look. cwnd reduction is capped to 2. 1 should put you
in quickack mode AND to go lower seemingly it's supposed to
then rely on the retransmit timer.

...

If the congestion window consists of only one MSS (maximum
   segment size), and the sending TCP receives an ECN-Echo ACK packet,
   then the sending TCP should in principle still reduce its congestion
   window in half. However, the value of the congestion window is
   bounded below by a value of one MSS.  If the sending TCP were to
   continue to send, using a congestion window of 1 MSS, this results in
   the transmission of one packet per round-trip time.  It is necessary
   to still reduce the sending rate of the TCP sender even further, on
   receipt of an ECN-Echo packet when the congestion window is one.

^^^^^^^^^^^^^^^^^^^^^^^^
We
   use the retransmit timer as a means of reducing the rate further in
   this circumstance.  Therefore, the sending TCP MUST reset the

^^^^^^^^^^^^^^^^^^^^^^^^
   retransmit timer on receiving the ECN-Echo packet when the congestion
   window is one.  The sending TCP will then be able to send a new
   packet only when the retransmit timer expires.


-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Ecn-sane] rfc3168 sec 6.1.2

[Jonathan Morton]

> On 29 Aug, 2019, at 5:08 am, Dave Taht <dave.taht@gmail.com> wrote:
> 
> It would explain a lot if this was not actually implemented in Linux.
> I'm afraid to look. cwnd reduction is capped to 2. 1 should put you
> in quickack mode AND to go lower seemingly it's supposed to
> then rely on the retransmit timer.

Nowadays the same effect can be obtained from the pacing timer.  Just set the CA scale factor to 50% to get an effective minimum cwnd of 1, or lower still if needed.

In most of our SCE testing, we're now setting the SS scale factor to 100% (the default is 200%, which means the cwnd is sent over half an RTT and the other half is idle) and the CA scale factor to 40% (default is 120%, so the effective minimum cwnd is actually 2.4 from a packet-pair standpoint).  See the last substantive slide in the IETF-105 SCE deck.

 - Jonathan Morton

Re: [Ecn-sane] rfc3168 sec 6.1.2

[Dave Taht]

On Thu, Aug 29, 2019 at 1:02 AM Jonathan Morton <chromatix99@gmail.com> wrote:
>
> > On 29 Aug, 2019, at 5:08 am, Dave Taht <dave.taht@gmail.com> wrote:
> >
> > It would explain a lot if this was not actually implemented in Linux.
> > I'm afraid to look. cwnd reduction is capped to 2. 1 should put you
> > in quickack mode AND to go lower seemingly it's supposed to
> > then rely on the retransmit timer.
>
> Nowadays the same effect can be obtained from the pacing timer.  Just set the CA scale factor to 50% to get an effective minimum cwnd of 1, or lower still if needed.

But that wouldn't trigger quickacks from the other side unless it's cwnd 1?

> In most of our SCE testing, we're now setting the SS scale factor to 100% (the default is 200%, which means the cwnd is sent over half an RTT and the other half is idle) and the CA scale factor to 40% (default is 120%, so the effective minimum cwnd is actually 2.4 from a packet-pair standpoint).  See the last substantive slide in the IETF-105 SCE deck.
>
>  - Jonathan Morton

I am leveraging hazy memories of old work a years back where I pounded
50 ? 100 ? flows
through a 100Mbit ethernet bottleneck, with a variety of aqm and tcp
ccs. I never got around to writing it up,
but what I observed was along the lines of:

A) fq_codel with drop had MUCH lower RTTs - and would trigger RTOs etc
- and interactive ssh sessions kept  working - which made me happier
than
B) cake (or fq_codel with ecn) hit, I don't remember, 40ms tcp delays.
> double that of drop is the
stat I remember
C) The workload was such that the babel protocol (1000?  routes - 4
packet non-ecn'd udp bursts) would eventually fail - dramatically, by
retracting the route I was on and thus acting as a circuit breaker on
all traffic, so I'd lose connectivit for 16 sec  - and failed much
more often in the latter case
D) The packet caps were capped at cwnd 2 or (4 with BBR) and I didn't
know enough about that until today
E) Head drop really was remarkablly better at keeping all the flows going
F) Pie hit its targets as did codel with drop, but with ecn.... ugh....

And at the time of all this carnage I basically said "ecn scares me
yet again", patched my babel daemons to use it, filed a bug on how we
think about micro flows wrongly here:
here:  https://github.com/tohojo/flent/issues/148 - wrote the ecn-sane
manefesto - and went off to play with my kid and boat.

Anyway, 100 flows, no delays, straight ethernet, and babel with 1000+
routes is easy to setup as a std test,
and I'd love it if y'all could have that in your testbed.

And:

cwnd 1 + pacing might help in this extreme scenarios. This last bit of
how rfc3168 ecn should be better handled was not in my head, I had
assumed til now that new research into subpacket windows was required.

Leveraging the retransmit timer, btw, would lighten the load on the
network a LOT, way back in 2001
when it was first thunk up. sally was a genius
-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Ecn-sane] rfc3168 sec 6.1.2

[Jeremy Harris]

On 29/08/2019 14:51, Dave Taht wrote:
> On Thu, Aug 29, 2019 at 1:02 AM Jonathan Morton <chromatix99@gmail.com> wrote:
>>
>>> On 29 Aug, 2019, at 5:08 am, Dave Taht <dave.taht@gmail.com> wrote:
>>>
>>> It would explain a lot if this was not actually implemented in Linux.
>>> I'm afraid to look. cwnd reduction is capped to 2. 1 should put you
>>> in quickack mode AND to go lower seemingly it's supposed to
>>> then rely on the retransmit timer.
>>
>> Nowadays the same effect can be obtained from the pacing timer.  Just set the CA scale factor to 50% to get an effective minimum cwnd of 1, or lower still if needed.
> 
> But that wouldn't trigger quickacks from the other side unless it's cwnd 1?

I've also seen suggestion of reducing the effective MSS for the
duration of really-low-cwnd.  Any comments on that?

-- 
Cheers,
  Jeremy

Re: [Ecn-sane] rfc3168 sec 6.1.2

[Jonathan Morton]

> On 29 Aug, 2019, at 4:51 pm, Dave Taht <dave.taht@gmail.com> wrote:
> 
> I am leveraging hazy memories of old work a years back where I pounded 50 ? 100 ? flows through a 100Mbit ethernet

At 100 flows, that gives you 1Mbps per flow fair share, so 80pps or 12.5ms between packets on each flow, assuming they're all saturating.  This also means you have a minimum sojourn time (for saturating flows) of 12.5ms, which is well above the Codel target, so Codel will always be in dropping-state and will continuously ramp up its signalling frequency (unless some mitigation is in place for this very situation, which there is in Cake).

Both Cake and fq_codel should still be able to prioritise sparse flows to sub-millisecond delays under these conditions.  They'll be pretty strict about what counts as "sparse" though.  Your individual keystrokes and echoes should get through quickly, but output from programs may end up waiting.

> A) fq_codel with drop had MUCH lower RTTs - and would trigger RTOs etc

RTOs are bad.  They indicate that the steady flow of traffic has broken down on that flow due to tail loss, which is a particular danger at very small cwnds.

Cake tries to avoid them by not dropping the last queued packet from any given flow.  Fq_codel doesn't have that protection, so in non-ECN mode it will drop way too many packets in a desperate (and misguided) attempt to maintain the target sojourn time.

What you need to understand here is that dropped packets increase *application* latency, even if they also reduce the delay to individual packets.  ECN doesn't incur that problem.

> B) cake (or fq_codel with ecn) hit, I don't remember, 40ms tcp delays.

A delay of 40ms suggests about 3 packets per flow are in the queue.  That's pretty close to the minimum cwnd of 2.  One would like to do better than that, of course, but options for doing so become limited.

I would expect SCE to do better at staying *at* the minimum cwnd in these conditions.  That by itself would reduce your delay to 25ms.  Combined with setting the CA pacing scale factor to 40%, that would also reduce the average packets per flow in the queue to 0.8.  I think that's independent of whether the receiver still acks only every other segment.  The delay on each flow would probably go down to about 10ms on average, but I'm not going to claim anything about the variance around that value.

Since 10ms is still well above the normal Codel target, SCE will be signalling 100% to these flows, and thus preventing them from increasing the cwnd from 2.

> C) The workload was such that the babel protocol (1000?  routes - 4
> packet non-ecn'd udp bursts) would eventually fail - dramatically, by
> retracting the route I was on and thus acting as a circuit breaker on
> all traffic, so I'd lose connectivit for 16 sec

That's a problem with Babel, not with ECN.  A robust routing protocol should not drop the last working route to any node, just because the link gets congested.  It *may* consider that link as non-preferred and seek alternative routes that are less congested, but it *must* keep the route open (if it is working at all) until such an alternative is found.

But you did find that turning on ECN for the routing protocol helped.  So the problem wasn't latency per se, but packet loss from the AQM over-reacting to that latency.

> Anyway, 100 flows, no delays, straight ethernet, and babel with 1000+ routes is easy to setup as a std test, and I'd love it if y'all could have that in your testbed.

Let's put it on the todo list.  Do you have a working script we can just use?

 - Jonathan Morton

Re: [Ecn-sane] rfc3168 sec 6.1.2

[Dave Taht]

On Thu, Aug 29, 2019 at 7:42 AM Jonathan Morton <chromatix99@gmail.com> wrote:
>
> > On 29 Aug, 2019, at 4:51 pm, Dave Taht <dave.taht@gmail.com> wrote:
> >
> > I am leveraging hazy memories of old work a years back where I pounded 50 ? 100 ? flows through a 100Mbit ethernet
>
> At 100 flows, that gives you 1Mbps per flow fair share, so 80pps or 12.5ms between packets on each flow, assuming they're all saturating.  This also means you have a minimum sojourn time (for saturating flows) of 12.5ms, which is well above the Codel target, so Codel will always be in dropping-state and will continuously ramp up its signalling frequency (unless some mitigation is in place for this very situation, which there is in Cake).
>
> Both Cake and fq_codel should still be able to prioritise sparse flows to sub-millisecond delays under these conditions.  They'll be pretty strict about what counts as "sparse" though.  Your individual keystrokes and echoes should get through quickly, but output from programs may end up waiting.
>
> > A) fq_codel with drop had MUCH lower RTTs - and would trigger RTOs etc
>
> RTOs are bad.  They indicate that the steady flow of traffic has broken down on that flow due to tail loss, which is a particular danger at very small cwnds.

They indicated that traffic has broken down for any of a zillion
reasons. RTO's for example, are what
gets tcp restarted after babel does the circuit breaker thing on this
test and restores it.

RTOs are Good. :)

> Cake tries to avoid them by not dropping the last queued packet from any given flow.  Fq_codel doesn't have that protection, so in non-ECN mode it will drop way too many packets in a desperate (and misguided) attempt to maintain the target sojourn time.

We are trying to encourage others to stop editorizing so much. As the
author of this behavior in fq_codel,
my reasoning at the time was that under conditions of overload that
there were usually packets "in the network", and keeping the last
packet in the queue scaled badly in terms of total RTT. Saying "go
away, come back later" was a totally reasonable response, baked into
TCPs since the very beginning.

I'm glad that cake and fq_codel have a different response curve here.
It's interesting. Catagorizing the
differences between approaches is good.

As best as I can recall I put this behavior into fq_codel after some
very similar testing back in 2012.


> What you need to understand here is that dropped packets increase *application* latency, even if they also reduce the delay to individual packets.  ECN doesn't incur that problem.

Well, let me point at my data here:
http://blog.cerowrt.org/post/ecn_fq_codel_wifi_airbook/

We need to be clear about what we consider an "application". I tend to
think about things more
as "human facing" or not, and optimize for humans first.

In this case dropped packets on a 2 second flow account for a maximum
of 16ms increase for FCT. Inperceptable. Compared to making room for
other packets from other flows at the point of contention
is a win for those other flows.

In particular (and perhaps we can show this with a heavy load test)
having shorter RTTs from drop makes it
faster for a new or existing flows to grab back bandwidth when part of
that load exits.

I've long bought the argument for human interactive flows that need a
reliable transport - that ecn is good - as we did in mosh. But (being
chicken) on doing it to everything, not so much.

Anyway, the cwnd 1 + retransmit (or pacing!) idea would hopefully
reduce the ecn'd RTTs to something
more comparable to the drop in this particular test, which would be a
step forward.

I'll get to your other points below, later.

> > B) cake (or fq_codel with ecn) hit, I don't remember, 40ms tcp delays.
>
> A delay of 40ms suggests about 3 packets per flow are in the queue.  That's pretty close to the minimum cwnd of 2.  One would like to do better than that, of course, but options for doing so become limited.
>
> I would expect SCE to do better at staying *at* the minimum cwnd in these conditions.  That by itself would reduce your delay to 25ms.  Combined with setting the CA pacing scale factor to 40%, that would also reduce the average packets per flow in the queue to 0.8.  I think that's independent of whether the receiver still acks only every other segment.  The delay on each flow would probably go down to about 10ms on average, but I'm not going to claim anything about the variance around that value.
>
> Since 10ms is still well above the normal Codel target, SCE will be signalling 100% to these flows, and thus preventing them from increasing the cwnd from 2.
>
> > C) The workload was such that the babel protocol (1000?  routes - 4
> > packet non-ecn'd udp bursts) would eventually fail - dramatically, by
> > retracting the route I was on and thus acting as a circuit breaker on
> > all traffic, so I'd lose connectivit for 16 sec
>
> That's a problem with Babel, not with ECN.  A robust routing protocol should not drop the last working route to any node, just because the link gets congested.  It *may* consider that link as non-preferred and seek alternative routes that are less congested, but it *must* keep the route open (if it is working at all) until such an alternative is found.
>
> But you did find that turning on ECN for the routing protocol helped.  So the problem wasn't latency per se, but packet loss from the AQM over-reacting to that latency.
>
> > Anyway, 100 flows, no delays, straight ethernet, and babel with 1000+ routes is easy to setup as a std test, and I'd love it if y'all could have that in your testbed.
>
> Let's put it on the todo list.  Do you have a working script we can just use?
>
>  - Jonathan Morton



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Ecn-sane] rfc3168 sec 6.1.2

[Dave Taht]

On Thu, Aug 29, 2019 at 12:10 PM Dave Taht <dave.taht@gmail.com> wrote:
>
> On Thu, Aug 29, 2019 at 7:42 AM Jonathan Morton <chromatix99@gmail.com> wrote:
> >
> > > On 29 Aug, 2019, at 4:51 pm, Dave Taht <dave.taht@gmail.com> wrote:
> > >
> > > I am leveraging hazy memories of old work a years back where I pounded 50 ? 100 ? flows through a 100Mbit ethernet
> >
> > At 100 flows, that gives you 1Mbps per flow fair share, so 80pps or 12.5ms between packets on each flow, assuming they're all saturating.  This also means you have a minimum sojourn time (for saturating flows) of 12.5ms, which is well above the Codel target, so Codel will always be in dropping-state and will continuously ramp up its signalling frequency (unless some mitigation is in place for this very situation, which there is in Cake).
> >
> > Both Cake and fq_codel should still be able to prioritise sparse flows to sub-millisecond delays under these conditions.  They'll be pretty strict about what counts as "sparse" though.  Your individual keystrokes and echoes should get through quickly, but output from programs may end up waiting.
> >
> > > A) fq_codel with drop had MUCH lower RTTs - and would trigger RTOs etc
> >
> > RTOs are bad.  They indicate that the steady flow of traffic has broken down on that flow due to tail loss, which is a particular danger at very small cwnds.
>
> They indicated that traffic has broken down for any of a zillion
> reasons. RTO's for example, are what
> gets tcp restarted after babel does the circuit breaker thing on this
> test and restores it.
>
> RTOs are Good. :)
>
> > Cake tries to avoid them by not dropping the last queued packet from any given flow.  Fq_codel doesn't have that protection, so in non-ECN mode it will drop way too many packets in a desperate (and misguided) attempt to maintain the target sojourn time.
>
> We are trying to encourage others to stop editorizing so much. As the
> author of this behavior in fq_codel,
> my reasoning at the time was that under conditions of overload that
> there were usually packets "in the network", and keeping the last
> packet in the queue scaled badly in terms of total RTT. Saying "go
> away, come back later" was a totally reasonable response, baked into
> TCPs since the very beginning.
>
> I'm glad that cake and fq_codel have a different response curve here.
> It's interesting. Catagorizing the
> differences between approaches is good.
>
> As best as I can recall I put this behavior into fq_codel after some
> very similar testing back in 2012.
>
>
> > What you need to understand here is that dropped packets increase *application* latency, even if they also reduce the delay to individual packets.  ECN doesn't incur that problem.
>
> Well, let me point at my data here:
> http://blog.cerowrt.org/post/ecn_fq_codel_wifi_airbook/
>
> We need to be clear about what we consider an "application". I tend to
> think about things more
> as "human facing" or not, and optimize for humans first.
>
> In this case dropped packets on a 2 second flow account for a maximum
> of 16ms increase for FCT. Inperceptable. Compared to making room for
> other packets from other flows at the point of contention
> is a win for those other flows.

And to wax philosophical (I'm trying really hard to keep my limbic system
out of things this time around!), in stuart's other example of a screen sharing
application, ecn is useful!

And he made use of tcp_notsent_lowat to skip a frame when congestion
indicators told the application to do so. very compelling example.

I think that starting to build charts of our different outlooks under
variing circumstances would help.

Me, I'm all about the latency, willing to do almost anything to hold overall
latencies to a minimum. I'd assert: on reliable transports you recover
from a short rtt loss
faster than if you get a swollen RTT and CE within that rtt (I DO note
that both sce and l4s change this equation!!!!) - and try to show that
-

and I'm generally willing to accept lots of loss on voice and gaming
traffic in exchange for low jitter.

If we can improve the tcps or quic in any way - drop, loss, ecn, sce,
improving rto behavior, reducing mss, cc behavior, even adding tachyon
support  - GREAT.

Anyway with more stuff in comparison tables, and maybe we could also
channel sally floyd and the l4s folk, for each remarkable
circumstance.

ok, I really gotta go


One of my puzzlements in life is that I really love that option and I imagine
it's not used as much as it could be.
> In particular (and perhaps we can show this with a heavy load test)
> having shorter RTTs from drop makes it
> faster for a new or existing flows to grab back bandwidth when part of
> that load exits.

> I've long bought the argument for human interactive flows that need a
> reliable transport - that ecn is good - as we did in mosh. But (being
> chicken) on doing it to everything, not so much.
>
> Anyway, the cwnd 1 + retransmit (or pacing!) idea would hopefully
> reduce the ecn'd RTTs to something
> more comparable to the drop in this particular test, which would be a
> step forward.
>
> I'll get to your other points below, later.
>
> > > B) cake (or fq_codel with ecn) hit, I don't remember, 40ms tcp delays.
> >
> > A delay of 40ms suggests about 3 packets per flow are in the queue.  That's pretty close to the minimum cwnd of 2.  One would like to do better than that, of course, but options for doing so become limited.
> >
> > I would expect SCE to do better at staying *at* the minimum cwnd in these conditions.  That by itself would reduce your delay to 25ms.  Combined with setting the CA pacing scale factor to 40%, that would also reduce the average packets per flow in the queue to 0.8.  I think that's independent of whether the receiver still acks only every other segment.  The delay on each flow would probably go down to about 10ms on average, but I'm not going to claim anything about the variance around that value.
> >
> > Since 10ms is still well above the normal Codel target, SCE will be signalling 100% to these flows, and thus preventing them from increasing the cwnd from 2.
> >
> > > C) The workload was such that the babel protocol (1000?  routes - 4
> > > packet non-ecn'd udp bursts) would eventually fail - dramatically, by
> > > retracting the route I was on and thus acting as a circuit breaker on
> > > all traffic, so I'd lose connectivit for 16 sec
> >
> > That's a problem with Babel, not with ECN.  A robust routing protocol should not drop the last working route to any node, just because the link gets congested.  It *may* consider that link as non-preferred and seek alternative routes that are less congested, but it *must* keep the route open (if it is working at all) until such an alternative is found.
> >
> > But you did find that turning on ECN for the routing protocol helped.  So the problem wasn't latency per se, but packet loss from the AQM over-reacting to that latency.
> >
> > > Anyway, 100 flows, no delays, straight ethernet, and babel with 1000+ routes is easy to setup as a std test, and I'd love it if y'all could have that in your testbed.
> >
> > Let's put it on the todo list.  Do you have a working script we can just use?
> >
> >  - Jonathan Morton
>
>
>
> --
>
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740