[Codel] [RFC PATCH] codel: ecn mark at target

[Codel] [RFC PATCH] codel: ecn mark at target

[Dave Täht]

From: Dave Taht <dave.taht@bufferbloat.net>

The consensus at ietf was that ecn marking should start at
target, and then the results fed into the codel drop scheduler.

While I agree with the latter, I feel that waiting an interval
before starting to mark will be more in-tune with the concept
of a sojourn time, and lead to better utilization.

As I am outnumbered and outgunned, do it at target.
---
 include/net/codel.h |   30 ++++++++++--------------------
 1 file changed, 10 insertions(+), 20 deletions(-)

diff --git a/include/net/codel.h b/include/net/codel.h
index 550debf..f06d99c 100644
--- a/include/net/codel.h
+++ b/include/net/codel.h
@@ -204,7 +204,7 @@ static codel_time_t codel_control_law(codel_time_t t,
 }
 
 
-static bool codel_should_drop(const struct sk_buff *skb,
+static bool codel_should_drop(struct sk_buff *skb,
 			      struct Qdisc *sch,
 			      struct codel_vars *vars,
 			      struct codel_params *params,
@@ -231,6 +231,9 @@ static bool codel_should_drop(const struct sk_buff *skb,
 		return false;
 	}
 	ok_to_drop = false;
+	/* Above target, start marking */
+	if (params->ecn && INET_ECN_set_ce(skb))
+		stats->ecn_mark++; 
 	if (vars->first_above_time == 0) {
 		/* just went above from below. If we stay above
 		 * for at least interval we'll say it's ok to drop
@@ -238,6 +241,7 @@ static bool codel_should_drop(const struct sk_buff *skb,
 		vars->first_above_time = now + params->interval;
 	} else if (codel_time_after(now, vars->first_above_time)) {
 		ok_to_drop = true;
+	/* dtaht: I think we should start marking here, instead */
 	}
 	return ok_to_drop;
 }
@@ -280,14 +284,6 @@ static struct sk_buff *codel_dequeue(struct Qdisc *sch,
 						* since there is no more divide
 						*/
 				codel_Newton_step(vars);
-				if (params->ecn && INET_ECN_set_ce(skb)) {
-					stats->ecn_mark++;
-					vars->drop_next =
-						codel_control_law(vars->drop_next,
-								  params->interval,
-								  vars->rec_inv_sqrt);
-					goto end;
-				}
 				qdisc_drop(skb, sch);
 				stats->drop_count++;
 				skb = dequeue_func(vars, sch);
@@ -305,16 +301,11 @@ static struct sk_buff *codel_dequeue(struct Qdisc *sch,
 			}
 		}
 	} else if (drop) {
-		if (params->ecn && INET_ECN_set_ce(skb)) {
-			stats->ecn_mark++;
-		} else {
-			qdisc_drop(skb, sch);
-			stats->drop_count++;
-
-			skb = dequeue_func(vars, sch);
-			drop = codel_should_drop(skb, sch, vars, params,
-						 stats, now);
-		}
+		qdisc_drop(skb, sch);
+		stats->drop_count++;
+		skb = dequeue_func(vars, sch);
+		drop = codel_should_drop(skb, sch, vars, params,
+					 stats, now);
 		vars->dropping = true;
 		/* if min went above target close to when we last went below it
 		 * assume that the drop rate that controlled the queue on the
@@ -336,7 +327,6 @@ static struct sk_buff *codel_dequeue(struct Qdisc *sch,
 		vars->drop_next = codel_control_law(now, params->interval,
 						    vars->rec_inv_sqrt);
 	}
-end:
 	return skb;
 }
 #endif
-- 
1.7.9.5

Re: [Codel] [RFC PATCH] codel: ecn mark at target

[Eric Dumazet]

On Fri, 2012-08-03 at 19:44 -0700, Dave Täht wrote:
> From: Dave Taht <dave.taht@bufferbloat.net>
> 
> The consensus at ietf was that ecn marking should start at
> target, and then the results fed into the codel drop scheduler.
> 
> While I agree with the latter, I feel that waiting an interval
> before starting to mark will be more in-tune with the concept
> of a sojourn time, and lead to better utilization.
> 
> As I am outnumbered and outgunned, do it at target.

Well, thats a huge way to favor non ECN flows against ECN flows.

Marking _all_ ECN enabled packets just because last packet sent had a
sojourn time above target is going to throttle ECN flows and let non ECN
flows going full speed and take whole bandwidth.

Doing so is a nice way to keep users switching to ECN one day.

IETF could just say : ECN is doomed, forget about it, dont even try. 

Re: [Codel] [RFC PATCH] codel: ecn mark at target

[Roger Jørgensen]

On Sat, Aug 4, 2012 at 4:44 AM, Dave Täht <dave.taht@bufferbloat.net> wrote:
> From: Dave Taht <dave.taht@bufferbloat.net>
>
> The consensus at ietf was that ecn marking should start at
> target, and then the results fed into the codel drop scheduler.
>
> While I agree with the latter, I feel that waiting an interval
> before starting to mark will be more in-tune with the concept
> of a sojourn time, and lead to better utilization.
>
> As I am outnumbered and outgunned, do it at target.

Well, what do you think is The Best way of doing it? Prove'em wrong if
you really think they are wrong;)



-- 

Roger Jorgensen           | ROJO9-RIPE
rogerj@gmail.com          | - IPv6 is The Key!
http://www.jorgensen.no   | roger@jorgensen.no

Re: [Codel] [RFC PATCH] codel: ecn mark at target

[Richard Scheffenegger]

Hi,

My 0.02 EUR:

mark every packet with a sojourn time above target is OK.

The reaction of the congestion controller to ECN marks need to be more 
differentiated; ECN can deliver much more fine-grained information about the 
current state of the network. If a congestion controller (e.g. legacy 3168 
ECN TCP) chooses to overshoot in it's reaction, thats a problem for that 
particular controller... But even if every packet of a window is marked, a 
legacy TCP will only reduce cwnd once per window, reacting the same as is a 
non-ECN had a single drop in that window...

DCTCP (ECN alpha/beta) does things more fine-grained, and would push legacy 
(non ECN) out of the way, while maintainig full bandwidth and very low 
sojourn time (queue occupancy).

Fixing the reaction of congestion controllers to ECN signals (deliberate 
plural) will be the next step in that area...

Best regards,
   Richard

----- Original Message ----- 
From: "Roger Jørgensen" <rogerj@gmail.com>
To: "Dave Täht" <dave.taht@bufferbloat.net>
Cc: <codel@lists.bufferbloat.net>
Sent: Saturday, August 04, 2012 9:00 AM
Subject: Re: [Codel] [RFC PATCH] codel: ecn mark at target


On Sat, Aug 4, 2012 at 4:44 AM, Dave Täht <dave.taht@bufferbloat.net> wrote:
> From: Dave Taht <dave.taht@bufferbloat.net>
>
> The consensus at ietf was that ecn marking should start at
> target, and then the results fed into the codel drop scheduler.
>
> While I agree with the latter, I feel that waiting an interval
> before starting to mark will be more in-tune with the concept
> of a sojourn time, and lead to better utilization.
>
> As I am outnumbered and outgunned, do it at target.

Well, what do you think is The Best way of doing it? Prove'em wrong if
you really think they are wrong;)



-- 

Roger Jorgensen           | ROJO9-RIPE
rogerj@gmail.com          | - IPv6 is The Key!
http://www.jorgensen.no   | roger@jorgensen.no
_______________________________________________
Codel mailing list
Codel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/codel 

Re: [Codel] [RFC PATCH] codel: ecn mark at target

[Eric Dumazet]

On Sat, 2012-08-04 at 15:38 +0200, Richard Scheffenegger wrote:
> Hi,
> 
> My 0.02 EUR:
> 
> mark every packet with a sojourn time above target is OK.
> 

Thats insane.

> The reaction of the congestion controller to ECN marks need to be more 
> differentiated; ECN can deliver much more fine-grained information about the 
> current state of the network. If a congestion controller (e.g. legacy 3168 
> ECN TCP) chooses to overshoot in it's reaction, thats a problem for that 
> particular controller... But even if every packet of a window is marked, a 
> legacy TCP will only reduce cwnd once per window, reacting the same as is a 
> non-ECN had a single drop in that window...

Thats would maybe work if all flows were ECN and kind.

If you have a mix, then ECN flows are completely shut off. And I knew it
before doing any experiment.

What makes you think non ECN flows will drop at least one packet, while
we ECN mark ECN flow ? How is cwnd 'known' by Codel exactly ?

I did following experiment :

link with HTB limit of 100Mbit/s
One netperf TCP_STREAM to a non ECN enabled local target, duration 100s
One netpert TCP_STREAM to a ECN enabled local target, duration 100s

1) Current Codel implementation

non ECN target A : 50.55 Mbit/s   (4336 dropped packets)
ECN target B     : 45.86 Mbits/s (3239 ecn marks)

2) Patch to mark all ecn-enabled packets above 'target'

non ECN target A : 95.43 Mbit/s (2957 drops)
ECN target B    :  0.97 Mbit/s (2500 ecn marks)

Yes, you see how wrong this idea is : Almost all packets to host B
are marked, so throughput is horrible.

It seems few people really understood CoDel 'target'. When a link
is used, almost all packets are above target. Thats OK. If it was not
OK, why would we use the sqrt(count) at all, I wonder.

Codel intent is not to mark/drop _all_ packets above 'target'.

ECN intent is to replace a drop by a mark, not marking packets that
would not have been dropped at all if they were not ECN enabled.

Re: [Codel] [RFC PATCH] codel: ecn mark at target

[Kathleen Nichols]

Yes, why would a single delay of more than target be considered
as a reason to take action? I thought Van did a very nice job of
explaining this last Monday.

On 8/3/12 11:45 PM, Eric Dumazet wrote:
> On Fri, 2012-08-03 at 19:44 -0700, Dave Täht wrote:
>> From: Dave Taht <dave.taht@bufferbloat.net>
>>
>> The consensus at ietf was that ecn marking should start at
>> target, and then the results fed into the codel drop scheduler.
>>
>> While I agree with the latter, I feel that waiting an interval
>> before starting to mark will be more in-tune with the concept
>> of a sojourn time, and lead to better utilization.
>>
>> As I am outnumbered and outgunned, do it at target.
> 
> Well, thats a huge way to favor non ECN flows against ECN flows.
> 
> Marking _all_ ECN enabled packets just because last packet sent had a
> sojourn time above target is going to throttle ECN flows and let non ECN
> flows going full speed and take whole bandwidth.
> 
> Doing so is a nice way to keep users switching to ECN one day.
> 
> IETF could just say : ECN is doomed, forget about it, dont even try. 
> 
> 
> 
> _______________________________________________
> Codel mailing list
> Codel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/codel
> 

Re: [Codel] [RFC PATCH] codel: ecn mark at target

[Andrew McGregor]

[-- Attachment #1: Type: text/plain, Size: 1687 bytes --]

Well, thanks Eric for trying it.

Hmm.  How was I that wrong?  Because I was supporting that idea.

Time to think.

On 4/08/2012, at 2:53 PM, Kathleen Nichols <nichols@pollere.com> wrote:

> 
> Yes, why would a single delay of more than target be considered
> as a reason to take action? I thought Van did a very nice job of
> explaining this last Monday.
> 
> On 8/3/12 11:45 PM, Eric Dumazet wrote:
>> On Fri, 2012-08-03 at 19:44 -0700, Dave Täht wrote:
>>> From: Dave Taht <dave.taht@bufferbloat.net>
>>> 
>>> The consensus at ietf was that ecn marking should start at
>>> target, and then the results fed into the codel drop scheduler.
>>> 
>>> While I agree with the latter, I feel that waiting an interval
>>> before starting to mark will be more in-tune with the concept
>>> of a sojourn time, and lead to better utilization.
>>> 
>>> As I am outnumbered and outgunned, do it at target.
>> 
>> Well, thats a huge way to favor non ECN flows against ECN flows.
>> 
>> Marking _all_ ECN enabled packets just because last packet sent had a
>> sojourn time above target is going to throttle ECN flows and let non ECN
>> flows going full speed and take whole bandwidth.
>> 
>> Doing so is a nice way to keep users switching to ECN one day.
>> 
>> IETF could just say : ECN is doomed, forget about it, dont even try. 
>> 
>> 
>> 
>> _______________________________________________
>> Codel mailing list
>> Codel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/codel
>> 
> 
> _______________________________________________
> Codel mailing list
> Codel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/codel


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2330 bytes --]

Re: [Codel] [RFC PATCH] codel: ecn mark at target

[Eric Dumazet]

On Sat, 2012-08-04 at 20:06 -0700, Andrew McGregor wrote:
> Well, thanks Eric for trying it.
> 
> Hmm.  How was I that wrong?  Because I was supporting that idea.
> 
> Time to think.

No problem Andrew ;)

Its seems ECN is not well enough understood.

ECN marking a packet has the same effect for the sender : reducing cwnd
exactly like a packet drop. Only difference is avoiding the
retransmit[s].

It cannot be used only to send a 'small' warning, while other competing
non ECN flows have no signal.

As far as packet schedulers are concerned, there should be no difference
in ECN marking and dropping a packet. I believe linux packet schedulers
are fine in this area.

Now, there are fundamental issues with ECN itself, out of Codel scope,
thats for sure.

How widely has been RFC 3540 deployed, anybody knows ?

Re: [Codel] [RFC PATCH] codel: ecn mark at target

[Andrew McGregor]

[-- Attachment #1: Type: text/plain, Size: 1780 bytes --]

Well, there's a lot of people at the IETF who really want to do other things with ECN, but it seems like the simple version is far too aggressive.

So, I think the desirable properties are something like:
1) Allow ECN flows to achieve the same or slightly higher throughput to maintain an incentive to deploy it.
2) Still drop ECN flows eventually to avoid too much queue buildup.
3) Account somehow for the fact that marking takes longer to control the queue (but we don't know how much longer).

Maybe mark ECN instead of dropping, but if we end up trying to mark/drop twice in one round, drop the later packets?

Oh, and ECN nonce deployment is negligible, to the extent that there are proposals in the IETF to reuse the bits for other things, and there is no pushback on that.

Andrew

On 4/08/2012, at 10:30 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:

> On Sat, 2012-08-04 at 20:06 -0700, Andrew McGregor wrote:
>> Well, thanks Eric for trying it.
>> 
>> Hmm.  How was I that wrong?  Because I was supporting that idea.
>> 
>> Time to think.
> 
> No problem Andrew ;)
> 
> Its seems ECN is not well enough understood.
> 
> ECN marking a packet has the same effect for the sender : reducing cwnd
> exactly like a packet drop. Only difference is avoiding the
> retransmit[s].
> 
> It cannot be used only to send a 'small' warning, while other competing
> non ECN flows have no signal.
> 
> As far as packet schedulers are concerned, there should be no difference
> in ECN marking and dropping a packet. I believe linux packet schedulers
> are fine in this area.
> 
> Now, there are fundamental issues with ECN itself, out of Codel scope,
> thats for sure.
> 
> How widely has been RFC 3540 deployed, anybody knows ?
> 
> 
> 


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2330 bytes --]

Re: [Codel] [RFC PATCH] codel: ecn mark at target

[Richard Scheffenegger]

Hi Eric,

----- Original Message ----- 
From: "Eric Dumazet" <eric.dumazet@gmail.com>
To: "Andrew McGregor" <andrewmcgr@gmail.com>
Cc: <codel@lists.bufferbloat.net>
Sent: Sunday, August 05, 2012 7:30 AM
Subject: Re: [Codel] [RFC PATCH] codel: ecn mark at target


> On Sat, 2012-08-04 at 20:06 -0700, Andrew McGregor wrote:
>> Well, thanks Eric for trying it.
>>
>> Hmm.  How was I that wrong?  Because I was supporting that idea.
>>
>> Time to think.
>
> No problem Andrew ;)
>
> Its seems ECN is not well enough understood.
>
> ECN marking a packet has the same effect for the sender : reducing cwnd
> exactly like a packet drop. Only difference is avoiding the
> retransmit[s].

That's true for the first mark; any subsequent mark (during the same window) 
should have no effect - thus a high marking rate (marking fraction per 
window) should not be that much worse... Of course, the queue can never know 
the effective window of the tcp stream it is marking...

As a test, when the marking is done really instead of drop, do you see 
fairness betwenn the ecn and legacy tcp flows? (if not, the ecn 
implementation may be faulty).

>
> It cannot be used only to send a 'small' warning, while other competing
> non ECN flows have no signal.
>
> As far as packet schedulers are concerned, there should be no difference
> in ECN marking and dropping a packet. I believe linux packet schedulers
> are fine in this area.
>
> Now, there are fundamental issues with ECN itself, out of Codel scope,
> thats for sure.
>
> How widely has been RFC 3540 deployed, anybody knows ?

Virtually not; even if the end hosts negotiate ECN, the network doesn't do 
any marking (rendering ecn mood).

See
Bauer and Beverly, "Measuring the current state of ECN support in server, 
clients, and routers",

http://mirrors.bufferbloat.net/Talks/AIMS2011/bauer-ecn-aims-2011.pdf


>
>
>
> _______________________________________________
> Codel mailing list
> Codel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/codel 

Re: [Codel] [RFC PATCH] codel: ecn mark at target

[Kathleen Nichols]

So you want to provide an incentive to deploy ECN. This also provides an
incentive to
scam the network. I think a bigger question is what is gained by doing
this? Is it more
important to deploy something called ECN than to have a well-functioning
network?

	Kathie

On 8/5/12 9:53 AM, Andrew McGregor wrote:
> Well, there's a lot of people at the IETF who really want to do other things with ECN, but it seems like the simple version is far too aggressive.
> 
> So, I think the desirable properties are something like:
> 1) Allow ECN flows to achieve the same or slightly higher throughput to maintain an incentive to deploy it.
> 2) Still drop ECN flows eventually to avoid too much queue buildup.
> 3) Account somehow for the fact that marking takes longer to control the queue (but we don't know how much longer).
> 
> Maybe mark ECN instead of dropping, but if we end up trying to mark/drop twice in one round, drop the later packets?
> 
> Oh, and ECN nonce deployment is negligible, to the extent that there are proposals in the IETF to reuse the bits for other things, and there is no pushback on that.
> 
> Andrew
> 
> On 4/08/2012, at 10:30 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> 
>> On Sat, 2012-08-04 at 20:06 -0700, Andrew McGregor wrote:
>>> Well, thanks Eric for trying it.
>>>
>>> Hmm.  How was I that wrong?  Because I was supporting that idea.
>>>
>>> Time to think.
>>
>> No problem Andrew ;)
>>
>> Its seems ECN is not well enough understood.
>>
>> ECN marking a packet has the same effect for the sender : reducing cwnd
>> exactly like a packet drop. Only difference is avoiding the
>> retransmit[s].
>>
>> It cannot be used only to send a 'small' warning, while other competing
>> non ECN flows have no signal.
>>
>> As far as packet schedulers are concerned, there should be no difference
>> in ECN marking and dropping a packet. I believe linux packet schedulers
>> are fine in this area.
>>
>> Now, there are fundamental issues with ECN itself, out of Codel scope,
>> thats for sure.
>>
>> How widely has been RFC 3540 deployed, anybody knows ?
>>
>>
>>

Re: [Codel] [RFC PATCH] codel: ecn mark at target

[Andrew McGregor]

[-- Attachment #1: Type: text/plain, Size: 3519 bytes --]

I don't think that's a real either/or question.  ECN itself does not mean that those flows get to scam the network; that's what my points 2 and 3 are for.  I guess that argues for the marking to be earlier than drop onset, but right now that's just a guess. Properly working ECN TCP should mean that with the right choice of marking pattern the result should be much the same as non-ECN TCP for fairness, but less bursty and with fewer delay events for the application.

ECN isn't all that useful for bulk transfer TCP, but for things like videoconferencing RTP it's really quite important to maintaining a good user experience. So is congestion control; these applications are better served by being consistently a bit slower than they are by bursting up and down, because frequent resolution or frame rate changes are really annoying to watch. ECN also serves many thin flow TCP applications well (SSH for example).

I have to think a bit more about what the right thing is; maybe marking once or twice each time codel enters the dropping state, and then dropping as usual. Don't take that as a real proposal, though, as this is a quick response and a solid proposal deserves a lot more thought.

Andrew

On 5/08/2012, at 9:58 AM, Kathleen Nichols <nichols@pollere.com> wrote:

> 
> So you want to provide an incentive to deploy ECN. This also provides an
> incentive to
> scam the network. I think a bigger question is what is gained by doing
> this? Is it more
> important to deploy something called ECN than to have a well-functioning
> network?
> 
> 	Kathie
> 
> On 8/5/12 9:53 AM, Andrew McGregor wrote:
>> Well, there's a lot of people at the IETF who really want to do other things with ECN, but it seems like the simple version is far too aggressive.
>> 
>> So, I think the desirable properties are something like:
>> 1) Allow ECN flows to achieve the same or slightly higher throughput to maintain an incentive to deploy it.
>> 2) Still drop ECN flows eventually to avoid too much queue buildup.
>> 3) Account somehow for the fact that marking takes longer to control the queue (but we don't know how much longer).
>> 
>> Maybe mark ECN instead of dropping, but if we end up trying to mark/drop twice in one round, drop the later packets?
>> 
>> Oh, and ECN nonce deployment is negligible, to the extent that there are proposals in the IETF to reuse the bits for other things, and there is no pushback on that.
>> 
>> Andrew
>> 
>> On 4/08/2012, at 10:30 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
>> 
>>> On Sat, 2012-08-04 at 20:06 -0700, Andrew McGregor wrote:
>>>> Well, thanks Eric for trying it.
>>>> 
>>>> Hmm.  How was I that wrong?  Because I was supporting that idea.
>>>> 
>>>> Time to think.
>>> 
>>> No problem Andrew ;)
>>> 
>>> Its seems ECN is not well enough understood.
>>> 
>>> ECN marking a packet has the same effect for the sender : reducing cwnd
>>> exactly like a packet drop. Only difference is avoiding the
>>> retransmit[s].
>>> 
>>> It cannot be used only to send a 'small' warning, while other competing
>>> non ECN flows have no signal.
>>> 
>>> As far as packet schedulers are concerned, there should be no difference
>>> in ECN marking and dropping a packet. I believe linux packet schedulers
>>> are fine in this area.
>>> 
>>> Now, there are fundamental issues with ECN itself, out of Codel scope,
>>> thats for sure.
>>> 
>>> How widely has been RFC 3540 deployed, anybody knows ?
>>> 
>>> 
>>> 
> 


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2330 bytes --]

Re: [Codel] [RFC PATCH] codel: ecn mark at target

[Eric Dumazet]

As far as CoDel is concerned, we could have two stages of control :

The current one, to mark/drop packet as specified by Kathleen & Van

Then, for marked (ecn) packets, pass a second codel stage, but dropping
packets this time, to make sure we dont allow queue to become too large.


On Sun, 2012-08-05 at 09:53 -0700, Andrew McGregor wrote:
> Well, there's a lot of people at the IETF who really want to do other things with ECN, but it seems like the simple version is far too aggressive.
> 
> So, I think the desirable properties are something like:
> 1) Allow ECN flows to achieve the same or slightly higher throughput to maintain an incentive to deploy it.
> 2) Still drop ECN flows eventually to avoid too much queue buildup.
> 3) Account somehow for the fact that marking takes longer to control the queue (but we don't know how much longer).
> 
> Maybe mark ECN instead of dropping, but if we end up trying to mark/drop twice in one round, drop the later packets?
> 
> Oh, and ECN nonce deployment is negligible, to the extent that there are proposals in the IETF to reuse the bits for other things, and there is no pushback on that.
> 
> Andrew
> 
> On 4/08/2012, at 10:30 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> 
> > On Sat, 2012-08-04 at 20:06 -0700, Andrew McGregor wrote:
> >> Well, thanks Eric for trying it.
> >> 
> >> Hmm.  How was I that wrong?  Because I was supporting that idea.
> >> 
> >> Time to think.
> > 
> > No problem Andrew ;)
> > 
> > Its seems ECN is not well enough understood.
> > 
> > ECN marking a packet has the same effect for the sender : reducing cwnd
> > exactly like a packet drop. Only difference is avoiding the
> > retransmit[s].
> > 
> > It cannot be used only to send a 'small' warning, while other competing
> > non ECN flows have no signal.
> > 
> > As far as packet schedulers are concerned, there should be no difference
> > in ECN marking and dropping a packet. I believe linux packet schedulers
> > are fine in this area.
> > 
> > Now, there are fundamental issues with ECN itself, out of Codel scope,
> > thats for sure.
> > 
> > How widely has been RFC 3540 deployed, anybody knows ?
> > 
> > 
> > 
> 

Re: [Codel] [RFC PATCH] codel: ecn mark at target

[Eric Dumazet]

On Sun, 2012-08-05 at 18:54 +0200, Richard Scheffenegger wrote:
> Hi Eric,
> 
> ----- Original Message ----- 
> From: "Eric Dumazet" <eric.dumazet@gmail.com>
> To: "Andrew McGregor" <andrewmcgr@gmail.com>
> Cc: <codel@lists.bufferbloat.net>
> Sent: Sunday, August 05, 2012 7:30 AM
> Subject: Re: [Codel] [RFC PATCH] codel: ecn mark at target
> 
> 
> > On Sat, 2012-08-04 at 20:06 -0700, Andrew McGregor wrote:
> >> Well, thanks Eric for trying it.
> >>
> >> Hmm.  How was I that wrong?  Because I was supporting that idea.
> >>
> >> Time to think.
> >
> > No problem Andrew ;)
> >
> > Its seems ECN is not well enough understood.
> >
> > ECN marking a packet has the same effect for the sender : reducing cwnd
> > exactly like a packet drop. Only difference is avoiding the
> > retransmit[s].
> 
> That's true for the first mark; any subsequent mark (during the same window) 
> should have no effect - thus a high marking rate (marking fraction per 
> window) should not be that much worse... Of course, the queue can never know 
> the effective window of the tcp stream it is marking...
> 

Once your cwnd is 1 packet, and RTT is 100ms, what can you get from
this, if all your packets have ECN mark ?

> As a test, when the marking is done really instead of drop, do you see 
> fairness betwenn the ecn and legacy tcp flows? (if not, the ecn 
> implementation may be faulty).

Yes its fine as mentioned in my test : codel , and I get 50/50 split
between my two flows.

It could be a flaw in linux implementation, I admit we had so many bugs
that it could very well be still buggy.

Re: [Codel] [RFC PATCH] codel: ecn mark at target

[Eric Dumazet]

On Sun, 2012-08-05 at 19:26 +0200, Eric Dumazet wrote:

> It could be a flaw in linux implementation, I admit we had so many bugs
> that it could very well be still buggy.

And at first glance, the following tcpdump seems suspect : We can see
all ACK are delayed by about 40 ms

19:30:54.367739 IP (tos 0x3,CE, ttl 64, id 52620, offset 0, flags [DF], proto TCP (6), length 1500)
    172.30.42.19.50341 > 172.30.42.18.50103: Flags [.], seq 156841:158289, ack 1, win 115, options [nop,nop,TS val 3115533 ecr 111212487], length 1448
19:30:54.406063 IP (tos 0x0, ttl 64, id 62099, offset 0, flags [DF], proto TCP (6), length 52)
    172.30.42.18.50103 > 172.30.42.19.50341: Flags [.E], cksum 0xac88 (incorrect -> 0xe2b7), ack 158289, win 551, options [nop,nop,TS val 111212491 ecr 3115533], length 0

19:30:54.407212 IP (tos 0x3,CE, ttl 64, id 52621, offset 0, flags [DF], proto TCP (6), length 1500)
    172.30.42.19.50341 > 172.30.42.18.50103: Flags [P.W], seq 158289:159737, ack 1, win 115, options [nop,nop,TS val 3115573 ecr 111212491], length 1448
19:30:54.446057 IP (tos 0x0, ttl 64, id 62100, offset 0, flags [DF], proto TCP (6), length 52)
    172.30.42.18.50103 > 172.30.42.19.50341: Flags [.E], cksum 0xac88 (incorrect -> 0xdce3), ack 159737, win 551, options [nop,nop,TS val 111212495 ecr 3115573], length 0



19:30:54.448209 IP (tos 0x3,CE, ttl 64, id 52622, offset 0, flags [DF], proto TCP (6), length 1500)
    172.30.42.19.50341 > 172.30.42.18.50103: Flags [.W], seq 159737:161185, ack 1, win 115, options [nop,nop,TS val 3115613 ecr 111212495], length 1448
19:30:54.486057 IP (tos 0x0, ttl 64, id 62101, offset 0, flags [DF], proto TCP (6), length 52)
    172.30.42.18.50103 > 172.30.42.19.50341: Flags [.E], cksum 0xac88 (incorrect -> 0xd70f), ack 161185, win 551, options [nop,nop,TS val 111212499 ecr 3115613], length 0

19:30:54.487443 IP (tos 0x3,CE, ttl 64, id 52623, offset 0, flags [DF], proto TCP (6), length 1500)
    172.30.42.19.50341 > 172.30.42.18.50103: Flags [.W], seq 161185:162633, ack 1, win 115, options [nop,nop,TS val 3115653 ecr 111212499], length 1448
19:30:54.526069 IP (tos 0x0, ttl 64, id 62102, offset 0, flags [DF], proto TCP (6), length 52)
    172.30.42.18.50103 > 172.30.42.19.50341: Flags [.E], cksum 0xac88 (incorrect -> 0xd13b), ack 162633, win 551, options [nop,nop,TS val 111212503 ecr 3115653], length 0

19:30:54.527241 IP (tos 0x2,ECT(0), ttl 64, id 52624, offset 0, flags [DF], proto TCP (6), length 1500)
    172.30.42.19.50341 > 172.30.42.18.50103: Flags [.W], seq 162633:164081, ack 1, win 115, options [nop,nop,TS val 3115693 ecr 111212503], length 1448
19:30:54.566049 IP (tos 0x0, ttl 64, id 62103, offset 0, flags [DF], proto TCP (6), length 52)
    172.30.42.18.50103 > 172.30.42.19.50341: Flags [.], cksum 0xac88 (incorrect -> 0xcba7), ack 164081, win 551, options [nop,nop,TS val 111212507 ecr 3115693], length 0

19:30:54.567809 IP (tos 0x3,CE, ttl 64, id 52625, offset 0, flags [DF], proto TCP (6), length 1500)
    172.30.42.19.50341 > 172.30.42.18.50103: Flags [.], seq 164081:165529, ack 1, win 115, options [nop,nop,TS val 3115733 ecr 111212507], length 1448
19:30:54.606067 IP (tos 0x0, ttl 64, id 62104, offset 0, flags [DF], proto TCP (6), length 52)
    172.30.42.18.50103 > 172.30.42.19.50341: Flags [.E], cksum 0xac88 (incorrect -> 0xc593), ack 165529, win 551, options [nop,nop,TS val 111212511 ecr 3115733], length 0
19:30:54.607088 IP (tos 0x2,ECT(0), ttl 64, id 52626, offset 0, flags [DF], proto TCP (6), length 1500)
    172.30.42.19.50341 > 172.30.42.18.50103: Flags [.W], seq 165529:166977, ack 1, win 115, options [nop,nop,TS val 3115773 ecr 111212511], length 1448
19:30:54.646056 IP (tos 0x0, ttl 64, id 62105, offset 0, flags [DF], proto TCP (6), length 52)
    172.30.42.18.50103 > 172.30.42.19.50341: Flags [.], cksum 0xac88 (incorrect -> 0xbfff), ack 166977, win 551, options [nop,nop,TS val 111212515 ecr 3115773], length 0
19:30:54.648105 IP (tos 0x3,CE, ttl 64, id 52627, offset 0, flags [DF], proto TCP (6), length 1500)
    172.30.42.19.50341 > 172.30.42.18.50103: Flags [.], seq 166977:168425, ack 1, win 115, options [nop,nop,TS val 3115813 ecr 111212515], length 1448

Re: [Codel] [RFC PATCH] codel: ecn mark at target

[Yuchung Cheng]

On Sun, Aug 5, 2012 at 10:35 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Sun, 2012-08-05 at 19:26 +0200, Eric Dumazet wrote:
>
>> It could be a flaw in linux implementation, I admit we had so many bugs
>> that it could very well be still buggy.
>
> And at first glance, the following tcpdump seems suspect : We can see
> all ACK are delayed by about 40 ms
but RFC 3168 (sec 6.1.3) does not mandate immediate ACKs for ECE
marked ones? is this because ECN response is per round-trip?



>
> 19:30:54.367739 IP (tos 0x3,CE, ttl 64, id 52620, offset 0, flags [DF], proto TCP (6), length 1500)
>     172.30.42.19.50341 > 172.30.42.18.50103: Flags [.], seq 156841:158289, ack 1, win 115, options [nop,nop,TS val 3115533 ecr 111212487], length 1448
> 19:30:54.406063 IP (tos 0x0, ttl 64, id 62099, offset 0, flags [DF], proto TCP (6), length 52)
>     172.30.42.18.50103 > 172.30.42.19.50341: Flags [.E], cksum 0xac88 (incorrect -> 0xe2b7), ack 158289, win 551, options [nop,nop,TS val 111212491 ecr 3115533], length 0
>
> 19:30:54.407212 IP (tos 0x3,CE, ttl 64, id 52621, offset 0, flags [DF], proto TCP (6), length 1500)
>     172.30.42.19.50341 > 172.30.42.18.50103: Flags [P.W], seq 158289:159737, ack 1, win 115, options [nop,nop,TS val 3115573 ecr 111212491], length 1448
> 19:30:54.446057 IP (tos 0x0, ttl 64, id 62100, offset 0, flags [DF], proto TCP (6), length 52)
>     172.30.42.18.50103 > 172.30.42.19.50341: Flags [.E], cksum 0xac88 (incorrect -> 0xdce3), ack 159737, win 551, options [nop,nop,TS val 111212495 ecr 3115573], length 0
>
>
>
> 19:30:54.448209 IP (tos 0x3,CE, ttl 64, id 52622, offset 0, flags [DF], proto TCP (6), length 1500)
>     172.30.42.19.50341 > 172.30.42.18.50103: Flags [.W], seq 159737:161185, ack 1, win 115, options [nop,nop,TS val 3115613 ecr 111212495], length 1448
> 19:30:54.486057 IP (tos 0x0, ttl 64, id 62101, offset 0, flags [DF], proto TCP (6), length 52)
>     172.30.42.18.50103 > 172.30.42.19.50341: Flags [.E], cksum 0xac88 (incorrect -> 0xd70f), ack 161185, win 551, options [nop,nop,TS val 111212499 ecr 3115613], length 0
>
> 19:30:54.487443 IP (tos 0x3,CE, ttl 64, id 52623, offset 0, flags [DF], proto TCP (6), length 1500)
>     172.30.42.19.50341 > 172.30.42.18.50103: Flags [.W], seq 161185:162633, ack 1, win 115, options [nop,nop,TS val 3115653 ecr 111212499], length 1448
> 19:30:54.526069 IP (tos 0x0, ttl 64, id 62102, offset 0, flags [DF], proto TCP (6), length 52)
>     172.30.42.18.50103 > 172.30.42.19.50341: Flags [.E], cksum 0xac88 (incorrect -> 0xd13b), ack 162633, win 551, options [nop,nop,TS val 111212503 ecr 3115653], length 0
>
> 19:30:54.527241 IP (tos 0x2,ECT(0), ttl 64, id 52624, offset 0, flags [DF], proto TCP (6), length 1500)
>     172.30.42.19.50341 > 172.30.42.18.50103: Flags [.W], seq 162633:164081, ack 1, win 115, options [nop,nop,TS val 3115693 ecr 111212503], length 1448
> 19:30:54.566049 IP (tos 0x0, ttl 64, id 62103, offset 0, flags [DF], proto TCP (6), length 52)
>     172.30.42.18.50103 > 172.30.42.19.50341: Flags [.], cksum 0xac88 (incorrect -> 0xcba7), ack 164081, win 551, options [nop,nop,TS val 111212507 ecr 3115693], length 0
>
> 19:30:54.567809 IP (tos 0x3,CE, ttl 64, id 52625, offset 0, flags [DF], proto TCP (6), length 1500)
>     172.30.42.19.50341 > 172.30.42.18.50103: Flags [.], seq 164081:165529, ack 1, win 115, options [nop,nop,TS val 3115733 ecr 111212507], length 1448
> 19:30:54.606067 IP (tos 0x0, ttl 64, id 62104, offset 0, flags [DF], proto TCP (6), length 52)
>     172.30.42.18.50103 > 172.30.42.19.50341: Flags [.E], cksum 0xac88 (incorrect -> 0xc593), ack 165529, win 551, options [nop,nop,TS val 111212511 ecr 3115733], length 0
> 19:30:54.607088 IP (tos 0x2,ECT(0), ttl 64, id 52626, offset 0, flags [DF], proto TCP (6), length 1500)
>     172.30.42.19.50341 > 172.30.42.18.50103: Flags [.W], seq 165529:166977, ack 1, win 115, options [nop,nop,TS val 3115773 ecr 111212511], length 1448
> 19:30:54.646056 IP (tos 0x0, ttl 64, id 62105, offset 0, flags [DF], proto TCP (6), length 52)
>     172.30.42.18.50103 > 172.30.42.19.50341: Flags [.], cksum 0xac88 (incorrect -> 0xbfff), ack 166977, win 551, options [nop,nop,TS val 111212515 ecr 3115773], length 0
> 19:30:54.648105 IP (tos 0x3,CE, ttl 64, id 52627, offset 0, flags [DF], proto TCP (6), length 1500)
>     172.30.42.19.50341 > 172.30.42.18.50103: Flags [.], seq 166977:168425, ack 1, win 115, options [nop,nop,TS val 3115813 ecr 111212515], length 1448
>
>
>
> _______________________________________________
> Codel mailing list
> Codel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/codel

Re: [Codel] [RFC PATCH] codel: ecn mark at target

[Eric Dumazet]

On Sun, 2012-08-05 at 11:14 -0700, Yuchung Cheng wrote:
> On Sun, Aug 5, 2012 at 10:35 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> > On Sun, 2012-08-05 at 19:26 +0200, Eric Dumazet wrote:
> >
> >> It could be a flaw in linux implementation, I admit we had so many bugs
> >> that it could very well be still buggy.
> >
> > And at first glance, the following tcpdump seems suspect : We can see
> > all ACK are delayed by about 40 ms
> but RFC 3168 (sec 6.1.3) does not mandate immediate ACKs for ECE
> marked ones? is this because ECN response is per round-trip?
> 

We should IMHO not delay ACKS, exactly like we react to a dropped
packet.

If not specified in RFC 3168, it seems a forgotten point.

Re: [Codel] [RFC PATCH] codel: ecn mark at target

[Eric Dumazet]

On Sun, 2012-08-05 at 20:40 +0200, Eric Dumazet wrote:
> On Sun, 2012-08-05 at 11:14 -0700, Yuchung Cheng wrote:
> > On Sun, Aug 5, 2012 at 10:35 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> > > On Sun, 2012-08-05 at 19:26 +0200, Eric Dumazet wrote:
> > >
> > >> It could be a flaw in linux implementation, I admit we had so many bugs
> > >> that it could very well be still buggy.
> > >
> > > And at first glance, the following tcpdump seems suspect : We can see
> > > all ACK are delayed by about 40 ms
> > but RFC 3168 (sec 6.1.3) does not mandate immediate ACKs for ECE
> > marked ones? is this because ECN response is per round-trip?
> > 
> 
> We should IMHO not delay ACKS, exactly like we react to a dropped
> packet.
> 
> If not specified in RFC 3168, it seems a forgotten point.

Following patch does the trick.

Delaying ACK while sender might have a cwnd = 1 is not nice at all.

I get better behavior, but still codel with Dave patch (mark all packets
if above 'target') is giving an unfair advantage to non ECN flow.

diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 2fd2bc9..b03f429 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -237,6 +237,7 @@ static inline void TCP_ECN_check_ce(struct tcp_sock *tp, const struct sk_buff *s
 			tcp_enter_quickack_mode((struct sock *)tp);
 		break;
 	case INET_ECN_CE:
+		inet_csk((struct sock *)tp)->icsk_ack.quick |= 1;
 		tp->ecn_flags |= TCP_ECN_DEMAND_CWR;
 		/* fallinto */
 	default:

Re: [Codel] [RFC PATCH] codel: ecn mark at target

[Richard Scheffenegger]

Well, as long as the window is large enough, the delayed ACKs shouldn't 
matter, even if the ECE is delayed for 1 segment; the argument about delayed 
ACKs when cwnd is 1 is also true for non-ECN flows; they would run better 
when every segment is acked individually; but can the receiver tell, if the 
sender is running at cwnd=1?

Perhaps, if it tracks the RTT of the flow (which has to work without TS, as 
they are undefined for pure ACKs), and the number of segments seen during 
one RTT...

(Perhaps another performance tweak for linux TCP...)

Best regards,
   Richard

----- Original Message ----- 
From: "Eric Dumazet" <eric.dumazet@gmail.com>
To: "Yuchung Cheng" <ycheng@google.com>
Cc: "Richard Scheffenegger" <rscheff@gmx.at>; <codel@lists.bufferbloat.net>
Sent: Sunday, August 05, 2012 8:40 PM
Subject: Re: [Codel] [RFC PATCH] codel: ecn mark at target


> On Sun, 2012-08-05 at 11:14 -0700, Yuchung Cheng wrote:
>> On Sun, Aug 5, 2012 at 10:35 AM, Eric Dumazet <eric.dumazet@gmail.com> 
>> wrote:
>> > On Sun, 2012-08-05 at 19:26 +0200, Eric Dumazet wrote:
>> >
>> >> It could be a flaw in linux implementation, I admit we had so many 
>> >> bugs
>> >> that it could very well be still buggy.
>> >
>> > And at first glance, the following tcpdump seems suspect : We can see
>> > all ACK are delayed by about 40 ms
>> but RFC 3168 (sec 6.1.3) does not mandate immediate ACKs for ECE
>> marked ones? is this because ECN response is per round-trip?
>>
>
> We should IMHO not delay ACKS, exactly like we react to a dropped
> packet.
>
> If not specified in RFC 3168, it seems a forgotten point.
>
>
> 

Re: [Codel] [RFC PATCH] codel: ecn mark at target

[Eric Dumazet]

On Mon, 2012-08-06 at 18:22 +0200, Richard Scheffenegger wrote:
> Well, as long as the window is large enough, the delayed ACKs shouldn't 
> matter, even if the ECE is delayed for 1 segment; the argument about delayed 
> ACKs when cwnd is 1 is also true for non-ECN flows; they would run better 
> when every segment is acked individually; but can the receiver tell, if the 
> sender is running at cwnd=1?
> 

It should just be safe instead than lazy.

Frankly what's the deal to send an ACK right now instead of in 40ms ?

> Perhaps, if it tracks the RTT of the flow (which has to work without TS, as 
> they are undefined for pure ACKs), and the number of segments seen during 
> one RTT...

Linux does have an heuristic to _not_ use delayed acks for the first
packets received (about 15 MSS), _because_ it assumes sender has a small
cwnd.

When receiving out of order packets, this logic is restarted, because we
want to send SACKS as soon as possible, not after a delayed ack.

Every time we suspect sender has a small cwnd, we enter the quickack
mode.

So when receiving CE segments, we should do the same. Thats a flaw in
current linux code to do nothing and expect more packets to come.

It just works because RTO triggers. A lot of bugs are hidden because of
various timers that take some emergency actions.

tcp_enter_quickack_mode() has all the needed tweaks.


diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 2fd2bc9..2e55337 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -237,6 +237,9 @@ static inline void TCP_ECN_check_ce(struct tcp_sock *tp, const struct sk_buff *s
 			tcp_enter_quickack_mode((struct sock *)tp);
 		break;
 	case INET_ECN_CE:
+		/* Better not delay acks, sender can have a very low cwnd */
+		tcp_enter_quickack_mode((struct sock *)tp);
+
 		tp->ecn_flags |= TCP_ECN_DEMAND_CWR;
 		/* fallinto */
 	default:

Re: [Codel] [RFC PATCH] codel: ecn mark at target

[Dave Taht]

On Mon, Aug 6, 2012 at 9:46 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote:

> Every time we suspect sender has a small cwnd, we enter the quickack
> mode.
>
> So when receiving CE segments, we should do the same. Thats a flaw in
> current linux code to do nothing and expect more packets to come.
>
> It just works because RTO triggers. A lot of bugs are hidden because of
> various timers that take some emergency actions.
>

This discussion is getting mildly off-track. My intent in posting this patch
was to prove how wrong the "ecn mark at target" idea was by example,
and in doing so, shed light on those new to codel, on how the algorithm
actually works, and to encourage those that didn't grok it, to read and
run the code in whatever scenarios would help more people to
grokking in fullness.

I hadn't expected to twiddle a bug!

In a previous mail eric had written:

>link with HTB limit of 100Mbit/s
>One netperf TCP_STREAM to a non ECN enabled local target, duration 100s
>One netpert TCP_STREAM to a ECN enabled local target, duration 100s

>1) Current Codel implementation

>non ECN target A : 50.55 Mbit/s   (4336 dropped packets)
>ECN target B     : 45.86 Mbits/s (3239 ecn marks)

My assumption is that with the ecn quickack fix in place, that these two
streams are now more closely equivalent in throughput.

>2) Patch to mark all ecn-enabled packets above 'target'

>non ECN target A : 95.43 Mbit/s (2957 drops)
>ECN target B    :  0.97 Mbit/s (2500 ecn marks)

And this was the result I'd expected from the patch regardless. While
I would expect the ecn quickack thing to help matters, with the
"ietf patch" - not mine! I just decided that a negative proof was the quickest
way to move forward! - I would still expect non-ecn marked streams dropped
via the codel drop scheduler to seriously  outcompete ecn streams
being marked at target.

but as I'm on the road, and away from my test rig, I lack numbers
to deal with. Eric, could you repost the results from these two
scenarios?

>It seems few people really understood CoDel 'target'. When a link
>is used, almost all packets are above target. Thats OK. If it was not
>OK, why would we use the sqrt(count) at all, I wonder.

Two of the really brilliant parts of codel are that A) it aims to *eventually*
hit an optimum target, while absorbing bursts, and B) does so gently
using the invsqrt trick to schedule drops at a decreasing rate until an
optimum is found, and to continually re-adjust under varying loads to
ensure a minimal queue length.

The "finding the optimum" portion of codel is still a matter
of research, with something like 6 different variants existing so far,
and the version published in the original paper being thoroughly
obsolete. The current ns2 code has some promising ideas in it,
and I went through the trouble of getting that, and what
variants I could find for ns2, up on github last week.

https://github.com/dtaht/ns2

The ns3 code is also on github, which has a fq_codel implementation
too that doesn't quite match the linux code.

If anyone would like commit access, let me know. The ns2 version now ALSO
contains a re-implementation of fq_codel, made lighter-weight
because it is packet oriented rather than byte-fair oriented.

NOTE:

I have an email about reconciling each of the variants of codel/fq_codel
in ns2,ns3, and linux stacked up behind this email, but  I'd like to keep
trying to nail this ECN issue to the ground, first.

My own concern with ECN is finding a way to use it effectively while making
it difficult to abuse in the real world.

>Codel intent is not to mark/drop _all_ packets above 'target'.

>ECN intent is to replace a drop by a mark, not marking packets that
>would not have been dropped at all if they were not ECN enabled.

I really, really, really would like more folk to run themselves through
the behavior of codel in various scenarios, particularly
under contention from multiple streams, and in the fq_codel case as well.

In part it's a shame
that we removed some of the scaffolding that existed in earlier, less
optimized versions of the Linux implementation, which lent more insight
as to what happened, when.

The ns3 code has remnants of that, tracking the major states in codel
in a way that's mildly easier to understand.

I've had to look at a lot of traces and a lot of captures to get a grip
one what actually happens in each variant.

-- 
Dave Täht

Re: [Codel] [RFC PATCH] codel: ecn mark at target

[Andrew McGregor]

[-- Attachment #1: Type: text/plain, Size: 835 bytes --]


On 6/08/2012, at 10:50 AM, Dave Taht <dave.taht@gmail.com> wrote:

> This discussion is getting mildly off-track. My intent in posting this patch
> was to prove how wrong the "ecn mark at target" idea was by example,
> and in doing so, shed light on those new to codel, on how the algorithm
> actually works, and to encourage those that didn't grok it, to read and
> run the code in whatever scenarios would help more people to
> grokking in fullness.
> 
> I hadn't expected to twiddle a bug!

Well, so drop at target is wrong wrt deployed TCPs.  Ok, fine.

So, instead, how about this: mark instead of dropping, but only for the first few iterations around the while loop in dequeue (so that huge backlogs can be drained).  The question then is, how many is a few?  I suppose that can be answered empirically.

Andrew

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2330 bytes --]

Re: [Codel] [RFC PATCH] codel: ecn mark at target

[Eric Dumazet]

On Mon, 2012-08-06 at 12:09 -0700, Andrew McGregor wrote:
> On 6/08/2012, at 10:50 AM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> > This discussion is getting mildly off-track. My intent in posting this patch
> > was to prove how wrong the "ecn mark at target" idea was by example,
> > and in doing so, shed light on those new to codel, on how the algorithm
> > actually works, and to encourage those that didn't grok it, to read and
> > run the code in whatever scenarios would help more people to
> > grokking in fullness.
> > 
> > I hadn't expected to twiddle a bug!
> 
> Well, so drop at target is wrong wrt deployed TCPs.  Ok, fine.
> 
> So, instead, how about this: mark instead of dropping, but only for
> the first few iterations around the while loop in dequeue (so that
> huge backlogs can be drained).  The question then is, how many is a
> few?  I suppose that can be answered empirically.
> 
> Andrew

Lets take the analogy with RED.

Once a MAX_THRESH was reached, it did a hard_mark.

So we could choose a threshold to drop instead of marking.

Possible easy choices :

1) A threshold on sojourn time. For example force_drop = 2 * target

2) A threshold on 'count'. For example force_drop if count > 1000

Re: [Codel] [RFC PATCH] codel: ecn mark at target

[Dave Taht]

On Mon, Aug 6, 2012 at 1:01 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Mon, 2012-08-06 at 12:09 -0700, Andrew McGregor wrote:
>> On 6/08/2012, at 10:50 AM, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> > This discussion is getting mildly off-track. My intent in posting this patch
>> > was to prove how wrong the "ecn mark at target" idea was by example,
>> > and in doing so, shed light on those new to codel, on how the algorithm
>> > actually works, and to encourage those that didn't grok it, to read and
>> > run the code in whatever scenarios would help more people to
>> > grokking in fullness.
>> >
>> > I hadn't expected to twiddle a bug!

I have put eric's ecn quickack patch into what will be the next
release of cerowrt.

I think I saw another related one go by, too.

>>
>> Well, so drop at target is wrong wrt deployed TCPs.  Ok, fine.
>>
>> So, instead, how about this: mark instead of dropping, but only for
>> the first few iterations around the while loop in dequeue (so that
>> huge backlogs can be drained).  The question then is, how many is a
>> few?  I suppose that can be answered empirically.
>>
>> Andrew
>
> Lets take the analogy with RED.
>
> Once a MAX_THRESH was reached, it did a hard_mark.
>
> So we could choose a threshold to drop instead of marking.
>
> Possible easy choices :
>
> 1) A threshold on sojourn time. For example force_drop = 2 * target

I tried this and the net result was that: as codel doesn't start the
drop scheduler until things are wildly out of wack, that a large
percentage of ecn marked packets would be dropped at the outset, and
only when things got closer to target would CE start to be asserted.

Fast convergence is kind of a feature but, arguably, to use ECN most
effectively would be to start off marking and switch to dropping if
marking was not being effective in reducing the the delay
sufficiently.

> 2) A threshold on 'count'. For example force_drop if count > 1000

This has the benefit of having not been tried. However how count
varies is presently subject to the implementation.

So, new proposal:

3) When entering the drop scheduler for the first time, CE mark rather
than drop on ecn enabled packets. Over time (an interval?) keep an
overall slope (ewma?) of the curve between ldelay and current delay.
If that overall slope is downwards toward the target, then great, ECN
marking is working, and we're eventually going to get back to target.
If it is upwards for greater than (x), then start dropping instead.

This has the desirable properties of:

A) Trying CE first
B) Continuing to apply CE if it works
C) In case of overload, drop

But you'll note that there is no code in this proposal, and what an
"overall slope" is, is a handwave.


>
>



-- 
Dave Täht
http://www.bufferbloat.net/projects/cerowrt/wiki - "3.3.8-6 is out
with fq_codel!"