[Ecn-sane] results of two simple ECN tests

[Ecn-sane] results of two simple ECN tests

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 973 bytes --]

Attached are some scripts that run two simple tests of ECN with veth devices, with and without ECN. The topology is:

client - middlebox (20Mbit htb+fq_codel egress both ways) - net (40ms netem delay both ways, i.e. 80ms RTT) - server

Here are some results from the APU2 with Debian 9 / kernel 4.9.0-8:

Test 1 (“One vs one”, two clients uploads competing, one flow each for 60 seconds, measure total data transferred):

	No ECN, 63.2 + 63.5 transferred = 126.7MB
	ECN, 63.2 + 61.5 transferred = 124.7MB

Test 2 (“One vs pulses”, client #1: upload for 60 seconds, client #2: 40x 1M uploads sequentially (iperf -n 1M), measure client #1 data transferred):

	No ECN, 63.2 MB transferred
	ECN, 65.0 MB transferred

Can anyone suggest changes to this test or a better test that would more clearly show the benefit of ECN? I guess we’d want more congestion and the cost of each lost packet to be higher, meaning higher RTTs and more clients?

Pete

[-- Attachment #2: veth_ecn.tar.gz --]
[-- Type: application/x-gzip, Size: 5391 bytes --]

Re: [Ecn-sane] results of two simple ECN tests

[Sebastian Moeller]

[-- Attachment #1: Type: text/plain, Size: 1185 bytes --]

Did you use SACK?

On February 17, 2019 12:26:51 PM GMT+01:00, Pete Heist <pete@heistp.net> wrote:
>Attached are some scripts that run two simple tests of ECN with veth
>devices, with and without ECN. The topology is:
>
>client - middlebox (20Mbit htb+fq_codel egress both ways) - net (40ms
>netem delay both ways, i.e. 80ms RTT) - server
>
>Here are some results from the APU2 with Debian 9 / kernel 4.9.0-8:
>
>Test 1 (“One vs one”, two clients uploads competing, one flow each for
>60 seconds, measure total data transferred):
>
>	No ECN, 63.2 + 63.5 transferred = 126.7MB
>	ECN, 63.2 + 61.5 transferred = 124.7MB
>
>Test 2 (“One vs pulses”, client #1: upload for 60 seconds, client #2:
>40x 1M uploads sequentially (iperf -n 1M), measure client #1 data
>transferred):
>
>	No ECN, 63.2 MB transferred
>	ECN, 65.0 MB transferred
>
>Can anyone suggest changes to this test or a better test that would
>more clearly show the benefit of ECN? I guess we’d want more congestion
>and the cost of each lost packet to be higher, meaning higher RTTs and
>more clients?
>
>Pete

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 1440 bytes --]

Re: [Ecn-sane] results of two simple ECN tests

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 1597 bytes --]

Yes, it's enabled by default. I think I'm just measuring the wrong thing. ECN seems to be about reducing TCP RTT and jitter, not increasing throughput per se. I'll rather compare packet captures with it on and off to look for an improvement in the TCP RTT spikes typically associated with drops.

> On 17 Feb 2019, at 14:02, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> Did you use SACK?
> 
>> On February 17, 2019 12:26:51 PM GMT+01:00, Pete Heist <pete@heistp.net> wrote:
>> Attached are some scripts that run two simple tests of ECN with veth devices, with and without ECN. The topology is:
>> 
>> client - middlebox (20Mbit htb+fq_codel egress both ways) - net (40ms netem delay both ways, i.e. 80ms RTT) - server
>> 
>> Here are some results from the APU2 with Debian 9 / kernel 4.9.0-8:
>> 
>> Test 1 (“One vs one”, two clients uploads competing, one flow each for 60 seconds, measure total data transferred):
>> 
>> 	No ECN, 63.2 + 63.5 transferred = 126.7MB
>> 	ECN, 63.2 + 61.5 transferred = 124.7MB
>> 
>> Test 2 (“One vs pulses”, client #1: upload for 60 seconds, client #2: 40x 1M uploads sequentially (iperf -n 1M), measure client #1 data transferred):
>> 
>> 	No ECN, 63.2 MB transferred
>> 	ECN, 65.0 MB transferred
>> 
>> Can anyone suggest changes to this test or a better test that would more clearly show the benefit of ECN? I guess we’d want more congestion and the cost of each lost packet to be higher, meaning higher RTTs and more clients?
>> 
>> Pete
> 
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 2054 bytes --]

Re: [Ecn-sane] results of two simple ECN tests

[Sebastian Moeller]

Hi Pete,


> On Feb 17, 2019, at 21:57, Pete Heist <pete@heistp.net> wrote:
> 
> Yes, it's enabled by default. I think I'm just measuring the wrong thing. ECN seems to be about reducing TCP RTT and jitter, not increasing throughput per se.

	But in your test, a reduced TCP RTT should result in a higher throughput, no?

> I'll rather compare packet captures with it on and off to look for an improvement in the TCP RTT spikes typically associated with drops.

Well, the big danger of dropping packets is that you might stall a flow (say, by dropping enough consecutive packets to drive the flow into RTO) something much less likely with SACK (at least that is my understanding of one of SACKs promises). For post-bottleneck shaping there is also the issue that ECN-marking an incoming packet will at least not have wasted the "transmit-slot" that was occupied for the transmit. But given that TCP was designed to interpret lost packets as a sign of having exceeded capacity I am not that amazed that it still does a decent job doing so ;) I believe there is an argument for giving ECN capable flows a lower marking probability than non-ECN flows would get a drop probability, but since that is easily gamed (end-points negotiating ECN but simply not slowing down on receiving marks) it is not an option for the wide-internet and hence ECN should not give much improvement in throughput (although it will reduce the number of retransmitted packets). I wonder how this would change if you would reconfigure the shaper to half the bandwidth in the middle of the 


> 
> On 17 Feb 2019, at 14:02, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
>> Did you use SACK?
>> 
>> On February 17, 2019 12:26:51 PM GMT+01:00, Pete Heist <pete@heistp.net> wrote:
>> Attached are some scripts that run two simple tests of ECN with veth devices, with and without ECN. The topology is:
>> 
>> client - middlebox (20Mbit htb+fq_codel egress both ways) - net (40ms netem delay both ways, i.e. 80ms RTT) - server
>> 
>> Here are some results from the APU2 with Debian 9 / kernel 4.9.0-8:
>> 
>> Test 1 (“One vs one”, two clients uploads competing, one flow each for 60 seconds, measure total data transferred):
>> 
>> 	No ECN, 63.2 + 63.5 transferred = 126.7MB
>> 	ECN, 63.2 + 61.5 transferred = 124.7MB
>> 
>> Test 2 (“One vs pulses”, client #1: upload for 60 seconds, client #2: 40x 1M uploads sequentially (iperf -n 1M), measure client #1 data transferred):
>> 
>> 	No ECN, 63.2 MB transferred
>> 	ECN, 65.0 MB transferred
>> 
>> Can anyone suggest changes to this test or a better test that would more clearly show the benefit of ECN? I guess we’d want more congestion and the cost of each lost packet to be higher, meaning higher RTTs and more clients?
>> 
>> Pete
>> 
>> -- 
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [Ecn-sane] results of two simple ECN tests

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 2676 bytes --]


> On Feb 17, 2019, at 10:07 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> 	But in your test, a reduced TCP RTT should result in a higher throughput, no?

Theoretically, but I think the difference may be only marginal at modern bandwidths. For example, during my 20Mbit, 10 second “one vs one” iperf test, 23770 segments are sent and only 9 are dropped. ECN saves those 9 segments, but that’s only 13626 bytes (0.038%), plus what ever side effects there may be. My current test with iperf isn’t sensitive enough to measure a corresponding difference in throughput.

Note that at https://en.wikipedia.org/wiki/Explicit_Congestion_Notification#Effects_on_performance <https://en.wikipedia.org/wiki/Explicit_Congestion_Notification#Effects_on_performance>, it claims "Effects of ECN on bulk throughput are less clear”, which references a 2003 paper "Marek Malowidzki, Simulation-based Study of ECN Performance in RED Networks”.

>> I'll rather compare packet captures with it on and off to look for an improvement in the TCP RTT spikes typically associated with drops.
> 
> Well, the big danger of dropping packets is that you might stall a flow (say, by dropping enough consecutive packets to drive the flow into RTO) something much less likely with SACK (at least that is my understanding of one of SACKs promises).

That may be, but my simple simulation doesn’t reproduce that case. I’ve updated it and made some TCP RTT graphs, which does show a clearer difference with ECN. All the files and pcaps are here:

https://www.heistp.net/downloads/veth_ecn/ <https://www.heistp.net/downloads/veth_ecn/>

Compare these two one-vs-one RTT graphs and the difference with ECN enabled can be seen:

https://www.heistp.net/downloads/veth_ecn/client_one_vs_one_noecn_rtt.png <https://www.heistp.net/downloads/veth_ecn/client_one_vs_one_noecn_rtt.png>

https://www.heistp.net/downloads/veth_ecn/client_one_vs_one_ecn_rtt.png <https://www.heistp.net/downloads/veth_ecn/client_one_vs_one_ecn_rtt.png>

Similar for one-vs-pulses:

https://www.heistp.net/downloads/veth_ecn/client_one_vs_pulses_noecn_rtt.png <https://www.heistp.net/downloads/veth_ecn/client_one_vs_pulses_noecn_rtt.png>

https://www.heistp.net/downloads/veth_ecn/client_one_vs_pulses_ecn_rtt.png <https://www.heistp.net/downloads/veth_ecn/client_one_vs_pulses_ecn_rtt.png>

The graphs of TCP window are also more appealing in the ECN case, at least.

Now, re-reading Dave’s posts about why the ECN-sane project was started, there appear to be some pathological cases. This simple test doesn’t get to those. For now just wanted to get in touch with some basics. :)


[-- Attachment #2: Type: text/html, Size: 4019 bytes --]

Re: [Ecn-sane] results of two simple ECN tests

[Dave Taht]

I appreciate the shot at going back to basics and the packet capture
driven tests.

ECN does help keep latency down at longer RTTs. But the bandwidth
improvements are rather minimal in this test case, and basically a
function of what would have been the drop rate. The pathological cases
I was discussing at one point (100 flows through a short low bandwidth
link) dramatically increase delay, cause starvation for other flows
(including new flows trying to start).