debloat-testing loadlatency test

[Jonathan Morton]

On 21 Mar, 2011, at 4:45 pm, Dave Täht wrote:

> I aborted the test when I got up this morning.

Yes, it looks as though it may have got stuck, possibly on a partly-mitigated race condition I found.  A full test run should never take that long, as there is an override condition which forces a scenario to complete after about 10 minutes.

It would probably be worth re-running this with better propagation conditions.  We've already established that even with high-quality physical links, the statistics are usually remarkably bad - I'd like to establish that high smoothness and responsiveness figures are actually possible.

At least you don't seem to have any actual dropped connections.  Here's a run between two Macs on GigE:

MinRTT: 0.0ms
Scenario 1: 0 uploads, 1 downloads... 107198 KiB/s down, 22.82 Hz smoothness
Scenario 2: 1 uploads, 0 downloads... 105388 KiB/s up, 31.00 Hz smoothness
Scenario 3: 0 uploads, 2 downloads... 110783 KiB/s down, 7.52 Hz smoothness
Scenario 4: 1 uploads, 1 downloads... 50998 KiB/s up, 85859 KiB/s down, 2.90 Hz smoothness
Scenario 5: 2 uploads, 0 downloads... 108230 KiB/s up, 2.73 Hz smoothness
Scenario 6: 0 uploads, 3 downloads... 112491 KiB/s down, 7.29 Hz smoothness
Scenario 7: 1 uploads, 2 downloads... 33296 KiB/s up, 100526 KiB/s down, 3.11 Hz smoothness
Scenario 8: 2 uploads, 1 downloads... 69995 KiB/s up, 66111 KiB/s down, 2.53 Hz smoothness
Scenario 9: 3 uploads, 0 downloads... 102502 KiB/s up, 4.96 Hz smoothness
Scenario 10: 0 uploads, 4 downloads... 111745 KiB/s down, 2.55 Hz smoothness
Scenario 11: 1 uploads, 3 downloads... 21816 KiB/s up, 104424 KiB/s down, 6.00 Hz smoothness
Scenario 12: 2 uploads, 2 downloads... 52828 KiB/s up, 93344 KiB/s down, 2.96 Hz smoothness
Scenario 13: 3 uploads, 1 downloads... 82564 KiB/s up, 52255 KiB/s down, 6.89 Hz smoothness
Scenario 14: 4 uploads, 0 downloads... 110715 KiB/s up, 2.57 Hz smoothness
Scenario 15: 0 uploads, 32 downloads... 0 KiB/s down, 0.00 Hz smoothness
Scenario 16: 1 uploads, 31 downloads... 14712 KiB/s up, 0 KiB/s down, 0.00 Hz smoothness
Scenario 17: 16 uploads, 16 downloads... 76252 KiB/s up, 85105 KiB/s down, 0.86 Hz smoothness
Scenario 18: 31 uploads, 1 downloads... 113567 KiB/s up, 5775 KiB/s down, 2.64 Hz smoothness
Scenario 19: 32 uploads, 0 downloads... 107671 KiB/s up, 2.55 Hz smoothness

OVERALL:
    Upload Capacity: 50450 KiB/s
  Download Capacity: 0 KiB/s
Link Responsiveness: 0 Hz
    Flow Smoothness: 0 Hz

In the above, during scenarios 15 and 16, the G5 (acting as the server) reported hard shutdowns of several spew() threads, indicating that the TCP/IP stack had cancelled the connections.  The MBP (as client) didn't hard-shutdown any connections. Shutdown connections are deliberately recorded as zeroes because they constitute a serious failure of the network, and the way the stats are combined ensures that this is reflected in the overall results.

It seems that OSX uses a rather aggressive TCP which can actually saturate GigE even with one connection.  The tradeoff is that with multiple flows in contention, they can be mutually unfair enough for some flows to be completely starved out by packet losses.  When that happens, the TCP retransmits, but the retransmissions also get lost at a rather high probability.  Eventually, the stack decides the flow is dead and cancels it.

It's not clear whether both ends are using ECN here (the G5 is restricted to OSX 10.5.x because 10.6 requires an Intel CPU), but either way it is clearly ineffective because there is no AQM on the GigE ports (or at least not the one in the G5).  ECN requires AQM to function.

 - Jonathan