Linux SCTP: what kind of performance should I expect from netperf?

Linux SCTP: what kind of performance should I expect from netperf?

[Dave Taht]

I was curious about sctp's performance characteristics on
AQM'd systems... so
I built netperf with sctp support, and ran a couple tests on
kernel 3.11...

+1: SCTP appears to work over IPv6
-1: Throughput is terrible


d@nuc:~/git/netperf$ netperf -6 -H snapon.lab.bufferbloat.net -t
SCTP_STREAM_MANY
SCTP 1-TO-MANY STREAM TEST from ::0 (::) port 0 AF_INET6 to
snapon.lab.bufferbloat.net () port 0 AF_INET6 : demo
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

212992 212992   4096    10.00       0.31
d@nuc:~/git/netperf$ netperf -6 -H snapon.lab.bufferbloat.net -t TCP_MAERTS
MIGRATED TCP MAERTS TEST from ::0 (::) port 0 AF_INET6 to
snapon.lab.bufferbloat.net () port 0 AF_INET6 : demo
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

 87380  16384  16384    10.00       7.65


-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: Linux SCTP: what kind of performance should I expect from netperf?

[Jesper Dangaard Brouer]

Hi Daniel,

Can you give some input on this thread?


On Tue, 18 Mar 2014 10:53:40 -0400 Dave Taht <dave.taht@gmail.com> wrote:

> I was curious about sctp's performance characteristics on
> AQM'd systems... so
> I built netperf with sctp support, and ran a couple tests on
> kernel 3.11...
> 
> +1: SCTP appears to work over IPv6
> -1: Throughput is terrible
> 
> 
> d@nuc:~/git/netperf$ netperf -6 -H snapon.lab.bufferbloat.net -t
> SCTP_STREAM_MANY
> SCTP 1-TO-MANY STREAM TEST from ::0 (::) port 0 AF_INET6 to
> snapon.lab.bufferbloat.net () port 0 AF_INET6 : demo
> Recv   Send    Send
> Socket Socket  Message  Elapsed
> Size   Size    Size     Time     Throughput
> bytes  bytes   bytes    secs.    10^6bits/sec
> 
> 212992 212992   4096    10.00       0.31
> d@nuc:~/git/netperf$ netperf -6 -H snapon.lab.bufferbloat.net -t TCP_MAERTS
> MIGRATED TCP MAERTS TEST from ::0 (::) port 0 AF_INET6 to
> snapon.lab.bufferbloat.net () port 0 AF_INET6 : demo
> Recv   Send    Send
> Socket Socket  Message  Elapsed
> Size   Size    Size     Time     Throughput
> bytes  bytes   bytes    secs.    10^6bits/sec
> 
>  87380  16384  16384    10.00       7.65
> 
> 



-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

Re: Linux SCTP: what kind of performance should I expect from netperf?

[Daniel Borkmann]

On 03/18/2014 04:16 PM, Jesper Dangaard Brouer wrote:
>
> Hi Daniel,
>
> Can you give some input on this thread?
>
> On Tue, 18 Mar 2014 10:53:40 -0400 Dave Taht <dave.taht@gmail.com> wrote:
>
>> I was curious about sctp's performance characteristics on
>> AQM'd systems... so
>> I built netperf with sctp support, and ran a couple tests on
>> kernel 3.11...
>>
>> +1: SCTP appears to work over IPv6
>> -1: Throughput is terrible

Yes, performance sucks so far (it's a known problem) and
we need to work on it ... ;-)

I presume one reason here could be as well that you need to
do crc32c checksumming on software (what does perf say?).

>> d@nuc:~/git/netperf$ netperf -6 -H snapon.lab.bufferbloat.net -t
>> SCTP_STREAM_MANY
>> SCTP 1-TO-MANY STREAM TEST from ::0 (::) port 0 AF_INET6 to
>> snapon.lab.bufferbloat.net () port 0 AF_INET6 : demo
>> Recv   Send    Send
>> Socket Socket  Message  Elapsed
>> Size   Size    Size     Time     Throughput
>> bytes  bytes   bytes    secs.    10^6bits/sec
>>
>> 212992 212992   4096    10.00       0.31
>> d@nuc:~/git/netperf$ netperf -6 -H snapon.lab.bufferbloat.net -t TCP_MAERTS
>> MIGRATED TCP MAERTS TEST from ::0 (::) port 0 AF_INET6 to
>> snapon.lab.bufferbloat.net () port 0 AF_INET6 : demo
>> Recv   Send    Send
>> Socket Socket  Message  Elapsed
>> Size   Size    Size     Time     Throughput
>> bytes  bytes   bytes    secs.    10^6bits/sec
>>
>>   87380  16384  16384    10.00       7.65

Re: Linux SCTP: what kind of performance should I expect from netperf?

[Dave Taht]

On Tue, Mar 18, 2014 at 11:49 AM, Daniel Borkmann <dborkman@redhat.com> wrote:
> On 03/18/2014 04:16 PM, Jesper Dangaard Brouer wrote:
>>
>>
>> Hi Daniel,
>>
>> Can you give some input on this thread?
>>
>> On Tue, 18 Mar 2014 10:53:40 -0400 Dave Taht <dave.taht@gmail.com> wrote:
>>
>>> I was curious about sctp's performance characteristics on
>>> AQM'd systems... so
>>> I built netperf with sctp support, and ran a couple tests on
>>> kernel 3.11...
>>>
>>> +1: SCTP appears to work over IPv6
>>> -1: Throughput is terrible
>
>
> Yes, performance sucks so far (it's a known problem) and
> we need to work on it ... ;-)


I want to make clear that I don't know diddly about SCTP. I DO grok
TCP fairly well...

I got interested in sctp again after hearing a proposal to make
it easier to fq chunks.

> I presume one reason here could be as well that you need to
> do crc32c checksumming on software (what does perf say?).

Well what I think I see is sctp not opening up a window for packets
in flight (as tcp would with cwnd), and basically ping-ponging
sends and acks over the 14ms RTT:

capture here

http://snapon.lab.bufferbloat.net/~d/sctp/

One of my concerns with all the tcp optimization work over the last 3
years was that it might have broken other protocols and stuff that
plugged into the congestion control api for Linux. For example, my
ledbat kernel module behaves similarly, never getting out of slow
start.

But not having a test for sctp in general (I used netperf, is there a better?).

So if SCTP not ramping up is a known problem I can go back to scratching
my head at the ledbat code.

I WAS quite delighted to see SCTP "just work" over ipv6. :)

>
>
>>> d@nuc:~/git/netperf$ netperf -6 -H snapon.lab.bufferbloat.net -t
>>> SCTP_STREAM_MANY
>>> SCTP 1-TO-MANY STREAM TEST from ::0 (::) port 0 AF_INET6 to
>>> snapon.lab.bufferbloat.net () port 0 AF_INET6 : demo
>>> Recv   Send    Send
>>> Socket Socket  Message  Elapsed
>>> Size   Size    Size     Time     Throughput
>>> bytes  bytes   bytes    secs.    10^6bits/sec
>>>
>>> 212992 212992   4096    10.00       0.31
>>> d@nuc:~/git/netperf$ netperf -6 -H snapon.lab.bufferbloat.net -t
>>> TCP_MAERTS
>>> MIGRATED TCP MAERTS TEST from ::0 (::) port 0 AF_INET6 to
>>> snapon.lab.bufferbloat.net () port 0 AF_INET6 : demo
>>> Recv   Send    Send
>>> Socket Socket  Message  Elapsed
>>> Size   Size    Size     Time     Throughput
>>> bytes  bytes   bytes    secs.    10^6bits/sec
>>>
>>>   87380  16384  16384    10.00       7.65



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: Linux SCTP: what kind of performance should I expect from netperf?

[Dave Taht]

On Tue, Mar 18, 2014 at 9:03 AM, Dave Taht <dave.taht@gmail.com> wrote:
> On Tue, Mar 18, 2014 at 11:49 AM, Daniel Borkmann <dborkman@redhat.com> wrote:
>> On 03/18/2014 04:16 PM, Jesper Dangaard Brouer wrote:
>>>
>>>
>>> Hi Daniel,
>>>
>>> Can you give some input on this thread?

It appears I despaired of SCTP's performance too early,
according to this commit, it just got vastly improved:

https://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=362d52040c71f6e8d8158be48c812d7729cb8df1

way to go daniel!

Have to write some tests for it now!

I do wish someone could get the ledbat linux kernel module up to parity with
at least the osx implementation and get it mainlined.

https://github.com/silviov/TCP-LEDBAT

>>>
>>> On Tue, 18 Mar 2014 10:53:40 -0400 Dave Taht <dave.taht@gmail.com> wrote:
>>>
>>>> I was curious about sctp's performance characteristics on
>>>> AQM'd systems... so
>>>> I built netperf with sctp support, and ran a couple tests on
>>>> kernel 3.11...
>>>>
>>>> +1: SCTP appears to work over IPv6
>>>> -1: Throughput is terrible
>>
>>
>> Yes, performance sucks so far (it's a known problem) and
>> we need to work on it ... ;-)
>
>
> I want to make clear that I don't know diddly about SCTP. I DO grok
> TCP fairly well...
>
> I got interested in sctp again after hearing a proposal to make
> it easier to fq chunks.
>
>> I presume one reason here could be as well that you need to
>> do crc32c checksumming on software (what does perf say?).
>
> Well what I think I see is sctp not opening up a window for packets
> in flight (as tcp would with cwnd), and basically ping-ponging
> sends and acks over the 14ms RTT:
>
> capture here
>
> http://snapon.lab.bufferbloat.net/~d/sctp/
>
> One of my concerns with all the tcp optimization work over the last 3
> years was that it might have broken other protocols and stuff that
> plugged into the congestion control api for Linux. For example, my
> ledbat kernel module behaves similarly, never getting out of slow
> start.
>
> But not having a test for sctp in general (I used netperf, is there a better?).
>
> So if SCTP not ramping up is a known problem I can go back to scratching
> my head at the ledbat code.
>
> I WAS quite delighted to see SCTP "just work" over ipv6. :)
>
>>
>>
>>>> d@nuc:~/git/netperf$ netperf -6 -H snapon.lab.bufferbloat.net -t
>>>> SCTP_STREAM_MANY
>>>> SCTP 1-TO-MANY STREAM TEST from ::0 (::) port 0 AF_INET6 to
>>>> snapon.lab.bufferbloat.net () port 0 AF_INET6 : demo
>>>> Recv   Send    Send
>>>> Socket Socket  Message  Elapsed
>>>> Size   Size    Size     Time     Throughput
>>>> bytes  bytes   bytes    secs.    10^6bits/sec
>>>>
>>>> 212992 212992   4096    10.00       0.31
>>>> d@nuc:~/git/netperf$ netperf -6 -H snapon.lab.bufferbloat.net -t
>>>> TCP_MAERTS
>>>> MIGRATED TCP MAERTS TEST from ::0 (::) port 0 AF_INET6 to
>>>> snapon.lab.bufferbloat.net () port 0 AF_INET6 : demo
>>>> Recv   Send    Send
>>>> Socket Socket  Message  Elapsed
>>>> Size   Size    Size     Time     Throughput
>>>> bytes  bytes   bytes    secs.    10^6bits/sec
>>>>
>>>>   87380  16384  16384    10.00       7.65
>
>
>
> --
> Dave Täht
>
> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html



-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article

Re: Linux SCTP: what kind of performance should I expect from netperf?

[Daniel Borkmann]

On 04/16/2014 03:23 AM, Dave Taht wrote:
...
> It appears I despaired of SCTP's performance too early,
> according to this commit, it just got vastly improved:
>
> https://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=362d52040c71f6e8d8158be48c812d7729cb8df1
>
> way to go daniel!
>
> Have to write some tests for it now!

Ok, thanks. That commit needs to be reworked in a different way
by the original authors or us so that it won't hurt performance.
The idea from the original commit is good, but the overall
implementation is not yet fully there it seems.

> I do wish someone could get the ledbat linux kernel module up to parity with
> at least the osx implementation and get it mainlined.
>
> https://github.com/silviov/TCP-LEDBAT

Sounds interesting!

Re: Linux SCTP: what kind of performance should I expect from netperf?

[Dave Taht]

On Wed, Apr 16, 2014 at 12:33 AM, Daniel Borkmann <dborkman@redhat.com> wrote:
> On 04/16/2014 03:23 AM, Dave Taht wrote:
> ...
>
>> It appears I despaired of SCTP's performance too early,
>> according to this commit, it just got vastly improved:
>>
>>
>> https://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=362d52040c71f6e8d8158be48c812d7729cb8df1
>>
>> way to go daniel!
>>
>> Have to write some tests for it now!
>
>
> Ok, thanks. That commit needs to be reworked in a different way
> by the original authors or us so that it won't hurt performance.
> The idea from the original commit is good, but the overall
> implementation is not yet fully there it seems.
>
>
>> I do wish someone could get the ledbat linux kernel module up to parity
>> with
>> at least the osx implementation and get it mainlined.
>>
>> https://github.com/silviov/TCP-LEDBAT
>
>
> Sounds interesting!


If it worked correctly... what appeared to happen at the time I'd
tested it is that it never really got into ramping up its window.

You could look at it and say "oh, yes, it's scavanging!" but no, it
was really stuck
at low bandwidth and not taking advantage of the path. Your fix to
sctp reminded me of that behavior of this code.

I figured it would be a fun weekend project to make tcp-ledbat on linux work,
something like 120 weekends ago. (The code has no owner anymore.)

certainly it would be awesome and useful if someone could make it work
- and at the time I last looked at it, which was pre-BQL, the linux
tcp stack was so messed up that I'd thought maybe the problem was
elsewhere. And there wasn't a lot of other ledbat code to compare it against,
which has changed.

But it does look like the code has a bug, see below.

netperf has the ability to select congestion control algos on the fly,
we have several tests in the netperf-wrapper suite that test the known
working tcp cc algorithms (reno, cubic, westwood, lp) against each other,
and I'd dearly like a test against uTP. and support for "SCTP_MAERTS"
in netperf itself.

syntax looks like this - 100mbit empty path:

root@comcast-gw:~# netperf -H 172.26.4.1 -t omni -- -K ledbat,ledbat
OMNI Send TEST from 0.0.0.0 () port 0 AF_INET to 172.26.4.1 () port 0
AF_INET : demo
Local       Remote      Local  Elapsed Throughput Throughput
Send Socket Recv Socket Send   Time               Units
Size        Size        Size   (sec)
Final       Final
65536       87380       65536  10.86   0.31       10^6bits/s

root@comcast-gw:~# netperf -H 172.26.4.1 -t omni -- -K lp,lp
OMNI Send TEST from 0.0.0.0 () port 0 AF_INET to 172.26.4.1 () port 0
AF_INET : demo
Local       Remote      Local  Elapsed Throughput Throughput
Send Socket Recv Socket Send   Time               Units
Size        Size        Size   (sec)
Final       Final
292128      650496      65536  10.02   94.12      10^6bits/s

so it's stuck at well below the capability of this path.

from a theoretical perspective, ledbat originally aimed at a target
delay of 25ms before backing off but that was unachievable in practice,
and they went to 100ms.

Now that we can reliably via various aqm techniques achieve target
delays well below that it would be cool to experiment with a delay based tcp
that could try to back off at 25ms or below...

(tom sawyer, trying to make painting a fence more attractive here)

-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article