* Linux SCTP: what kind of performance should I expect from netperf? @ 2014-03-18 14:53 Dave Taht 2014-03-18 15:16 ` Jesper Dangaard Brouer 0 siblings, 1 reply; 7+ messages in thread From: Dave Taht @ 2014-03-18 14:53 UTC (permalink / raw) To: bloat-devel, netperf-dev I was curious about sctp's performance characteristics on AQM'd systems... so I built netperf with sctp support, and ran a couple tests on kernel 3.11... +1: SCTP appears to work over IPv6 -1: Throughput is terrible d@nuc:~/git/netperf$ netperf -6 -H snapon.lab.bufferbloat.net -t SCTP_STREAM_MANY SCTP 1-TO-MANY STREAM TEST from ::0 (::) port 0 AF_INET6 to snapon.lab.bufferbloat.net () port 0 AF_INET6 : demo Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 212992 212992 4096 10.00 0.31 d@nuc:~/git/netperf$ netperf -6 -H snapon.lab.bufferbloat.net -t TCP_MAERTS MIGRATED TCP MAERTS TEST from ::0 (::) port 0 AF_INET6 to snapon.lab.bufferbloat.net () port 0 AF_INET6 : demo Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 16384 16384 10.00 7.65 -- Dave Täht Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Linux SCTP: what kind of performance should I expect from netperf? 2014-03-18 14:53 Linux SCTP: what kind of performance should I expect from netperf? Dave Taht @ 2014-03-18 15:16 ` Jesper Dangaard Brouer 2014-03-18 15:49 ` Daniel Borkmann 0 siblings, 1 reply; 7+ messages in thread From: Jesper Dangaard Brouer @ 2014-03-18 15:16 UTC (permalink / raw) To: Dave Taht, Daniel Borkmann; +Cc: netperf-dev, bloat-devel Hi Daniel, Can you give some input on this thread? On Tue, 18 Mar 2014 10:53:40 -0400 Dave Taht <dave.taht@gmail.com> wrote: > I was curious about sctp's performance characteristics on > AQM'd systems... so > I built netperf with sctp support, and ran a couple tests on > kernel 3.11... > > +1: SCTP appears to work over IPv6 > -1: Throughput is terrible > > > d@nuc:~/git/netperf$ netperf -6 -H snapon.lab.bufferbloat.net -t > SCTP_STREAM_MANY > SCTP 1-TO-MANY STREAM TEST from ::0 (::) port 0 AF_INET6 to > snapon.lab.bufferbloat.net () port 0 AF_INET6 : demo > Recv Send Send > Socket Socket Message Elapsed > Size Size Size Time Throughput > bytes bytes bytes secs. 10^6bits/sec > > 212992 212992 4096 10.00 0.31 > d@nuc:~/git/netperf$ netperf -6 -H snapon.lab.bufferbloat.net -t TCP_MAERTS > MIGRATED TCP MAERTS TEST from ::0 (::) port 0 AF_INET6 to > snapon.lab.bufferbloat.net () port 0 AF_INET6 : demo > Recv Send Send > Socket Socket Message Elapsed > Size Size Size Time Throughput > bytes bytes bytes secs. 10^6bits/sec > > 87380 16384 16384 10.00 7.65 > > -- Best regards, Jesper Dangaard Brouer MSc.CS, Sr. Network Kernel Developer at Red Hat Author of http://www.iptv-analyzer.org LinkedIn: http://www.linkedin.com/in/brouer ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Linux SCTP: what kind of performance should I expect from netperf? 2014-03-18 15:16 ` Jesper Dangaard Brouer @ 2014-03-18 15:49 ` Daniel Borkmann 2014-03-18 16:03 ` Dave Taht 0 siblings, 1 reply; 7+ messages in thread From: Daniel Borkmann @ 2014-03-18 15:49 UTC (permalink / raw) To: Jesper Dangaard Brouer; +Cc: netperf-dev, bloat-devel On 03/18/2014 04:16 PM, Jesper Dangaard Brouer wrote: > > Hi Daniel, > > Can you give some input on this thread? > > On Tue, 18 Mar 2014 10:53:40 -0400 Dave Taht <dave.taht@gmail.com> wrote: > >> I was curious about sctp's performance characteristics on >> AQM'd systems... so >> I built netperf with sctp support, and ran a couple tests on >> kernel 3.11... >> >> +1: SCTP appears to work over IPv6 >> -1: Throughput is terrible Yes, performance sucks so far (it's a known problem) and we need to work on it ... ;-) I presume one reason here could be as well that you need to do crc32c checksumming on software (what does perf say?). >> d@nuc:~/git/netperf$ netperf -6 -H snapon.lab.bufferbloat.net -t >> SCTP_STREAM_MANY >> SCTP 1-TO-MANY STREAM TEST from ::0 (::) port 0 AF_INET6 to >> snapon.lab.bufferbloat.net () port 0 AF_INET6 : demo >> Recv Send Send >> Socket Socket Message Elapsed >> Size Size Size Time Throughput >> bytes bytes bytes secs. 10^6bits/sec >> >> 212992 212992 4096 10.00 0.31 >> d@nuc:~/git/netperf$ netperf -6 -H snapon.lab.bufferbloat.net -t TCP_MAERTS >> MIGRATED TCP MAERTS TEST from ::0 (::) port 0 AF_INET6 to >> snapon.lab.bufferbloat.net () port 0 AF_INET6 : demo >> Recv Send Send >> Socket Socket Message Elapsed >> Size Size Size Time Throughput >> bytes bytes bytes secs. 10^6bits/sec >> >> 87380 16384 16384 10.00 7.65 ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Linux SCTP: what kind of performance should I expect from netperf? 2014-03-18 15:49 ` Daniel Borkmann @ 2014-03-18 16:03 ` Dave Taht 2014-04-16 1:23 ` Dave Taht 0 siblings, 1 reply; 7+ messages in thread From: Dave Taht @ 2014-03-18 16:03 UTC (permalink / raw) To: Daniel Borkmann; +Cc: bloat-devel On Tue, Mar 18, 2014 at 11:49 AM, Daniel Borkmann <dborkman@redhat.com> wrote: > On 03/18/2014 04:16 PM, Jesper Dangaard Brouer wrote: >> >> >> Hi Daniel, >> >> Can you give some input on this thread? >> >> On Tue, 18 Mar 2014 10:53:40 -0400 Dave Taht <dave.taht@gmail.com> wrote: >> >>> I was curious about sctp's performance characteristics on >>> AQM'd systems... so >>> I built netperf with sctp support, and ran a couple tests on >>> kernel 3.11... >>> >>> +1: SCTP appears to work over IPv6 >>> -1: Throughput is terrible > > > Yes, performance sucks so far (it's a known problem) and > we need to work on it ... ;-) I want to make clear that I don't know diddly about SCTP. I DO grok TCP fairly well... I got interested in sctp again after hearing a proposal to make it easier to fq chunks. > I presume one reason here could be as well that you need to > do crc32c checksumming on software (what does perf say?). Well what I think I see is sctp not opening up a window for packets in flight (as tcp would with cwnd), and basically ping-ponging sends and acks over the 14ms RTT: capture here http://snapon.lab.bufferbloat.net/~d/sctp/ One of my concerns with all the tcp optimization work over the last 3 years was that it might have broken other protocols and stuff that plugged into the congestion control api for Linux. For example, my ledbat kernel module behaves similarly, never getting out of slow start. But not having a test for sctp in general (I used netperf, is there a better?). So if SCTP not ramping up is a known problem I can go back to scratching my head at the ledbat code. I WAS quite delighted to see SCTP "just work" over ipv6. :) > > >>> d@nuc:~/git/netperf$ netperf -6 -H snapon.lab.bufferbloat.net -t >>> SCTP_STREAM_MANY >>> SCTP 1-TO-MANY STREAM TEST from ::0 (::) port 0 AF_INET6 to >>> snapon.lab.bufferbloat.net () port 0 AF_INET6 : demo >>> Recv Send Send >>> Socket Socket Message Elapsed >>> Size Size Size Time Throughput >>> bytes bytes bytes secs. 10^6bits/sec >>> >>> 212992 212992 4096 10.00 0.31 >>> d@nuc:~/git/netperf$ netperf -6 -H snapon.lab.bufferbloat.net -t >>> TCP_MAERTS >>> MIGRATED TCP MAERTS TEST from ::0 (::) port 0 AF_INET6 to >>> snapon.lab.bufferbloat.net () port 0 AF_INET6 : demo >>> Recv Send Send >>> Socket Socket Message Elapsed >>> Size Size Size Time Throughput >>> bytes bytes bytes secs. 10^6bits/sec >>> >>> 87380 16384 16384 10.00 7.65 -- Dave Täht Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Linux SCTP: what kind of performance should I expect from netperf? 2014-03-18 16:03 ` Dave Taht @ 2014-04-16 1:23 ` Dave Taht 2014-04-16 7:33 ` Daniel Borkmann 0 siblings, 1 reply; 7+ messages in thread From: Dave Taht @ 2014-04-16 1:23 UTC (permalink / raw) To: Daniel Borkmann; +Cc: bloat-devel On Tue, Mar 18, 2014 at 9:03 AM, Dave Taht <dave.taht@gmail.com> wrote: > On Tue, Mar 18, 2014 at 11:49 AM, Daniel Borkmann <dborkman@redhat.com> wrote: >> On 03/18/2014 04:16 PM, Jesper Dangaard Brouer wrote: >>> >>> >>> Hi Daniel, >>> >>> Can you give some input on this thread? It appears I despaired of SCTP's performance too early, according to this commit, it just got vastly improved: https://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=362d52040c71f6e8d8158be48c812d7729cb8df1 way to go daniel! Have to write some tests for it now! I do wish someone could get the ledbat linux kernel module up to parity with at least the osx implementation and get it mainlined. https://github.com/silviov/TCP-LEDBAT >>> >>> On Tue, 18 Mar 2014 10:53:40 -0400 Dave Taht <dave.taht@gmail.com> wrote: >>> >>>> I was curious about sctp's performance characteristics on >>>> AQM'd systems... so >>>> I built netperf with sctp support, and ran a couple tests on >>>> kernel 3.11... >>>> >>>> +1: SCTP appears to work over IPv6 >>>> -1: Throughput is terrible >> >> >> Yes, performance sucks so far (it's a known problem) and >> we need to work on it ... ;-) > > > I want to make clear that I don't know diddly about SCTP. I DO grok > TCP fairly well... > > I got interested in sctp again after hearing a proposal to make > it easier to fq chunks. > >> I presume one reason here could be as well that you need to >> do crc32c checksumming on software (what does perf say?). > > Well what I think I see is sctp not opening up a window for packets > in flight (as tcp would with cwnd), and basically ping-ponging > sends and acks over the 14ms RTT: > > capture here > > http://snapon.lab.bufferbloat.net/~d/sctp/ > > One of my concerns with all the tcp optimization work over the last 3 > years was that it might have broken other protocols and stuff that > plugged into the congestion control api for Linux. For example, my > ledbat kernel module behaves similarly, never getting out of slow > start. > > But not having a test for sctp in general (I used netperf, is there a better?). > > So if SCTP not ramping up is a known problem I can go back to scratching > my head at the ledbat code. > > I WAS quite delighted to see SCTP "just work" over ipv6. :) > >> >> >>>> d@nuc:~/git/netperf$ netperf -6 -H snapon.lab.bufferbloat.net -t >>>> SCTP_STREAM_MANY >>>> SCTP 1-TO-MANY STREAM TEST from ::0 (::) port 0 AF_INET6 to >>>> snapon.lab.bufferbloat.net () port 0 AF_INET6 : demo >>>> Recv Send Send >>>> Socket Socket Message Elapsed >>>> Size Size Size Time Throughput >>>> bytes bytes bytes secs. 10^6bits/sec >>>> >>>> 212992 212992 4096 10.00 0.31 >>>> d@nuc:~/git/netperf$ netperf -6 -H snapon.lab.bufferbloat.net -t >>>> TCP_MAERTS >>>> MIGRATED TCP MAERTS TEST from ::0 (::) port 0 AF_INET6 to >>>> snapon.lab.bufferbloat.net () port 0 AF_INET6 : demo >>>> Recv Send Send >>>> Socket Socket Message Elapsed >>>> Size Size Size Time Throughput >>>> bytes bytes bytes secs. 10^6bits/sec >>>> >>>> 87380 16384 16384 10.00 7.65 > > > > -- > Dave Täht > > Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html -- Dave Täht NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Linux SCTP: what kind of performance should I expect from netperf? 2014-04-16 1:23 ` Dave Taht @ 2014-04-16 7:33 ` Daniel Borkmann 2014-04-16 15:30 ` Dave Taht 0 siblings, 1 reply; 7+ messages in thread From: Daniel Borkmann @ 2014-04-16 7:33 UTC (permalink / raw) To: Dave Taht; +Cc: bloat-devel On 04/16/2014 03:23 AM, Dave Taht wrote: ... > It appears I despaired of SCTP's performance too early, > according to this commit, it just got vastly improved: > > https://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=362d52040c71f6e8d8158be48c812d7729cb8df1 > > way to go daniel! > > Have to write some tests for it now! Ok, thanks. That commit needs to be reworked in a different way by the original authors or us so that it won't hurt performance. The idea from the original commit is good, but the overall implementation is not yet fully there it seems. > I do wish someone could get the ledbat linux kernel module up to parity with > at least the osx implementation and get it mainlined. > > https://github.com/silviov/TCP-LEDBAT Sounds interesting! ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Linux SCTP: what kind of performance should I expect from netperf? 2014-04-16 7:33 ` Daniel Borkmann @ 2014-04-16 15:30 ` Dave Taht 0 siblings, 0 replies; 7+ messages in thread From: Dave Taht @ 2014-04-16 15:30 UTC (permalink / raw) To: Daniel Borkmann; +Cc: bloat-devel On Wed, Apr 16, 2014 at 12:33 AM, Daniel Borkmann <dborkman@redhat.com> wrote: > On 04/16/2014 03:23 AM, Dave Taht wrote: > ... > >> It appears I despaired of SCTP's performance too early, >> according to this commit, it just got vastly improved: >> >> >> https://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=362d52040c71f6e8d8158be48c812d7729cb8df1 >> >> way to go daniel! >> >> Have to write some tests for it now! > > > Ok, thanks. That commit needs to be reworked in a different way > by the original authors or us so that it won't hurt performance. > The idea from the original commit is good, but the overall > implementation is not yet fully there it seems. > > >> I do wish someone could get the ledbat linux kernel module up to parity >> with >> at least the osx implementation and get it mainlined. >> >> https://github.com/silviov/TCP-LEDBAT > > > Sounds interesting! If it worked correctly... what appeared to happen at the time I'd tested it is that it never really got into ramping up its window. You could look at it and say "oh, yes, it's scavanging!" but no, it was really stuck at low bandwidth and not taking advantage of the path. Your fix to sctp reminded me of that behavior of this code. I figured it would be a fun weekend project to make tcp-ledbat on linux work, something like 120 weekends ago. (The code has no owner anymore.) certainly it would be awesome and useful if someone could make it work - and at the time I last looked at it, which was pre-BQL, the linux tcp stack was so messed up that I'd thought maybe the problem was elsewhere. And there wasn't a lot of other ledbat code to compare it against, which has changed. But it does look like the code has a bug, see below. netperf has the ability to select congestion control algos on the fly, we have several tests in the netperf-wrapper suite that test the known working tcp cc algorithms (reno, cubic, westwood, lp) against each other, and I'd dearly like a test against uTP. and support for "SCTP_MAERTS" in netperf itself. syntax looks like this - 100mbit empty path: root@comcast-gw:~# netperf -H 172.26.4.1 -t omni -- -K ledbat,ledbat OMNI Send TEST from 0.0.0.0 () port 0 AF_INET to 172.26.4.1 () port 0 AF_INET : demo Local Remote Local Elapsed Throughput Throughput Send Socket Recv Socket Send Time Units Size Size Size (sec) Final Final 65536 87380 65536 10.86 0.31 10^6bits/s root@comcast-gw:~# netperf -H 172.26.4.1 -t omni -- -K lp,lp OMNI Send TEST from 0.0.0.0 () port 0 AF_INET to 172.26.4.1 () port 0 AF_INET : demo Local Remote Local Elapsed Throughput Throughput Send Socket Recv Socket Send Time Units Size Size Size (sec) Final Final 292128 650496 65536 10.02 94.12 10^6bits/s so it's stuck at well below the capability of this path. from a theoretical perspective, ledbat originally aimed at a target delay of 25ms before backing off but that was unachievable in practice, and they went to 100ms. Now that we can reliably via various aqm techniques achieve target delays well below that it would be cool to experiment with a delay based tcp that could try to back off at 25ms or below... (tom sawyer, trying to make painting a fence more attractive here) -- Dave Täht NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2014-04-16 15:30 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-03-18 14:53 Linux SCTP: what kind of performance should I expect from netperf? Dave Taht 2014-03-18 15:16 ` Jesper Dangaard Brouer 2014-03-18 15:49 ` Daniel Borkmann 2014-03-18 16:03 ` Dave Taht 2014-04-16 1:23 ` Dave Taht 2014-04-16 7:33 ` Daniel Borkmann 2014-04-16 15:30 ` Dave Taht
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox