Re: oprofiling is much saner looking now with rc6-smoketest

[Simon Barber <simon@superduper.net>]

Why is conntrack even getting involved?

Simon

On 08/30/2011 06:01 PM, Rick Jones wrote:
> On 08/30/2011 05:32 PM, Dave Taht wrote:
>> I get about 190Mbit/sec from netperf now, on GigE, with oprofiling
>> enabled, driver buffers of 4, txqueue of 8, cerowrt default iptables
>> rules, AND web10g patched into kernel 3.0.3.
>>
>> This is much saner than rc3, and judging from the csum_partial and
>> copy_user being roughly equal, there isn't much left to be gained...
>  >
>> Nice work.
>>
>> (Without oprofiling, and without web10g and with tcp cubic I can get
>> past 250Mbit)
>>
>>
>> CPU: MIPS 24K, speed 0 MHz (estimated)
>> Counted INSTRUCTIONS events (Instructions completed) with a unit mask
>> of 0x00 (No unit mask) count 100000
>> samples % app name symbol name
>> -------------------------------------------------------------------------------
>>
>> 17277 13.8798 vmlinux csum_partial
>> 17277 100.000 vmlinux csum_partial [self]
>> -------------------------------------------------------------------------------
>>
>> 16607 13.3415 vmlinux __copy_user
>> 16607 100.000 vmlinux __copy_user [self]
>> -------------------------------------------------------------------------------
>>
>> 11913 9.5705 ip_tables /ip_tables
>> 11913 100.000 ip_tables /ip_tables [self]
>> -------------------------------------------------------------------------------
>>
>> 8949 7.1893 nf_conntrack /nf_conntrack
>> 8949 100.000 nf_conntrack /nf_conntrack [self]
>>
>> In this case I was going from laptop - gige - through another
>> rc6-smoketest router - to_this_box's internal lan port.
>>
>> It bugs me that iptables and conntrack eat so much cpu for what
>> is an internal-only connection, e.g. one that
>> doesn't need conntracking.
>
> The csum_partial is a bit surprising - I thought every NIC and its dog
> offered CKO these days - or is that something happening with
> ip_tables/contrack? I also thought that Linux used an integrated
> copy/checksum in at least one direction, or did that go away when CKO
> became prevalent?
>
> If this is inbound, and there is just plain checksumming and not
> anything funny from conntrack, I would have expected checksum to be much
> larger than copy. Checksum (in the inbound direction) will take the
> cache misses and the copy would not. Unless... the data cache of the
> processor is getting completely trashed - say from the netserver running
> on the router not keeping up with the inbound data fully and so the copy
> gets "far away" from the checksum verification.
>
> Does perf/perf_events (whatever the followon to perfmon2 is called) have
> support for the CPU used in the device? (Assuming it even has a PMU to
> be queried in the first place)
>
>> That said, I understand that people like their statistics, and me,
>> I'm trying to make split-tcp work better, ultimately, one day....
>>
>> I'm going to rerun this without the fw rules next.
>
> It would be interesting to see if the csum time goes away. Long ago and
> far away when I was beating on a 32-core system with aggregate netperf
> TCP_RR and enabling or not FW rules, conntrack had a non-trivial effect
> indeed on performance.
>
> http://markmail.org/message/exjtzel7vq2ugt66#query:netdev%20conntrack%20rick%20jones%2032%20netperf+page:1+mid:s5v5kylvmlfrpb7a+state:results
>
>
> I think will get to the start of that thread. The subject is '32 core
> net-next stack/netfilter "scaling"'
>
> rick jones
> _______________________________________________
> Bloat-devel mailing list
> Bloat-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat-devel