From: "Thomas Rosenstein" <thomas.rosenstein@creamfinance.com>
To: "Jesper Dangaard Brouer" <brouer@redhat.com>
Cc: Bufferbloat <bloat@lists.bufferbloat.net>
Subject: Re: [Bloat] Router congestion, slow ping/ack times with kernel 5.4.60
Date: Mon, 16 Nov 2020 13:49:32 +0100 [thread overview]
Message-ID: <F3BDC5B8-6A84-4004-AFA0-F33A9D9EF81D@creamfinance.com> (raw)
In-Reply-To: <20201116133438.19459da5@carbon>
On 16 Nov 2020, at 13:34, Jesper Dangaard Brouer wrote:
> On Wed, 04 Nov 2020 16:23:12 +0100
> Thomas Rosenstein via Bloat <bloat@lists.bufferbloat.net> wrote:
>
> [...]
>> I have multiple routers which connect to multiple upstream providers,
>> I
>> have noticed a high latency shift in icmp (and generally all
>> connection)
>> if I run b2 upload-file --threads 40 (and I can reproduce this)
>>
>> What options do I have to analyze why this happens?
>>
>> General Info:
>>
>> Routers are connected between each other with 10G Mellanox Connect-X
>> cards via 10G SPF+ DAC cables via a 10G Switch from fs.com
>> Latency generally is around 0.18 ms between all routers (4).
>> Throughput is 9.4 Gbit/s with 0 retransmissions when tested with
>> iperf3.
>> 2 of the 4 routers are connected upstream with a 1G connection
>> (separate
>> port, same network card)
>> All routers have the full internet routing tables, i.e. 80k entries
>> for
>> IPv6 and 830k entries for IPv4
>> Conntrack is disabled (-j NOTRACK)
>> Kernel 5.4.60 (custom)
>> 2x Xeon X5670 @ 2.93 Ghz
>
> I think I have spotted your problem... This CPU[1] Xeon X5670 is more
> than 10 years old! It basically corresponds to the machines I used
> for
> my presentation at LinuxCon 2009 see slides[2]. Only with large
> frames
> and with massive scaling across all CPUs was I able to get close to
> 10Gbit/s through these machines. And on top I had to buy low-latency
> RAM memory-blocks to make it happen.
>
> As you can see on my slides[2], memory bandwidth and PCIe speeds was
> at
> the limit for making it possible on the hardware level. I had to run
> DDR3 memory at 1333MHz and tune the QuickPath Interconnect (QPI) to
> 6.4GT/s (default 4.8GT/s).
>
> This generation Motherboards had both PCIe gen-1 and gen-2 slots.
> Only
> the PCIe gen-2 slots had barely enough bandwidth. Maybe you
> physically
> placed NIC in PCIe gen-1 slot?
>
> On top of this, you also have a NUMA system, 2x Xeon X5670, which can
> result is A LOT of "funny" issue, that is really hard to
> troubleshoot...
>
Yes, I'm aware of the limits of what to expect, but as we agree 60 tcp
streams with not even 200 Mbits shouldn't overload the PCIex bus or the
cpus.
Also, don't forget, no issues with Kernel 3.10.
PCI slot is a Gen2, x8, so more than enough bandwidth there luckily ;)
But yes, they are quite old...
>
> [1]
> https://ark.intel.com/content/www/us/en/ark/products/47920/intel-xeon-processor-x5670-12m-cache-2-93-ghz-6-40-gt-s-intel-qpi.html
>
> [2]
> https://people.netfilter.org/hawk/presentations/LinuxCon2009/LinuxCon2009_JesperDangaardBrouer_final.pdf
>
> --
> Best regards,
> Jesper Dangaard Brouer
> MSc.CS, Principal Kernel Engineer at Red Hat
> LinkedIn: http://www.linkedin.com/in/brouer
prev parent reply other threads:[~2020-11-16 12:49 UTC|newest]
Thread overview: 47+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-04 15:23 Thomas Rosenstein
2020-11-04 16:10 ` Toke Høiland-Jørgensen
2020-11-04 16:24 ` Thomas Rosenstein
2020-11-05 0:10 ` Toke Høiland-Jørgensen
2020-11-05 8:48 ` Thomas Rosenstein
2020-11-05 11:21 ` Toke Høiland-Jørgensen
2020-11-05 12:22 ` Thomas Rosenstein
2020-11-05 12:38 ` Toke Høiland-Jørgensen
2020-11-05 12:41 ` Thomas Rosenstein
2020-11-05 12:47 ` Toke Høiland-Jørgensen
2020-11-05 13:33 ` Jesper Dangaard Brouer
2020-11-06 8:48 ` Thomas Rosenstein
2020-11-06 10:53 ` Jesper Dangaard Brouer
2020-11-06 9:18 ` Thomas Rosenstein
2020-11-06 11:18 ` Jesper Dangaard Brouer
2020-11-06 11:37 ` Thomas Rosenstein
2020-11-06 11:45 ` Toke Høiland-Jørgensen
2020-11-06 12:01 ` Thomas Rosenstein
2020-11-06 12:53 ` Jesper Dangaard Brouer
2020-11-06 14:13 ` Jesper Dangaard Brouer
2020-11-06 17:04 ` Thomas Rosenstein
2020-11-06 20:19 ` Jesper Dangaard Brouer
2020-11-07 12:37 ` Thomas Rosenstein
2020-11-07 12:40 ` Jan Ceuleers
2020-11-07 12:43 ` Thomas Rosenstein
2020-11-07 13:00 ` Thomas Rosenstein
2020-11-09 8:24 ` Jesper Dangaard Brouer
2020-11-09 10:09 ` Thomas Rosenstein
2020-11-09 11:40 ` Jesper Dangaard Brouer
2020-11-09 11:51 ` Toke Høiland-Jørgensen
2020-11-09 12:25 ` Thomas Rosenstein
2020-11-09 14:33 ` Thomas Rosenstein
2020-11-12 10:05 ` Jesper Dangaard Brouer
2020-11-12 11:26 ` Thomas Rosenstein
2020-11-12 13:31 ` Jesper Dangaard Brouer
2020-11-12 13:42 ` Thomas Rosenstein
2020-11-12 15:42 ` Jesper Dangaard Brouer
2020-11-13 6:31 ` Thomas Rosenstein
2020-11-16 11:56 ` Jesper Dangaard Brouer
2020-11-16 12:05 ` Thomas Rosenstein
2020-11-09 16:39 ` Thomas Rosenstein
2020-11-07 13:33 ` Thomas Rosenstein
2020-11-07 16:46 ` Jesper Dangaard Brouer
2020-11-07 17:01 ` Thomas Rosenstein
2020-11-07 17:26 ` Sebastian Moeller
2020-11-16 12:34 ` Jesper Dangaard Brouer
2020-11-16 12:49 ` Thomas Rosenstein [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://lists.bufferbloat.net/postorius/lists/bloat.lists.bufferbloat.net/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=F3BDC5B8-6A84-4004-AFA0-F33A9D9EF81D@creamfinance.com \
--to=thomas.rosenstein@creamfinance.com \
--cc=bloat@lists.bufferbloat.net \
--cc=brouer@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox