General list for discussing Bufferbloat
 help / color / mirror / Atom feed
From: "Thomas Rosenstein" <thomas.rosenstein@creamfinance.com>
To: "Jesper Dangaard Brouer" <brouer@redhat.com>
Cc: Bloat <bloat@lists.bufferbloat.net>
Subject: Re: [Bloat] Router congestion, slow ping/ack times with kernel 5.4.60
Date: Sat, 07 Nov 2020 13:37:01 +0100	[thread overview]
Message-ID: <F2A84998-D280-45E2-A38E-62C76D96676C@creamfinance.com> (raw)
In-Reply-To: <20201106211940.4c30ccc9@carbon>



On 6 Nov 2020, at 21:19, Jesper Dangaard Brouer wrote:

> On Fri, 06 Nov 2020 18:04:49 +0100
> "Thomas Rosenstein" <thomas.rosenstein@creamfinance.com> wrote:
>
>> On 6 Nov 2020, at 15:13, Jesper Dangaard Brouer wrote:
>>
>>
>> I'm using ping on IPv4, but I'll try to see if IPv6 makes any
>> difference!
>
> I think you misunderstand me.  I'm not asking you to use ping6. The
> gobgpd daemon updates will both update IPv4 and IPv6 routes, right.
> Updating IPv6 routes are more problematic than IPv4 routes.  The IPv6
> route tables update can potentially stall softirq from running, which
> was the latency tool was measuring... and it did show some outliers.

yes I did, I assumed the latency would be introduced in the traffic path 
by the lock.
Nonetheless, I tested it and no difference :)

>
>
>>> Have you tried to use 'perf record' to observe that is happening on
>>> the system while these latency incidents happen?  (let me know if 
>>> you
>>> want some cmdline hints)
>>
>> Haven't tried this yet. If you have some hints what events to monitor
>> I'll take them!
>
> Okay to record everything (-a) on the system and save call-graph (-g),
> and run for 5 seconds (via profiling the sleep function).
>
>  # perf record -g -a  sleep 5
>
> To view the result the simply use the 'perf report', but likely you
> want to use option --no-children as you are profiling the kernel (and
> not a userspace program you want to have grouped 'children' by).  I
> also include the CPU column via '--sort cpu,comm,dso,symbol' and you
> can select/zoom-in-on a specific CPU via '-C zero-indexed-cpu-num'.
>
>  # perf report --sort cpu,comm,dso,symbol --no-children
>
> When we ask you to provide the output, you can use the --stdio option,
> and provide txt-info via a pastebin link as it is very long.

Here is the output from kernel 3.10_1127 (I updated to the really newest 
in that branch):  https://pastebin.com/5mxirXPw
Here is the output from kernel 5.9.4: https://pastebin.com/KDZ2Ei2F

I have noticed that the delays are directly related to the traffic 
flows, see below.

These tests are WITHOUT gobgpd running, so no updates to the route 
table, but the route tables are fully populated.
Also, it's ONLY outgoing traffic, the return packets are coming in on 
another router.

I have then cleared the routing tables, and the issue persists, table 
has only 78 entries.

40 threads -> sometimes higher rtt times: https://pastebin.com/Y9nd0h4h
60 threads -> always high rtt times: https://pastebin.com/JFvhtLrH

So it definitly gets worse the more connections there are.

I have also tried to reproduce the issue with the kernel on a virtual 
hyper-v machine, there I don't have any adverse effects.
But it's not 100% the same, since MASQ happens on it .. will restructure 
a bit to get a similar representation

I also suspected now that -j NOTRACK would be an issue, removed that 
too, no change. (it's anyways async routing)

Additionally I have quit all applications except for sshd, no change!



>
> -- 
> Best regards,
>   Jesper Dangaard Brouer
>   MSc.CS, Principal Kernel Engineer at Red Hat
>   LinkedIn: http://www.linkedin.com/in/brouer

  reply	other threads:[~2020-11-07 12:37 UTC|newest]

Thread overview: 47+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-11-04 15:23 Thomas Rosenstein
2020-11-04 16:10 ` Toke Høiland-Jørgensen
2020-11-04 16:24   ` Thomas Rosenstein
2020-11-05  0:10     ` Toke Høiland-Jørgensen
2020-11-05  8:48       ` Thomas Rosenstein
2020-11-05 11:21         ` Toke Høiland-Jørgensen
2020-11-05 12:22           ` Thomas Rosenstein
2020-11-05 12:38             ` Toke Høiland-Jørgensen
2020-11-05 12:41               ` Thomas Rosenstein
2020-11-05 12:47                 ` Toke Høiland-Jørgensen
2020-11-05 13:33             ` Jesper Dangaard Brouer
2020-11-06  8:48               ` Thomas Rosenstein
2020-11-06 10:53                 ` Jesper Dangaard Brouer
2020-11-06  9:18               ` Thomas Rosenstein
2020-11-06 11:18                 ` Jesper Dangaard Brouer
2020-11-06 11:37                   ` Thomas Rosenstein
2020-11-06 11:45                     ` Toke Høiland-Jørgensen
2020-11-06 12:01                       ` Thomas Rosenstein
2020-11-06 12:53                       ` Jesper Dangaard Brouer
2020-11-06 14:13                         ` Jesper Dangaard Brouer
2020-11-06 17:04                           ` Thomas Rosenstein
2020-11-06 20:19                             ` Jesper Dangaard Brouer
2020-11-07 12:37                               ` Thomas Rosenstein [this message]
2020-11-07 12:40                                 ` Jan Ceuleers
2020-11-07 12:43                                   ` Thomas Rosenstein
2020-11-07 13:00                                   ` Thomas Rosenstein
2020-11-09  8:24                                     ` Jesper Dangaard Brouer
2020-11-09 10:09                                       ` Thomas Rosenstein
2020-11-09 11:40                                         ` Jesper Dangaard Brouer
2020-11-09 11:51                                           ` Toke Høiland-Jørgensen
2020-11-09 12:25                                           ` Thomas Rosenstein
2020-11-09 14:33                                           ` Thomas Rosenstein
2020-11-12 10:05                                             ` Jesper Dangaard Brouer
2020-11-12 11:26                                               ` Thomas Rosenstein
2020-11-12 13:31                                                 ` Jesper Dangaard Brouer
2020-11-12 13:42                                                   ` Thomas Rosenstein
2020-11-12 15:42                                                     ` Jesper Dangaard Brouer
2020-11-13  6:31                                                       ` Thomas Rosenstein
2020-11-16 11:56                                                         ` Jesper Dangaard Brouer
2020-11-16 12:05                                                           ` Thomas Rosenstein
2020-11-09 16:39                                           ` Thomas Rosenstein
2020-11-07 13:33                                 ` Thomas Rosenstein
2020-11-07 16:46                                 ` Jesper Dangaard Brouer
2020-11-07 17:01                                   ` Thomas Rosenstein
2020-11-07 17:26                                     ` Sebastian Moeller
2020-11-16 12:34 ` Jesper Dangaard Brouer
2020-11-16 12:49   ` Thomas Rosenstein

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://lists.bufferbloat.net/postorius/lists/bloat.lists.bufferbloat.net/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=F2A84998-D280-45E2-A38E-62C76D96676C@creamfinance.com \
    --to=thomas.rosenstein@creamfinance.com \
    --cc=bloat@lists.bufferbloat.net \
    --cc=brouer@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox