[LibreQoS] Fwd: mlx5 XDP redirect leaking memory on kernel 6.3

[LibreQoS] Fwd: mlx5 XDP redirect leaking memory on kernel 6.3

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 2668 bytes --]

Not sure what driver our friends in NZ are using...

---------- Forwarded message ---------
From: Jesper Dangaard Brouer <jbrouer@redhat.com>
Date: Tue, May 23, 2023, 9:55 AM
Subject: mlx5 XDP redirect leaking memory on kernel 6.3
To: Dragos Tatulea <dtatulea@nvidia.com>, Saeed Mahameed <saeed@kernel.org>,
Saeed Mahameed <saeedm@nvidia.com>, Tariq Toukan <tariqt@nvidia.com>, Tariq
Toukan <ttoukan.linux@gmail.com>, Netdev <netdev@vger.kernel.org>, Yunsheng
Lin <linyunsheng@huawei.com>
Cc: <brouer@redhat.com>, <atzin@redhat.com>, <mkabat@redhat.com>, <
kheib@redhat.com>, Jiri Benc <jbenc@redhat.com>, bpf <bpf@vger.kernel.org>,
Felix Maurer <fmaurer@redhat.com>, Alexander Duyck <
alexander.duyck@gmail.com>, Ilias Apalodimas <ilias.apalodimas@linaro.org>,
Lorenzo Bianconi <lorenzo@kernel.org>, Maxim Mikityanskiy <
maxtram95@gmail.com>



When the mlx5 driver runs an XDP program doing XDP_REDIRECT, then memory
is getting leaked. Other XDP actions, like XDP_DROP, XDP_PASS and XDP_TX
works correctly. I tested both redirecting back out same mlx5 device and
cpumap redirect (with XDP_PASS), which both cause leaking.

After removing the XDP prog, which also cause the page_pool to be
released by mlx5, then the leaks are visible via the page_pool periodic
inflight reports. I have this bpftrace[1] tool that I also use to detect
the problem faster (not waiting 60 sec for a report).

  [1]
https://github.com/xdp-project/xdp-project/blob/master/areas/mem/bpftrace/page_pool_track_shutdown01.bt

I've been debugging and reading through the code for a couple of days,
but I've not found the root-cause, yet. I would appreciate new ideas
where to look and fresh eyes on the issue.

To Lin, it looks like mlx5 uses PP_FLAG_PAGE_FRAG, and my current
suspicion is that mlx5 driver doesn't fully release the bias count (hint
see MLX5E_PAGECNT_BIAS_MAX).

--Jesper


Extra info about my device.  Providing these as mlx5 driver can have
different allocation modes depending on HW and device priv-flags setup.

$ ethtool --show-priv-flags mlx5p1
Private flags for mlx5p1:
rx_cqe_moder       : on
tx_cqe_moder       : off
rx_cqe_compress    : off
rx_striding_rq     : on
rx_no_csum_complete: off
xdp_tx_mpwqe       : on
skb_tx_mpwqe       : on
tx_port_ts         : off

$ ethtool -i mlx5p1
driver: mlx5_core
version: 6.4.0-rc2-net-next-vm-lock-dbg+
firmware-version: 16.23.1020 (MT_0000000009)
expansion-rom-version:
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

$ lspci -v | grep 03:00.0
03:00.0 Ethernet controller: Mellanox Technologies MT28800 Family
[ConnectX-5 Ex]

[-- Attachment #2: Type: text/html, Size: 4248 bytes --]

[LibreQoS] Fwd: mlx5 XDP redirect leaking memory on kernel 6.3

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 2733 bytes --]

---------- Forwarded message ---------
From: Dragos Tatulea <dtatulea@nvidia.com>
Date: Tue, May 23, 2023, 10:36 AM
Subject: Re: mlx5 XDP redirect leaking memory on kernel 6.3
To: Tariq Toukan <tariqt@nvidia.com>, ttoukan.linux@gmail.com <
ttoukan.linux@gmail.com>, jbrouer@redhat.com <jbrouer@redhat.com>, Saeed
Mahameed <saeedm@nvidia.com>, saeed@kernel.org <saeed@kernel.org>,
linyunsheng@huawei.com <linyunsheng@huawei.com>, netdev@vger.kernel.org <
netdev@vger.kernel.org>
Cc: maxtram95@gmail.com <maxtram95@gmail.com>, lorenzo@kernel.org <
lorenzo@kernel.org>, alexander.duyck@gmail.com <alexander.duyck@gmail.com>,
kheib@redhat.com <kheib@redhat.com>, ilias.apalodimas@linaro.org <
ilias.apalodimas@linaro.org>, mkabat@redhat.com <mkabat@redhat.com>,
brouer@redhat.com <brouer@redhat.com>, atzin@redhat.com <atzin@redhat.com>,
fmaurer@redhat.com <fmaurer@redhat.com>, bpf@vger.kernel.org <
bpf@vger.kernel.org>, jbenc@redhat.com <jbenc@redhat.com>



On Tue, 2023-05-23 at 17:55 +0200, Jesper Dangaard Brouer wrote:
>
> When the mlx5 driver runs an XDP program doing XDP_REDIRECT, then memory
> is getting leaked. Other XDP actions, like XDP_DROP, XDP_PASS and XDP_TX
> works correctly. I tested both redirecting back out same mlx5 device and
> cpumap redirect (with XDP_PASS), which both cause leaking.
>
> After removing the XDP prog, which also cause the page_pool to be
> released by mlx5, then the leaks are visible via the page_pool periodic
> inflight reports. I have this bpftrace[1] tool that I also use to detect
> the problem faster (not waiting 60 sec for a report).
>
>   [1]
>
https://github.com/xdp-project/xdp-project/blob/master/areas/mem/bpftrace/page_pool_track_shutdown01.bt
>
> I've been debugging and reading through the code for a couple of days,
> but I've not found the root-cause, yet. I would appreciate new ideas
> where to look and fresh eyes on the issue.
>
>
> To Lin, it looks like mlx5 uses PP_FLAG_PAGE_FRAG, and my current
> suspicion is that mlx5 driver doesn't fully release the bias count (hint
> see MLX5E_PAGECNT_BIAS_MAX).
>

Thanks for the report Jesper. Incidentally I've just picked up this issue
today
as well.

On XDP redirect and tx, the page is set to skip the bias counter release
with
the expectation that page_pool_put_defragged_page will be called from [1].
But,
as I found out now, during XDP redirect only one fragment of the page is
released in xdp core [2]. This is where the leak is coming from.

We'll provide a fix soon.

[1]
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c#n665

[2]
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/net/core/xdp.c#n390

Thanks,
Dragos

[-- Attachment #2: Type: text/html, Size: 5310 bytes --]

[LibreQoS] Fwd: mlx5 XDP redirect leaking memory on kernel 6.3

[Dave Taht]

not that this applies to our mlx problem... probably...

---------- Forwarded message ---------
From: Jesper Dangaard Brouer <jbrouer@redhat.com>
Date: Thu, Jul 13, 2023 at 8:59 AM
Subject: Re: mlx5 XDP redirect leaking memory on kernel 6.3
To: Dragos Tatulea <dtatulea@nvidia.com>, Tariq Toukan
<tariqt@nvidia.com>, jbrouer@redhat.com <jbrouer@redhat.com>, Saeed
Mahameed <saeedm@nvidia.com>, saeed@kernel.org <saeed@kernel.org>,
netdev@vger.kernel.org <netdev@vger.kernel.org>, Greg KH
<gregkh@linuxfoundation.org>
Cc: <brouer@redhat.com>, maxtram95@gmail.com <maxtram95@gmail.com>,
lorenzo@kernel.org <lorenzo@kernel.org>, alexander.duyck@gmail.com
<alexander.duyck@gmail.com>, kheib@redhat.com <kheib@redhat.com>,
ilias.apalodimas@linaro.org <ilias.apalodimas@linaro.org>,
mkabat@redhat.com <mkabat@redhat.com>, atzin@redhat.com
<atzin@redhat.com>, fmaurer@redhat.com <fmaurer@redhat.com>,
bpf@vger.kernel.org <bpf@vger.kernel.org>, jbenc@redhat.com
<jbenc@redhat.com>, linyunsheng@huawei.com <linyunsheng@huawei.com>,
ttoukan.linux@gmail.com <ttoukan.linux@gmail.com>




On 13/07/2023 12.11, Dragos Tatulea wrote:
> Gi Jesper,
> On Thu, 2023-07-13 at 11:20 +0200, Jesper Dangaard Brouer wrote:
>> Hi Dragos,
>>
>> Below you promised to work on a fix for XDP redirect memory leak...
>> What is the status?
>>
> The fix got merged into net a week ago:
> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/drivers/net/ethernet/mellanox/mlx5/core?id=7abd955a58fb0fcd4e756fa2065c03ae488fcfa7
>
> Just forgot to follow up on this thread. Sorry about that...
>

Good to see it being fixed in net.git commit:
  7abd955a58fb ("net/mlx5e: RX, Fix page_pool page fragment tracking for
XDP")

This need to be backported into stable tree 6.3, but I can see 6.3.13 is
marked EOL (End-of-Life).
Can we still get this fix applied? (Cc. GregKH)

--Jesper

>
>> On 23/05/2023 18.35, Dragos Tatulea wrote:
>>>
>>> On Tue, 2023-05-23 at 17:55 +0200, Jesper Dangaard Brouer wrote:
>>>>
>>>> When the mlx5 driver runs an XDP program doing XDP_REDIRECT, then memory
>>>> is getting leaked. Other XDP actions, like XDP_DROP, XDP_PASS and XDP_TX
>>>> works correctly. I tested both redirecting back out same mlx5 device and
>>>> cpumap redirect (with XDP_PASS), which both cause leaking.
>>>>
>>>> After removing the XDP prog, which also cause the page_pool to be
>>>> released by mlx5, then the leaks are visible via the page_pool periodic
>>>> inflight reports. I have this bpftrace[1] tool that I also use to detect
>>>> the problem faster (not waiting 60 sec for a report).
>>>>
>>>>     [1]
>>>> https://github.com/xdp-project/xdp-project/blob/master/areas/mem/bpftrace/page_pool_track_shutdown01.bt
>>>>
>>>> I've been debugging and reading through the code for a couple of days,
>>>> but I've not found the root-cause, yet. I would appreciate new ideas
>>>> where to look and fresh eyes on the issue.
>>>>
>>>>
>>>> To Lin, it looks like mlx5 uses PP_FLAG_PAGE_FRAG, and my current
>>>> suspicion is that mlx5 driver doesn't fully release the bias count (hint
>>>> see MLX5E_PAGECNT_BIAS_MAX).
>>>>
>>>
>>> Thanks for the report Jesper. Incidentally I've just picked up this issue
>>> today
>>> as well.
>>>
>>> On XDP redirect and tx, the page is set to skip the bias counter release
>>> with
>>> the expectation that page_pool_put_defragged_page will be called from [1].
>>> But,
>>> as I found out now, during XDP redirect only one fragment of the page is
>>> released in xdp core [2]. This is where the leak is coming from.
>>>
>>> We'll provide a fix soon.
>>>
>>> [1]
>>> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c#n665
>>>
>>> [2]
>>> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/net/core/xdp.c#n390
>>>
>>> Thanks,
>>> Dragos
>>>
>>>
>>
>




-- 
Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/
Dave Täht CSO, LibreQos