[LibreQoS] Fwd: mlx5 XDP redirect leaking memory on kernel 6.3

[Dave Taht <dave.taht@gmail.com>]

not that this applies to our mlx problem... probably...

---------- Forwarded message ---------
From: Jesper Dangaard Brouer <jbrouer@redhat.com>
Date: Thu, Jul 13, 2023 at 8:59 AM
Subject: Re: mlx5 XDP redirect leaking memory on kernel 6.3
To: Dragos Tatulea <dtatulea@nvidia.com>, Tariq Toukan
<tariqt@nvidia.com>, jbrouer@redhat.com <jbrouer@redhat.com>, Saeed
Mahameed <saeedm@nvidia.com>, saeed@kernel.org <saeed@kernel.org>,
netdev@vger.kernel.org <netdev@vger.kernel.org>, Greg KH
<gregkh@linuxfoundation.org>
Cc: <brouer@redhat.com>, maxtram95@gmail.com <maxtram95@gmail.com>,
lorenzo@kernel.org <lorenzo@kernel.org>, alexander.duyck@gmail.com
<alexander.duyck@gmail.com>, kheib@redhat.com <kheib@redhat.com>,
ilias.apalodimas@linaro.org <ilias.apalodimas@linaro.org>,
mkabat@redhat.com <mkabat@redhat.com>, atzin@redhat.com
<atzin@redhat.com>, fmaurer@redhat.com <fmaurer@redhat.com>,
bpf@vger.kernel.org <bpf@vger.kernel.org>, jbenc@redhat.com
<jbenc@redhat.com>, linyunsheng@huawei.com <linyunsheng@huawei.com>,
ttoukan.linux@gmail.com <ttoukan.linux@gmail.com>




On 13/07/2023 12.11, Dragos Tatulea wrote:
> Gi Jesper,
> On Thu, 2023-07-13 at 11:20 +0200, Jesper Dangaard Brouer wrote:
>> Hi Dragos,
>>
>> Below you promised to work on a fix for XDP redirect memory leak...
>> What is the status?
>>
> The fix got merged into net a week ago:
> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/drivers/net/ethernet/mellanox/mlx5/core?id=7abd955a58fb0fcd4e756fa2065c03ae488fcfa7
>
> Just forgot to follow up on this thread. Sorry about that...
>

Good to see it being fixed in net.git commit:
  7abd955a58fb ("net/mlx5e: RX, Fix page_pool page fragment tracking for
XDP")

This need to be backported into stable tree 6.3, but I can see 6.3.13 is
marked EOL (End-of-Life).
Can we still get this fix applied? (Cc. GregKH)

--Jesper

>
>> On 23/05/2023 18.35, Dragos Tatulea wrote:
>>>
>>> On Tue, 2023-05-23 at 17:55 +0200, Jesper Dangaard Brouer wrote:
>>>>
>>>> When the mlx5 driver runs an XDP program doing XDP_REDIRECT, then memory
>>>> is getting leaked. Other XDP actions, like XDP_DROP, XDP_PASS and XDP_TX
>>>> works correctly. I tested both redirecting back out same mlx5 device and
>>>> cpumap redirect (with XDP_PASS), which both cause leaking.
>>>>
>>>> After removing the XDP prog, which also cause the page_pool to be
>>>> released by mlx5, then the leaks are visible via the page_pool periodic
>>>> inflight reports. I have this bpftrace[1] tool that I also use to detect
>>>> the problem faster (not waiting 60 sec for a report).
>>>>
>>>>     [1]
>>>> https://github.com/xdp-project/xdp-project/blob/master/areas/mem/bpftrace/page_pool_track_shutdown01.bt
>>>>
>>>> I've been debugging and reading through the code for a couple of days,
>>>> but I've not found the root-cause, yet. I would appreciate new ideas
>>>> where to look and fresh eyes on the issue.
>>>>
>>>>
>>>> To Lin, it looks like mlx5 uses PP_FLAG_PAGE_FRAG, and my current
>>>> suspicion is that mlx5 driver doesn't fully release the bias count (hint
>>>> see MLX5E_PAGECNT_BIAS_MAX).
>>>>
>>>
>>> Thanks for the report Jesper. Incidentally I've just picked up this issue
>>> today
>>> as well.
>>>
>>> On XDP redirect and tx, the page is set to skip the bias counter release
>>> with
>>> the expectation that page_pool_put_defragged_page will be called from [1].
>>> But,
>>> as I found out now, during XDP redirect only one fragment of the page is
>>> released in xdp core [2]. This is where the leak is coming from.
>>>
>>> We'll provide a fix soon.
>>>
>>> [1]
>>> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c#n665
>>>
>>> [2]
>>> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/net/core/xdp.c#n390
>>>
>>> Thanks,
>>> Dragos
>>>
>>>
>>
>




-- 
Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/
Dave Täht CSO, LibreQos