<div dir="auto">Not sure what driver our friends in NZ are using...</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">---------- Forwarded message ---------<br>From: <strong class="gmail_sendername" dir="auto">Jesper Dangaard Brouer</strong> <span dir="auto"><<a href="mailto:jbrouer@redhat.com">jbrouer@redhat.com</a>></span><br>Date: Tue, May 23, 2023, 9:55 AM<br>Subject: mlx5 XDP redirect leaking memory on kernel 6.3<br>To: Dragos Tatulea <<a href="mailto:dtatulea@nvidia.com">dtatulea@nvidia.com</a>>, Saeed Mahameed <<a href="mailto:saeed@kernel.org">saeed@kernel.org</a>>, Saeed Mahameed <<a href="mailto:saeedm@nvidia.com">saeedm@nvidia.com</a>>, Tariq Toukan <<a href="mailto:tariqt@nvidia.com">tariqt@nvidia.com</a>>, Tariq Toukan <<a href="mailto:ttoukan.linux@gmail.com">ttoukan.linux@gmail.com</a>>, Netdev <<a href="mailto:netdev@vger.kernel.org">netdev@vger.kernel.org</a>>, Yunsheng Lin <<a href="mailto:linyunsheng@huawei.com">linyunsheng@huawei.com</a>><br>Cc: <<a href="mailto:brouer@redhat.com">brouer@redhat.com</a>>, <<a href="mailto:atzin@redhat.com">atzin@redhat.com</a>>, <<a href="mailto:mkabat@redhat.com">mkabat@redhat.com</a>>, <<a href="mailto:kheib@redhat.com">kheib@redhat.com</a>>, Jiri Benc <<a href="mailto:jbenc@redhat.com">jbenc@redhat.com</a>>, bpf <<a href="mailto:bpf@vger.kernel.org">bpf@vger.kernel.org</a>>, Felix Maurer <<a href="mailto:fmaurer@redhat.com">fmaurer@redhat.com</a>>, Alexander Duyck <<a href="mailto:alexander.duyck@gmail.com">alexander.duyck@gmail.com</a>>, Ilias Apalodimas <<a href="mailto:ilias.apalodimas@linaro.org">ilias.apalodimas@linaro.org</a>>, Lorenzo Bianconi <<a href="mailto:lorenzo@kernel.org">lorenzo@kernel.org</a>>, Maxim Mikityanskiy <<a href="mailto:maxtram95@gmail.com">maxtram95@gmail.com</a>><br></div><br><br><br>
When the mlx5 driver runs an XDP program doing XDP_REDIRECT, then memory<br>
is getting leaked. Other XDP actions, like XDP_DROP, XDP_PASS and XDP_TX<br>
works correctly. I tested both redirecting back out same mlx5 device and<br>
cpumap redirect (with XDP_PASS), which both cause leaking.<br>
<br>
After removing the XDP prog, which also cause the page_pool to be<br>
released by mlx5, then the leaks are visible via the page_pool periodic<br>
inflight reports. I have this bpftrace[1] tool that I also use to detect<br>
the problem faster (not waiting 60 sec for a report).<br>
<br>
[1] <br>
<a href="https://github.com/xdp-project/xdp-project/blob/master/areas/mem/bpftrace/page_pool_track_shutdown01.bt" rel="noreferrer noreferrer" target="_blank">https://github.com/xdp-project/xdp-project/blob/master/areas/mem/bpftrace/page_pool_track_shutdown01.bt</a><br>
<br>
I've been debugging and reading through the code for a couple of days,<br>
but I've not found the root-cause, yet. I would appreciate new ideas<br>
where to look and fresh eyes on the issue.<br>
<br>
To Lin, it looks like mlx5 uses PP_FLAG_PAGE_FRAG, and my current<br>
suspicion is that mlx5 driver doesn't fully release the bias count (hint<br>
see MLX5E_PAGECNT_BIAS_MAX).<br>
<br>
--Jesper<br>
<br>
<br>
Extra info about my device. Providing these as mlx5 driver can have <br>
different allocation modes depending on HW and device priv-flags setup.<br>
<br>
$ ethtool --show-priv-flags mlx5p1<br>
Private flags for mlx5p1:<br>
rx_cqe_moder : on<br>
tx_cqe_moder : off<br>
rx_cqe_compress : off<br>
rx_striding_rq : on<br>
rx_no_csum_complete: off<br>
xdp_tx_mpwqe : on<br>
skb_tx_mpwqe : on<br>
tx_port_ts : off<br>
<br>
$ ethtool -i mlx5p1<br>
driver: mlx5_core<br>
version: 6.4.0-rc2-net-next-vm-lock-dbg+<br>
firmware-version: 16.23.1020 (MT_0000000009)<br>
expansion-rom-version:<br>
bus-info: 0000:03:00.0<br>
supports-statistics: yes<br>
supports-test: yes<br>
supports-eeprom-access: no<br>
supports-register-dump: no<br>
supports-priv-flags: yes<br>
<br>
$ lspci -v | grep 03:00.0<br>
03:00.0 Ethernet controller: Mellanox Technologies MT28800 Family <br>
[ConnectX-5 Ex]<br>
<br>
<br>
</div>