From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf1-x132.google.com (mail-lf1-x132.google.com [IPv6:2a00:1450:4864:20::132]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 4452A3B2A4 for ; Mon, 17 Jun 2024 12:07:55 -0400 (EDT) Received: by mail-lf1-x132.google.com with SMTP id 2adb3069b0e04-52c84a21c62so5110495e87.1 for ; Mon, 17 Jun 2024 09:07:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1718640473; x=1719245273; darn=lists.bufferbloat.net; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=Qy6algBEVpMyUx/KNWU3UTiIKZ64+N2e3Balkt5QDZc=; b=XHdOkgL3b0MIQubqNnFs1+4721bN6A+nOBRY7Y3T1ZUgcJimPXE/P0cumlE54BAlCQ I3tbSFohXQ9W9c+Xxodj6CxrxP6if05WKQsQGayjKV365FB+qNnEIslXTkWbd21xjCUL uSIcNRSrUOawdlYgmnm2a/LH9dW3+cuYCzbqu91aA0NURUD8BW96Vws4fknAkpPwdQv+ qjY4g1XvXZU3nQsJHCrAOOSin1BV9v0BX/cFypb0zZPS9vJZbp6+4ALhAYaiX83KZiFK 5xIq2R2R8Ir/YUCYMVtskOiZ2fm2Pa+eHNeBx/IiecOuoYwn+c1ftuGMuptpGEJk8rpx Q/JA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718640473; x=1719245273; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Qy6algBEVpMyUx/KNWU3UTiIKZ64+N2e3Balkt5QDZc=; b=w3PLgMBPzqucXhn7z+XLpAJoTCkfXJM4X8LRXBSmt7NOSBqup3G2zaIusNBp9akVVt qcT+WqKbc4QN/10uHG9Q512JZ9keJdKJv58cVFyuOll08zB5k1oRF6aB4CXddc0PlgSq qULhoJ67ngXJaOCRktyoYsDuBXE4smr+uplSHYOhsBT624tofxncwHwdR8tdI5ENej0Y MRhiscBgVsVMCT6I4xFfgjSCoixS+lrlqXcfJBvDcpKeLW1uxjXMQ49gJNh8/II4apfk rtr0q7QP6fClg1ZM92R6seusiNY8D6+CEAPwBj4JGf8zi0AMdHos92jrlnaOWZGC/s+r vn0g== X-Gm-Message-State: AOJu0Yy3g/UCKmgr6Uy90p4FxITY8cC1a0+zY4saOiOu7QSnP1LEp+K8 hC16CFLUQdIQVZh9M0v7mpmD7eFzCdYzZ32vSKIZkG4a0WGMutE7sJ6g3y21hTAYPvvOgzFWj18 A/IPQyKy2SQf2/4W/XJ7nPe8BwiG8qg== X-Google-Smtp-Source: AGHT+IE11uLFi5xjFAQkNxVCkDzV6QNzwNhER7TiEX0b6tZGSiwNew8iUPsDYqSdnVRv5yR0oEq17htlroQlQWg2mYY= X-Received: by 2002:a05:6512:4028:b0:52c:80f6:d384 with SMTP id 2adb3069b0e04-52ca6e55f77mr10244427e87.3.1718640473229; Mon, 17 Jun 2024 09:07:53 -0700 (PDT) MIME-Version: 1.0 References: <20240612170851.1004604-1-jiri@resnulli.us> In-Reply-To: From: Dave Taht Date: Mon, 17 Jun 2024 09:07:42 -0700 Message-ID: To: bloat Content-Type: multipart/alternative; boundary="000000000000e751f1061b182a91" Subject: [Bloat] Fwd: [PATCH net-next v2] virtio_net: add support for Byte Queue Limits X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Jun 2024 16:07:55 -0000 --000000000000e751f1061b182a91 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I am so happy to see this patch land finally! ---------- Forwarded message --------- From: Jason Xing Date: Mon, Jun 17, 2024, 8:48=E2=80=AFAM Subject: Re: [PATCH net-next v2] virtio_net: add support for Byte Queue Limits To: Jiri Pirko Cc: , , , , , , < jasowang@redhat.com>, , < virtualization@lists.linux.dev>, , , = < hawk@kernel.org>, , , < hengqi@linux.alibaba.com> On Mon, Jun 17, 2024 at 5:15=E2=80=AFPM Jiri Pirko wrote= : > > Fri, Jun 14, 2024 at 11:54:04AM CEST, kerneljasonxing@gmail.com wrote: > >Hello Jiri, > > > >On Thu, Jun 13, 2024 at 1:08=E2=80=AFAM Jiri Pirko wr= ote: > >> > >> From: Jiri Pirko > >> > >> Add support for Byte Queue Limits (BQL). > >> > >> Tested on qemu emulated virtio_net device with 1, 2 and 4 queues. > >> Tested with fq_codel and pfifo_fast. Super netperf with 50 threads is > >> running in background. Netperf TCP_RR results: > >> > >> NOBQL FQC 1q: 159.56 159.33 158.50 154.31 agv: 157.925 > >> NOBQL FQC 2q: 184.64 184.96 174.73 174.15 agv: 179.62 > >> NOBQL FQC 4q: 994.46 441.96 416.50 499.56 agv: 588.12 > >> NOBQL PFF 1q: 148.68 148.92 145.95 149.48 agv: 148.2575 > >> NOBQL PFF 2q: 171.86 171.20 170.42 169.42 agv: 170.725 > >> NOBQL PFF 4q: 1505.23 1137.23 2488.70 3507.99 agv: 2159.7875 > >> BQL FQC 1q: 1332.80 1297.97 1351.41 1147.57 agv: 1282.4375 > >> BQL FQC 2q: 768.30 817.72 864.43 974.40 agv: 856.2125 > >> BQL FQC 4q: 945.66 942.68 878.51 822.82 agv: 897.4175 > >> BQL PFF 1q: 149.69 151.49 149.40 147.47 agv: 149.5125 > >> BQL PFF 2q: 2059.32 798.74 1844.12 381.80 agv: 1270.995 > >> BQL PFF 4q: 1871.98 4420.02 4916.59 13268.16 agv: 6119.1875 > > > >I cannot get such a huge improvement when I was doing multiple tests > >between two VMs. I'm pretty sure the BQL feature is working, but the > >numbers look the same with/without BQL. > > > >VM 1 (client): > >16 cpus, x86_64, 4 queues, the latest net-next kernel with/without > >this patch, pfifo_fast, napi_tx=3Dtrue, napi_weight=3D128 > > > >VM 2 (server): > >16 cpus, aarch64, 4 queues, the latest net-next kernel without this > >patch, pfifo_fast > > > >What the 'ping' command shows to me between two VMs is : rtt > >min/avg/max/mdev =3D 0.233/0.257/0.300/0.024 ms > > > >I started 50 netperfs to communicate the other side with the following command: > >#!/bin/bash > > > >for i in $(seq 5000 5050); > >do > >netperf -p $i -H [ip addr] -l 60 -t TCP_RR -- -r 64,64 > /dev/null 2>&1 = & > >done > > > >The results are around 30423.62 txkB/s. If I remove '-r 64 64', they > >are still the same/similar. > > You have to stress the line by parallel TCP_STREAM instances (50 in my > case). For consistent results, use -p portnum,locport to specify the > local port. Thanks. Even though the results of TCP_RR mode vary sometimes, I can see a big improvement in the total value of those results under such circumstances. With BQL, the throughput is 2159.17 Without BQL, it's 1099.33 Please feel free to add the tag: Tested-by: Jason Xing Thanks, Jason --000000000000e751f1061b182a91 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I am so happy to see this patch land finally!

---------- F= orwarded message ---------
From: Jason Xing <kerneljasonxing@gmail.com>
Date: Mo= n, Jun 17, 2024, 8:48=E2=80=AFAM
Subject: Re: [PATCH net-next v2] virtio= _net: add support for Byte Queue Limits
To: Jiri Pirko <jiri@resnulli.us>
Cc: <netdev@vger.kernel.org>, <davem@davemloft.net>, <edumazet@google.com>, <kuba@kernel.org>, <pabeni@redhat.com>, <mst@redhat.com>, <jaso= wang@redhat.com>, <xuanzhuo@linux.alibaba.com>, <virtualization@lists.linux.dev>, <ast@kernel.org>, <daniel@iogearbox.net>, <hawk@kernel.org>, <john.fastabend@gmail.com>, <dave.taht@gmail.com>, <hengqi@linux.alibaba.com>


On Mon, Ju= n 17, 2024 at 5:15=E2=80=AFPM Jiri Pirko <jiri@resnulli.us> wrote:<= br> >
> Fri, Jun 14, 2024 at 11:54:04AM CEST, kerneljasonxing@gmail.com= wrote:
> >Hello Jiri,
> >
> >On Thu, Jun 13, 2024 at 1:08=E2=80=AFAM Jiri Pirko <jiri@resnulli= .us> wrote:
> >>
> >> From: Jiri Pirko <jiri@nvidia.com>
> >>
> >> Add support for Byte Queue Limits (BQL).
> >>
> >> Tested on qemu emulated virtio_net device with 1, 2 and 4 que= ues.
> >> Tested with fq_codel and pfifo_fast. Super netperf with 50 th= reads is
> >> running in background. Netperf TCP_RR results:
> >>
> >> NOBQL FQC 1q:=C2=A0 159.56=C2=A0 159.33=C2=A0 158.50=C2=A0 15= 4.31=C2=A0 =C2=A0 agv: 157.925
> >> NOBQL FQC 2q:=C2=A0 184.64=C2=A0 184.96=C2=A0 174.73=C2=A0 17= 4.15=C2=A0 =C2=A0 agv: 179.62
> >> NOBQL FQC 4q:=C2=A0 994.46=C2=A0 441.96=C2=A0 416.50=C2=A0 49= 9.56=C2=A0 =C2=A0 agv: 588.12
> >> NOBQL PFF 1q:=C2=A0 148.68=C2=A0 148.92=C2=A0 145.95=C2=A0 14= 9.48=C2=A0 =C2=A0 agv: 148.2575
> >> NOBQL PFF 2q:=C2=A0 171.86=C2=A0 171.20=C2=A0 170.42=C2=A0 16= 9.42=C2=A0 =C2=A0 agv: 170.725
> >> NOBQL PFF 4q: 1505.23 1137.23 2488.70 3507.99=C2=A0 =C2=A0 ag= v: 2159.7875
> >>=C2=A0 =C2=A0BQL FQC 1q: 1332.80 1297.97 1351.41 1147.57=C2=A0= =C2=A0 agv: 1282.4375
> >>=C2=A0 =C2=A0BQL FQC 2q:=C2=A0 768.30=C2=A0 817.72=C2=A0 864.4= 3=C2=A0 974.40=C2=A0 =C2=A0 agv: 856.2125
> >>=C2=A0 =C2=A0BQL FQC 4q:=C2=A0 945.66=C2=A0 942.68=C2=A0 878.5= 1=C2=A0 822.82=C2=A0 =C2=A0 agv: 897.4175
> >>=C2=A0 =C2=A0BQL PFF 1q:=C2=A0 149.69=C2=A0 151.49=C2=A0 149.4= 0=C2=A0 147.47=C2=A0 =C2=A0 agv: 149.5125
> >>=C2=A0 =C2=A0BQL PFF 2q: 2059.32=C2=A0 798.74 1844.12=C2=A0 38= 1.80=C2=A0 =C2=A0 agv: 1270.995
> >>=C2=A0 =C2=A0BQL PFF 4q: 1871.98 4420.02 4916.59 13268.16=C2= =A0 =C2=A0agv: 6119.1875
> >
> >I cannot get such a huge improvement when I was doing multiple tes= ts
> >between two VMs. I'm pretty sure the BQL feature is working, b= ut the
> >numbers look the same with/without BQL.
> >
> >VM 1 (client):
> >16 cpus, x86_64, 4 queues, the latest net-next kernel with/without=
> >this patch, pfifo_fast, napi_tx=3Dtrue, napi_weight=3D128
> >
> >VM 2 (server):
> >16 cpus, aarch64, 4 queues, the latest net-next kernel without thi= s
> >patch, pfifo_fast
> >
> >What the 'ping' command shows to me between two VMs is : r= tt
> >min/avg/max/mdev =3D 0.233/0.257/0.300/0.024 ms
> >
> >I started 50 netperfs to communicate the other side with the follo= wing command:
> >#!/bin/bash
> >
> >for i in $(seq 5000 5050);
> >do
> >netperf -p $i -H [ip addr] -l 60 -t TCP_RR -- -r 64,64 > /dev/n= ull 2>&1 &
> >done
> >
> >The results are around 30423.62 txkB/s. If I remove '-r 64 64&= #39;, they
> >are still the same/similar.
>
> You have to stress the line by parallel TCP_STREAM instances (50 in my=
> case). For consistent results, use -p portnum,locport to specify the > local port.

Thanks. Even though the results of TCP_RR mode vary sometimes, I can
see a big improvement in the total value of those results under such
circumstances.
With BQL, the throughput is 2159.17
Without BQL, it's 1099.33

Please feel free to add the tag:
Tested-by: Jason Xing <kerneljasonxing@gmail.com>

Thanks,
Jason
--000000000000e751f1061b182a91--