<div dir="ltr"><div>Toke,</div><div><br></div><div>Thank you very much for pointing me in the right direction.</div><div>I am having some fun in the lab tinkering with the 'mq' qdisc and Jesper's xdp-cpumap-tc.</div><div>It seems I will need to use iptables or nftables to filter packets to corresponding queues, since mq apparently cannot have u32 filters on its root.</div><div>I will try to familiarize myself with iptables and nftables, and hopefully get it working soon and report back. Thank you!<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 15, 2021 at 5:30 AM Toke Høiland-Jørgensen <<a href="mailto:toke@toke.dk">toke@toke.dk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Robert Chacon <<a href="mailto:robert.chacon@jackrabbitwireless.com" target="_blank">robert.chacon@jackrabbitwireless.com</a>> writes:<br>
<br>
>> Cool! What kind of performance are you seeing? The README mentions being<br>
>> limited by the BPF hash table size, but can you actually shape 2000<br>
>> customers on one machine? On what kind of hardware and at what rate(s)?<br>
><br>
> On our production network our peak throughput is 1.5Gbps from 200 clients,<br>
> and it works very well.<br>
> We use a simple consumer-class AMD 2700X CPU in production because<br>
> utilization of the shaper VM is ~15% at 1.5Gbps load.<br>
> Customers get reliably capped within ±2Mbps of their allocated htb/fq_codel<br>
> bandwidth, which is very helpful to control network congestion.<br>
><br>
> Here are some graphs from RRUL performed on our test bench hypervisor:<br>
> <a href="https://raw.githubusercontent.com/rchac/LibreQoS/main/docs/fq_codel_1000_subs_4G.png" rel="noreferrer" target="_blank">https://raw.githubusercontent.com/rchac/LibreQoS/main/docs/fq_codel_1000_subs_4G.png</a><br>
> In that example, bandwidth for the "subscriber" client VM was set to 4Gbps.<br>
> 1000 IPv4 IPs and 1000 IPv6 IPs were in the filter hash table of LibreQoS.<br>
> The test bench server has an AMD 3900X running Ubuntu in Proxmox. 4Gbps<br>
> utilizes 10% of the VM's 12 cores. Paravirtualized VirtIO network drivers<br>
> are used and most offloading types are enabled.<br>
> In our setup, VM networking multiqueue isn't enabled (it kept disrupting<br>
> traffic flow), so 6Gbps is probably the most it can achieve like this. Our<br>
> qdiscs in this VM may be limited to one core because of that.<br>
<br>
I suspect the issue you had with multiqueue is that it requires per-CPU<br>
partitioning on a per-customer base to work well. This is possible to do<br>
with XDP, as Jesper demonstrates here:<br>
<br>
<a href="https://github.com/netoptimizer/xdp-cpumap-tc" rel="noreferrer" target="_blank">https://github.com/netoptimizer/xdp-cpumap-tc</a><br>
<br>
With this it should be possible to scale the hardware queues across<br>
multiple CPUs properly, and you should be able to go to much higher<br>
rates by just throwing more CPU cores at it. At least on bare metal; not<br>
sure if the VM virt-drivers have the needed support yet...<br>
<br>
-Toke<br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr">
<table style="width:450px;font-size:10pt;font-family:Arial,sans-serif" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td style="font-size:10pt;font-family:Arial,sans-serif;border-right:3px solid rgb(33,150,243);width:180px;padding-right:10px;vertical-align:top" valign="top">
<img alt="photograph" style="border: 0px none; height: auto; width: 120px;" src="https://jackrabbitwireless.com/wp-content/uploads/2018/11/cropped-jackrabbitwirelesslogo-yellow1.png" width="120" border="0"><p style="margin-top:32px;margin-bottom:0px;line-height:1"><b><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(33,150,243)">Robert Chacón<br></span></b>
<span style="font-family:Arial,sans-serif;font-size:9pt;color:rgb(0,0,0)">Owner</span>
</p>
</td>
<td valign="top">
<table cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td style="font-size:10pt;color:rgb(68,68,68);font-family:Arial,sans-serif;padding-bottom:5px;padding-left:30px;vertical-align:top;line-height:1.3" valign="top">
<span><span style="color:rgb(0,0,0)"><b>M</b></span><span style="font-size:9pt;font-family:Arial,sans-serif;color:rgb(0,0,0)"> (915) 730-1472<br></span></span>
<span><span style="color:rgb(0,0,0)"><b>E</b></span><span style="font-size:9pt;font-family:Arial,sans-serif;color:rgb(0,0,0)"> <a href="mailto:robert.chacon@jackrabbitwireless.com" target="_blank">robert.chacon@jackrabbitwireless.com</a></span></span>
</td>
</tr>
<tr>
<td style="font-size:10pt;color:rgb(0,121,172);font-family:Arial,sans-serif;padding-bottom:5px;padding-left:30px;vertical-align:top;line-height:1.3" valign="top">
<b style="font-family:Arial,sans-serif;font-size:9pt;color:rgb(0,0,0)">JackRabbit Wireless LLC</b>
<span>
<br>
<span style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(0,0,0)">P.O. Box 222111<span><br></span></span>
<span style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(0,0,0)">El Paso, TX 79913</span>
</span>
</td>
</tr>
<tr>
<td style="font-size:10pt;font-family:Arial,sans-serif;padding-bottom:5px;padding-top:5px;padding-left:30px;vertical-align:top;color:rgb(0,121,172)" valign="top">
<a href="http://jackrabbitwireless.com" target="_blank"><b style="font-size:9pt;font-family:Arial,sans-serif;color:rgb(33,150,243)">jackrabbitwireless.com</b></a><br><br>
</td>
</tr>
</tbody>
</table>
</td></tr>
</tbody>
</table>
</div></div>