<div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif">These are just the fq settings as they get applied when having fq as default qdiscs. I guess there are room for improvements on those default settings depending on use case.</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">For future reference: should I increase the limit on drops or is it okay as it is?</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 26 January 2017 at 00:53, Eric Dumazet <span dir="ltr"><<a href="mailto:eric.dumazet@gmail.com" target="_blank">eric.dumazet@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Thu, 2017-01-26 at 00:47 +0100, Hans-Kristian Bakke wrote:<br>
><br>
><br>
> I did record the qdisc settings, but I didn't capture the stats, but<br>
> throttling is definitively active when I watch the tc -s stats in<br>
> realtime when testing (looking at tun1)<br>
><br>
><br>
> tc -s qdisc show<br>
> qdisc noqueue 0: dev lo root refcnt 2<br>
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)<br>
> backlog 0b 0p requeues 0<br>
> qdisc fq 8007: dev eth0 root refcnt 2 limit 10000p flow_limit 100p<br>
> buckets 1024 orphan_mask 1023 quantum 3028 initial_quantum 15140<br>
> refill_delay 40.0ms<br>
> Sent 1420855729 bytes 969198 pkt (dropped 134, overlimits 0 requeues<br>
> 0)<br>
> backlog 0b 0p requeues 0<br>
> 124 flows (123 inactive, 0 throttled)<br>
> 0 gc, 0 highprio, 3 throttled, 3925 ns latency, 134 flows_plimit<br>
<br>
</span>You seem to hit the "flow_limit 100" maybe because all packets are going<br>
through a single encap flow. ( 134 drops )<br>
<span class=""><br>
> qdisc fq 8008: dev tun1 root refcnt 2 limit 10000p flow_limit 100p<br>
> buckets 1024 orphan_mask 1023 quantum 3000 initial_quantum 15000<br>
> refill_delay 40.0ms<br>
> Sent 1031289740 bytes 741181 pkt (dropped 0, overlimits 0 requeues 0)<br>
> backlog 101616b 3p requeues 0<br>
> 16 flows (15 inactive, 1 throttled), next packet delay 351937 ns<br>
> 0 gc, 0 highprio, 58377 throttled, 12761 ns latency<br>
> <br>
><br>
<br>
</span>Looks good, although latency seems a bit high, thanks !<br>
<div class="HOEnZb"><div class="h5">><br>
><br>
> On 26 January 2017 at 00:33, Eric Dumazet <<a href="mailto:eric.dumazet@gmail.com">eric.dumazet@gmail.com</a>><br>
> wrote:<br>
><br>
> On Thu, 2017-01-26 at 00:04 +0100, Hans-Kristian Bakke wrote:<br>
> > I can do that. I guess I should do the capture from tun1 as<br>
> that is<br>
> > the place that the tcp-traffic is visible? My non-virtual<br>
> nic is only<br>
> > seeing OpenVPN encapsulated UDP-traffic.<br>
> ><br>
><br>
> But is FQ installed at the point TCP sockets are ?<br>
><br>
> You should give us "tc -s qdisc show xxx" so that we can<br>
> check if<br>
> pacing (throttling) actually happens.<br>
><br>
><br>
> > On 25 January 2017 at 23:48, Neal Cardwell<br>
> <<a href="mailto:ncardwell@google.com">ncardwell@google.com</a>><br>
> > wrote:<br>
> > On Wed, Jan 25, 2017 at 5:38 PM, Hans-Kristian Bakke<br>
> > <<a href="mailto:hkbakke@gmail.com">hkbakke@gmail.com</a>> wrote:<br>
> > Actually.. the 1-4 mbit/s results with fq<br>
> sporadically<br>
> > appears again as I keep testing but it is<br>
> most likely<br>
> > caused by all the unknowns between me an my<br>
> > testserver. But still, changing to<br>
> pfifo_qdisc seems<br>
> > to normalize the throughput again with BBR,<br>
> could this<br>
> > be one of those times where BBR and pacing<br>
> actually is<br>
> > getting hurt for playing nice in some very<br>
> variable<br>
> > bottleneck on the way?<br>
> ><br>
> ><br>
> > Possibly. Would you be able to take a tcpdump trace<br>
> of each<br>
> > trial (headers only would be ideal), and post on a<br>
> web site<br>
> > somewhere a pcap trace for one of the slow trials?<br>
> ><br>
> ><br>
> > For example:<br>
> ><br>
> ><br>
> > tcpdump -n -w /tmp/out.pcap -s 120 -i eth0 -c<br>
> 1000000 &<br>
> ><br>
> ><br>
> ><br>
> > thanks,<br>
> > neal<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > On 25 January 2017 at 23:01, Neal Cardwell<br>
> > <<a href="mailto:ncardwell@google.com">ncardwell@google.com</a>> wrote:<br>
> > On Wed, Jan 25, 2017 at 3:54 PM,<br>
> Hans-Kristian<br>
> > Bakke <<a href="mailto:hkbakke@gmail.com">hkbakke@gmail.com</a>> wrote:<br>
> > Hi<br>
> ><br>
> ><br>
> > Kernel 4.9 finally landed in<br>
> Debian<br>
> > testing so I could finally<br>
> test BBR in<br>
> > a real life environment that<br>
> I have<br>
> > struggled with getting any<br>
> kind of<br>
> > performance out of.<br>
> ><br>
> ><br>
> > The challenge at hand is UDP<br>
> based<br>
> > OpenVPN through europe at<br>
> around 35 ms<br>
> > rtt to my VPN-provider with<br>
> plenty of<br>
> > available bandwith available<br>
> in both<br>
> > ends and everything<br>
> completely unknown<br>
> > in between. After tuning the<br>
> > UDP-buffers up to make room<br>
> for my 500<br>
> > mbit/s symmetrical bandwith<br>
> at 35 ms<br>
> > the download part seemed to<br>
> work<br>
> > nicely at an unreliable 150<br>
> to 300<br>
> > mbit/s, while the upload was<br>
> stuck at<br>
> > 30 to 60 mbit/s.<br>
> ><br>
> ><br>
> > Just by activating BBR the<br>
> bandwith<br>
> > instantly shot up to around<br>
> 150 mbit/s<br>
> > using a fat tcp test to a<br>
> public<br>
> > iperf3 server located near<br>
> my VPN exit<br>
> > point in the Netherlands.<br>
> Replace BBR<br>
> > with qubic again and the<br>
> performance<br>
> > is once again all over the<br>
> place<br>
> > ranging from very bad to<br>
> bad, but<br>
> > never better than 1/3 of<br>
> BBRs "steady<br>
> > state". In other words<br>
> "instant WIN!"<br>
> ><br>
> ><br>
> > Glad to hear it. Thanks for the test<br>
> report!<br>
> ><br>
> > However, seeing the<br>
> requirement of fq<br>
> > and pacing for BBR and<br>
> noticing that I<br>
> > am running pfifo_fast within<br>
> a VM with<br>
> > virtio NIC on a Proxmox VE<br>
> host with<br>
> > fq_codel on all physical<br>
> interfaces, I<br>
> > was surprised to see that it<br>
> worked so<br>
> > well.<br>
> > I then replaced pfifo_fast<br>
> with fq and<br>
> > the performance went right<br>
> down to<br>
> > only 1-4 mbit/s from around<br>
> 150<br>
> > mbit/s. Removing the fq<br>
> again regained<br>
> > the performance at once.<br>
> ><br>
> ><br>
> > I have got some questions to<br>
> you guys<br>
> > that know a lot more than me<br>
> about<br>
> > these things:<br>
> > 1. Do fq (and fq_codel) even<br>
> work<br>
> > reliably in a VM? What is<br>
> the best<br>
> > choice for default qdisc to<br>
> use in a<br>
> > VM in general?<br>
> ><br>
> ><br>
> > Eric covered this one. We are not<br>
> aware of<br>
> > specific issues with fq in VM<br>
> environments.<br>
> > And we have tested that fq works<br>
> sufficiently<br>
> > well on Google Cloud VMs.<br>
> ><br>
> > 2. Why do BBR immediately<br>
> "fix" all my<br>
> > issues with upload through<br>
> that<br>
> > "unreliable" big BDP link<br>
> with<br>
> > pfifo_fast when fq pacing is<br>
> a<br>
> > requirement?<br>
> ><br>
> ><br>
> > For BBR, pacing is part of the<br>
> design in order<br>
> > to make BBR more "gentle" in terms<br>
> of the rate<br>
> > at which it sends, in order to put<br>
> less<br>
> > pressure on buffers and keep packet<br>
> loss<br>
> > lower. This is particularly<br>
> important when a<br>
> > BBR flow is restarting from idle. In<br>
> this case<br>
> > BBR starts with a full cwnd, and it<br>
> counts on<br>
> > pacing to pace out the packets at<br>
> the<br>
> > estimated bandwidth, so that the<br>
> queue can<br>
> > stay relatively short and yet the<br>
> pipe can be<br>
> > filled immediately.<br>
> ><br>
> ><br>
> > Running BBR without pacing makes BBR<br>
> more<br>
> > aggressive, particularly in<br>
> restarting from<br>
> > idle, but also in the steady state,<br>
> where BBR<br>
> > tries to use pacing to keep the<br>
> queue short.<br>
> ><br>
> ><br>
> > For bulk transfer tests with one<br>
> flow, running<br>
> > BBR without pacing will likely cause<br>
> higher<br>
> > queues and loss rates at the<br>
> bottleneck, which<br>
> > may negatively impact other traffic<br>
> sharing<br>
> > that bottleneck.<br>
> ><br>
> > 3. Could fq_codel on the<br>
> physical host<br>
> > be the reason that it still<br>
> works?<br>
> ><br>
> ><br>
> > Nope, fq_codel does not implement<br>
> pacing.<br>
> ><br>
> > 4. Do BBR _only_ work with<br>
> fq pacing<br>
> > or could fq_codel be used as<br>
> a<br>
> > replacement?<br>
> ><br>
> ><br>
> > Nope, BBR needs pacing to work<br>
> correctly, and<br>
> > currently fq is the only Linux qdisc<br>
> that<br>
> > implements pacing.<br>
> ><br>
> > 5. Is BBR perhaps modified<br>
> to do the<br>
> > right thing without having<br>
> to change<br>
> > the qdisc in the current<br>
> kernel 4.9?<br>
> ><br>
> ><br>
> > Nope. Linux 4.9 contains the initial<br>
> public<br>
> > release of BBR from September 2016.<br>
> And there<br>
> > have been no code changes since then<br>
> (just<br>
> > expanded comments).<br>
> ><br>
> ><br>
> > Thanks for the test report!<br>
> ><br>
> ><br>
> > neal<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
><br>
> > ______________________________<wbr>_________________<br>
> > Bloat mailing list<br>
> > <a href="mailto:Bloat@lists.bufferbloat.net">Bloat@lists.bufferbloat.net</a><br>
> > <a href="https://lists.bufferbloat.net/listinfo/bloat" rel="noreferrer" target="_blank">https://lists.bufferbloat.net/<wbr>listinfo/bloat</a><br>
><br>
><br>
><br>
><br>
><br>
<br>
<br>
</div></div></blockquote></div><br></div>