[Bloat] Can't Run Tests Against netperf.bufferbloat.net

Sun Feb 9 14:08:28 EST 2020

OK, so I did a few test results from flent-atlanta.taht.net. They are here:

http://flent-atlanta.taht.net/~d/atl.tgz

Trying to run it at close to a gbit, there are all sorts of
interesting behavior between the two dcs. Using sqm + sch_fq I got
mildly saner results,
but... well, download the above and scratch your head some....

On Sun, Feb 9, 2020 at 8:31 AM Dave Taht <dave at taht.net> wrote:
>
>
> I note that I totally missed this thread somehow.
>
> I have about 12 (rather undermaintained) servers in linode's cloud these
> days that have netperf and irtt on them, all over the world, singapore,
> mumbai, etc, etc....
>
> The monthly transfer allotment is ~28TB across them all. So far this
> month we've used up ~1.2TB, principally from flent-fremont.bufferbloat.net.
>
> The current cost, which includes bufferbloat.net and taht.net, is
>  180/month.  I just vector a portion of the donations to patreon over
>  there and don't think about it much. All of these boxes could use love
>  and an upgrade (softwarewise), especially bufferbloat.net, which could
>  also get a downgrade cpuwise. The investment of time (days) to do that,
>  vs the ongoing extra cost of a 2 cpu machine (15/dollars/month) has not
>  seemed worth it.
>
> I would like it very much if we were more serious about a published
> reliable test infrastructure for flent. However, it was also my hope
> that flent would get mostly used inside people's firewalls on their own
> testbeds while developing new gear and drivers and network designs, as
> it's so easy to do so (apt-get install irtt netperf flent etc) nowadays.
>
> One of my concerns has long been that I have no idea what lies
> underneath these cloudy vms (anywhere) that is actually doing the rate
> limiting (anyone? AWS? linode?), and have kind of wished we could go
> back to at least one piece of bare metal on the internet as we had when
> isc.org was still donating services to us. Back then vm jitter was
> measured in the 10s of ms using xen... and we have no means to compare
> vm performance vs bare metal performance with all this finicy stuff,
> such as pacing, SCE/L4S etc. The google folk are perpetually publishing
> ce_threshold set at 260us, which is just impossible (IMHO) in a vm....
>
> The "nanodes" in this cloud are limited to 100Mbit, somehow.
>
> I've occasionally priced out what it would take to get a 10Gbit in
> hurricane's cages in california - 900/month just for transit when on
> special, 2k a month otherwise.... Half a cage was another 400 or so. I
> already have a spare /24 and BGP AS and hardware I could put in it,
> but...
>
> Rich Brown <richb.hanover at gmail.com> writes:
>
> > Update: (I thought I had sent the previous message yesterday. My mistake.)
> >
> > I now have atl3.richb-hanover.com running a netperf server. it's a
> > stock Ubuntu 18.04.4 LTS - uname -a shows: Linux atl3
> > 4.15.0-76-generic #86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 2020 x86_64
> > x86_64 x86_64 GNU/Linux. I have installed netperf 2.6.0, and little
> > else.
>
> I imagine fq_codel is the underlying qdisc?
>
> irtt would be nice.
>
> At some point, somehow, using some tool, we need to start supporting QUIC.
>
> >
> > Next steps:
> >
> > 1) Please hammer on the server to see if it's a suitable replacement
> > for the canonical "netperf.bufferbloat.net". Please feel free to check
> > both its ability to handle traffic as well as any security surprises
> > you discover...
>
> Thank you so much for maintaining this server for so long and I had no
> idea before this thread that it was being so hard hit.
>
>
> > 2) I welcome suggestions for configuring the server's TCP stack to be
> > most useful for researchers. fq_codel, bbr, - I'm open to your
> > thoughts.
>
> I have generally made available cubic, reno, bbr, and dctcp. I was
> always rather reluctant to publish where I'd turned on dctcp given that
> it only recently gained a response to packet loss. In fact, reluctant to
> publish anything in the cloud.
>
> I had (briefly) an SCE capable machine up in singapore. It would be
> good if more folk could try cubic-sce, reno-sce, dctcp-sce, bbr-sce, and
> for that matter, play with l4s in the same dc on an equivalent machine,
> but that involves making custom kernels for each at present. I'm also
> dying to try out the ebpf + etx stuff google is presenting at netdevconf....
>
> >
> > 3) It's not too soon for advice on an iptables strategy for limiting
> > the access/bandwidth/traffic to people who're abusing the service...
>
> I'd love to have that too!
>
> > Once we have all this in place, we can change the netperf.bufferbloat.net name to point to this server. Thanks.
> >
> > Rich
> >
> >> On Feb 8, 2020, at 5:35 PM, Rich Brown <richb.hanover at gmail.com> wrote:
> >>
> >> Toke and Jesper,
> >>
> >> Thanks both for these responses.
> >>
> >> netperf.bufferbloat.net is running an OpenVZ VPS with a 3.10
> >> kernel. Tech support at Ramnode tells me that I need to get to a KVM
> >> instance in order to use ipset and other fancy kernel stuff.
> >>
> >> Here's my plan:
> >>
> >> 1) Unless anyone can recommend a better hosting service ...
> >>
> >> 2) Over the weekend, I'll stand up a new KVM server at Ramnode. They
> >> offer a 2GB RAM, 2 core, 65 GB SSD, with 3TB per month of
> >> data. It'll cost $10/month: adding 2x1TB at $4/month brings it to a
> >> total of $18/month, about what the current server costs. I can get
> >> Ubuntu 18.04 LTS as a standard install.
> >>
> >> 3) While that's in-flight I would request that an iptables expert on
> >> the list recommend a better strategy. (I was just makin' stuff up in
> >> the current setup - as you could tell :-)
> >>
> >> 4) I'd also accept any thoughts about tc commands for setting up the
> >> networking on the host to work best as a netperf server. (Maybe
> >> enable fq_codel or better...)
> >>
> >> Thanks
> >>
> >> Rich
> >>
> >>> On Feb 7, 2020, at 7:02 AM, Jesper Dangaard Brouer <brouer at redhat.com> wrote:
> >>>
> >>> On Thu, 6 Feb 2020 18:47:06 -0500
> >>> Rich Brown <richb.hanover at gmail.com> wrote:
> >>>
> >>>>> On Feb 6, 2020, at 12:00 PM, Matt Taggart wrote:
> >>>>>
> >>>>> This smells like a munin or smokeping plugin (or some other sort of
> >>>>> monitoring) gathering data for graphing.
> >>>>
> >>>> Yup. That is a real possibility. The question is what we do about it.
> >>>>
> >>>> If I understood, we left it at:
> >>>>
> >>>> 1) Toke was going to look into some way to spread the
> >>>> 'netperf.bufferbloat.net' load across several of our netperf servers.
> >>>>
> >>>> 2) Can someone give me advice about iptables/tc/? to identify IP
> >>>> addresses that make "too many" connections and either shut them off
> >>>> or dial their bandwidth back to a 3 or 5 kbps?
> >>>
> >>> Look at man iptables-extensions and find "connlimit" and "recent".
> >>>
> >>>
> >>>> (If you're terminally curious, Line 5 of
> >>>> https://github.com/richb-hanover/netperfclean/blob/master/addtoblacklist.sh
> >>>> shows the current iptables command to drop connections from "heavy
> >>>> users" identified in the findunfilteredips.sh script. You can read
> >>>> the current iptables rules at:
> >>>> https://github.com/richb-hanover/netperfclean/blob/master/iptables.txt)
> >>>
> >>> Sorry but this is a wrong approach.  Creating an iptables rule per
> >>> source IP-address, will (as you also demonstrate) give you a VERY long
> >>> list of rules (which is evaluated sequentially by the kernel).
> >>>
> >>> This should instead be solved by using an ipset (howto a match from
> >>> iptables see man iptables-extensions(8) and "set").  And use the
> >>> cmdline tool ipset to add and remove entries.
> >>>
> >>> --
> >>> Best regards,
> >>> Jesper Dangaard Brouer
> >>> MSc.CS, Principal Kernel Engineer at Red Hat
> >>> LinkedIn: http://www.linkedin.com/in/brouer
> >>>
> >>
> >
> > _______________________________________________
> > Bloat mailing list
> > Bloat at lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
> _______________________________________________
> Bloat mailing list
> Bloat at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

-- 
Make Music, Not War

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-435-0729