[Bloat] Can't Run Tests Against netperf.bufferbloat.net

[Toke Høiland-Jørgensen]

Rich Brown <richb.hanover at gmail.com> writes:

> Hi all,
>
> Thanks for the note. Yes, the netperf server at
> netperf.bufferbloat.net <http://netperf.bufferbloat.net/> is turned
> off. The VPS that runs it is consuming its bandwidth limit (4TB per
> month) at an ever-increasing rate. When that happens, my hosting
> service (Ramnode.com <http://ramnode.com/> - good guys, stable
> hosting, great tech support service) automatically turns off the VPS
> 'til the start of the next month.

Right, good to know; thanks for the explanation! And for taking on
running this service! I think we should try to find something more
sustainable, though, see below:

> In the distant past, the 4 TB sufficed for the entire month. More
> recently, I would occasionally get a 90% warning by the 25th or 26th
> of the month. Last month, I hit 90% on Jan 6th(!) so I shut off the
> netperf server so I could continue to work on it.
>
> I briefly turned on netserver on the VPS today. At 08:15 today, the
> VPS control panel showed: 181.7 MB of 3.9 TB Used. At 09:16 today, it
> showed 46.3 GB. 46 GBytes in 1 hour => ~30 TB/month (!)
>
> I'm going to appeal to the group's collective wisdom to find a better
> solution.
>
> Current mitigations:
>
> - I use iptables to log all netperf connections. I see a pattern of
> certain IP addresses that seem to be firing off a test every five
> minutes, 24 x 7 for days at a time.
>
> - A few times a month, I run a script (see findunfilteredips.sh in
> https://github.com/richb-hanover/netperfclean
> <https://github.com/richb-hanover/netperfclean>) that scans the log
> files to count netperf connections and to block devices (using
> iptables) that have made more than 5,000 connections in the last seven
> days. This helps, but only delays the inevitable.
>
> Potential (additional) mitigations:
>
> - We could change DNS to spread the load of netperf.bufferbloat.net
> <http://netperf.bufferbloat.net/> across our fleet of servers.
> (Researchers who need consistent results could still choose a specific
> server: netperf-east, netperf-west, etc.)

I think we should definitely do this, maybe even do something
geoip-based to select the "closest". I'll look into options...

> - I could automate the current script to look for heavy users every
> day or two.

Personally I think it would be fine if you just ban heavy users with
reckless abandon :)

> - Maybe I'm doing iptables imperfectly - comments appreciated.
>
> - I have toyed with the notion of tweaking the iptables rules to
> throttle heavy users (over a certain number of tests/connections per
> time-period). That way, the 24x7 people would receive, say 3kbps
> instead of the actual link speed. There are a couple difficulties:

> 	a) I don't want to inconvenience actual researchers/bufferbloat
> 	testers. When I test a connection, I typically make 3-10 tests
> 	in rapid succession before I go away. This looks an awful lot
> 	like the 24x7 folks, except that real testers stop after 15
> 	minutes. Could iptables be tweaked to tell one from the other?
>
> 	b) When I looked into this, I realized I might need to move the
> 	VPS from OpenVZ (which has limited iptables capabilities - no
> 	'ipset' for example) to KVM (which is full virtualization).

What about just giving each IP a bandwidth cap? It may need more kernel
capabilities to support this efficiently, though; so moving to KVM may
be necessary. What kernel version does you openvz host have?

> - I could just buy more bandwidth. Currently, I pay $194/year for this
> server with the 4TB limit. Additional bandwidth on this provider is
> $48/year per additional TB. But 30 TB/month would be pricey.
>
> - I could move to a different hosting service where bandwidth is
> cheaper. (Any recommendations?)

Rather than spend more money bankrolling a free service, I think we
should see if we can find some way to sponsor this from an organisation
that doesn't pay per the byte. Anyone who knows someone at a university
in the US? I'll see if Red Hat can do anything for us...

-Toke