From: "Toke Høiland-Jørgensen" <toke@toke.dk>
To: Rich Brown <richb.hanover@gmail.com>
Cc: Taran Lynn <taranlynn0@gmail.com>, bloat@lists.bufferbloat.net
Subject: Re: [Bloat] Can't Run Tests Against netperf.bufferbloat.net
Date: Wed, 05 Feb 2020 17:12:00 +0100 [thread overview]
Message-ID: <871rr9dne7.fsf@toke.dk> (raw)
In-Reply-To: <07A876F7-97C3-45D2-9950-82B7E44E5641@gmail.com>
Rich Brown <richb.hanover@gmail.com> writes:
> Hi all,
>
> Thanks for the note. Yes, the netperf server at
> netperf.bufferbloat.net <http://netperf.bufferbloat.net/> is turned
> off. The VPS that runs it is consuming its bandwidth limit (4TB per
> month) at an ever-increasing rate. When that happens, my hosting
> service (Ramnode.com <http://ramnode.com/> - good guys, stable
> hosting, great tech support service) automatically turns off the VPS
> 'til the start of the next month.
Right, good to know; thanks for the explanation! And for taking on
running this service! I think we should try to find something more
sustainable, though, see below:
> In the distant past, the 4 TB sufficed for the entire month. More
> recently, I would occasionally get a 90% warning by the 25th or 26th
> of the month. Last month, I hit 90% on Jan 6th(!) so I shut off the
> netperf server so I could continue to work on it.
>
> I briefly turned on netserver on the VPS today. At 08:15 today, the
> VPS control panel showed: 181.7 MB of 3.9 TB Used. At 09:16 today, it
> showed 46.3 GB. 46 GBytes in 1 hour => ~30 TB/month (!)
>
> I'm going to appeal to the group's collective wisdom to find a better
> solution.
>
> Current mitigations:
>
> - I use iptables to log all netperf connections. I see a pattern of
> certain IP addresses that seem to be firing off a test every five
> minutes, 24 x 7 for days at a time.
>
> - A few times a month, I run a script (see findunfilteredips.sh in
> https://github.com/richb-hanover/netperfclean
> <https://github.com/richb-hanover/netperfclean>) that scans the log
> files to count netperf connections and to block devices (using
> iptables) that have made more than 5,000 connections in the last seven
> days. This helps, but only delays the inevitable.
>
> Potential (additional) mitigations:
>
> - We could change DNS to spread the load of netperf.bufferbloat.net
> <http://netperf.bufferbloat.net/> across our fleet of servers.
> (Researchers who need consistent results could still choose a specific
> server: netperf-east, netperf-west, etc.)
I think we should definitely do this, maybe even do something
geoip-based to select the "closest". I'll look into options...
> - I could automate the current script to look for heavy users every
> day or two.
Personally I think it would be fine if you just ban heavy users with
reckless abandon :)
> - Maybe I'm doing iptables imperfectly - comments appreciated.
>
> - I have toyed with the notion of tweaking the iptables rules to
> throttle heavy users (over a certain number of tests/connections per
> time-period). That way, the 24x7 people would receive, say 3kbps
> instead of the actual link speed. There are a couple difficulties:
> a) I don't want to inconvenience actual researchers/bufferbloat
> testers. When I test a connection, I typically make 3-10 tests
> in rapid succession before I go away. This looks an awful lot
> like the 24x7 folks, except that real testers stop after 15
> minutes. Could iptables be tweaked to tell one from the other?
>
> b) When I looked into this, I realized I might need to move the
> VPS from OpenVZ (which has limited iptables capabilities - no
> 'ipset' for example) to KVM (which is full virtualization).
What about just giving each IP a bandwidth cap? It may need more kernel
capabilities to support this efficiently, though; so moving to KVM may
be necessary. What kernel version does you openvz host have?
> - I could just buy more bandwidth. Currently, I pay $194/year for this
> server with the 4TB limit. Additional bandwidth on this provider is
> $48/year per additional TB. But 30 TB/month would be pricey.
>
> - I could move to a different hosting service where bandwidth is
> cheaper. (Any recommendations?)
Rather than spend more money bankrolling a free service, I think we
should see if we can find some way to sponsor this from an organisation
that doesn't pay per the byte. Anyone who knows someone at a university
in the US? I'll see if Red Hat can do anything for us...
-Toke
next prev parent reply other threads:[~2020-02-05 16:12 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-02-05 2:18 Taran Lynn
2020-02-05 8:15 ` Toke Høiland-Jørgensen
2020-02-05 14:49 ` Rich Brown
2020-02-05 16:12 ` Toke Høiland-Jørgensen [this message]
2020-02-05 21:55 ` Matt Taggart
2020-02-05 8:57 ` Sebastian Moeller
[not found] <mailman.5.1581008402.30625.bloat@lists.bufferbloat.net>
2020-02-06 23:47 ` Rich Brown
2020-02-07 11:11 ` Toke Høiland-Jørgensen
2020-02-07 12:02 ` Jesper Dangaard Brouer
2020-02-08 22:35 ` Rich Brown
2020-02-08 23:17 ` Rich Brown
2020-02-09 16:31 ` Dave Taht
2020-02-09 19:08 ` Dave Taht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://lists.bufferbloat.net/postorius/lists/bloat.lists.bufferbloat.net/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=871rr9dne7.fsf@toke.dk \
--to=toke@toke.dk \
--cc=bloat@lists.bufferbloat.net \
--cc=richb.hanover@gmail.com \
--cc=taranlynn0@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox