From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.toke.dk (mail.toke.dk [45.145.95.4]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 7D4B23B29D for ; Wed, 5 Feb 2020 11:12:02 -0500 (EST) From: Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=toke.dk; s=20161023; t=1580919121; bh=VxQY+khm3zf8mDo6m5T0DziEkKmDXnVqCPYuVX9Q+Fo=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=m1cY+s3iveeqIIHq9EKontimGvQs7jfIydvpRsQgy2/2rFSskbrCUgTdBoBzgPai7 8APrM+pLYyu11Ku1V0KzejoExFFU7jQEb7onUVK5ljsnb45qUWRFOz03v78mWuIdKG mPb6AQcd0/jnvgSsIxtoZdXrF6UFRGcWLA2x0Iz08T8J4/t5h+LRpDkhSuAdNb1f4l eBWmozqk6NQrOKlLGo3Ma7t6dCutjC6Veqzm5F1uvEc3A9nUgs/PDWAKT+YYVtfX5z lVmPvTBNFt03TOiL8NLX7Om9V6EQcEmQU8VU300P7KH/ktzIPhH5Axp3TfawbJpV+c sC4lmSpK3kw/A== To: Rich Brown Cc: Taran Lynn , bloat@lists.bufferbloat.net In-Reply-To: <07A876F7-97C3-45D2-9950-82B7E44E5641@gmail.com> References: <9b84ced6-dc62-90ee-33c8-807c5c0a4a17@gmail.com> <87o8ude9gh.fsf@toke.dk> <07A876F7-97C3-45D2-9950-82B7E44E5641@gmail.com> Date: Wed, 05 Feb 2020 17:12:00 +0100 X-Clacks-Overhead: GNU Terry Pratchett Message-ID: <871rr9dne7.fsf@toke.dk> MIME-Version: 1.0 Content-Type: text/plain Subject: Re: [Bloat] Can't Run Tests Against netperf.bufferbloat.net X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Feb 2020 16:12:02 -0000 Rich Brown writes: > Hi all, > > Thanks for the note. Yes, the netperf server at > netperf.bufferbloat.net is turned > off. The VPS that runs it is consuming its bandwidth limit (4TB per > month) at an ever-increasing rate. When that happens, my hosting > service (Ramnode.com - good guys, stable > hosting, great tech support service) automatically turns off the VPS > 'til the start of the next month. Right, good to know; thanks for the explanation! And for taking on running this service! I think we should try to find something more sustainable, though, see below: > In the distant past, the 4 TB sufficed for the entire month. More > recently, I would occasionally get a 90% warning by the 25th or 26th > of the month. Last month, I hit 90% on Jan 6th(!) so I shut off the > netperf server so I could continue to work on it. > > I briefly turned on netserver on the VPS today. At 08:15 today, the > VPS control panel showed: 181.7 MB of 3.9 TB Used. At 09:16 today, it > showed 46.3 GB. 46 GBytes in 1 hour => ~30 TB/month (!) > > I'm going to appeal to the group's collective wisdom to find a better > solution. > > Current mitigations: > > - I use iptables to log all netperf connections. I see a pattern of > certain IP addresses that seem to be firing off a test every five > minutes, 24 x 7 for days at a time. > > - A few times a month, I run a script (see findunfilteredips.sh in > https://github.com/richb-hanover/netperfclean > ) that scans the log > files to count netperf connections and to block devices (using > iptables) that have made more than 5,000 connections in the last seven > days. This helps, but only delays the inevitable. > > Potential (additional) mitigations: > > - We could change DNS to spread the load of netperf.bufferbloat.net > across our fleet of servers. > (Researchers who need consistent results could still choose a specific > server: netperf-east, netperf-west, etc.) I think we should definitely do this, maybe even do something geoip-based to select the "closest". I'll look into options... > - I could automate the current script to look for heavy users every > day or two. Personally I think it would be fine if you just ban heavy users with reckless abandon :) > - Maybe I'm doing iptables imperfectly - comments appreciated. > > - I have toyed with the notion of tweaking the iptables rules to > throttle heavy users (over a certain number of tests/connections per > time-period). That way, the 24x7 people would receive, say 3kbps > instead of the actual link speed. There are a couple difficulties: > a) I don't want to inconvenience actual researchers/bufferbloat > testers. When I test a connection, I typically make 3-10 tests > in rapid succession before I go away. This looks an awful lot > like the 24x7 folks, except that real testers stop after 15 > minutes. Could iptables be tweaked to tell one from the other? > > b) When I looked into this, I realized I might need to move the > VPS from OpenVZ (which has limited iptables capabilities - no > 'ipset' for example) to KVM (which is full virtualization). What about just giving each IP a bandwidth cap? It may need more kernel capabilities to support this efficiently, though; so moving to KVM may be necessary. What kernel version does you openvz host have? > - I could just buy more bandwidth. Currently, I pay $194/year for this > server with the 4TB limit. Additional bandwidth on this provider is > $48/year per additional TB. But 30 TB/month would be pricey. > > - I could move to a different hosting service where bandwidth is > cheaper. (Any recommendations?) Rather than spend more money bankrolling a free service, I think we should see if we can find some way to sponsor this from an organisation that doesn't pay per the byte. Anyone who knows someone at a university in the US? I'll see if Red Hat can do anything for us... -Toke