From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qk1-x744.google.com (mail-qk1-x744.google.com [IPv6:2607:f8b0:4864:20::744]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 811733B29D for ; Wed, 5 Feb 2020 09:50:02 -0500 (EST) Received: by mail-qk1-x744.google.com with SMTP id v195so2018630qkb.11 for ; Wed, 05 Feb 2020 06:50:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=4mfsbHBOBQtFmYZ2nT6OVswUCITQFzZMuTVBHe2H94Y=; b=uJVKs5BmPxdADpCIhljbakOWiqDaFsRL/2r9LipGFj05aWiVZQqyd10FisAf5dnFXm Z6mZr2KkYaJ6o0XZTtMVRg7ccH6N/9IUJtfTZpsGHU4PyqxH3SgZlu9oEtwtfSgX3DCY C3mJ4sk5SpeNsRN3bK1nGFpa52wX8+GaPNjGPg+tDEJR5/f9YnUSdzXMOSb1GwfXF76y hO2E/LbDFP7EgQYZemZpyuhATKF/eAKg1m0fwTO1FWWo+pnwZr2BgnWFlpqlVDX3aRKa Juuo1t3AMNMRPgezMnHuVGavJAEgNHmpbTe+POtO/IvCaMHuEoU94x16C6U8m8DSBTSI CrBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=4mfsbHBOBQtFmYZ2nT6OVswUCITQFzZMuTVBHe2H94Y=; b=Jg5hzcSy/XLK9G43sURZkQ+Hd4E9En5/lwtZ61xmgylCRfFCrAfsOCZVUOpdraC07O o/Uk41DtjZE8dPA9N2cnvKP0cIL14mbg0Exui5kl0Wr2pHxCt0GIgdBEiX3egpWKo4jM Cb2uJx8vkb8kDe1O6au1Y3UtJt+2MK+Ffh5681C7U2kbz5qr6mGMFBNGfA2C3p8d5ufQ GNYYcvh3KU2k3kIh35IipCSNij2Zpj0Z1FvjM4BpDroKr8Q5bbG7C25Hc+V7TqgCEj8A unRhHRPEcasnQQ2on9HccQWi4AFegkUeTIEhqSjcu6ps7j5FQ4dzQqHJymgSqI6CYoyX Uo7Q== X-Gm-Message-State: APjAAAU0+HyTS49TewKkbX3TBdj3FDUiXngqUQ2m0mp0djlIXHhk+fxg CUn1mjF1tFzj1BEWdcOjRFZb/q3TF8Y= X-Google-Smtp-Source: APXvYqwNrK9PI2IMlq/7Ymuju11NKw9PhzPTwGwr/zEF2wlzNOn5Ag7T1ASjaVpE7u55xLQed/bHhQ== X-Received: by 2002:a37:a513:: with SMTP id o19mr13903545qke.13.1580914201929; Wed, 05 Feb 2020 06:50:01 -0800 (PST) Received: from ?IPv6:2001:470:8c46:0:e57d:726c:6028:4317? ([2001:470:8c46:0:e57d:726c:6028:4317]) by smtp.gmail.com with ESMTPSA id n3sm13171639qkn.105.2020.02.05.06.50.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Feb 2020 06:50:00 -0800 (PST) From: Rich Brown Message-Id: <07A876F7-97C3-45D2-9950-82B7E44E5641@gmail.com> Content-Type: multipart/alternative; boundary="Apple-Mail=_57D07184-7492-4583-BD25-F8D93CD8A540" Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3601.0.10\)) Date: Wed, 5 Feb 2020 09:49:59 -0500 In-Reply-To: <87o8ude9gh.fsf@toke.dk> Cc: Taran Lynn , bloat@lists.bufferbloat.net To: =?utf-8?Q?Toke_H=C3=B8iland-J=C3=B8rgensen?= References: <9b84ced6-dc62-90ee-33c8-807c5c0a4a17@gmail.com> <87o8ude9gh.fsf@toke.dk> X-Mailer: Apple Mail (2.3601.0.10) Subject: Re: [Bloat] Can't Run Tests Against netperf.bufferbloat.net X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Feb 2020 14:50:02 -0000 --Apple-Mail=_57D07184-7492-4583-BD25-F8D93CD8A540 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi all, Thanks for the note. Yes, the netperf server at netperf.bufferbloat.net = is turned off. The VPS that runs it is = consuming its bandwidth limit (4TB per month) at an ever-increasing = rate. When that happens, my hosting service (Ramnode.com = - good guys, stable hosting, great tech support = service) automatically turns off the VPS 'til the start of the next = month. In the distant past, the 4 TB sufficed for the entire month. More = recently, I would occasionally get a 90% warning by the 25th or 26th of = the month. Last month, I hit 90% on Jan 6th(!) so I shut off the netperf = server so I could continue to work on it. I briefly turned on netserver on the VPS today. At 08:15 today, the VPS = control panel showed: 181.7 MB of 3.9 TB Used. At 09:16 today, it showed = 46.3 GB. 46 GBytes in 1 hour =3D> ~30 TB/month (!)=20 I'm going to appeal to the group's collective wisdom to find a better = solution. Current mitigations: - I use iptables to log all netperf connections. I see a pattern of = certain IP addresses that seem to be firing off a test every five = minutes, 24 x 7 for days at a time. - A few times a month, I run a script (see findunfilteredips.sh in = https://github.com/richb-hanover/netperfclean = ) that scans the log = files to count netperf connections and to block devices (using iptables) = that have made more than 5,000 connections in the last seven days. This = helps, but only delays the inevitable. Potential (additional) mitigations: - We could change DNS to spread the load of netperf.bufferbloat.net = across our fleet of servers. = (Researchers who need consistent results could still choose a specific = server: netperf-east, netperf-west, etc.) - I could automate the current script to look for heavy users every day = or two. - Maybe I'm doing iptables imperfectly - comments appreciated. - I have toyed with the notion of tweaking the iptables rules to = throttle heavy users (over a certain number of tests/connections per = time-period). That way, the 24x7 people would receive, say 3kbps instead = of the actual link speed. There are a couple difficulties: a) I don't want to inconvenience actual researchers/bufferbloat = testers. When I test a connection, I typically make 3-10 tests in rapid = succession before I go away. This looks an awful lot like the 24x7 = folks, except that real testers stop after 15 minutes. Could iptables be = tweaked to tell one from the other? b) When I looked into this, I realized I might need to move the = VPS from OpenVZ (which has limited iptables capabilities - no 'ipset' = for example) to KVM (which is full virtualization). - I could just buy more bandwidth. Currently, I pay $194/year for this = server with the 4TB limit. Additional bandwidth on this provider is = $48/year per additional TB. But 30 TB/month would be pricey. - I could move to a different hosting service where bandwidth is = cheaper. (Any recommendations?) - Other thoughts?=20 Thanks. Rich > On Feb 5, 2020, at 3:15 AM, Toke H=C3=B8iland-J=C3=B8rgensen = wrote: >=20 > Taran Lynn writes: >=20 >> All of my attempts to run netperf (and flent) against >> netperf.bufferbloat.net have failed for the past couple of weeks. >> Netperf seems to work fine between my desktop and laptop, so I don't >> think the issue is on my end. Can anyone verify if the server is up? >=20 > Rich, I think this one is yours? Did netserver die or something? >=20 > -Toke --Apple-Mail=_57D07184-7492-4583-BD25-F8D93CD8A540 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi = all,

Thanks for the = note. Yes, the netperf server at netperf.bufferbloat.net is turned off. The VPS that = runs it is consuming its bandwidth limit (4TB per month) at an = ever-increasing rate. When that happens, my hosting service (Ramnode.com - good guys, = stable hosting, great tech support service) automatically turns off the = VPS 'til the start of the next month.

In the distant past, the 4 TB sufficed = for the entire month. More recently, I would occasionally get a 90% = warning by the 25th or 26th of the month. Last month, I hit 90% on Jan = 6th(!) so I shut off the netperf server so I could continue to work on = it.

I = briefly turned on netserver on the VPS today. At 08:15 today, the VPS = control panel showed: 181.7 MB of 3.9 TB Used. = At 09:16 today, it showed 46.3 GB. 46 = GBytes in 1 hour =3D> ~30 TB/month (!) 

I'm going to appeal to = the group's collective wisdom to find a better solution.

Current = mitigations:

- = I use iptables to log all netperf connections. I see a pattern of = certain IP addresses that seem to be firing off a test every five = minutes, 24 x 7 for days at a time.

- A few times a month, I run a script = (see findunfilteredips.sh in https://github.com/richb-hanover/netperfclean) that scans = the log files to count netperf connections and to block devices (using = iptables) that have made more than 5,000 connections in the last seven = days. This helps, but only delays the inevitable.
Potential (additional) = mitigations:

- = We could change DNS to spread the load of netperf.bufferbloat.net across our fleet of servers. = (Researchers who need consistent results could still choose a specific = server: netperf-east, netperf-west, etc.)

- I could automate the current script = to look for heavy users every day or two.

- Maybe I'm doing iptables imperfectly = - comments appreciated.

- I have toyed with the notion of tweaking the iptables rules = to throttle heavy users (over a certain number of tests/connections per = time-period). That way, the 24x7 people would receive, say 3kbps instead = of the actual link speed. There are a couple difficulties:
= a) I don't want to inconvenience actual researchers/bufferbloat = testers. When I test a connection, I typically make 3-10 tests in rapid = succession before I go away. This looks an awful lot like the 24x7 = folks, except that real testers stop after 15 minutes. Could iptables be = tweaked to tell one from the other?
b) When I = looked into this, I realized I might need to move the VPS from OpenVZ = (which has limited iptables capabilities - no 'ipset' for example) to = KVM (which is full virtualization).

- I could just buy more bandwidth. = Currently, I pay $194/year for this server with the 4TB limit. = Additional bandwidth on this provider is $48/year per additional TB. But = 30 TB/month would be pricey.

- I could move to a different hosting = service where bandwidth is cheaper. (Any recommendations?)

- Other = thoughts? 

Thanks.

Rich


On Feb 5, 2020, at 3:15 AM, Toke H=C3=B8iland-J=C3=B8rgensen = <toke@toke.dk> = wrote:

Taran Lynn <taranlynn0@gmail.com> writes:

All of my attempts to = run netperf (and flent) against
netperf.bufferbloat.net have failed for the past couple = of weeks.
Netperf seems to work fine between my desktop = and laptop, so I don't
think the issue is on my end. Can = anyone verify if the server is up?

Rich, I think this one is yours? Did netserver die or = something?

-Toke

= --Apple-Mail=_57D07184-7492-4583-BD25-F8D93CD8A540--