[Bloat] netperf server news

[Bloat] netperf server news

[Rich Brown]

To the Bloat list,

I had some time, so I looked into what it might take to keep the netperf.bufferbloat.net server on-line in the face of an unwitting "DDoS" attack - automated scripts that run tests every 5 minutes 24x7. The problem was that these tests would blow through my 4TB/month bandwidth allocation in a few days.

In the past, I had been irregularly running a set of scripts to count incoming netperf connections and blacklist (in iptables) those whose counts were too high. This wasn't good enough: it wasn't keeping up with the tidal wave of connections.

Last week, I revised those scripts to work as a cron job. The current parameters are: run the script every hour; process the last two days' of kern.log files; look for > 500 connections; drop those addresses in iptables.

There are currently 479 addresses blacklisted in iptables (that explains why the bandwidth was being consumed so quickly). There are only a few new addresses being added per day, so it seems that we have flushed out most of the abusers.

My questions for this august group:

1) The server at netperf.bufferbloat.net is up and running. I get full rate speed from my 7mbps DSL circuit, but that's not much of a test. I would be interested to hear your results.

2) The current threshold comes from this estimate: most speed tests use 10 connections: 5 connections up and 5 down. So 500 connections would permit about 50 tests over the course of two days. Is that enough for "real research"? (If you need more, I can add your address to my whitelist file...)

3) I would be pleased to get comments on the set of scripts. I'm a newbie at iptables, so it wouldn't hurt to have someone else check the rules I devised. See the README at https://github.com/richb-hanover/netperfclean

Thanks.

Rich

Re: [Bloat] netperf server news

[Sebastian Moeller]

Dear Rich,

first, thanks for supplying that service.

> On Oct 6, 2020, at 12:52, Rich Brown <richb.hanover@gmail.com> wrote:
> 
> To the Bloat list,
> 
> I had some time, so I looked into what it might take to keep the netperf.bufferbloat.net server on-line in the face of an unwitting "DDoS" attack - automated scripts that run tests every 5 minutes 24x7. The problem was that these tests would blow through my 4TB/month bandwidth allocation in a few days.
> 
> In the past, I had been irregularly running a set of scripts to count incoming netperf connections and blacklist (in iptables) those whose counts were too high. This wasn't good enough: it wasn't keeping up with the tidal wave of connections.
> 
> Last week, I revised those scripts to work as a cron job. The current parameters are: run the script every hour; process the last two days' of kern.log files; look for > 500 connections; drop those addresses in iptables.
> 
> There are currently 479 addresses blacklisted in iptables (that explains why the bandwidth was being consumed so quickly). There are only a few new addresses being added per day, so it seems that we have flushed out most of the abusers.
> 
> My questions for this august group:
> 
> 1) The server at netperf.bufferbloat.net is up and running. I get full rate speed from my 7mbps DSL circuit, but that's not much of a test. I would be interested to hear your results.

From work:

bash-3.2$ ./betterspeedtest.sh
2020-10-06 14:46:19 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (60 seconds in each direction)
.
 Download:  Mbps
  Latency: (in msec, 1 pings, 0.00% packet loss)
      Min: 6.868 
    10pct: 0.000 
   Median: 0.000 
      Avg: 6.868 
    90pct: 0.000 
      Max: 6.868
.............................................................
   Upload:  309.67 Mbps
  Latency: (in msec, 61 pings, 0.00% packet loss)
      Min: 6.644 
    10pct: 6.730 
   Median: 7.289 
      Avg: 7.385 
    90pct: 7.941 
      Max: 9.980
Press any key to continue...

bash-3.2$ ./betterspeedtest.sh
2020-10-06 14:49:33 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (60 seconds in each direction)
................................................................................
 Download:  0 Mbps
  Latency: (in msec, 80 pings, 0.00% packet loss)
      Min: 6.583 
    10pct: 6.637 
   Median: 6.674 
      Avg: 6.694 
    90pct: 6.743 
      Max: 7.204
................................................................................
   Upload:  0 Mbps
  Latency: (in msec, 80 pings, 0.00% packet loss)
      Min: 6.555 
    10pct: 6.622 
   Median: 6.667 
      Avg: 6.687 
    90pct: 6.742 
      Max: 7.218
Press any key to continue...

So there seems to be an issue with the Download test, from home I currently get 0/0 for both Upload/download.... Maybe I just made it on the block list (not that I remember trying to reach that server in the last weeks at all).

Running flent's rrul_cs8 manually against netperf.bufferbloat.net gave me around 80/25 which seems  believable.


> 
> 2) The current threshold comes from this estimate: most speed tests use 10 connections: 5 connections up and 5 down. So 500 connections would permit about 50 tests over the course of two days. Is that enough for "real research"? (If you need more, I can add your address to my whitelist file...)

	I think 50 tests is quite generous, that is more than one test every hour for two days ;)

> 
> 3) I would be pleased to get comments on the set of scripts. I'm a newbie at iptables, so it wouldn't hurt to have someone else check the rules I devised. See the README at https://github.com/richb-hanover/netperfclean

	Outside of my area of expertise....

Best Reards
	Sebastian

> 
> Thanks.
> 
> Rich
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] netperf server news

[Colin Dearborn]

1): This hits what I'd expect my connection to be (or close enough anyway):
./betterspeedtest.sh
2020-10-06 12:53:54 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (60 seconds in each direction)
.............................................................
 Download: 926.27 Mbps
  Latency: (in msec, 61 pings, 0.00% packet loss)
      Min: 23.200
    10pct: 25.500
   Median: 31.100
      Avg: 30.503
    90pct: 34.100
      Max: 38.800
...............................................................
   Upload: 103.50 Mbps
  Latency: (in msec, 63 pings, 0.00% packet loss)
      Min: 22.700
    10pct: 23.900
   Median: 28.600
      Avg: 30.233
    90pct: 33.400
      Max: 112.000

2) That sounds like plenty to me.


-----Original Message-----
From: Bloat <bloat-bounces@lists.bufferbloat.net> On Behalf Of Rich Brown
Sent: October 6, 2020 4:53 AM
To: bloat <bloat@lists.bufferbloat.net>
Cc: Richard E. Brown <richb.hanover@gmail.com>
Subject: [Bloat] netperf server news

  CAUTION: This email is from an external source. Do not click links or open attachments unless you recognize the sender and know the content is safe.

To the Bloat list,

I had some time, so I looked into what it might take to keep the netperf.bufferbloat.net server on-line in the face of an unwitting "DDoS" attack - automated scripts that run tests every 5 minutes 24x7. The problem was that these tests would blow through my 4TB/month bandwidth allocation in a few days.

In the past, I had been irregularly running a set of scripts to count incoming netperf connections and blacklist (in iptables) those whose counts were too high. This wasn't good enough: it wasn't keeping up with the tidal wave of connections.

Last week, I revised those scripts to work as a cron job. The current parameters are: run the script every hour; process the last two days' of kern.log files; look for > 500 connections; drop those addresses in iptables.

There are currently 479 addresses blacklisted in iptables (that explains why the bandwidth was being consumed so quickly). There are only a few new addresses being added per day, so it seems that we have flushed out most of the abusers.

My questions for this august group:

1) The server at netperf.bufferbloat.net is up and running. I get full rate speed from my 7mbps DSL circuit, but that's not much of a test. I would be interested to hear your results.

2) The current threshold comes from this estimate: most speed tests use 10 connections: 5 connections up and 5 down. So 500 connections would permit about 50 tests over the course of two days. Is that enough for "real research"? (If you need more, I can add your address to my whitelist file...)

3) I would be pleased to get comments on the set of scripts. I'm a newbie at iptables, so it wouldn't hurt to have someone else check the rules I devised. See the README at https://github.com/richb-hanover/netperfclean

Thanks.

Rich

_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] netperf server news

[Rich Brown]

Thanks for the feedback. Some responses:

1) I'm glad that people are seeing reasonable speeds from the VPS. (I don't know what I can do to make it go faster, so I'm relieved...)

2) I don't think I posed the right question for the number-of-tests threshold. (Most of the responses were like, "Sure, that sounds like enough..." Let me reframe the question: 

	In your normal testing/troubleshooting process, what is the maximum number of tests YOU might need to run in any two-day period?

3) If you can't get through to netperf.bufferbloat.net, send me your IP address because it might have been blacklisted.

Thanks!

Rich


> On Oct 6, 2020, at 6:52 AM, Rich Brown <richb.hanover@gmail.com> wrote:
> 
> To the Bloat list,
> 
> I had some time, so I looked into what it might take to keep the netperf.bufferbloat.net server on-line in the face of an unwitting "DDoS" attack - automated scripts that run tests every 5 minutes 24x7. The problem was that these tests would blow through my 4TB/month bandwidth allocation in a few days.
> 
> In the past, I had been irregularly running a set of scripts to count incoming netperf connections and blacklist (in iptables) those whose counts were too high. This wasn't good enough: it wasn't keeping up with the tidal wave of connections.
> 
> Last week, I revised those scripts to work as a cron job. The current parameters are: run the script every hour; process the last two days' of kern.log files; look for > 500 connections; drop those addresses in iptables.
> 
> There are currently 479 addresses blacklisted in iptables (that explains why the bandwidth was being consumed so quickly). There are only a few new addresses being added per day, so it seems that we have flushed out most of the abusers.
> 
> My questions for this august group:
> 
> 1) The server at netperf.bufferbloat.net is up and running. I get full rate speed from my 7mbps DSL circuit, but that's not much of a test. I would be interested to hear your results.
> 
> 2) The current threshold comes from this estimate: most speed tests use 10 connections: 5 connections up and 5 down. So 500 connections would permit about 50 tests over the course of two days. Is that enough for "real research"? (If you need more, I can add your address to my whitelist file...)
> 
> 3) I would be pleased to get comments on the set of scripts. I'm a newbie at iptables, so it wouldn't hurt to have someone else check the rules I devised. See the README at https://github.com/richb-hanover/netperfclean
> 
> Thanks.
> 
> Rich
> 

Re: [Bloat] netperf server news

[Dave Collier-Brown]

[-- Attachment #1: Type: text/plain, Size: 4151 bytes --]

Number?

One or two that were complete and successful.  Maybe five unsuccessful tries.

More of a sequence than a time-period: u{1-5}s{1-2}

--daev


On 2020-10-06 4:40 p.m., Rich Brown wrote:

Thanks for the feedback. Some responses:

1) I'm glad that people are seeing reasonable speeds from the VPS. (I don't know what I can do to make it go faster, so I'm relieved...)

2) I don't think I posed the right question for the number-of-tests threshold. (Most of the responses were like, "Sure, that sounds like enough..." Let me reframe the question:

        In your normal testing/troubleshooting process, what is the maximum number of tests YOU might need to run in any two-day period?

3) If you can't get through to netperf.bufferbloat.net, send me your IP address because it might have been blacklisted.

Thanks!

Rich




On Oct 6, 2020, at 6:52 AM, Rich Brown <richb.hanover@gmail.com><mailto:richb.hanover@gmail.com> wrote:

To the Bloat list,

I had some time, so I looked into what it might take to keep the netperf.bufferbloat.net server on-line in the face of an unwitting "DDoS" attack - automated scripts that run tests every 5 minutes 24x7. The problem was that these tests would blow through my 4TB/month bandwidth allocation in a few days.

In the past, I had been irregularly running a set of scripts to count incoming netperf connections and blacklist (in iptables) those whose counts were too high. This wasn't good enough: it wasn't keeping up with the tidal wave of connections.

Last week, I revised those scripts to work as a cron job. The current parameters are: run the script every hour; process the last two days' of kern.log files; look for > 500 connections; drop those addresses in iptables.

There are currently 479 addresses blacklisted in iptables (that explains why the bandwidth was being consumed so quickly). There are only a few new addresses being added per day, so it seems that we have flushed out most of the abusers.

My questions for this august group:

1) The server at netperf.bufferbloat.net is up and running. I get full rate speed from my 7mbps DSL circuit, but that's not much of a test. I would be interested to hear your results.

2) The current threshold comes from this estimate: most speed tests use 10 connections: 5 connections up and 5 down. So 500 connections would permit about 50 tests over the course of two days. Is that enough for "real research"? (If you need more, I can add your address to my whitelist file...)

3) I would be pleased to get comments on the set of scripts. I'm a newbie at iptables, so it wouldn't hurt to have someone else check the rules I devised. See the README at https://github.com/richb-hanover/netperfclean

Thanks.

Rich




_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net<mailto:Bloat@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/bloat


--
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
dave.collier-brown@indexexchange.com<mailto:dave.collier-brown@indexexchange.com> |              -- Mark Twain



CONFIDENTIALITY NOTICE AND DISCLAIMER : This telecommunication, including any and all attachments, contains confidential information intended only for the person(s) to whom it is addressed. Any dissemination, distribution, copying or disclosure is strictly prohibited and is not a waiver of confidentiality. If you have received this telecommunication in error, please notify the sender immediately by return electronic mail and delete the message from your inbox and deleted items folders. This telecommunication does not constitute an express or implied agreement to conduct transactions by electronic means, nor does it constitute a contract offer, a contract amendment or an acceptance of a contract offer. Contract terms contained in this telecommunication are subject to legal review and the completion of formal documentation and are not binding until same is confirmed in writing and has been signed by an authorized signatory.

[-- Attachment #2: Type: text/html, Size: 5177 bytes --]

Re: [Bloat] netperf server news

[Kenneth Porter]

--On Tuesday, October 06, 2020 7:52 AM -0400 Rich Brown 
<richb.hanover@gmail.com> wrote:

> 3) I would be pleased to get comments on the set of scripts. I'm a newbie
> at iptables, so it wouldn't hurt to have someone else check the rules I
> devised. See the README at https://github.com/richb-hanover/netperfclean

A couple of alternatives to custom scripts are fail2ban and the 
rate-limiting modules available for iptables such as hashlimit and recent. 
I haven't used fail2ban for rate-limiting so I'm not sure if it's the right 
tool for that but it monitors log files to add iptables rules for 
short-term banning. It's not hard to add your own log monitoring rule. I 
haven't used the iptables modules but they look like a natural solution for 
this.

<https://poorlydocumented.com/2017/08/understanding-iptables-hashlimit-module/>

<https://serverfault.com/questions/682045/source-ip-rate-limiting-in-iptables-hashlimit-vs-recent>

Instead of using a unique iptables rule for each blocklist member, I 
suggest using an ipset. (I use firewalld as a front-end to iptables so I 
let it manage my ipsets, but you can also install ipset's service for use 
with raw iptables to save and restore the sets across boots.) Your block 
rule could be as simple as this:

iptables -I INPUT 1 -p tcp --dport netperf -m set --match-set 
NetPerfAbusers src -m conntrack --ctstate NEW -j DROP

Re: [Bloat] netperf server news

[Kenneth Porter]

--On Wednesday, October 07, 2020 3:23 PM -0400 Rich Brown 
<richb.hanover@gmail.com> wrote:

> I'm also aware of ipset, which I take to be an optimized alternative to
> searching a long set of iptables rules (true?) I don't believe that my
> OpenVZ VPS has kernel support for this, so as long as the
> long-list-of-rules seems to work well, I'm going to leave it alone.

A quick google of "OpenVZ ipset" turned up a thread from 3 years ago 
suggesting it's in their kernel:

<https://forum.openvz.org/index.php?t=rview&goto=53549&th=13604>

Note that ipset operates in addition to iptables. Other kernel subsystems 
can also use them. iptables has a module to query an ipset.

500 rules is a lot to search linearly. I'd think a hash table would give 
much superior performance. Note that every "good" packet has to check ALL 
the blocking rules to be approved.

I use ipsets to block probes to my mail servers from outside the country 
and from cloud services. I've managed to find a few sources of lists for 
those. I also use ipset with fail2ban.

The only complicated part is how to handle reboots or other service 
restarts. I use firewalld which does its own ipset management so I put the 
permanent lists there. (I have scripts to convert the cloud lists to a 
firewalld's XML format for its ipset storage.) fail2ban keeps its own block 
database in a sqlite file and tears down and recreates its ipsets on 
restart.

Re: [Bloat] netperf server news

[Rich Brown]

> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Tue, 06 Oct 2020 19:39:54 -0700
> From: Kenneth Porter <shiva@sewingwitch.com>
> To: bloat <bloat@lists.bufferbloat.net>
> Subject: Re: [Bloat] netperf server news
> Message-ID: <38F0B196CFEA470FEEBE0520@[172.27.17.193]>
> Content-Type: text/plain; charset=us-ascii; format=flowed
> 
> --On Tuesday, October 06, 2020 7:52 AM -0400 Rich Brown 
> <richb.hanover@gmail.com> wrote:
> 
>> 3) I would be pleased to get comments on the set of scripts. I'm a newbie
>> at iptables, so it wouldn't hurt to have someone else check the rules I
>> devised. See the README at https://github.com/richb-hanover/netperfclean
> 
> A couple of alternatives to custom scripts are fail2ban and the 
> rate-limiting modules available for iptables such as hashlimit and recent. 
> I haven't used fail2ban for rate-limiting so I'm not sure if it's the right 
> tool for that but it monitors log files to add iptables rules for 
> short-term banning. It's not hard to add your own log monitoring rule. I 
> haven't used the iptables modules but they look like a natural solution for 
> this.
> 
> <https://poorlydocumented.com/2017/08/understanding-iptables-hashlimit-module/>
> 
> <https://serverfault.com/questions/682045/source-ip-rate-limiting-in-iptables-hashlimit-vs-recent>
> 
> Instead of using a unique iptables rule for each blocklist member, I 
> suggest using an ipset. (I use firewalld as a front-end to iptables so I 
> let it manage my ipsets, but you can also install ipset's service for use 
> with raw iptables to save and restore the sets across boots.) Your block 
> rule could be as simple as this:
> 
> iptables -I INPUT 1 -p tcp --dport netperf -m set --match-set 
> NetPerfAbusers src -m conntrack --ctstate NEW -j DROP

Thanks for these thoughts. I looked briefly at rate-limiting schemes, but didn't see a good way for them to distinguish good users from bad:

- Good users (who are setting up their SQM, or testing various algorithms) run a test (that creates 10 connections in ~10-60 seconds), tweak a parameter, then re-run that test, repeating until they're happy.

- Bad users who test every five minutes 24x7 create 10 connections every 300 seconds - a slower "rate" of establishing new connections than the good guys.

The primary characteristic that distinguishes the good guys from the bad is that good guys *stop.*

So, my reasoning goes, I need to look at a longer time window and set a limit on the number of connections over the course of a day or two (not minutes or hours). And that's the genesis of my question to the group:

	What is *your* pattern of testing? How many successive tests are you likely to make over the course of a day? 

I'm also aware of ipset, which I take to be an optimized alternative to searching a long set of iptables rules (true?) I don't believe that my OpenVZ VPS has kernel support for this, so as long as the long-list-of-rules seems to work well, I'm going to leave it alone.

That's my thinking, but please let me know if I'm missing something. Thanks again.