[Bloat] Can't Run Tests Against netperf.bufferbloat.net

[Bloat] Can't Run Tests Against netperf.bufferbloat.net

[Taran Lynn]

All of my attempts to run netperf (and flent) against
netperf.bufferbloat.net have failed for the past couple of weeks.
Netperf seems to work fine between my desktop and laptop, so I don't
think the issue is on my end. Can anyone verify if the server is up?

Re: [Bloat] Can't Run Tests Against netperf.bufferbloat.net

[Toke Høiland-Jørgensen]

Taran Lynn <taranlynn0@gmail.com> writes:

> All of my attempts to run netperf (and flent) against
> netperf.bufferbloat.net have failed for the past couple of weeks.
> Netperf seems to work fine between my desktop and laptop, so I don't
> think the issue is on my end. Can anyone verify if the server is up?

Rich, I think this one is yours? Did netserver die or something?

-Toke

Re: [Bloat] Can't Run Tests Against netperf.bufferbloat.net

[Rich Brown]

[-- Attachment #1: Type: text/plain, Size: 3575 bytes --]

Hi all,

Thanks for the note. Yes, the netperf server at netperf.bufferbloat.net <http://netperf.bufferbloat.net/> is turned off. The VPS that runs it is consuming its bandwidth limit (4TB per month) at an ever-increasing rate. When that happens, my hosting service (Ramnode.com <http://ramnode.com/> - good guys, stable hosting, great tech support service) automatically turns off the VPS 'til the start of the next month.

In the distant past, the 4 TB sufficed for the entire month. More recently, I would occasionally get a 90% warning by the 25th or 26th of the month. Last month, I hit 90% on Jan 6th(!) so I shut off the netperf server so I could continue to work on it.

I briefly turned on netserver on the VPS today. At 08:15 today, the VPS control panel showed: 181.7 MB of 3.9 TB Used. At 09:16 today, it showed 46.3 GB. 46 GBytes in 1 hour => ~30 TB/month (!) 

I'm going to appeal to the group's collective wisdom to find a better solution.

Current mitigations:

- I use iptables to log all netperf connections. I see a pattern of certain IP addresses that seem to be firing off a test every five minutes, 24 x 7 for days at a time.

- A few times a month, I run a script (see findunfilteredips.sh in https://github.com/richb-hanover/netperfclean <https://github.com/richb-hanover/netperfclean>) that scans the log files to count netperf connections and to block devices (using iptables) that have made more than 5,000 connections in the last seven days. This helps, but only delays the inevitable.

Potential (additional) mitigations:

- We could change DNS to spread the load of netperf.bufferbloat.net <http://netperf.bufferbloat.net/> across our fleet of servers. (Researchers who need consistent results could still choose a specific server: netperf-east, netperf-west, etc.)

- I could automate the current script to look for heavy users every day or two.

- Maybe I'm doing iptables imperfectly - comments appreciated.

- I have toyed with the notion of tweaking the iptables rules to throttle heavy users (over a certain number of tests/connections per time-period). That way, the 24x7 people would receive, say 3kbps instead of the actual link speed. There are a couple difficulties:
	a) I don't want to inconvenience actual researchers/bufferbloat testers. When I test a connection, I typically make 3-10 tests in rapid succession before I go away. This looks an awful lot like the 24x7 folks, except that real testers stop after 15 minutes. Could iptables be tweaked to tell one from the other?
	b) When I looked into this, I realized I might need to move the VPS from OpenVZ (which has limited iptables capabilities - no 'ipset' for example) to KVM (which is full virtualization).

- I could just buy more bandwidth. Currently, I pay $194/year for this server with the 4TB limit. Additional bandwidth on this provider is $48/year per additional TB. But 30 TB/month would be pricey.

- I could move to a different hosting service where bandwidth is cheaper. (Any recommendations?)

- Other thoughts? 

Thanks.

Rich


> On Feb 5, 2020, at 3:15 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Taran Lynn <taranlynn0@gmail.com> writes:
> 
>> All of my attempts to run netperf (and flent) against
>> netperf.bufferbloat.net have failed for the past couple of weeks.
>> Netperf seems to work fine between my desktop and laptop, so I don't
>> think the issue is on my end. Can anyone verify if the server is up?
> 
> Rich, I think this one is yours? Did netserver die or something?
> 
> -Toke


[-- Attachment #2: Type: text/html, Size: 5726 bytes --]

Re: [Bloat] Can't Run Tests Against netperf.bufferbloat.net

[Toke Høiland-Jørgensen]

Rich Brown <richb.hanover@gmail.com> writes:

> Hi all,
>
> Thanks for the note. Yes, the netperf server at
> netperf.bufferbloat.net <http://netperf.bufferbloat.net/> is turned
> off. The VPS that runs it is consuming its bandwidth limit (4TB per
> month) at an ever-increasing rate. When that happens, my hosting
> service (Ramnode.com <http://ramnode.com/> - good guys, stable
> hosting, great tech support service) automatically turns off the VPS
> 'til the start of the next month.

Right, good to know; thanks for the explanation! And for taking on
running this service! I think we should try to find something more
sustainable, though, see below:

> In the distant past, the 4 TB sufficed for the entire month. More
> recently, I would occasionally get a 90% warning by the 25th or 26th
> of the month. Last month, I hit 90% on Jan 6th(!) so I shut off the
> netperf server so I could continue to work on it.
>
> I briefly turned on netserver on the VPS today. At 08:15 today, the
> VPS control panel showed: 181.7 MB of 3.9 TB Used. At 09:16 today, it
> showed 46.3 GB. 46 GBytes in 1 hour => ~30 TB/month (!)
>
> I'm going to appeal to the group's collective wisdom to find a better
> solution.
>
> Current mitigations:
>
> - I use iptables to log all netperf connections. I see a pattern of
> certain IP addresses that seem to be firing off a test every five
> minutes, 24 x 7 for days at a time.
>
> - A few times a month, I run a script (see findunfilteredips.sh in
> https://github.com/richb-hanover/netperfclean
> <https://github.com/richb-hanover/netperfclean>) that scans the log
> files to count netperf connections and to block devices (using
> iptables) that have made more than 5,000 connections in the last seven
> days. This helps, but only delays the inevitable.
>
> Potential (additional) mitigations:
>
> - We could change DNS to spread the load of netperf.bufferbloat.net
> <http://netperf.bufferbloat.net/> across our fleet of servers.
> (Researchers who need consistent results could still choose a specific
> server: netperf-east, netperf-west, etc.)

I think we should definitely do this, maybe even do something
geoip-based to select the "closest". I'll look into options...

> - I could automate the current script to look for heavy users every
> day or two.

Personally I think it would be fine if you just ban heavy users with
reckless abandon :)

> - Maybe I'm doing iptables imperfectly - comments appreciated.
>
> - I have toyed with the notion of tweaking the iptables rules to
> throttle heavy users (over a certain number of tests/connections per
> time-period). That way, the 24x7 people would receive, say 3kbps
> instead of the actual link speed. There are a couple difficulties:

> 	a) I don't want to inconvenience actual researchers/bufferbloat
> 	testers. When I test a connection, I typically make 3-10 tests
> 	in rapid succession before I go away. This looks an awful lot
> 	like the 24x7 folks, except that real testers stop after 15
> 	minutes. Could iptables be tweaked to tell one from the other?
>
> 	b) When I looked into this, I realized I might need to move the
> 	VPS from OpenVZ (which has limited iptables capabilities - no
> 	'ipset' for example) to KVM (which is full virtualization).

What about just giving each IP a bandwidth cap? It may need more kernel
capabilities to support this efficiently, though; so moving to KVM may
be necessary. What kernel version does you openvz host have?

> - I could just buy more bandwidth. Currently, I pay $194/year for this
> server with the 4TB limit. Additional bandwidth on this provider is
> $48/year per additional TB. But 30 TB/month would be pricey.
>
> - I could move to a different hosting service where bandwidth is
> cheaper. (Any recommendations?)

Rather than spend more money bankrolling a free service, I think we
should see if we can find some way to sponsor this from an organisation
that doesn't pay per the byte. Anyone who knows someone at a university
in the US? I'll see if Red Hat can do anything for us...

-Toke

Re: [Bloat] Can't Run Tests Against netperf.bufferbloat.net

[Matt Taggart]

On 2/5/20 6:49 AM, Rich Brown wrote:

> - I use iptables to log all netperf connections. I see a pattern of 
> certain IP addresses that seem to be firing off a test every five 
> minutes, 24 x 7 for days at a time.

This smells like a munin or smokeping plugin (or some other sort of 
monitoring) gathering data for graphing.

I have wanted something like this to graph bandwidth, latency, packet 
loss over time. If we could come up with a lighter weight way of doing 
this (or using a speedtest.net type service) that would be nice to have.

I recently went from having a comcast cable modem where the bottleneck 
was in the modem, to a centurylink gig fiber where my bandwidth depends 
on what the neighbors are doing, which makes tuning SQM difficult... 
Thus my desire to start monitoring things.

-- 
Matt Taggart
matt@lackof.org

Re: [Bloat] Can't Run Tests Against netperf.bufferbloat.net

[Sebastian Moeller]

Maybe related?
https://forum.openwrt.org/t/speedtest-new-package-to-measure-network-performance/24647/97?u=moeller0

Best Regards
	Sebastian

> On Feb 5, 2020, at 03:18, Taran Lynn <taranlynn0@gmail.com> wrote:
> 
> 
> All of my attempts to run netperf (and flent) against
> netperf.bufferbloat.net have failed for the past couple of weeks.
> Netperf seems to work fine between my desktop and laptop, so I don't
> think the issue is on my end. Can anyone verify if the server is up?
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Can't Run Tests Against netperf.bufferbloat.net

[Rich Brown]

[-- Attachment #1: Type: text/plain, Size: 967 bytes --]


> On Feb 6, 2020, at 12:00 PM, Matt Taggart wrote:
> 
> This smells like a munin or smokeping plugin (or some other sort of 
> monitoring) gathering data for graphing.

Yup. That is a real possibility. The question is what we do about it.

If I understood, we left it at:

1) Toke was going to look into some way to spread the 'netperf.bufferbloat.net' load across several of our netperf servers.

2) Can someone give me advice about iptables/tc/? to identify IP addresses that make "too many" connections and either shut them off or dial their bandwidth back to a 3 or 5 kbps? 

(If you're terminally curious, Line 5 of https://github.com/richb-hanover/netperfclean/blob/master/addtoblacklist.sh shows the current iptables command to drop connections from "heavy users" identified in the findunfilteredips.sh script. You can read the current iptables rules at: https://github.com/richb-hanover/netperfclean/blob/master/iptables.txt)

Thanks.

Rich

[-- Attachment #2: Type: text/html, Size: 3611 bytes --]

Re: [Bloat] Can't Run Tests Against netperf.bufferbloat.net

[Toke Høiland-Jørgensen]

Rich Brown <richb.hanover@gmail.com> writes:

>> On Feb 6, 2020, at 12:00 PM, Matt Taggart wrote:
>> 
>> This smells like a munin or smokeping plugin (or some other sort of 
>> monitoring) gathering data for graphing.
>
> Yup. That is a real possibility. The question is what we do about it.
>
> If I understood, we left it at:
>
> 1) Toke was going to look into some way to spread the
> 'netperf.bufferbloat.net' load across several of our netperf servers.
>
> 2) Can someone give me advice about iptables/tc/? to identify IP
> addresses that make "too many" connections and either shut them off or
> dial their bandwidth back to a 3 or 5 kbps?

Not sure if it's possible to do with iptables, but it should be fairly
straight-forward to write an eBPF filtering program that will lock out
an IP after it has used a certain amount of bandwidth. That was the
reason for my question about your kernel version; need to figure out
what types of BPF support you have available :)

-Toke

Re: [Bloat] Can't Run Tests Against netperf.bufferbloat.net

[Jesper Dangaard Brouer]

On Thu, 6 Feb 2020 18:47:06 -0500
Rich Brown <richb.hanover@gmail.com> wrote:

> > On Feb 6, 2020, at 12:00 PM, Matt Taggart wrote:
> > 
> > This smells like a munin or smokeping plugin (or some other sort of 
> > monitoring) gathering data for graphing.  
> 
> Yup. That is a real possibility. The question is what we do about it.
> 
> If I understood, we left it at:
> 
> 1) Toke was going to look into some way to spread the
> 'netperf.bufferbloat.net' load across several of our netperf servers.
> 
> 2) Can someone give me advice about iptables/tc/? to identify IP
> addresses that make "too many" connections and either shut them off
> or dial their bandwidth back to a 3 or 5 kbps? 

Look at man iptables-extensions and find "connlimit" and "recent".

 
> (If you're terminally curious, Line 5 of
> https://github.com/richb-hanover/netperfclean/blob/master/addtoblacklist.sh
> shows the current iptables command to drop connections from "heavy
> users" identified in the findunfilteredips.sh script. You can read
> the current iptables rules at:
> https://github.com/richb-hanover/netperfclean/blob/master/iptables.txt)

Sorry but this is a wrong approach.  Creating an iptables rule per
source IP-address, will (as you also demonstrate) give you a VERY long
list of rules (which is evaluated sequentially by the kernel).

This should instead be solved by using an ipset (howto a match from
iptables see man iptables-extensions(8) and "set").  And use the
cmdline tool ipset to add and remove entries.

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

Re: [Bloat] Can't Run Tests Against netperf.bufferbloat.net

[Rich Brown]

Toke and Jesper,

Thanks both for these responses. 

netperf.bufferbloat.net is running an OpenVZ VPS with a 3.10 kernel. Tech support at Ramnode tells me that I need to get to a KVM instance in order to use ipset and other fancy kernel stuff.

Here's my plan:

1) Unless anyone can recommend a better hosting service ...

2) Over the weekend, I'll stand up a new KVM server at Ramnode. They offer a 2GB RAM, 2 core, 65 GB SSD, with 3TB per month of data. It'll cost $10/month: adding 2x1TB at $4/month brings it to a total of $18/month, about what the current server costs. I can get Ubuntu 18.04 LTS as a standard install.

3) While that's in-flight I would request that an iptables expert on the list recommend a better strategy. (I was just makin' stuff up in the current setup - as you could tell :-)

4) I'd also accept any thoughts about tc commands for setting up the networking on the host to work best as a netperf server. (Maybe enable fq_codel or better...) 

Thanks

Rich

> On Feb 7, 2020, at 7:02 AM, Jesper Dangaard Brouer <brouer@redhat.com> wrote:
> 
> On Thu, 6 Feb 2020 18:47:06 -0500
> Rich Brown <richb.hanover@gmail.com> wrote:
> 
>>> On Feb 6, 2020, at 12:00 PM, Matt Taggart wrote:
>>> 
>>> This smells like a munin or smokeping plugin (or some other sort of 
>>> monitoring) gathering data for graphing.  
>> 
>> Yup. That is a real possibility. The question is what we do about it.
>> 
>> If I understood, we left it at:
>> 
>> 1) Toke was going to look into some way to spread the
>> 'netperf.bufferbloat.net' load across several of our netperf servers.
>> 
>> 2) Can someone give me advice about iptables/tc/? to identify IP
>> addresses that make "too many" connections and either shut them off
>> or dial their bandwidth back to a 3 or 5 kbps? 
> 
> Look at man iptables-extensions and find "connlimit" and "recent".
> 
> 
>> (If you're terminally curious, Line 5 of
>> https://github.com/richb-hanover/netperfclean/blob/master/addtoblacklist.sh
>> shows the current iptables command to drop connections from "heavy
>> users" identified in the findunfilteredips.sh script. You can read
>> the current iptables rules at:
>> https://github.com/richb-hanover/netperfclean/blob/master/iptables.txt)
> 
> Sorry but this is a wrong approach.  Creating an iptables rule per
> source IP-address, will (as you also demonstrate) give you a VERY long
> list of rules (which is evaluated sequentially by the kernel).
> 
> This should instead be solved by using an ipset (howto a match from
> iptables see man iptables-extensions(8) and "set").  And use the
> cmdline tool ipset to add and remove entries.
> 
> -- 
> Best regards,
>  Jesper Dangaard Brouer
>  MSc.CS, Principal Kernel Engineer at Red Hat
>  LinkedIn: http://www.linkedin.com/in/brouer
> 

Re: [Bloat] Can't Run Tests Against netperf.bufferbloat.net

[Rich Brown]

Update: (I thought I had sent the previous message yesterday. My mistake.)

I now have atl3.richb-hanover.com running a netperf server. it's a stock Ubuntu 18.04.4 LTS - uname -a shows: Linux atl3 4.15.0-76-generic #86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux. I have installed netperf 2.6.0, and little else.

Next steps:

1) Please hammer on the server to see if it's a suitable replacement for the canonical "netperf.bufferbloat.net". Please feel free to check both its ability to handle traffic as well as any security surprises you discover...

2) I welcome suggestions for configuring the server's TCP stack to be most useful for researchers. fq_codel, bbr, - I'm open to your thoughts. 

3) It's not too soon for advice on an iptables strategy for limiting the access/bandwidth/traffic to people who're abusing the service...

Once we have all this in place, we can change the netperf.bufferbloat.net name to point to this server. Thanks.

Rich

> On Feb 8, 2020, at 5:35 PM, Rich Brown <richb.hanover@gmail.com> wrote:
> 
> Toke and Jesper,
> 
> Thanks both for these responses. 
> 
> netperf.bufferbloat.net is running an OpenVZ VPS with a 3.10 kernel. Tech support at Ramnode tells me that I need to get to a KVM instance in order to use ipset and other fancy kernel stuff.
> 
> Here's my plan:
> 
> 1) Unless anyone can recommend a better hosting service ...
> 
> 2) Over the weekend, I'll stand up a new KVM server at Ramnode. They offer a 2GB RAM, 2 core, 65 GB SSD, with 3TB per month of data. It'll cost $10/month: adding 2x1TB at $4/month brings it to a total of $18/month, about what the current server costs. I can get Ubuntu 18.04 LTS as a standard install.
> 
> 3) While that's in-flight I would request that an iptables expert on the list recommend a better strategy. (I was just makin' stuff up in the current setup - as you could tell :-)
> 
> 4) I'd also accept any thoughts about tc commands for setting up the networking on the host to work best as a netperf server. (Maybe enable fq_codel or better...) 
> 
> Thanks
> 
> Rich
> 
>> On Feb 7, 2020, at 7:02 AM, Jesper Dangaard Brouer <brouer@redhat.com> wrote:
>> 
>> On Thu, 6 Feb 2020 18:47:06 -0500
>> Rich Brown <richb.hanover@gmail.com> wrote:
>> 
>>>> On Feb 6, 2020, at 12:00 PM, Matt Taggart wrote:
>>>> 
>>>> This smells like a munin or smokeping plugin (or some other sort of 
>>>> monitoring) gathering data for graphing.  
>>> 
>>> Yup. That is a real possibility. The question is what we do about it.
>>> 
>>> If I understood, we left it at:
>>> 
>>> 1) Toke was going to look into some way to spread the
>>> 'netperf.bufferbloat.net' load across several of our netperf servers.
>>> 
>>> 2) Can someone give me advice about iptables/tc/? to identify IP
>>> addresses that make "too many" connections and either shut them off
>>> or dial their bandwidth back to a 3 or 5 kbps? 
>> 
>> Look at man iptables-extensions and find "connlimit" and "recent".
>> 
>> 
>>> (If you're terminally curious, Line 5 of
>>> https://github.com/richb-hanover/netperfclean/blob/master/addtoblacklist.sh
>>> shows the current iptables command to drop connections from "heavy
>>> users" identified in the findunfilteredips.sh script. You can read
>>> the current iptables rules at:
>>> https://github.com/richb-hanover/netperfclean/blob/master/iptables.txt)
>> 
>> Sorry but this is a wrong approach.  Creating an iptables rule per
>> source IP-address, will (as you also demonstrate) give you a VERY long
>> list of rules (which is evaluated sequentially by the kernel).
>> 
>> This should instead be solved by using an ipset (howto a match from
>> iptables see man iptables-extensions(8) and "set").  And use the
>> cmdline tool ipset to add and remove entries.
>> 
>> -- 
>> Best regards,
>> Jesper Dangaard Brouer
>> MSc.CS, Principal Kernel Engineer at Red Hat
>> LinkedIn: http://www.linkedin.com/in/brouer
>> 
> 

Re: [Bloat] Can't Run Tests Against netperf.bufferbloat.net

[Dave Taht]

I note that I totally missed this thread somehow.

I have about 12 (rather undermaintained) servers in linode's cloud these
days that have netperf and irtt on them, all over the world, singapore,
mumbai, etc, etc....

The monthly transfer allotment is ~28TB across them all. So far this
month we've used up ~1.2TB, principally from flent-fremont.bufferbloat.net.

The current cost, which includes bufferbloat.net and taht.net, is
 180/month.  I just vector a portion of the donations to patreon over
 there and don't think about it much. All of these boxes could use love
 and an upgrade (softwarewise), especially bufferbloat.net, which could
 also get a downgrade cpuwise. The investment of time (days) to do that,
 vs the ongoing extra cost of a 2 cpu machine (15/dollars/month) has not
 seemed worth it.
 
I would like it very much if we were more serious about a published
reliable test infrastructure for flent. However, it was also my hope
that flent would get mostly used inside people's firewalls on their own
testbeds while developing new gear and drivers and network designs, as
it's so easy to do so (apt-get install irtt netperf flent etc) nowadays.

One of my concerns has long been that I have no idea what lies
underneath these cloudy vms (anywhere) that is actually doing the rate
limiting (anyone? AWS? linode?), and have kind of wished we could go
back to at least one piece of bare metal on the internet as we had when
isc.org was still donating services to us. Back then vm jitter was
measured in the 10s of ms using xen... and we have no means to compare
vm performance vs bare metal performance with all this finicy stuff,
such as pacing, SCE/L4S etc. The google folk are perpetually publishing
ce_threshold set at 260us, which is just impossible (IMHO) in a vm....

The "nanodes" in this cloud are limited to 100Mbit, somehow.

I've occasionally priced out what it would take to get a 10Gbit in
hurricane's cages in california - 900/month just for transit when on
special, 2k a month otherwise.... Half a cage was another 400 or so. I
already have a spare /24 and BGP AS and hardware I could put in it,
but...

Rich Brown <richb.hanover@gmail.com> writes:

> Update: (I thought I had sent the previous message yesterday. My mistake.)
>
> I now have atl3.richb-hanover.com running a netperf server. it's a
> stock Ubuntu 18.04.4 LTS - uname -a shows: Linux atl3
> 4.15.0-76-generic #86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 2020 x86_64
> x86_64 x86_64 GNU/Linux. I have installed netperf 2.6.0, and little
> else.

I imagine fq_codel is the underlying qdisc?

irtt would be nice.

At some point, somehow, using some tool, we need to start supporting QUIC.

>
> Next steps:
>
> 1) Please hammer on the server to see if it's a suitable replacement
> for the canonical "netperf.bufferbloat.net". Please feel free to check
> both its ability to handle traffic as well as any security surprises
> you discover...

Thank you so much for maintaining this server for so long and I had no
idea before this thread that it was being so hard hit. 


> 2) I welcome suggestions for configuring the server's TCP stack to be
> most useful for researchers. fq_codel, bbr, - I'm open to your
> thoughts.

I have generally made available cubic, reno, bbr, and dctcp. I was
always rather reluctant to publish where I'd turned on dctcp given that
it only recently gained a response to packet loss. In fact, reluctant to
publish anything in the cloud.

I had (briefly) an SCE capable machine up in singapore. It would be
good if more folk could try cubic-sce, reno-sce, dctcp-sce, bbr-sce, and
for that matter, play with l4s in the same dc on an equivalent machine,
but that involves making custom kernels for each at present. I'm also
dying to try out the ebpf + etx stuff google is presenting at netdevconf....

>
> 3) It's not too soon for advice on an iptables strategy for limiting
> the access/bandwidth/traffic to people who're abusing the service...

I'd love to have that too!

> Once we have all this in place, we can change the netperf.bufferbloat.net name to point to this server. Thanks.
>
> Rich
>
>> On Feb 8, 2020, at 5:35 PM, Rich Brown <richb.hanover@gmail.com> wrote:
>> 
>> Toke and Jesper,
>> 
>> Thanks both for these responses. 
>> 
>> netperf.bufferbloat.net is running an OpenVZ VPS with a 3.10
>> kernel. Tech support at Ramnode tells me that I need to get to a KVM
>> instance in order to use ipset and other fancy kernel stuff.
>> 
>> Here's my plan:
>> 
>> 1) Unless anyone can recommend a better hosting service ...
>> 
>> 2) Over the weekend, I'll stand up a new KVM server at Ramnode. They
>> offer a 2GB RAM, 2 core, 65 GB SSD, with 3TB per month of
>> data. It'll cost $10/month: adding 2x1TB at $4/month brings it to a
>> total of $18/month, about what the current server costs. I can get
>> Ubuntu 18.04 LTS as a standard install.
>> 
>> 3) While that's in-flight I would request that an iptables expert on
>> the list recommend a better strategy. (I was just makin' stuff up in
>> the current setup - as you could tell :-)
>> 
>> 4) I'd also accept any thoughts about tc commands for setting up the
>> networking on the host to work best as a netperf server. (Maybe
>> enable fq_codel or better...)
>> 
>> Thanks
>> 
>> Rich
>> 
>>> On Feb 7, 2020, at 7:02 AM, Jesper Dangaard Brouer <brouer@redhat.com> wrote:
>>> 
>>> On Thu, 6 Feb 2020 18:47:06 -0500
>>> Rich Brown <richb.hanover@gmail.com> wrote:
>>> 
>>>>> On Feb 6, 2020, at 12:00 PM, Matt Taggart wrote:
>>>>> 
>>>>> This smells like a munin or smokeping plugin (or some other sort of 
>>>>> monitoring) gathering data for graphing.  
>>>> 
>>>> Yup. That is a real possibility. The question is what we do about it.
>>>> 
>>>> If I understood, we left it at:
>>>> 
>>>> 1) Toke was going to look into some way to spread the
>>>> 'netperf.bufferbloat.net' load across several of our netperf servers.
>>>> 
>>>> 2) Can someone give me advice about iptables/tc/? to identify IP
>>>> addresses that make "too many" connections and either shut them off
>>>> or dial their bandwidth back to a 3 or 5 kbps? 
>>> 
>>> Look at man iptables-extensions and find "connlimit" and "recent".
>>> 
>>> 
>>>> (If you're terminally curious, Line 5 of
>>>> https://github.com/richb-hanover/netperfclean/blob/master/addtoblacklist.sh
>>>> shows the current iptables command to drop connections from "heavy
>>>> users" identified in the findunfilteredips.sh script. You can read
>>>> the current iptables rules at:
>>>> https://github.com/richb-hanover/netperfclean/blob/master/iptables.txt)
>>> 
>>> Sorry but this is a wrong approach.  Creating an iptables rule per
>>> source IP-address, will (as you also demonstrate) give you a VERY long
>>> list of rules (which is evaluated sequentially by the kernel).
>>> 
>>> This should instead be solved by using an ipset (howto a match from
>>> iptables see man iptables-extensions(8) and "set").  And use the
>>> cmdline tool ipset to add and remove entries.
>>> 
>>> -- 
>>> Best regards,
>>> Jesper Dangaard Brouer
>>> MSc.CS, Principal Kernel Engineer at Red Hat
>>> LinkedIn: http://www.linkedin.com/in/brouer
>>> 
>> 
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Can't Run Tests Against netperf.bufferbloat.net

[Dave Taht]

OK, so I did a few test results from flent-atlanta.taht.net. They are here:

http://flent-atlanta.taht.net/~d/atl.tgz

Trying to run it at close to a gbit, there are all sorts of
interesting behavior between the two dcs. Using sqm + sch_fq I got
mildly saner results,
but... well, download the above and scratch your head some....



On Sun, Feb 9, 2020 at 8:31 AM Dave Taht <dave@taht.net> wrote:
>
>
> I note that I totally missed this thread somehow.
>
> I have about 12 (rather undermaintained) servers in linode's cloud these
> days that have netperf and irtt on them, all over the world, singapore,
> mumbai, etc, etc....
>
> The monthly transfer allotment is ~28TB across them all. So far this
> month we've used up ~1.2TB, principally from flent-fremont.bufferbloat.net.
>
> The current cost, which includes bufferbloat.net and taht.net, is
>  180/month.  I just vector a portion of the donations to patreon over
>  there and don't think about it much. All of these boxes could use love
>  and an upgrade (softwarewise), especially bufferbloat.net, which could
>  also get a downgrade cpuwise. The investment of time (days) to do that,
>  vs the ongoing extra cost of a 2 cpu machine (15/dollars/month) has not
>  seemed worth it.
>
> I would like it very much if we were more serious about a published
> reliable test infrastructure for flent. However, it was also my hope
> that flent would get mostly used inside people's firewalls on their own
> testbeds while developing new gear and drivers and network designs, as
> it's so easy to do so (apt-get install irtt netperf flent etc) nowadays.
>
> One of my concerns has long been that I have no idea what lies
> underneath these cloudy vms (anywhere) that is actually doing the rate
> limiting (anyone? AWS? linode?), and have kind of wished we could go
> back to at least one piece of bare metal on the internet as we had when
> isc.org was still donating services to us. Back then vm jitter was
> measured in the 10s of ms using xen... and we have no means to compare
> vm performance vs bare metal performance with all this finicy stuff,
> such as pacing, SCE/L4S etc. The google folk are perpetually publishing
> ce_threshold set at 260us, which is just impossible (IMHO) in a vm....
>
> The "nanodes" in this cloud are limited to 100Mbit, somehow.
>
> I've occasionally priced out what it would take to get a 10Gbit in
> hurricane's cages in california - 900/month just for transit when on
> special, 2k a month otherwise.... Half a cage was another 400 or so. I
> already have a spare /24 and BGP AS and hardware I could put in it,
> but...
>
> Rich Brown <richb.hanover@gmail.com> writes:
>
> > Update: (I thought I had sent the previous message yesterday. My mistake.)
> >
> > I now have atl3.richb-hanover.com running a netperf server. it's a
> > stock Ubuntu 18.04.4 LTS - uname -a shows: Linux atl3
> > 4.15.0-76-generic #86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 2020 x86_64
> > x86_64 x86_64 GNU/Linux. I have installed netperf 2.6.0, and little
> > else.
>
> I imagine fq_codel is the underlying qdisc?
>
> irtt would be nice.
>
> At some point, somehow, using some tool, we need to start supporting QUIC.
>
> >
> > Next steps:
> >
> > 1) Please hammer on the server to see if it's a suitable replacement
> > for the canonical "netperf.bufferbloat.net". Please feel free to check
> > both its ability to handle traffic as well as any security surprises
> > you discover...
>
> Thank you so much for maintaining this server for so long and I had no
> idea before this thread that it was being so hard hit.
>
>
> > 2) I welcome suggestions for configuring the server's TCP stack to be
> > most useful for researchers. fq_codel, bbr, - I'm open to your
> > thoughts.
>
> I have generally made available cubic, reno, bbr, and dctcp. I was
> always rather reluctant to publish where I'd turned on dctcp given that
> it only recently gained a response to packet loss. In fact, reluctant to
> publish anything in the cloud.
>
> I had (briefly) an SCE capable machine up in singapore. It would be
> good if more folk could try cubic-sce, reno-sce, dctcp-sce, bbr-sce, and
> for that matter, play with l4s in the same dc on an equivalent machine,
> but that involves making custom kernels for each at present. I'm also
> dying to try out the ebpf + etx stuff google is presenting at netdevconf....
>
> >
> > 3) It's not too soon for advice on an iptables strategy for limiting
> > the access/bandwidth/traffic to people who're abusing the service...
>
> I'd love to have that too!
>
> > Once we have all this in place, we can change the netperf.bufferbloat.net name to point to this server. Thanks.
> >
> > Rich
> >
> >> On Feb 8, 2020, at 5:35 PM, Rich Brown <richb.hanover@gmail.com> wrote:
> >>
> >> Toke and Jesper,
> >>
> >> Thanks both for these responses.
> >>
> >> netperf.bufferbloat.net is running an OpenVZ VPS with a 3.10
> >> kernel. Tech support at Ramnode tells me that I need to get to a KVM
> >> instance in order to use ipset and other fancy kernel stuff.
> >>
> >> Here's my plan:
> >>
> >> 1) Unless anyone can recommend a better hosting service ...
> >>
> >> 2) Over the weekend, I'll stand up a new KVM server at Ramnode. They
> >> offer a 2GB RAM, 2 core, 65 GB SSD, with 3TB per month of
> >> data. It'll cost $10/month: adding 2x1TB at $4/month brings it to a
> >> total of $18/month, about what the current server costs. I can get
> >> Ubuntu 18.04 LTS as a standard install.
> >>
> >> 3) While that's in-flight I would request that an iptables expert on
> >> the list recommend a better strategy. (I was just makin' stuff up in
> >> the current setup - as you could tell :-)
> >>
> >> 4) I'd also accept any thoughts about tc commands for setting up the
> >> networking on the host to work best as a netperf server. (Maybe
> >> enable fq_codel or better...)
> >>
> >> Thanks
> >>
> >> Rich
> >>
> >>> On Feb 7, 2020, at 7:02 AM, Jesper Dangaard Brouer <brouer@redhat.com> wrote:
> >>>
> >>> On Thu, 6 Feb 2020 18:47:06 -0500
> >>> Rich Brown <richb.hanover@gmail.com> wrote:
> >>>
> >>>>> On Feb 6, 2020, at 12:00 PM, Matt Taggart wrote:
> >>>>>
> >>>>> This smells like a munin or smokeping plugin (or some other sort of
> >>>>> monitoring) gathering data for graphing.
> >>>>
> >>>> Yup. That is a real possibility. The question is what we do about it.
> >>>>
> >>>> If I understood, we left it at:
> >>>>
> >>>> 1) Toke was going to look into some way to spread the
> >>>> 'netperf.bufferbloat.net' load across several of our netperf servers.
> >>>>
> >>>> 2) Can someone give me advice about iptables/tc/? to identify IP
> >>>> addresses that make "too many" connections and either shut them off
> >>>> or dial their bandwidth back to a 3 or 5 kbps?
> >>>
> >>> Look at man iptables-extensions and find "connlimit" and "recent".
> >>>
> >>>
> >>>> (If you're terminally curious, Line 5 of
> >>>> https://github.com/richb-hanover/netperfclean/blob/master/addtoblacklist.sh
> >>>> shows the current iptables command to drop connections from "heavy
> >>>> users" identified in the findunfilteredips.sh script. You can read
> >>>> the current iptables rules at:
> >>>> https://github.com/richb-hanover/netperfclean/blob/master/iptables.txt)
> >>>
> >>> Sorry but this is a wrong approach.  Creating an iptables rule per
> >>> source IP-address, will (as you also demonstrate) give you a VERY long
> >>> list of rules (which is evaluated sequentially by the kernel).
> >>>
> >>> This should instead be solved by using an ipset (howto a match from
> >>> iptables see man iptables-extensions(8) and "set").  And use the
> >>> cmdline tool ipset to add and remove entries.
> >>>
> >>> --
> >>> Best regards,
> >>> Jesper Dangaard Brouer
> >>> MSc.CS, Principal Kernel Engineer at Red Hat
> >>> LinkedIn: http://www.linkedin.com/in/brouer
> >>>
> >>
> >
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
Make Music, Not War

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-435-0729