[Bloat] What's a good non-intrusive way to look at bloat (and perhaps things like gout (:-))

[Bloat] What's a good non-intrusive way to look at bloat (and perhaps things like gout (:-))

[Dave Collier-Brown]

[-- Attachment #1: Type: text/plain, Size: 2051 bytes --]

We've good tools to measure network performance under stress, by the simple expedient of stressing it, but is there a good approach I could recommend to my company to monitor a bunch of reasonably modern links,  without the measurement significantly affecting their state?

I don't mind increasing bandwidth usage, but I'm downright grumpy about adding to the service time: I have a transaction that times out for gross slowness if it takes much more that an tenth of a second, and it involves a scatter-gather interaction with at least 10 customers in that time.

I'm topically interested in bloat, but really we should understand "everything" about our links. If they can get the bloats like cattle, they can probably get the gout, like King Henry the Eighth (;-))

My platform is Centos 8, and I have lots of Smarter Colleagues to help.

--dave

--
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
dave.collier-brown@indexexchange.com<mailto:dave.collier-brown@indexexchange.com> |              -- Mark Twain



CONFIDENTIALITY NOTICE AND DISCLAIMER : This telecommunication, including any and all attachments, contains confidential information intended only for the person(s) to whom it is addressed. Any dissemination, distribution, copying or disclosure is strictly prohibited and is not a waiver of confidentiality. If you have received this telecommunication in error, please notify the sender immediately by return electronic mail and delete the message from your inbox and deleted items folders. This telecommunication does not constitute an express or implied agreement to conduct transactions by electronic means, nor does it constitute a contract offer, a contract amendment or an acceptance of a contract offer. Contract terms contained in this telecommunication are subject to legal review and the completion of formal documentation and are not binding until same is confirmed in writing and has been signed by an authorized signatory.

[-- Attachment #2: Type: text/html, Size: 2466 bytes --]

Re: [Bloat] What's a good non-intrusive way to look at bloat (and perhaps things like gout (:-))

[Jonathan Morton]

> On 4 Jun, 2020, at 1:21 am, Dave Collier-Brown <dave.collier-brown@indexexchange.com> wrote:
> 
> We've good tools to measure network performance under stress, by the simple expedient of stressing it, but is there a good approach I could recommend to my company to monitor a bunch of reasonably modern links,  without the measurement significantly affecting their state?
> 
> I don't mind increasing bandwidth usage, but I'm downright grumpy about adding to the service time: I have a transaction that times out for gross slowness if it takes much more that an tenth of a second, and it involves a scatter-gather interaction with at least 10 customers in that time.
> 
> I'm topically interested in bloat, but really we should understand "everything" about our links. If they can get the bloats like cattle, they can probably get the gout, like King Henry the Eighth (;-))
> 
> My platform is Centos 8, and I have lots of Smarter Colleagues to help.

My first advice would be to browse pollere.net for tools - like pping (passive ping), which monitors the latency of flows in transit.  That should give you some interesting information without adding any load at all.  There is also connmon (https://github.com/pollere/connmon).

 - Jonathan Morton

Re: [Bloat] What's a good non-intrusive way to look at bloat (and perhaps things like gout (:-))

[Toke Høiland-Jørgensen]

Jonathan Morton <chromatix99@gmail.com> writes:

>> On 4 Jun, 2020, at 1:21 am, Dave Collier-Brown <dave.collier-brown@indexexchange.com> wrote:
>> 
>> We've good tools to measure network performance under stress, by the
>> simple expedient of stressing it, but is there a good approach I
>> could recommend to my company to monitor a bunch of reasonably modern
>> links, without the measurement significantly affecting their state?
>> 
>> I don't mind increasing bandwidth usage, but I'm downright grumpy
>> about adding to the service time: I have a transaction that times out
>> for gross slowness if it takes much more that an tenth of a second,
>> and it involves a scatter-gather interaction with at least 10
>> customers in that time.
>> 
>> I'm topically interested in bloat, but really we should understand
>> "everything" about our links. If they can get the bloats like cattle,
>> they can probably get the gout, like King Henry the Eighth (;-))
>> 
>> My platform is Centos 8, and I have lots of Smarter Colleagues to
>> help.
>
> My first advice would be to browse pollere.net for tools - like pping
> (passive ping), which monitors the latency of flows in transit. That
> should give you some interesting information without adding any load
> at all. There is also connmon (https://github.com/pollere/connmon).

Ah, good idea, totally forgot about Kathy's tools! :)

I figure one could probably implement something like connmon in eBPF (as
an XDP or TC hook program) and have it run as an always-on monitor with
fairly low overhead. Dave, if you have development resources to throw at
this, I'll be happy to help with pointers on how to get the eBPF bits
working. I believe CentOS 8.2+ should have the needed kernel support...

Of course, you could also just use the connmon utility as-is if you have
CPU cycles to spare for the extra overhead (it looks like it's using
libpcap to capture the packets and process them in userspace).

-Toke

Re: [Bloat] What's a good non-intrusive way to look at bloat (and perhaps things like gout (:-))

[Dave Collier-Brown]

[-- Attachment #1: Type: text/plain, Size: 3420 bytes --]

Thanks, that sounds like a good long-term approach. I know our CTO loves eBPF! As and introduction I'll start prothletising pping (;-))

--dave

On 2020-06-04 6:56 a.m., Toke Høiland-Jørgensen wrote:

Jonathan Morton <chromatix99@gmail.com><mailto:chromatix99@gmail.com> writes:



On 4 Jun, 2020, at 1:21 am, Dave Collier-Brown <dave.collier-brown@indexexchange.com><mailto:dave.collier-brown@indexexchange.com> wrote:

We've good tools to measure network performance under stress, by the
simple expedient of stressing it, but is there a good approach I
could recommend to my company to monitor a bunch of reasonably modern
links, without the measurement significantly affecting their state?

I don't mind increasing bandwidth usage, but I'm downright grumpy
about adding to the service time: I have a transaction that times out
for gross slowness if it takes much more that an tenth of a second,
and it involves a scatter-gather interaction with at least 10
customers in that time.

I'm topically interested in bloat, but really we should understand
"everything" about our links. If they can get the bloats like cattle,
they can probably get the gout, like King Henry the Eighth (;-))

My platform is Centos 8, and I have lots of Smarter Colleagues to
help.



My first advice would be to browse pollere.net for tools - like pping
(passive ping), which monitors the latency of flows in transit. That
should give you some interesting information without adding any load
at all. There is also connmon (https://github.com/pollere/connmon).



Ah, good idea, totally forgot about Kathy's tools! :)

I figure one could probably implement something like connmon in eBPF (as
an XDP or TC hook program) and have it run as an always-on monitor with
fairly low overhead. Dave, if you have development resources to throw at
this, I'll be happy to help with pointers on how to get the eBPF bits
working. I believe CentOS 8.2+ should have the needed kernel support...

Of course, you could also just use the connmon utility as-is if you have
CPU cycles to spare for the extra overhead (it looks like it's using
libpcap to capture the packets and process them in userspace).

-Toke


--
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
dave.collier-brown@indexexchange.com<mailto:dave.collier-brown@indexexchange.com> |              -- Mark Twain



CONFIDENTIALITY NOTICE AND DISCLAIMER : This telecommunication, including any and all attachments, contains confidential information intended only for the person(s) to whom it is addressed. Any dissemination, distribution, copying or disclosure is strictly prohibited and is not a waiver of confidentiality. If you have received this telecommunication in error, please notify the sender immediately by return electronic mail and delete the message from your inbox and deleted items folders. This telecommunication does not constitute an express or implied agreement to conduct transactions by electronic means, nor does it constitute a contract offer, a contract amendment or an acceptance of a contract offer. Contract terms contained in this telecommunication are subject to legal review and the completion of formal documentation and are not binding until same is confirmed in writing and has been signed by an authorized signatory.

[-- Attachment #2: Type: text/html, Size: 4364 bytes --]