[Bloat] What's a good non-intrusive way to look at bloat (and perhaps things like gout (:-))

[Toke Høiland-Jørgensen]

Jonathan Morton <chromatix99 at gmail.com> writes:

>> On 4 Jun, 2020, at 1:21 am, Dave Collier-Brown <dave.collier-brown at indexexchange.com> wrote:
>> 
>> We've good tools to measure network performance under stress, by the
>> simple expedient of stressing it, but is there a good approach I
>> could recommend to my company to monitor a bunch of reasonably modern
>> links, without the measurement significantly affecting their state?
>> 
>> I don't mind increasing bandwidth usage, but I'm downright grumpy
>> about adding to the service time: I have a transaction that times out
>> for gross slowness if it takes much more that an tenth of a second,
>> and it involves a scatter-gather interaction with at least 10
>> customers in that time.
>> 
>> I'm topically interested in bloat, but really we should understand
>> "everything" about our links. If they can get the bloats like cattle,
>> they can probably get the gout, like King Henry the Eighth (;-))
>> 
>> My platform is Centos 8, and I have lots of Smarter Colleagues to
>> help.
>
> My first advice would be to browse pollere.net for tools - like pping
> (passive ping), which monitors the latency of flows in transit. That
> should give you some interesting information without adding any load
> at all. There is also connmon (https://github.com/pollere/connmon).

Ah, good idea, totally forgot about Kathy's tools! :)

I figure one could probably implement something like connmon in eBPF (as
an XDP or TC hook program) and have it run as an always-on monitor with
fairly low overhead. Dave, if you have development resources to throw at
this, I'll be happy to help with pointers on how to get the eBPF bits
working. I believe CentOS 8.2+ should have the needed kernel support...

Of course, you could also just use the connmon utility as-is if you have
CPU cycles to spare for the extra overhead (it looks like it's using
libpcap to capture the packets and process them in userspace).

-Toke