[LibreQoS] Open source Netflow analysis for monitoring AS-to-AS traffic

[Dave Taht]

We are in the process of adding netflow collection to libreqos. Any
potential testers using any of these backends described below out there?

On Thu, Mar 28, 2024, 5:02 PM Brian Knight via NANOG <nanog at nanog.org>
wrote:

> Thanks to all who took the time to comment and make suggestions.
>
> To summarize the private messages, one respondent suggested Argus as a
> collector. Another mentioned that they are still using AS-Stats.
>
> I'm drawn to Akvorado. I like the self-contained nature of the
> application. NF collector, database, and modern web GUI are all bundled in
> one docker container. The full-featured demo <https://demo.akvorado.net/>
> is fantastic. That the app can enrich the Netflow data with BMP is an added
> bonus.
>
> The best part is, the GUI has the report viz I need, and it is actually
> the default visualization in the demo. It also has the graph types that I
> didn't know I needed, like the Sankey graph.
>
> FlowViewer looks interesting as well. I suspect getting the reports right
> may take some time, given the amount of GUI filtering options.
>
> pmacct and Argus seem to be capable tools that have been around for a long
> time, but I haven't seen a concise stack building guide to get Netflow data
> into a good GUI using these. Looks like there are some older Docker images
> available for both. I could write my own SQL or roll my own stack, but I'd
> much rather spend my time on other things.
>
> I appreciate the conversation around sFlow. I actually wasn't aware that
> XR supported it. AS path probably doesn't add a whole lot of value given
> that I'm focused on flows across our IP transit circuits. I'm able to
> determine my next AS hop simply by looking at the flow's associated tuple
> of (flow exporter, interface). I can use other tools like RouteViews or
> RIPE's RIS to determine the destination AS's upstreams if needed. The rest
> of the path is probably not too helpful for determining peering
> opportunities.
>
> I think I'm going to get Akvorado running in my environment. If that
> doesn't pan out, I'll likely go back to AS-Stats.
>
> Can those running Akvorado comment on their system specs? The only spec
> I've seen is a mention in this blog post
> <https://vincent.bernat.ch/en/blog/2022-akvorado-flow-collector>:
> "Akvorado is performant enough to handle 100 000 flows per second with 64
> GB of RAM and 24 vCPU. With 2 TB of disk, you should expect to keep data
> for a few years."
>
> Thanks again all,
>
> -Brian
>
>
> On 2024-03-26 19:04, Brian Knight via NANOG wrote:
>
> What's presently the most commonly used open source toolset for monitoring
> AS-to-AS traffic?
>
> I want to see with which ASes I am exchanging the most traffic across my
> transits and IX links. I want to look for opportunities to peer so I can
> better sell expansion of peering to upper management.
>
> Our routers are mostly $VENDOR_C_XR so Netflow support is key.
>
> In the past, I've used AS-Stats <https://github.com/manuelkasper/AS-Stats>
> for this purpose. However, it is particularly CPU and disk IO intensive.
> Also, it has not been actively maintained since 2017.
>
> InfluxDB wants to sell me
> <https://www.influxdata.com/what-are-netflow-and-sflow/> on Telegraf +
> InfluxDB + Chronograf + Kapacitor, but I can't find any clear guide on what
> hardware I would need for that, never mind how to set up the software. It
> does appear to have an open source option, however.
>
> pmacct seems to be good at gathering Netflow, but doesn't seem to analyze
> data. I don't see any concise howto guides for setting this up for my
> purpose, however.
>
> I'm aware Kentik does this very well, but I have no budget at the moment,
> my testing window is longer than the 30 day trial, and we are not prepared
> to share our Netflow data with a third party.
>
> Elastiflow <https://www.elastiflow.com/> appears to have been open source
> <https://github.com/robcowart/elastiflow?tab=readme-ov-file> at one time
> in the past, but no longer. Since it too appears to be hosted, I have the
> same objections as I do with Kentik above.
>
> On-list and off-list replies are welcome.
>
> Thanks,
>
> -Brian
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/libreqos/attachments/20240608/3a40c36d/attachment.html>