From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 91A963B2A4 for ; Sat, 8 Jun 2024 22:55:54 -0400 (EDT) Received: by mail-wm1-x334.google.com with SMTP id 5b1f17b1804b1-4217d808034so4732605e9.3 for ; Sat, 08 Jun 2024 19:55:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717901753; x=1718506553; darn=lists.bufferbloat.net; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=QBUiCMqP+hFh5EZJ5Kp6ggCEbvndVv4nY7TwAw63Y1Y=; b=FGJHVgeDVTQB1dHYAM4K5lhVzciBdsRLUsJn67g4e7j0QxGcm2MBhq22H0VsaA1nJm JT8kE9vjJ/Cxs4EhgZPVCIpZBYGBsrllpgIqMeOH0gFWYiCq3khSCx/ma3xIL6D5gD/k NmZAFYoSGEfM0A57zr4Ewe2UjmRJWiWJal+rekq9KkfCPMWu1IjlNkpz182DYQF0CbAq fK7/NHgWr9d+JJRIno4X+XrUNt0iaMw/ELugY1kaJnYONejW3D5xCaqQ9ufkD4MF7XNf K4t00OXvvo2WHIltPbVDz7u4WqoyCTKk+9hQLHqXfBmLdI6H4YyoHi7yGdulh4vGtYt8 aedg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717901753; x=1718506553; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=QBUiCMqP+hFh5EZJ5Kp6ggCEbvndVv4nY7TwAw63Y1Y=; b=TzNVyuBaX+hx10j1HZfaq/kIUA0IF65klRPHAQ3mYnP+eN/yzTVqVty/FWz/ikXHei 3DE8+fgmmxkBNbz8X1KmEkAyUinUF3d+BWKf9NNeA9rSc78fILgzT4ZFfE5C1kgNgavE M4dg6RKaoRaOMxFpJ0dcnoYjQqziKWQkxjU+17COvE1V1NMtbo01r0JW+dW9MCanGMzO orBnhCmhwpjO73cDMLnb3Xs9dYPlHh7ww2fNEx6WjPOLCRG3CoAHpRVR2okqJ8EzLm3X y8AEhjnUajLSYC9poyfNzsJuWuXMsYVNXApcd2j9bJ4PKfwmAKGKHBsmMHlSiX3HBZS7 tR7A== X-Forwarded-Encrypted: i=1; AJvYcCVqGDDgFDXSmCyyHegYq/Vzof8uBAKCXlfDd/QxNzdehK2tM/Nr69nj5Po9VMoebNoOy6M6tXRphoCdRnBceR0b3r5OqvJj4tOGNf2cjvw= X-Gm-Message-State: AOJu0YzffSYQJwb6U5vgweTbyRX/YvYKhLKsCEVGPWCT7KKJTSq/yxtt VYP9ACZ14oRZdaLnhuejSTi1kQ3sxJsDvHWucDC7B9VLbfJJulGfc6InHfkxlbesTd8YLvs9Nbb 0r6QrkDtuDEdFnyzgFwK0I1moY3E= X-Google-Smtp-Source: AGHT+IFmKrIAXJV42rRUk27SRDb+nDgnEGm2gB/dHe0l4yCgEtrrGg+nXdmMTM258g0Xj3bnXKDu1+GegV4sPSYjLl4= X-Received: by 2002:a05:600c:4f84:b0:421:80d2:9db1 with SMTP id 5b1f17b1804b1-42180d2a37emr10034085e9.25.1717901753250; Sat, 08 Jun 2024 19:55:53 -0700 (PDT) MIME-Version: 1.0 References: <999d91a84220961140c60df66336b47e@mail.knight-networks.com> In-Reply-To: <999d91a84220961140c60df66336b47e@mail.knight-networks.com> From: Dave Taht Date: Sat, 8 Jun 2024 19:55:41 -0700 Message-ID: To: Brian Knight Cc: John Stitt , Andrew Hoyos , Marinos Dimolianis , NANOG , libreqos Content-Type: multipart/alternative; boundary="000000000000c2f95e061a6c2b0d" Subject: Re: [LibreQoS] Open source Netflow analysis for monitoring AS-to-AS traffic X-BeenThere: libreqos@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Many ISPs need the kinds of quality shaping cake can do List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Jun 2024 02:55:54 -0000 --000000000000c2f95e061a6c2b0d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable We are in the process of adding netflow collection to libreqos. Any potential testers using any of these backends described below out there? On Thu, Mar 28, 2024, 5:02=E2=80=AFPM Brian Knight via NANOG wrote: > Thanks to all who took the time to comment and make suggestions. > > To summarize the private messages, one respondent suggested Argus as a > collector. Another mentioned that they are still using AS-Stats. > > I'm drawn to Akvorado. I like the self-contained nature of the > application. NF collector, database, and modern web GUI are all bundled i= n > one docker container. The full-featured demo > is fantastic. That the app can enrich the Netflow data with BMP is an add= ed > bonus. > > The best part is, the GUI has the report viz I need, and it is actually > the default visualization in the demo. It also has the graph types that I > didn't know I needed, like the Sankey graph. > > FlowViewer looks interesting as well. I suspect getting the reports right > may take some time, given the amount of GUI filtering options. > > pmacct and Argus seem to be capable tools that have been around for a lon= g > time, but I haven't seen a concise stack building guide to get Netflow da= ta > into a good GUI using these. Looks like there are some older Docker image= s > available for both. I could write my own SQL or roll my own stack, but I'= d > much rather spend my time on other things. > > I appreciate the conversation around sFlow. I actually wasn't aware that > XR supported it. AS path probably doesn't add a whole lot of value given > that I'm focused on flows across our IP transit circuits. I'm able to > determine my next AS hop simply by looking at the flow's associated tuple > of (flow exporter, interface). I can use other tools like RouteViews or > RIPE's RIS to determine the destination AS's upstreams if needed. The res= t > of the path is probably not too helpful for determining peering > opportunities. > > I think I'm going to get Akvorado running in my environment. If that > doesn't pan out, I'll likely go back to AS-Stats. > > Can those running Akvorado comment on their system specs? The only spec > I've seen is a mention in this blog post > : > "Akvorado is performant enough to handle 100=E2=80=AF000 flows per second= with 64 > GB of RAM and 24 vCPU. With=E2=80=AF2=E2=80=AFTB of disk, you should expe= ct to keep data > for a few years." > > Thanks again all, > > -Brian > > > On 2024-03-26 19:04, Brian Knight via NANOG wrote: > > What's presently the most commonly used open source toolset for monitorin= g > AS-to-AS traffic? > > I want to see with which ASes I am exchanging the most traffic across my > transits and IX links. I want to look for opportunities to peer so I can > better sell expansion of peering to upper management. > > Our routers are mostly $VENDOR_C_XR so Netflow support is key. > > In the past, I've used AS-Stats > for this purpose. However, it is particularly CPU and disk IO intensive. > Also, it has not been actively maintained since 2017. > > InfluxDB wants to sell me > on Telegraf + > InfluxDB + Chronograf + Kapacitor, but I can't find any clear guide on wh= at > hardware I would need for that, never mind how to set up the software. It > does appear to have an open source option, however. > > pmacct seems to be good at gathering Netflow, but doesn't seem to analyze > data. I don't see any concise howto guides for setting this up for my > purpose, however. > > I'm aware Kentik does this very well, but I have no budget at the moment, > my testing window is longer than the 30 day trial, and we are not prepare= d > to share our Netflow data with a third party. > > Elastiflow appears to have been open source > at one tim= e > in the past, but no longer. Since it too appears to be hosted, I have the > same objections as I do with Kentik above. > > On-list and off-list replies are welcome. > > Thanks, > > -Brian > > > > --000000000000c2f95e061a6c2b0d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
We are in the process of adding netflow collection t= o libreqos. Any potential testers using any of these backends described bel= ow out there?

On Thu, Mar 28, 2024, 5:02=E2=80=AFPM Brian Knight via NANOG <= ;nanog@nanog.org> wrote:

Thanks to all who took the time to comment and make suggestions.

To summarize the private messages, one respondent suggested Argus as a c= ollector. Another mentioned that they are still using AS-Stats.

I'm drawn to Akvorado. I like the self-contained nature of the appli= cation. NF collector, database, and modern web GUI are all bundled in one d= ocker container. The full-featured demo is fantastic.=C2=A0That the ap= p can enrich the Netflow data with BMP is an added bonus.

The best part is, the GUI has the report viz I need, and it is actually = the default visualization in the demo. It also has the graph types that I d= idn't know I needed, like the Sankey graph.

FlowViewer looks interesting as well. I suspect getting the reports righ= t may take some time, given the amount of GUI filtering options.

pmacct and Argus seem to be capable tools that have been around for a lo= ng time, but I haven't seen a concise stack building guide to get Netfl= ow data into a good GUI using these. Looks like there are some older Docker= images available for both. I could write my own SQL or roll my own stack, = but I'd much rather spend my time on other things.

I appreciate the conversation around sFlow. I actually wasn't aware = that XR supported it. AS path probably doesn't add a whole lot of value= given that I'm focused on flows across our IP transit circuits. I'= m able to determine my next AS hop simply by looking at the flow's asso= ciated tuple of (flow exporter, interface). I can use other tools like Rout= eViews or RIPE's RIS to determine the destination AS's upstreams if= needed. The rest of the path is probably not too helpful for determining p= eering opportunities.

I think I'm going to get Akvorado running in my environment. If that= doesn't pan out, I'll likely go back to AS-Stats.

Can those running Akvorado comment on their system specs? The only spec = I've seen is a mention in this blo= g post: "Akvorado is performant enough to handle 100=E2=80=AF000 f= lows per second with 64 GB of RAM and 24 vCPU. With=E2=80=AF2=E2=80=AFTB of= disk, you should expect to keep data for a few years."

Thanks again all,

-Brian


On 2024-03-26 19:04, Brian Knight via NANOG wrote:

What's presentl= y the most commonly used open source toolset for monitoring AS-to-AS traffi= c?

I want to see with which ASes I am exchanging the most traffic = across my transits and IX links. I want to look for opportunities to peer s= o I can better sell expansion of peering to upper management.
=C2=A0
Our routers are mos= tly $VENDOR_C_XR so Netflow support is key.

In the past, I've = used AS-Stats for this purpose. However, it is particula= rly CPU and disk IO intensive. Also, it has not been actively maintained si= nce 2017.

InfluxDB wants to sell me on Telegraf + InfluxDB + Chronograf + Kapacitor, but I can't find an= y clear guide on what hardware I would need for that, never mind how to set= up the software. It does appear to have an open source option, however.
=C2=A0
pmacct seems to be = good at gathering Netflow, but doesn't seem to analyze data. I don'= t see any concise howto guides for setting this up for my purpose, however.=
=C2=A0
I'm aware Kenti= k does this very well, but I have no budget at the moment, my testing windo= w is longer than the 30 day trial, and we are not prepared to share our Net= flow data with a third party.
=C2=A0
Elastiflow ap= pears to have been open source at = one time in the past, but no longer. Since it too appears to be hosted, I h= ave the same objections as I do with Kentik above.
=C2=A0
On-list and off-lis= t replies are welcome.
=C2=A0
Thanks,
=C2=A0
-Brian
=C2=A0


--000000000000c2f95e061a6c2b0d--