From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pg1-x531.google.com (mail-pg1-x531.google.com [IPv6:2607:f8b0:4864:20::531]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id A6E483B2A4 for ; Wed, 9 Nov 2022 11:20:36 -0500 (EST) Received: by mail-pg1-x531.google.com with SMTP id o13so7328046pgu.7 for ; Wed, 09 Nov 2022 08:20:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=oZK3A/7oGsbRXeVD1r41VYe6R47MDHGMsKTQ+njlwXU=; b=ah8jauDK1u9pU5sogOIwsjJFJMt/Cwq1fYPpF5NeKkkrrCD/QD7pu7fhxAFW2ugpvw tj9BOpTvwENz7N8klG1F2xMXJJ9B1Sco2yuvxQh7zCnRQv53qhA13F2LSNJp8P0sCLlT 2HiH5ExkS0j1+evjbGnFbJrHhIOt/Nc3Lm3jjIYKHShP0D094/5sA7cn5jdXayDVOit/ 3DCqo1csraChk7zEBSUjG9V0kACMmSHFGRp3A6i/WNEgjaZiPhDbh49tA4HQWGvkZ/QZ wqnGx9s7CxpjXG/lRWaCYiPejoqDySok1hYq0gD8qFORaV+ouMhQaBh3EVbJ15dSh7Uk 8M4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=oZK3A/7oGsbRXeVD1r41VYe6R47MDHGMsKTQ+njlwXU=; b=md8zYVupX2Z65rItPEXQkBuACbStM10RmKGIVua7TmLpS1ncjNOuUBhP5YqvHd/6Iq jdlMX7HWZtjTLd4TGVDA7sgGcVvHkhG35SBSImaacKHx7ITrwTrctnT0c63/nAHZUyp/ W08J6qaGr9CdaraPkLWPrN2SQDr85Uf1sYFoEPeGzTPR1bJheDtdd8RjkhDLfD4SHgDM QSp6cASP/oqg9OB0TCFLaaAYvMHe0BHYHhL5X1xpUcTicQavEWH2A4GE3j9qZJvSb0aD BPc4RJysflwm3MJKUte56RMAFT+DFIIopISnwrHZTN04LqbDKg4BeKQzh+Ql/NPdakdm lh5w== X-Gm-Message-State: ACrzQf2sck+OKcddrDGqumL13MRzSekqjqe0CjYuRyqh8RGzcMDd8noA WA2sQvXvFE5h/6PiRwUmklVhCuM+nzjHGJbnP7FMA6ny X-Google-Smtp-Source: AMsMyM5JHizpZqfqh4uNxCr9dmrw+sL//EwlwXj3g2BmfHIIpP8ctFtp0Kf/qqT3LLkFKRrGte8kptbyAB7IYpWKRVk= X-Received: by 2002:aa7:96c7:0:b0:56b:c569:99c with SMTP id h7-20020aa796c7000000b0056bc569099cmr61399868pfq.4.1668010835238; Wed, 09 Nov 2022 08:20:35 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Herbert Wolverson Date: Wed, 9 Nov 2022 10:20:24 -0600 Message-ID: Cc: libreqos Content-Type: multipart/alternative; boundary="00000000000050e5d205ed0c0911" Subject: Re: [LibreQoS] Tracking unknown IPs (maybe for 1.4?) X-BeenThere: libreqos@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Many ISPs need the kinds of quality shaping cake can do List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Nov 2022 16:20:36 -0000 --00000000000050e5d205ed0c0911 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable We have a bespoke solution to do similar (I keep meaning to make it more generic and open source it). The basic operation is (using our Mimosa devices as an example; it's actually a lot more complicated than that since we have everything from apartment complexes with Ethernet jacks to regular Ubiquiti devices in bridge mode): - A server contains an instance of FreeRADIUS. - Periodically, a script runs and queries UISP. It finds client sites with a device matching the type "Mimosa C5x" and an "other device" entit= led "Service IP". - A radius record is then added for the CPE (using the MAC and IP from the Mimosa device), placing the Mimosa CPE on the IP address in = the "mimosa" record. - A second radius record is added that matches Option 82 headers to see that a request passed through the Mimosa. This will always hand out the IP address from the "Service IP" record. - The radius database is refreshed with this information. - When a CPE comes online, the DHCP server sends a RADIUS request. If the MAC address matches a RADIUS record, the device is assigned to the C= PE address from the RADIUS records. - When a customer's device sends a DHCP request, it passes through the CPE. The CPE's "option 82" support decorates the DHCP request with the CPE's MAC address in a header. This then matches the second rule in Radi= us, ensuring that no matter what device the customer plugged in - it gets th= e Service IP. This then dovetails into the QoS - because we can be 100% sure that the customer's router has the IP address of the Service IP record in their client site. It took about an afternoon to setup, and is really nice. We have additional rules like "place unknown CPEs into a block that is redirected to a page reminding the installer to call in the account for setup", special handling of suspended accounts and similar. People have been *begging* Ubiquiti to a) support option 82 properly on the M5/M2 line (I have a 10 year old request still unanswered!), and b) provide some sort of RADIUS setup baked into UISP. The latter won't happen, because it reduces vendor lock-in. But it's really easy to setup, and use UISP as a "source of truth". (Obviously, when your clients are bridged you need to take precautions - client isolation, switch port isolation and DHCP snooping) On Wed, Nov 9, 2022 at 10:07 AM dan wrote: > How are you linking UISP to RADIUS? > > On Sat, Nov 5, 2022 at 10:29 AM Robert Chac=C3=B3n via LibreQoS < > libreqos@lists.bufferbloat.net> wrote: > >> In our particular case we use RADIUS tied to UISP so we don't have the >> immediate need, but I think it's an important feature to add. >> >> Perhaps cpumap-pping can have a feature to define "shaped subnets" durin= g >> the filter setup, and then we could query cpumap-pping for a JSON output= of >> IPs detected in traffic that are in the "shaped subnets" groups, but not >> defined in the hash map. >> >> Curious to hear what others think here. Would others need this in order >> to adopt LibreQoS? >> >> >> On Sat, Nov 5, 2022 at 7:33 AM Herbert Wolverson via LibreQoS < >> libreqos@lists.bufferbloat.net> wrote: >> >>> As we approach the v1.3 pre-release feature freeze, I've been thinking = a >>> little bit about nice things to have. One thing I found useful in both >>> BracketQoS and Preseem was the ability to grab a list of IP addresses t= hat >>> had been through the shaper, but weren't mapped to a queue (obviously, = only >>> from within the "allowed IP" range - we're not trying to map the Intern= et!). >>> >>> In Preseem, there's a link to download a CSV file containing all the >>> unmapped IP addresses and how much traffic they have consumed. BracketQ= oS >>> (pre cpumap-pping) has a report showing the IPs (no traffic). >>> >>> *Why is this useful?* >>> >>> Knowing which local IP addresses were processed but not mapped lets you >>> find: >>> >>> * the times that a device was installed, but the on-boarding process >>> wasn't completed. Yes, that shouldn't happen. And - unfortunately - it >>> occasionally does. If you're using RADIUS-based authentication, it's re= ally >>> difficult for this to happen - but not everyone is. >>> * If there's a bug in your shaper integration, it's helpful to see >>> "oops, I put X on the default" >>> * Just occasionally, you get a customer who needs a special setup; it's >>> helpful to see that it worked. >>> >>> *Current Status* >>> >>> Before cpumap-pping, Bracket was grabbing them by reading the pping >>> output and listing addresses that didn't match a shaping rule. That doe= sn't >>> work now: >>> >>> * xdp_pping is spitting out TC handles, rather than IP addresses. >>> * With a default rule in place, and handling for IPv6 and IPv4 subnets, >>> an IP address might not exactly match an entry (requires an LPM trie >>> lookup) - and IPs matching a default rule (::/0 or 0.0.0.0/0) will >>> always come back with the "default" handle. >>> >>> It's currently pretty tricky to do. >>> >>> So I'm curious; would others like to see this? I have a few ideas for >>> how to make it work, but don't want to start serious planning/design if= I'm >>> the only one who wants the feature. >>> _______________________________________________ >>> LibreQoS mailing list >>> LibreQoS@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/libreqos >>> >> >> >> -- >> Robert Chac=C3=B3n >> CEO | JackRabbit Wireless LLC >> Dev | LibreQoS.io >> >> _______________________________________________ >> LibreQoS mailing list >> LibreQoS@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/libreqos >> > --00000000000050e5d205ed0c0911 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
We have a bespoke solution to do similar (I keep mean= ing to make it more generic and open source it). The basic operation is (us= ing our Mimosa devices as an example; it's actually a lot more complica= ted than that since we have everything from apartment complexes with Ethern= et jacks to regular Ubiquiti devices in bridge mode):

<= div>
  • A server contains an instance of FreeRADIUS.
  • Periodical= ly, a script runs and queries UISP. It finds client sites with a device mat= ching the type "Mimosa C5x" and an "other device" entit= led "Service IP".
    • A radius record is then added for t= he CPE (using the MAC and IP from the Mimosa device), placing the Mimosa CP= E on the IP address in the "mimosa" record.
    • A second radi= us record is added that matches Option 82 headers to see that a request pas= sed through the Mimosa. This will always hand out the IP address from the &= quot;Service IP" record.
    • The radius database is refreshed with= this information.
  • When a CPE comes online, the DHCP server se= nds a RADIUS request. If the MAC address matches a RADIUS record, the devic= e is assigned to the CPE address from the RADIUS records.
  • When a cu= stomer's device sends a DHCP request, it passes through the CPE. The CP= E's "option 82" support decorates the DHCP request with the C= PE's MAC address in a header. This then matches the second rule in Radi= us, ensuring that no matter what device the customer plugged in - it gets t= he Service IP.
This then dovetails into the QoS - because we = can be 100% sure that the customer's router has the IP address of the S= ervice IP record in their client site.

It took abo= ut an afternoon to setup, and is really nice. We have additional rules like= "place unknown CPEs into a block that is redirected to a page remindi= ng the installer to call in the account for setup", special handling o= f suspended accounts and similar.

People have been= begging Ubiquiti to a) support option 82 properly on the M5/M2 line= (I have a 10 year old request still unanswered!), and b) provide some sort= of RADIUS setup baked into UISP. The latter won't happen, because it r= educes vendor lock-in. But it's really easy to setup, and use UISP as a= "source of truth". (Obviously, when your clients are bridged you= need to take precautions - client isolation, switch port isolation and DHC= P snooping)


On Wed, Nov 9, 2022 at 10:07 AM d= an <dandenson@gmail.com> w= rote:
How are you linking UISP to RADIUS?

On Sat, Nov 5, 2022 at 10:29 AM= Robert Chac=C3=B3n via LibreQoS <libreqos@lists.bufferbloat.net> wrote:=
In our particular case we use RADIUS tied to UISP so we d= on't have the immediate need, but I think it's an important feature= to add.

Perhaps cpumap-pping can have a feature t= o define "shaped subnets" during the filter setup, and then we co= uld query cpumap-pping for a JSON output of IPs detected in traffic that ar= e in the "shaped subnets" groups, but not defined in the hash map= .

Curious to hear what others think here. Would ot= hers need this in order to adopt LibreQoS?

<= br>
On Sat,= Nov 5, 2022 at 7:33 AM Herbert Wolverson via LibreQoS <libreqos@lists.bufferbl= oat.net> wrote:
As we approach the v1.3 pre-release feature fr= eeze, I've been thinking a little bit about nice things to have. One th= ing I found useful in both BracketQoS and Preseem was the ability to grab a= list of IP addresses that had been through the shaper, but weren't map= ped to a queue (obviously, only from within the "allowed IP" rang= e - we're not trying to map the Internet!).

In= Preseem, there's a link to download a CSV file containing all the unma= pped IP addresses and how much traffic they have consumed. BracketQoS (pre = cpumap-pping) has a report showing the IPs (no traffic).

=
*Why is this useful?*

Knowing which loc= al IP addresses were processed but not mapped lets you find:

=
* the times that a device was installed, but the on-boarding pro= cess wasn't completed. Yes, that shouldn't happen. And - unfortunat= ely - it occasionally does. If you're using RADIUS-based authentication= , it's really difficult for this to happen - but not everyone is.
=
* If there's a bug in your shaper integration, it's helpful to= see "oops, I put X on the default"
* Just occasionally= , you get a customer who needs a special setup; it's helpful to see tha= t it worked.

*Current Status*
Before cpumap-pping, Bracket was grabbing them by reading the = pping output and listing addresses that didn't match a shaping rule. Th= at doesn't work now:

* xdp_pping is spitting o= ut TC handles, rather than IP addresses.
* With a default rule in= place, and handling for IPv6 and IPv4 subnets, an IP address might not exa= ctly match an entry (requires an LPM trie lookup) - and IPs matching a defa= ult rule (::/0 or 0.0.0.0/0<= /a>) will always come back with the "default" handle.

<= div>So I'm curious; would others like to see this? I have a few ideas f= or how to make it work, but don't want to start serious planning/design= if I'm the only one who wants the feature.
_______________________________________________
LibreQoS mailing list
LibreQo= S@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/libreqos


--
Robert Chac=C3=B3n

<= /div>
_______________________________________________
LibreQoS mailing list
LibreQo= S@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/libreqos
--00000000000050e5d205ed0c0911--