Re: [LibreQoS] Integration system, aka fun with graph theory

["Robert Chacón" <robert.chacon@jackrabbitwireless.com>]

[-- Attachment #1.1: Type: text/plain, Size: 11603 bytes --]

Awesome work. It succeeded in building the topology and creating
ShapedDevices.csv for my network. It even graphed it perfectly. Nice!
I notice that in ShapedDevices.csv it does add CPE radios (which in our
case we don't shape - they are in bridge mode) with IPv4 and IPv6s both
being empty lists [].
This is not necessarily bad, but it may lead to empty leaf classes being
created on LibreQoS.py runs. Not a huge deal, it just makes the minor class
counter increment toward the 32k limit faster.
Do you think perhaps we should check:
*if (len(IPv4) == 0) and (len(IPv6) == 0):*
*   # Skip adding this entry to ShapedDevices.csv*
Or something similar around line 329 of integrationCommon.py?
Open to your suggestions there.



On Fri, Oct 28, 2022 at 1:55 PM Herbert Wolverson via LibreQoS <
libreqos@lists.bufferbloat.net> wrote:

> One more update, and I'm going to sleep until "pick up daughter" time. :-)
>
> The tree at
> https://github.com/thebracket/LibreQoS/tree/integration-common-graph can
> now build a network.json, ShapedDevices.csv, and
> integrationUISPBandwidth.csv and follows pretty much the same logic as the
> previous importer - other than using data links to build the hierarchy and
> letting (requiring, currently) you specify the root node. It's handling our
> bizarre UISP setup pretty well now - so if anyone wants to test it (I
> recommend just running integrationUISP.py and checking the output rather
> than throwing it into production), I'd appreciate any feedback.
>
> Still on my list: handling the Mikrotik IPv6 connections, and exceptionCPE
> and site exclusion.
>
> If you want the pretty graphics, you need to "pip install graphviz" and
> "sudo apt install graphviz". It *should* detect that these aren't present
> and not try to draw pictures, otherwise.
>
> On Fri, Oct 28, 2022 at 2:06 PM Robert Chacón <
> robert.chacon@jackrabbitwireless.com> wrote:
>
>> Wow. This is very nicely done. Awesome work!
>>
>> On Fri, Oct 28, 2022 at 11:44 AM Herbert Wolverson via LibreQoS <
>> libreqos@lists.bufferbloat.net> wrote:
>>
>>> The integration is coming along nicely. Some progress updates:
>>>
>>>    - You can specify a variable in ispConfig.py named "uispSite". This
>>>    sets where in the topology you want the tree to start. This has two
>>>    purposes:
>>>       - It's hard to be psychic and know for sure where the shaper is
>>>       in the network.
>>>       - You could run multiple shapers at different egress points, with
>>>       failover - and rebuild the entire topology from the point of view of a
>>>       network node.
>>>    - "Child node with children" are now automatically converted into a
>>>    "(Generated Site) name" site, and their children rearranged. This:
>>>       - Allows you to set the "site" bandwidth independently of the
>>>       client site bandwidth.
>>>       - Makes for easier trees, because we're inserting the site that
>>>       really should be there.
>>>    - Network.json generation (not the shaped devices file yet) is
>>>    automatically generated from a tree, once PrepareTree() and
>>>    createNetworkJson() are called.
>>>       - There's a unit test that generates the network.example.json
>>>       file and compares it with the original to ensure that they match.
>>>    - Unit test coverage hits every function in the graph system, now.
>>>
>>> I'm liking this setup. With the non-vendor-specific logic contained
>>> inside the NetworkGraph type, the actual UISP code to generate the example
>>> tree is down to 65
>>> lines of code, including comments. That'll grow a bit as I re-insert
>>> some automatic speed limit determination, AP/Site speed overrides (
>>> i.e. the integrationUISPbandwidths.csv file). Still pretty clean.
>>>
>>> Creating the network.example.json file only requires:
>>> from integrationCommon import NetworkGraph, NetworkNode, NodeType
>>>         import json
>>>         net = NetworkGraph()
>>>         net.addRawNode(NetworkNode("Site_1", "Site_1", "", NodeType.site,
>>> 1000, 1000))
>>>         net.addRawNode(NetworkNode("Site_2", "Site_2", "", NodeType.site,
>>> 500, 500))
>>>         net.addRawNode(NetworkNode("AP_A", "AP_A", "Site_1", NodeType.ap,
>>> 500, 500))
>>>         net.addRawNode(NetworkNode("Site_3", "Site_3", "Site_1",
>>> NodeType.site, 500, 500))
>>>         net.addRawNode(NetworkNode("PoP_5", "PoP_5", "Site_3", NodeType.
>>> site, 200, 200))
>>>         net.addRawNode(NetworkNode("AP_9", "AP_9", "PoP_5", NodeType.ap,
>>> 120, 120))
>>>         net.addRawNode(NetworkNode("PoP_6", "PoP_6", "PoP_5", NodeType.
>>> site, 60, 60))
>>>         net.addRawNode(NetworkNode("AP_11", "AP_11", "PoP_6", NodeType.
>>> ap, 30, 30))
>>>         net.addRawNode(NetworkNode("PoP_1", "PoP_1", "Site_2", NodeType.
>>> site, 200, 200))
>>>         net.addRawNode(NetworkNode("AP_7", "AP_7", "PoP_1", NodeType.ap,
>>> 100, 100))
>>>         net.addRawNode(NetworkNode("AP_1", "AP_1", "Site_2", NodeType.ap,
>>> 150, 150))
>>>         net.prepareTree()
>>>         net.createNetworkJson()
>>>
>>> (The id and name fields are duplicated right now, I'm using readable
>>> names to keep me sane. The third string is the parent, and the last two
>>> numbers are bandwidth limits)
>>> The nice, readable format being:
>>> NetworkNode(id="Site_1", displayName="Site_1", parentId="", type=
>>> NodeType.site, download=1000, upload=1000)
>>>
>>> That in turns gives you the example network:
>>> [image: image.png]
>>>
>>>
>>> On Fri, Oct 28, 2022 at 7:40 AM Herbert Wolverson <herberticus@gmail.com>
>>> wrote:
>>>
>>>> Dave: I love those Gource animations! Game development is my other
>>>> hobby, I could easily get lost for weeks tweaking the shaders to make the
>>>> glow "just right". :-)
>>>>
>>>> Dan: Discovery would be nice, but I don't think we're ready to look in
>>>> that direction yet. I'm trying to build a "common grammar" to make it
>>>> easier to express network layout from integrations; that would be another
>>>> form/layer of integration and a lot easier to work with once there's a
>>>> solid foundation. Preseem does some of this (admittedly over-eagerly;
>>>> nothing needs to query SNMP that often!), and the SNMP route is quite
>>>> remarkably convoluted. Their support turned on a few "extra" modules to
>>>> deal with things like PMP450 clients that change MAC when you put them in
>>>> bridge mode vs NAT mode (and report the bridge mode CPE in some places
>>>> either way), Elevate CPEs that almost but not quite make sense. Robert's
>>>> code has the beginnings of some of this, scanning Mikrotik routers for IPv6
>>>> allocations by MAC (this is also the hardest part for me to test, since I
>>>> don't have any v6 to test, currently).
>>>>
>>>> We tend to use UISP as the "source of truth" and treat it like a
>>>> database for a ton of external tools (mostly ones we've created).
>>>>
>>>> On Thu, Oct 27, 2022 at 7:27 PM dan <dandenson@gmail.com> wrote:
>>>>
>>>>> we're pretty similar in that we've made UISP a mess.  Multiple paths
>>>>> to a pop.  multiple pops on the network.  failover between pops.  Lots of
>>>>> 'other' devices. handing out /29 etc to customers.
>>>>>
>>>>> Some sort of discovery would be nice.  Ideally though, pulling
>>>>> something from SNMP or router APIs etc to build the paths, but having a
>>>>> 'network elements' list with each of the links described.  ie, backhaul 12
>>>>> has MACs ..01 and ...02 at 300x100 and then build the topology around that
>>>>> from discovery.
>>>>>
>>>>> I've also thought about doing routine trace routes or watching TTLs or
>>>>> something like that to get some indication that topology has changed and
>>>>> then do another discovery and potential tree rebuild.
>>>>>
>>>>> On Thu, Oct 27, 2022 at 3:48 PM Robert Chacón via LibreQoS <
>>>>> libreqos@lists.bufferbloat.net> wrote:
>>>>>
>>>>>> This is awesome! Way to go here. Thank you for contributing this.
>>>>>> Being able to map out these complex integrations will help ISPs a
>>>>>> ton, and I really like that it is sharing common features between the
>>>>>> Splynx and UISP integrations.
>>>>>>
>>>>>> Thanks,
>>>>>> Robert
>>>>>>
>>>>>> On Thu, Oct 27, 2022 at 3:33 PM Herbert Wolverson via LibreQoS <
>>>>>> libreqos@lists.bufferbloat.net> wrote:
>>>>>>
>>>>>>> So I've been doing some work on getting UISP integration (and
>>>>>>> integrations in general) to work a bit more smoothly.
>>>>>>>
>>>>>>> I started by implementing a graph structure that mirrors both the
>>>>>>> networks and sites system. It's not done yet, but the basics are coming
>>>>>>> together nicely. You can see my progress so far at:
>>>>>>> https://github.com/thebracket/LibreQoS/tree/integration-common-graph
>>>>>>>
>>>>>>> Our UISP instance is a *great* testcase for torturing the system. I
>>>>>>> even found a case of UISP somehow auto-generating a circular portion of the
>>>>>>> tree. We have:
>>>>>>>
>>>>>>>    - Non Ubiquiti devices as "other devices"
>>>>>>>    - Sections that need shaping by subnet (e.g. "all of
>>>>>>>    192.168.1.0/24 shared 100 mbit")
>>>>>>>    - Bridge mode devices using Option 82 to always allocate the
>>>>>>>    same IP, with a "service IP" entry
>>>>>>>    - Various bits of infrastructure mapped
>>>>>>>    - Sites that go to client sites, which go to other client sites
>>>>>>>
>>>>>>> In other words, over the years we've unleashed a bit of a monster.
>>>>>>> Cleaning it up is a useful talk, but I wanted the integration to be able to
>>>>>>> handle pathological cases like us!
>>>>>>>
>>>>>>> So I fed our network into the current graph generator, and used
>>>>>>> graphviz to spit out a directed graph:
>>>>>>> [image: image.png]
>>>>>>> That doesn't include client sites! Legend:
>>>>>>>
>>>>>>>
>>>>>>>    - Green = the root site.
>>>>>>>    - Red = a site
>>>>>>>    - Blue = an access point
>>>>>>>    - Magenta = a client site that has children
>>>>>>>
>>>>>>> So the part in "common" is designed heavily to reduce repetition.
>>>>>>> When it's done, you should be able to feed in sites, APs, clients, devices,
>>>>>>> etc. in a pretty flexible manner. Given how much code is shared between the
>>>>>>> UISP and Splynx integration code, I'm pretty sure both will be cut to a
>>>>>>> tiny fraction of the total code. :-)
>>>>>>>
>>>>>>> I can't post the full tree, it's full of client names.
>>>>>>> _______________________________________________
>>>>>>> LibreQoS mailing list
>>>>>>> LibreQoS@lists.bufferbloat.net
>>>>>>> https://lists.bufferbloat.net/listinfo/libreqos
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Robert Chacón
>>>>>> CEO | JackRabbit Wireless LLC <http://jackrabbitwireless.com>
>>>>>> _______________________________________________
>>>>>> LibreQoS mailing list
>>>>>> LibreQoS@lists.bufferbloat.net
>>>>>> https://lists.bufferbloat.net/listinfo/libreqos
>>>>>>
>>>>> _______________________________________________
>>> LibreQoS mailing list
>>> LibreQoS@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/libreqos
>>>
>>
>>
>> --
>> Robert Chacón
>> CEO | JackRabbit Wireless LLC <http://jackrabbitwireless.com>
>> Dev | LibreQoS.io
>>
>> _______________________________________________
> LibreQoS mailing list
> LibreQoS@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/libreqos
>


-- 
Robert Chacón
CEO | JackRabbit Wireless LLC <http://jackrabbitwireless.com>
Dev | LibreQoS.io

[-- Attachment #1.2: Type: text/html, Size: 28658 bytes --]

[-- Attachment #2: image.png --]
[-- Type: image/png, Size: 573568 bytes --]

[-- Attachment #3: image.png --]
[-- Type: image/png, Size: 115596 bytes --]