One more update, and I'm going to sleep until "pick up daughter" time. :-) The tree at https://github.com/thebracket/LibreQoS/tree/integration-common-graph can now build a network.json, ShapedDevices.csv, and integrationUISPBandwidth.csv and follows pretty much the same logic as the previous importer - other than using data links to build the hierarchy and letting (requiring, currently) you specify the root node. It's handling our bizarre UISP setup pretty well now - so if anyone wants to test it (I recommend just running integrationUISP.py and checking the output rather than throwing it into production), I'd appreciate any feedback. Still on my list: handling the Mikrotik IPv6 connections, and exceptionCPE and site exclusion. If you want the pretty graphics, you need to "pip install graphviz" and "sudo apt install graphviz". It *should* detect that these aren't present and not try to draw pictures, otherwise. On Fri, Oct 28, 2022 at 2:06 PM Robert Chacón < robert.chacon@jackrabbitwireless.com> wrote: > Wow. This is very nicely done. Awesome work! > > On Fri, Oct 28, 2022 at 11:44 AM Herbert Wolverson via LibreQoS < > libreqos@lists.bufferbloat.net> wrote: > >> The integration is coming along nicely. Some progress updates: >> >> - You can specify a variable in ispConfig.py named "uispSite". This >> sets where in the topology you want the tree to start. This has two >> purposes: >> - It's hard to be psychic and know for sure where the shaper is in >> the network. >> - You could run multiple shapers at different egress points, with >> failover - and rebuild the entire topology from the point of view of a >> network node. >> - "Child node with children" are now automatically converted into a >> "(Generated Site) name" site, and their children rearranged. This: >> - Allows you to set the "site" bandwidth independently of the >> client site bandwidth. >> - Makes for easier trees, because we're inserting the site that >> really should be there. >> - Network.json generation (not the shaped devices file yet) is >> automatically generated from a tree, once PrepareTree() and >> createNetworkJson() are called. >> - There's a unit test that generates the network.example.json file >> and compares it with the original to ensure that they match. >> - Unit test coverage hits every function in the graph system, now. >> >> I'm liking this setup. With the non-vendor-specific logic contained >> inside the NetworkGraph type, the actual UISP code to generate the example >> tree is down to 65 >> lines of code, including comments. That'll grow a bit as I re-insert some >> automatic speed limit determination, AP/Site speed overrides ( >> i.e. the integrationUISPbandwidths.csv file). Still pretty clean. >> >> Creating the network.example.json file only requires: >> from integrationCommon import NetworkGraph, NetworkNode, NodeType >> import json >> net = NetworkGraph() >> net.addRawNode(NetworkNode("Site_1", "Site_1", "", NodeType.site, >> 1000, 1000)) >> net.addRawNode(NetworkNode("Site_2", "Site_2", "", NodeType.site, >> 500, 500)) >> net.addRawNode(NetworkNode("AP_A", "AP_A", "Site_1", NodeType.ap, >> 500, 500)) >> net.addRawNode(NetworkNode("Site_3", "Site_3", "Site_1", NodeType >> .site, 500, 500)) >> net.addRawNode(NetworkNode("PoP_5", "PoP_5", "Site_3", NodeType. >> site, 200, 200)) >> net.addRawNode(NetworkNode("AP_9", "AP_9", "PoP_5", NodeType.ap, >> 120, 120)) >> net.addRawNode(NetworkNode("PoP_6", "PoP_6", "PoP_5", NodeType. >> site, 60, 60)) >> net.addRawNode(NetworkNode("AP_11", "AP_11", "PoP_6", NodeType.ap, >> 30, 30)) >> net.addRawNode(NetworkNode("PoP_1", "PoP_1", "Site_2", NodeType. >> site, 200, 200)) >> net.addRawNode(NetworkNode("AP_7", "AP_7", "PoP_1", NodeType.ap, >> 100, 100)) >> net.addRawNode(NetworkNode("AP_1", "AP_1", "Site_2", NodeType.ap, >> 150, 150)) >> net.prepareTree() >> net.createNetworkJson() >> >> (The id and name fields are duplicated right now, I'm using readable >> names to keep me sane. The third string is the parent, and the last two >> numbers are bandwidth limits) >> The nice, readable format being: >> NetworkNode(id="Site_1", displayName="Site_1", parentId="", type=NodeType >> .site, download=1000, upload=1000) >> >> That in turns gives you the example network: >> [image: image.png] >> >> >> On Fri, Oct 28, 2022 at 7:40 AM Herbert Wolverson >> wrote: >> >>> Dave: I love those Gource animations! Game development is my other >>> hobby, I could easily get lost for weeks tweaking the shaders to make the >>> glow "just right". :-) >>> >>> Dan: Discovery would be nice, but I don't think we're ready to look in >>> that direction yet. I'm trying to build a "common grammar" to make it >>> easier to express network layout from integrations; that would be another >>> form/layer of integration and a lot easier to work with once there's a >>> solid foundation. Preseem does some of this (admittedly over-eagerly; >>> nothing needs to query SNMP that often!), and the SNMP route is quite >>> remarkably convoluted. Their support turned on a few "extra" modules to >>> deal with things like PMP450 clients that change MAC when you put them in >>> bridge mode vs NAT mode (and report the bridge mode CPE in some places >>> either way), Elevate CPEs that almost but not quite make sense. Robert's >>> code has the beginnings of some of this, scanning Mikrotik routers for IPv6 >>> allocations by MAC (this is also the hardest part for me to test, since I >>> don't have any v6 to test, currently). >>> >>> We tend to use UISP as the "source of truth" and treat it like a >>> database for a ton of external tools (mostly ones we've created). >>> >>> On Thu, Oct 27, 2022 at 7:27 PM dan wrote: >>> >>>> we're pretty similar in that we've made UISP a mess. Multiple paths to >>>> a pop. multiple pops on the network. failover between pops. Lots of >>>> 'other' devices. handing out /29 etc to customers. >>>> >>>> Some sort of discovery would be nice. Ideally though, pulling >>>> something from SNMP or router APIs etc to build the paths, but having a >>>> 'network elements' list with each of the links described. ie, backhaul 12 >>>> has MACs ..01 and ...02 at 300x100 and then build the topology around that >>>> from discovery. >>>> >>>> I've also thought about doing routine trace routes or watching TTLs or >>>> something like that to get some indication that topology has changed and >>>> then do another discovery and potential tree rebuild. >>>> >>>> On Thu, Oct 27, 2022 at 3:48 PM Robert Chacón via LibreQoS < >>>> libreqos@lists.bufferbloat.net> wrote: >>>> >>>>> This is awesome! Way to go here. Thank you for contributing this. >>>>> Being able to map out these complex integrations will help ISPs a ton, >>>>> and I really like that it is sharing common features between the Splynx and >>>>> UISP integrations. >>>>> >>>>> Thanks, >>>>> Robert >>>>> >>>>> On Thu, Oct 27, 2022 at 3:33 PM Herbert Wolverson via LibreQoS < >>>>> libreqos@lists.bufferbloat.net> wrote: >>>>> >>>>>> So I've been doing some work on getting UISP integration (and >>>>>> integrations in general) to work a bit more smoothly. >>>>>> >>>>>> I started by implementing a graph structure that mirrors both the >>>>>> networks and sites system. It's not done yet, but the basics are coming >>>>>> together nicely. You can see my progress so far at: >>>>>> https://github.com/thebracket/LibreQoS/tree/integration-common-graph >>>>>> >>>>>> Our UISP instance is a *great* testcase for torturing the system. I >>>>>> even found a case of UISP somehow auto-generating a circular portion of the >>>>>> tree. We have: >>>>>> >>>>>> - Non Ubiquiti devices as "other devices" >>>>>> - Sections that need shaping by subnet (e.g. "all of >>>>>> 192.168.1.0/24 shared 100 mbit") >>>>>> - Bridge mode devices using Option 82 to always allocate the same >>>>>> IP, with a "service IP" entry >>>>>> - Various bits of infrastructure mapped >>>>>> - Sites that go to client sites, which go to other client sites >>>>>> >>>>>> In other words, over the years we've unleashed a bit of a monster. >>>>>> Cleaning it up is a useful talk, but I wanted the integration to be able to >>>>>> handle pathological cases like us! >>>>>> >>>>>> So I fed our network into the current graph generator, and used >>>>>> graphviz to spit out a directed graph: >>>>>> [image: image.png] >>>>>> That doesn't include client sites! Legend: >>>>>> >>>>>> >>>>>> - Green = the root site. >>>>>> - Red = a site >>>>>> - Blue = an access point >>>>>> - Magenta = a client site that has children >>>>>> >>>>>> So the part in "common" is designed heavily to reduce repetition. >>>>>> When it's done, you should be able to feed in sites, APs, clients, devices, >>>>>> etc. in a pretty flexible manner. Given how much code is shared between the >>>>>> UISP and Splynx integration code, I'm pretty sure both will be cut to a >>>>>> tiny fraction of the total code. :-) >>>>>> >>>>>> I can't post the full tree, it's full of client names. >>>>>> _______________________________________________ >>>>>> LibreQoS mailing list >>>>>> LibreQoS@lists.bufferbloat.net >>>>>> https://lists.bufferbloat.net/listinfo/libreqos >>>>>> >>>>> >>>>> >>>>> -- >>>>> Robert Chacón >>>>> CEO | JackRabbit Wireless LLC >>>>> _______________________________________________ >>>>> LibreQoS mailing list >>>>> LibreQoS@lists.bufferbloat.net >>>>> https://lists.bufferbloat.net/listinfo/libreqos >>>>> >>>> _______________________________________________ >> LibreQoS mailing list >> LibreQoS@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/libreqos >> > > > -- > Robert Chacón > CEO | JackRabbit Wireless LLC > Dev | LibreQoS.io > >