Awesome work. It succeeded in building the topology and creating ShapedDevices.csv for my network. It even graphed it perfectly. Nice! I notice that in ShapedDevices.csv it does add CPE radios (which in our case we don't shape - they are in bridge mode) with IPv4 and IPv6s both being empty lists []. This is not necessarily bad, but it may lead to empty leaf classes being created on LibreQoS.py runs. Not a huge deal, it just makes the minor class counter increment toward the 32k limit faster. Do you think perhaps we should check: *if (len(IPv4) == 0) and (len(IPv6) == 0):* * # Skip adding this entry to ShapedDevices.csv* Or something similar around line 329 of integrationCommon.py? Open to your suggestions there. On Fri, Oct 28, 2022 at 1:55 PM Herbert Wolverson via LibreQoS < libreqos@lists.bufferbloat.net> wrote: > One more update, and I'm going to sleep until "pick up daughter" time. :-) > > The tree at > https://github.com/thebracket/LibreQoS/tree/integration-common-graph can > now build a network.json, ShapedDevices.csv, and > integrationUISPBandwidth.csv and follows pretty much the same logic as the > previous importer - other than using data links to build the hierarchy and > letting (requiring, currently) you specify the root node. It's handling our > bizarre UISP setup pretty well now - so if anyone wants to test it (I > recommend just running integrationUISP.py and checking the output rather > than throwing it into production), I'd appreciate any feedback. > > Still on my list: handling the Mikrotik IPv6 connections, and exceptionCPE > and site exclusion. > > If you want the pretty graphics, you need to "pip install graphviz" and > "sudo apt install graphviz". It *should* detect that these aren't present > and not try to draw pictures, otherwise. > > On Fri, Oct 28, 2022 at 2:06 PM Robert Chacón < > robert.chacon@jackrabbitwireless.com> wrote: > >> Wow. This is very nicely done. Awesome work! >> >> On Fri, Oct 28, 2022 at 11:44 AM Herbert Wolverson via LibreQoS < >> libreqos@lists.bufferbloat.net> wrote: >> >>> The integration is coming along nicely. Some progress updates: >>> >>> - You can specify a variable in ispConfig.py named "uispSite". This >>> sets where in the topology you want the tree to start. This has two >>> purposes: >>> - It's hard to be psychic and know for sure where the shaper is >>> in the network. >>> - You could run multiple shapers at different egress points, with >>> failover - and rebuild the entire topology from the point of view of a >>> network node. >>> - "Child node with children" are now automatically converted into a >>> "(Generated Site) name" site, and their children rearranged. This: >>> - Allows you to set the "site" bandwidth independently of the >>> client site bandwidth. >>> - Makes for easier trees, because we're inserting the site that >>> really should be there. >>> - Network.json generation (not the shaped devices file yet) is >>> automatically generated from a tree, once PrepareTree() and >>> createNetworkJson() are called. >>> - There's a unit test that generates the network.example.json >>> file and compares it with the original to ensure that they match. >>> - Unit test coverage hits every function in the graph system, now. >>> >>> I'm liking this setup. With the non-vendor-specific logic contained >>> inside the NetworkGraph type, the actual UISP code to generate the example >>> tree is down to 65 >>> lines of code, including comments. That'll grow a bit as I re-insert >>> some automatic speed limit determination, AP/Site speed overrides ( >>> i.e. the integrationUISPbandwidths.csv file). Still pretty clean. >>> >>> Creating the network.example.json file only requires: >>> from integrationCommon import NetworkGraph, NetworkNode, NodeType >>> import json >>> net = NetworkGraph() >>> net.addRawNode(NetworkNode("Site_1", "Site_1", "", NodeType.site, >>> 1000, 1000)) >>> net.addRawNode(NetworkNode("Site_2", "Site_2", "", NodeType.site, >>> 500, 500)) >>> net.addRawNode(NetworkNode("AP_A", "AP_A", "Site_1", NodeType.ap, >>> 500, 500)) >>> net.addRawNode(NetworkNode("Site_3", "Site_3", "Site_1", >>> NodeType.site, 500, 500)) >>> net.addRawNode(NetworkNode("PoP_5", "PoP_5", "Site_3", NodeType. >>> site, 200, 200)) >>> net.addRawNode(NetworkNode("AP_9", "AP_9", "PoP_5", NodeType.ap, >>> 120, 120)) >>> net.addRawNode(NetworkNode("PoP_6", "PoP_6", "PoP_5", NodeType. >>> site, 60, 60)) >>> net.addRawNode(NetworkNode("AP_11", "AP_11", "PoP_6", NodeType. >>> ap, 30, 30)) >>> net.addRawNode(NetworkNode("PoP_1", "PoP_1", "Site_2", NodeType. >>> site, 200, 200)) >>> net.addRawNode(NetworkNode("AP_7", "AP_7", "PoP_1", NodeType.ap, >>> 100, 100)) >>> net.addRawNode(NetworkNode("AP_1", "AP_1", "Site_2", NodeType.ap, >>> 150, 150)) >>> net.prepareTree() >>> net.createNetworkJson() >>> >>> (The id and name fields are duplicated right now, I'm using readable >>> names to keep me sane. The third string is the parent, and the last two >>> numbers are bandwidth limits) >>> The nice, readable format being: >>> NetworkNode(id="Site_1", displayName="Site_1", parentId="", type= >>> NodeType.site, download=1000, upload=1000) >>> >>> That in turns gives you the example network: >>> [image: image.png] >>> >>> >>> On Fri, Oct 28, 2022 at 7:40 AM Herbert Wolverson >>> wrote: >>> >>>> Dave: I love those Gource animations! Game development is my other >>>> hobby, I could easily get lost for weeks tweaking the shaders to make the >>>> glow "just right". :-) >>>> >>>> Dan: Discovery would be nice, but I don't think we're ready to look in >>>> that direction yet. I'm trying to build a "common grammar" to make it >>>> easier to express network layout from integrations; that would be another >>>> form/layer of integration and a lot easier to work with once there's a >>>> solid foundation. Preseem does some of this (admittedly over-eagerly; >>>> nothing needs to query SNMP that often!), and the SNMP route is quite >>>> remarkably convoluted. Their support turned on a few "extra" modules to >>>> deal with things like PMP450 clients that change MAC when you put them in >>>> bridge mode vs NAT mode (and report the bridge mode CPE in some places >>>> either way), Elevate CPEs that almost but not quite make sense. Robert's >>>> code has the beginnings of some of this, scanning Mikrotik routers for IPv6 >>>> allocations by MAC (this is also the hardest part for me to test, since I >>>> don't have any v6 to test, currently). >>>> >>>> We tend to use UISP as the "source of truth" and treat it like a >>>> database for a ton of external tools (mostly ones we've created). >>>> >>>> On Thu, Oct 27, 2022 at 7:27 PM dan wrote: >>>> >>>>> we're pretty similar in that we've made UISP a mess. Multiple paths >>>>> to a pop. multiple pops on the network. failover between pops. Lots of >>>>> 'other' devices. handing out /29 etc to customers. >>>>> >>>>> Some sort of discovery would be nice. Ideally though, pulling >>>>> something from SNMP or router APIs etc to build the paths, but having a >>>>> 'network elements' list with each of the links described. ie, backhaul 12 >>>>> has MACs ..01 and ...02 at 300x100 and then build the topology around that >>>>> from discovery. >>>>> >>>>> I've also thought about doing routine trace routes or watching TTLs or >>>>> something like that to get some indication that topology has changed and >>>>> then do another discovery and potential tree rebuild. >>>>> >>>>> On Thu, Oct 27, 2022 at 3:48 PM Robert Chacón via LibreQoS < >>>>> libreqos@lists.bufferbloat.net> wrote: >>>>> >>>>>> This is awesome! Way to go here. Thank you for contributing this. >>>>>> Being able to map out these complex integrations will help ISPs a >>>>>> ton, and I really like that it is sharing common features between the >>>>>> Splynx and UISP integrations. >>>>>> >>>>>> Thanks, >>>>>> Robert >>>>>> >>>>>> On Thu, Oct 27, 2022 at 3:33 PM Herbert Wolverson via LibreQoS < >>>>>> libreqos@lists.bufferbloat.net> wrote: >>>>>> >>>>>>> So I've been doing some work on getting UISP integration (and >>>>>>> integrations in general) to work a bit more smoothly. >>>>>>> >>>>>>> I started by implementing a graph structure that mirrors both the >>>>>>> networks and sites system. It's not done yet, but the basics are coming >>>>>>> together nicely. You can see my progress so far at: >>>>>>> https://github.com/thebracket/LibreQoS/tree/integration-common-graph >>>>>>> >>>>>>> Our UISP instance is a *great* testcase for torturing the system. I >>>>>>> even found a case of UISP somehow auto-generating a circular portion of the >>>>>>> tree. We have: >>>>>>> >>>>>>> - Non Ubiquiti devices as "other devices" >>>>>>> - Sections that need shaping by subnet (e.g. "all of >>>>>>> 192.168.1.0/24 shared 100 mbit") >>>>>>> - Bridge mode devices using Option 82 to always allocate the >>>>>>> same IP, with a "service IP" entry >>>>>>> - Various bits of infrastructure mapped >>>>>>> - Sites that go to client sites, which go to other client sites >>>>>>> >>>>>>> In other words, over the years we've unleashed a bit of a monster. >>>>>>> Cleaning it up is a useful talk, but I wanted the integration to be able to >>>>>>> handle pathological cases like us! >>>>>>> >>>>>>> So I fed our network into the current graph generator, and used >>>>>>> graphviz to spit out a directed graph: >>>>>>> [image: image.png] >>>>>>> That doesn't include client sites! Legend: >>>>>>> >>>>>>> >>>>>>> - Green = the root site. >>>>>>> - Red = a site >>>>>>> - Blue = an access point >>>>>>> - Magenta = a client site that has children >>>>>>> >>>>>>> So the part in "common" is designed heavily to reduce repetition. >>>>>>> When it's done, you should be able to feed in sites, APs, clients, devices, >>>>>>> etc. in a pretty flexible manner. Given how much code is shared between the >>>>>>> UISP and Splynx integration code, I'm pretty sure both will be cut to a >>>>>>> tiny fraction of the total code. :-) >>>>>>> >>>>>>> I can't post the full tree, it's full of client names. >>>>>>> _______________________________________________ >>>>>>> LibreQoS mailing list >>>>>>> LibreQoS@lists.bufferbloat.net >>>>>>> https://lists.bufferbloat.net/listinfo/libreqos >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Robert Chacón >>>>>> CEO | JackRabbit Wireless LLC >>>>>> _______________________________________________ >>>>>> LibreQoS mailing list >>>>>> LibreQoS@lists.bufferbloat.net >>>>>> https://lists.bufferbloat.net/listinfo/libreqos >>>>>> >>>>> _______________________________________________ >>> LibreQoS mailing list >>> LibreQoS@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/libreqos >>> >> >> >> -- >> Robert Chacón >> CEO | JackRabbit Wireless LLC >> Dev | LibreQoS.io >> >> _______________________________________________ > LibreQoS mailing list > LibreQoS@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/libreqos > -- Robert Chacón CEO | JackRabbit Wireless LLC Dev | LibreQoS.io