Wow. This is very nicely done. Awesome work! On Fri, Oct 28, 2022 at 11:44 AM Herbert Wolverson via LibreQoS < libreqos@lists.bufferbloat.net> wrote: > The integration is coming along nicely. Some progress updates: > > - You can specify a variable in ispConfig.py named "uispSite". This > sets where in the topology you want the tree to start. This has two > purposes: > - It's hard to be psychic and know for sure where the shaper is in > the network. > - You could run multiple shapers at different egress points, with > failover - and rebuild the entire topology from the point of view of a > network node. > - "Child node with children" are now automatically converted into a > "(Generated Site) name" site, and their children rearranged. This: > - Allows you to set the "site" bandwidth independently of the > client site bandwidth. > - Makes for easier trees, because we're inserting the site that > really should be there. > - Network.json generation (not the shaped devices file yet) is > automatically generated from a tree, once PrepareTree() and > createNetworkJson() are called. > - There's a unit test that generates the network.example.json file > and compares it with the original to ensure that they match. > - Unit test coverage hits every function in the graph system, now. > > I'm liking this setup. With the non-vendor-specific logic contained inside > the NetworkGraph type, the actual UISP code to generate the example tree is > down to 65 > lines of code, including comments. That'll grow a bit as I re-insert some > automatic speed limit determination, AP/Site speed overrides ( > i.e. the integrationUISPbandwidths.csv file). Still pretty clean. > > Creating the network.example.json file only requires: > from integrationCommon import NetworkGraph, NetworkNode, NodeType > import json > net = NetworkGraph() > net.addRawNode(NetworkNode("Site_1", "Site_1", "", NodeType.site, > 1000, 1000)) > net.addRawNode(NetworkNode("Site_2", "Site_2", "", NodeType.site, > 500, 500)) > net.addRawNode(NetworkNode("AP_A", "AP_A", "Site_1", NodeType.ap, > 500, 500)) > net.addRawNode(NetworkNode("Site_3", "Site_3", "Site_1", NodeType. > site, 500, 500)) > net.addRawNode(NetworkNode("PoP_5", "PoP_5", "Site_3", NodeType. > site, 200, 200)) > net.addRawNode(NetworkNode("AP_9", "AP_9", "PoP_5", NodeType.ap, > 120, 120)) > net.addRawNode(NetworkNode("PoP_6", "PoP_6", "PoP_5", NodeType. > site, 60, 60)) > net.addRawNode(NetworkNode("AP_11", "AP_11", "PoP_6", NodeType.ap, > 30, 30)) > net.addRawNode(NetworkNode("PoP_1", "PoP_1", "Site_2", NodeType. > site, 200, 200)) > net.addRawNode(NetworkNode("AP_7", "AP_7", "PoP_1", NodeType.ap, > 100, 100)) > net.addRawNode(NetworkNode("AP_1", "AP_1", "Site_2", NodeType.ap, > 150, 150)) > net.prepareTree() > net.createNetworkJson() > > (The id and name fields are duplicated right now, I'm using readable names > to keep me sane. The third string is the parent, and the last two numbers > are bandwidth limits) > The nice, readable format being: > NetworkNode(id="Site_1", displayName="Site_1", parentId="", type=NodeType. > site, download=1000, upload=1000) > > That in turns gives you the example network: > [image: image.png] > > > On Fri, Oct 28, 2022 at 7:40 AM Herbert Wolverson > wrote: > >> Dave: I love those Gource animations! Game development is my other hobby, >> I could easily get lost for weeks tweaking the shaders to make the glow >> "just right". :-) >> >> Dan: Discovery would be nice, but I don't think we're ready to look in >> that direction yet. I'm trying to build a "common grammar" to make it >> easier to express network layout from integrations; that would be another >> form/layer of integration and a lot easier to work with once there's a >> solid foundation. Preseem does some of this (admittedly over-eagerly; >> nothing needs to query SNMP that often!), and the SNMP route is quite >> remarkably convoluted. Their support turned on a few "extra" modules to >> deal with things like PMP450 clients that change MAC when you put them in >> bridge mode vs NAT mode (and report the bridge mode CPE in some places >> either way), Elevate CPEs that almost but not quite make sense. Robert's >> code has the beginnings of some of this, scanning Mikrotik routers for IPv6 >> allocations by MAC (this is also the hardest part for me to test, since I >> don't have any v6 to test, currently). >> >> We tend to use UISP as the "source of truth" and treat it like a database >> for a ton of external tools (mostly ones we've created). >> >> On Thu, Oct 27, 2022 at 7:27 PM dan wrote: >> >>> we're pretty similar in that we've made UISP a mess. Multiple paths to >>> a pop. multiple pops on the network. failover between pops. Lots of >>> 'other' devices. handing out /29 etc to customers. >>> >>> Some sort of discovery would be nice. Ideally though, pulling something >>> from SNMP or router APIs etc to build the paths, but having a 'network >>> elements' list with each of the links described. ie, backhaul 12 has MACs >>> ..01 and ...02 at 300x100 and then build the topology around that from >>> discovery. >>> >>> I've also thought about doing routine trace routes or watching TTLs or >>> something like that to get some indication that topology has changed and >>> then do another discovery and potential tree rebuild. >>> >>> On Thu, Oct 27, 2022 at 3:48 PM Robert Chacón via LibreQoS < >>> libreqos@lists.bufferbloat.net> wrote: >>> >>>> This is awesome! Way to go here. Thank you for contributing this. >>>> Being able to map out these complex integrations will help ISPs a ton, >>>> and I really like that it is sharing common features between the Splynx and >>>> UISP integrations. >>>> >>>> Thanks, >>>> Robert >>>> >>>> On Thu, Oct 27, 2022 at 3:33 PM Herbert Wolverson via LibreQoS < >>>> libreqos@lists.bufferbloat.net> wrote: >>>> >>>>> So I've been doing some work on getting UISP integration (and >>>>> integrations in general) to work a bit more smoothly. >>>>> >>>>> I started by implementing a graph structure that mirrors both the >>>>> networks and sites system. It's not done yet, but the basics are coming >>>>> together nicely. You can see my progress so far at: >>>>> https://github.com/thebracket/LibreQoS/tree/integration-common-graph >>>>> >>>>> Our UISP instance is a *great* testcase for torturing the system. I >>>>> even found a case of UISP somehow auto-generating a circular portion of the >>>>> tree. We have: >>>>> >>>>> - Non Ubiquiti devices as "other devices" >>>>> - Sections that need shaping by subnet (e.g. "all of 192.168.1.0/24 >>>>> shared 100 mbit") >>>>> - Bridge mode devices using Option 82 to always allocate the same >>>>> IP, with a "service IP" entry >>>>> - Various bits of infrastructure mapped >>>>> - Sites that go to client sites, which go to other client sites >>>>> >>>>> In other words, over the years we've unleashed a bit of a monster. >>>>> Cleaning it up is a useful talk, but I wanted the integration to be able to >>>>> handle pathological cases like us! >>>>> >>>>> So I fed our network into the current graph generator, and used >>>>> graphviz to spit out a directed graph: >>>>> [image: image.png] >>>>> That doesn't include client sites! Legend: >>>>> >>>>> >>>>> - Green = the root site. >>>>> - Red = a site >>>>> - Blue = an access point >>>>> - Magenta = a client site that has children >>>>> >>>>> So the part in "common" is designed heavily to reduce repetition. When >>>>> it's done, you should be able to feed in sites, APs, clients, devices, etc. >>>>> in a pretty flexible manner. Given how much code is shared between the UISP >>>>> and Splynx integration code, I'm pretty sure both will be cut to a tiny >>>>> fraction of the total code. :-) >>>>> >>>>> I can't post the full tree, it's full of client names. >>>>> _______________________________________________ >>>>> LibreQoS mailing list >>>>> LibreQoS@lists.bufferbloat.net >>>>> https://lists.bufferbloat.net/listinfo/libreqos >>>>> >>>> >>>> >>>> -- >>>> Robert Chacón >>>> CEO | JackRabbit Wireless LLC >>>> _______________________________________________ >>>> LibreQoS mailing list >>>> LibreQoS@lists.bufferbloat.net >>>> https://lists.bufferbloat.net/listinfo/libreqos >>>> >>> _______________________________________________ > LibreQoS mailing list > LibreQoS@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/libreqos > -- Robert Chacón CEO | JackRabbit Wireless LLC Dev | LibreQoS.io