<div dir="ltr"><div>>
You talking about the relevant rfc? <br></div><div><br></div><div>In this case, the "6 to 4" refers to some integration code that was already present - named "mikrotikFindIpv6.py". I probably should've made that more clear. It connects to Mikrotik routers, and performs a MAC address search in their DHCPv6 tables - finding known MAC addresses and providing the allocated IPv6 address-space. Looks like a handy tool, and a good work-around for UISP (Ubiquiti's combined management and CRM tool) only kind-of supporting IPv6. The database format supports v6 addresses, but it doesn't consistently put any data in there; worse, it doesn't show it on-screen when it has it!</div><div><br></div><div>>
Seems to be a need for some level of exclusions for device type, e.g.
(at least per your report), don't run ack-filter on a cambium path. <br></div><div><br></div><div>I agree with that longer-term. For now, I'm trying to get the existing integrations up-to-speed and easy to work with. The whole "build on a good foundation" thing. That's one thing I've learned the hard way over the decades; it's a *lot* easier to shoot for the moon if you take the time to come up with a good launch platform!</div><div><br></div><div>Longer-term, it's looking more and more like we'll need a more robust discovery system. I've some ideas, but they are way too formative to be useful yet. Some early thinking: there's a big disparity between what the various back-ends WISPs (and ISPs in general) are using to manage and monitor their networks, and the systems that handle CRM (billing, ticketing, customer interaction, etc.). Spylnx and its ilk are great billing systems, but don't really know a lot about your network arrangement - it wouldn't surprise me if there are Spylnx and VISP users who also have UISP (just the network management mode) going as well. On the other extreme, PowerCode tries to write directly to your Mikrotik routers and wants to know everything right down to your underwear colour.</div><div><br></div><div>In my mind:<br></div><div>* Step 1 (we're nearly there!) is to build a good foundation for representing an IPv4/IPv6 network, that's really agnostic to all the crazy things a WISP may be doing. It should automate all the tedious parts (figuring out a tree from a soup of sites, access points, users - rearranging the tree to have a "starting point", emitting the various control files, etc.), be easy enough to use that someone could say "wow, I need to support my management system" and be able to do so with a little bit of hand-holding - encouraging participation.</div><div>* Step 2 would be to provide some great manual tools for the DIY crowd, and some really good documentation to make their life easy.</div><div>* Step 3 is some kind of way to mix-and-match systems. Say you have Splynx AND the management part of UISP. Wouldn't it be great if Spylnx could provide all of the "plan" data, and the data be provided from UISP's management"? It seems like that's quite do-able with a little work. We may need to think about a management GUI at this point, just to help hold hands a bit.</div><div>* Step 4 would be something Dan keeps asking about, ways to query hardware that exists and build some topology around it. That would be great, and is quite the undertaking (best tackled incrementally, and in a modular fashion, IMHO).</div><div><br></div><div>This is still just the musings of a sleep-deprived brain. :-)</div><div><br></div><div>>
Is there any particularly common set of radius servers in use? <br></div><div><br></div><div>It seems like when I poke deeply enough, most people are running FreeRADIUS or something vendor-supplied (which is sometimes FreeRADIUS with a badge on it). Then there's crazy people paying $10k for super high-end RADIUS servers that aren't actually much better than the free ones. RADIUS is a tough one, because LibreQoS isn't really well placed to directly utilize it. Typically, RADIUS is basically a "yes or no" box, with options attached. RADIUS queries happen on network entry (either as part of the admissions process, part of the Ethernet security step, or from the DHCP server) and the reply is basically "yes, you're admitted - these are your options". The problem is, Libre doesn't necessarily see any of that - it's inside the network. That's why we have API dependencies, even though Spylnx and VISP are basically a really big billing system that comes bundled with a RADIUS server. (Unfortunately, Mikrotik interprets the RADIUS replies to make a simple queue on the router that made the request - you can script that, but it gets messy fast).</div><div><br></div><div>I'll answer the second email in a bit.<br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Oct 29, 2022 at 2:18 PM Dave Taht <<a href="mailto:dave.taht@gmail.com">dave.taht@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Oct 29, 2022 at 8:57 AM Herbert Wolverson via LibreQoS <<a href="mailto:libreqos@lists.bufferbloat.net" target="_blank">libreqos@lists.bufferbloat.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Alright, the UISP side of the common integrations is pretty much feature complete. I'll update the tracking issue in a bit.</div><div><ul><li>Per your suggestion, devices with no IP addresses (v4 or v6) are not added.</li></ul></div></div></blockquote><div>Every device that is ipv6-ready comes up with a link-local address derived from the mac like fe80::6f16:fa94:f32b:e2e</div><div>Some actually will accept things like ssh to that address</div><div>Not that this is necessarily relevant to this bit of code. Dr irrelevant I am today.<br></div><div>(in the context of babel, at least, you can route ipv4 and ipv6 without either an ipv6 or ipv4 address, and hnetd configure)<br></div><div><br></div><div>I am kind of curious as to what weird configuration protocols are in common use today</div><div><br></div><div>Painfully common are "smart switches" that don't listen to dhcp by default AND come up on 192.168.1.1<br></div><div>ubnt comes up on 192.168.1.20 by defualt</div><div>a lot of cpe comes up on 192.168.1.100 (like cable and starlink)</div><div>I've seen stuff that uses ancient ieee protocols</div><div>bootp and tftp are still things</div><div><br></div><div>I've always kind of wanted a daemon on every device that would probe all possible ip addresses with a ttl of 2, to find rogue</div><div>devices etc. <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><ul><li>Mikrotik "4 to 6" mapping is implemented. I put it in the "common" side of things, so it can be used in other integrations also. I don't have a setup on which to test it, but if I'm reading the code right then the unit test is testing it appropriately.</li></ul></div></div></blockquote><div><br></div><div>You talking about the relevant rfc?<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><ul><li>excludeSites is supported as a common API feature. If a node is added with a name that matches an excluded site, it won't be added. The tree builder is smart enough to replace invalid "parentId" references with the shaper root, so if you have other tree items that rely on this site - they will be added to the tree. Was that the intent? (It looks pretty useful; we have a child site down the tree with a HUGE amount of load, and bumping it to the top-level with excludeSites would probably help our load balancing quite a bit)</li><ul><li>If the intent was to exclude the site and everything underneath it, I'd have to rework things a bit. Let me know; it wasn't quite clear.<br></li></ul><li>exceptionCPEs is also supported as a common API feature. It simply overrides the "parentId'' of incoming nodes with the new parent. Another potentially useful feature; if I got excludeSites the wrong away around, I'd add a "my_big_site":"" entry to push it to the top.</li></ul></div></div></blockquote><div><br></div><div>Seems to be a need for some level of exclusions for device type, e.g. (at least per your report), don't run ack-filter on a cambium path.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><ul><li>UISP integration now supports a "flat" topology option (set via uispStrategy = "flat" in ispConfig). I expanded <a href="http://ispConfig.example.py" target="_blank">ispConfig.example.py</a> to include this entry.<br></li></ul><div>I'll look and see how much of the Spylnx code I can shorten with the new API; I don't have a Spylnx setup to test against, making that tricky. I <i>think</i> the new API should shorten things a lot. I think routers act as node parents, with clients underneath them? Otherwise, a "flat" setup should be a little shorter (the CSV code can be replaced with a call to the graph builder). Most of the Spylnx (and VISP) users I've talked to layer MPLS+VPLS to pretend to have a big, flat network and then connect via a RADIUS call in the DHCP server; </div></div></div></blockquote><div><br></div><div>Is there any particularly common set of radius servers in use?<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>I've always assumed that's because those systems prefer the telecom model of "pretend everything is equal" to trying to model topology.*<br></div></div></div></blockquote><div><br></div><div>Except the billing. Always the billing. Our tuesday golden plate special is you can download all the pr0n from our special partner netblix for 24 hours a week! 9.95!</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div></div><div><br></div><div>I need to clean things up a bit (there's still a bit of duplicated code, and I believe in the DRY principle - don't repeat yourself; Dave Thomas - my boss at PragProg - coined the term in The Pragmatic Programmer, and I feel obliged to use it everywhere!), and do a quick rebase (I accidentally parented the branch off of a branch instead of main) - but I think I can have this as a PR for you on Monday.</div><div><br></div><div>* - The first big wireless network I setup used a Motorola WiMAX setup. They <i>required</i> that every single AP share two VLANs (management and bearer) with every other AP - all the way to the core. It kinda worked once they remembered client isolation was a thing in a patch... Then again, their installation instructions included connecting two ports of a router together with a jumper cable, because their localhost implementation didn't quite work. :-|<br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Oct 28, 2022 at 4:15 PM Robert Chacón <<a href="mailto:robert.chacon@jackrabbitwireless.com" target="_blank">robert.chacon@jackrabbitwireless.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Awesome work. It succeeded in building the topology and creating ShapedDevices.csv for my network. It even graphed it perfectly. Nice!<br></div><div>I notice that in ShapedDevices.csv it does add CPE radios (which in our case we don't shape - they are in bridge mode) with IPv4 and IPv6s both being empty lists [].</div><div>This is not necessarily bad, but it may lead to empty leaf classes being created on LibreQoS.py runs. Not a huge deal, it just makes the minor class counter increment toward the 32k limit faster.<br></div><div>Do you think perhaps we should check:</div><div><b>if (len(IPv4) == 0) and (len(IPv6) == 0):</b></div><div><b> # Skip adding this entry to ShapedDevices.csv</b></div><div>Or something similar around line 329 of integrationCommon.py?</div><div>Open to your suggestions there.</div><div><br></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Oct 28, 2022 at 1:55 PM Herbert Wolverson via LibreQoS <<a href="mailto:libreqos@lists.bufferbloat.net" target="_blank">libreqos@lists.bufferbloat.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>One more update, and I'm going to sleep until "pick up daughter" time. :-)</div><div><br></div><div>The tree at <a href="https://github.com/thebracket/LibreQoS/tree/integration-common-graph" target="_blank">https://github.com/thebracket/LibreQoS/tree/integration-common-graph</a> can now build a network.json, ShapedDevices.csv, and integrationUISPBandwidth.csv and follows pretty much the same logic as the previous importer - other than using data links to build the hierarchy and letting (requiring, currently) you specify the root node. It's handling our bizarre UISP setup pretty well now - so if anyone wants to test it (I recommend just running integrationUISP.py and checking the output rather than throwing it into production), I'd appreciate any feedback.</div><div><br></div><div>Still on my list: handling the Mikrotik IPv6 connections, and exceptionCPE and site exclusion.</div><div><br></div><div>If you want the pretty graphics, you need to "pip install graphviz" and "sudo apt install graphviz". It *should* detect that these aren't present and not try to draw pictures, otherwise.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Oct 28, 2022 at 2:06 PM Robert Chacón <<a href="mailto:robert.chacon@jackrabbitwireless.com" target="_blank">robert.chacon@jackrabbitwireless.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Wow. This is very nicely done. Awesome work!</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Oct 28, 2022 at 11:44 AM Herbert Wolverson via LibreQoS <<a href="mailto:libreqos@lists.bufferbloat.net" target="_blank">libreqos@lists.bufferbloat.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>The integration is coming along nicely. Some progress updates:</div><div><ul><li>You can specify a variable in ispConfig.py named "uispSite". This sets where in the topology you want the tree to start. This has two purposes:</li><ul><li>It's hard to be psychic and know for sure where the shaper is in the network.</li><li>You could run multiple shapers at different egress points, with failover - and rebuild the entire topology from the point of view of a network node.</li></ul><li>"Child node with children" are now automatically converted into a "(Generated Site) name" site, and their children rearranged. This:</li><ul><li>Allows you to set the "site" bandwidth independently of the client site bandwidth.</li><li>Makes for easier trees, because we're inserting the site that really should be there.</li></ul><li>Network.json generation (not the shaped devices file yet) is automatically generated from a tree, once PrepareTree() and createNetworkJson() are called.</li><ul><li>There's a unit test that generates the network.example.json file and compares it with the original to ensure that they match.</li></ul><li>Unit test coverage hits every function in the graph system, now.</li></ul><div>I'm liking this setup. With the non-vendor-specific logic contained inside the NetworkGraph type, the actual UISP code to generate the example tree is down to 65</div><div>lines of code, including comments. That'll grow a bit as I re-insert some automatic speed limit determination, AP/Site speed overrides (</div><div>i.e. the integrationUISPbandwidths.csv file). Still pretty clean.</div><div><br></div><div>Creating the network.example.json file only requires:</div><div>
<div style="color:rgb(212,212,212);background-color:rgb(30,30,30);font-family:Consolas,"Courier New",monospace;font-weight:normal;font-size:14px;line-height:19px;white-space:pre-wrap"><div><span style="color:rgb(197,134,192)">from</span><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(78,201,176)">integrationCommon</span><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(197,134,192)">import</span><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(78,201,176)">NetworkGraph</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(78,201,176)">NetworkNode</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(78,201,176)">NodeType</span></div><div><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(197,134,192)">import</span><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(78,201,176)">json</span></div><div><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(156,220,254)">net</span><span style="color:rgb(212,212,212)"> = </span><span style="color:rgb(78,201,176)">NetworkGraph</span><span style="color:rgb(212,212,212)">()</span></div><div><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(156,220,254)">net</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(220,220,170)">addRawNode</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(78,201,176)">NetworkNode</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(206,145,120)">"Site_1"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(206,145,120)">"Site_1"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(206,145,120)">""</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(78,201,176)">NodeType</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(79,193,255)">site</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(181,206,168)">1000</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(181,206,168)">1000</span><span style="color:rgb(212,212,212)">))</span></div><div><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(156,220,254)">net</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(220,220,170)">addRawNode</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(78,201,176)">NetworkNode</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(206,145,120)">"Site_2"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(206,145,120)">"Site_2"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(206,145,120)">""</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(78,201,176)">NodeType</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(79,193,255)">site</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(181,206,168)">500</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(181,206,168)">500</span><span style="color:rgb(212,212,212)">))</span></div><div><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(156,220,254)">net</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(220,220,170)">addRawNode</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(78,201,176)">NetworkNode</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(206,145,120)">"AP_A"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(206,145,120)">"AP_A"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(206,145,120)">"Site_1"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(78,201,176)">NodeType</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(79,193,255)">ap</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(181,206,168)">500</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(181,206,168)">500</span><span style="color:rgb(212,212,212)">))</span></div><div><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(156,220,254)">net</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(220,220,170)">addRawNode</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(78,201,176)">NetworkNode</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(206,145,120)">"Site_3"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(206,145,120)">"Site_3"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(206,145,120)">"Site_1"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(78,201,176)">NodeType</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(79,193,255)">site</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(181,206,168)">500</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(181,206,168)">500</span><span style="color:rgb(212,212,212)">))</span></div><div><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(156,220,254)">net</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(220,220,170)">addRawNode</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(78,201,176)">NetworkNode</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(206,145,120)">"PoP_5"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(206,145,120)">"PoP_5"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(206,145,120)">"Site_3"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(78,201,176)">NodeType</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(79,193,255)">site</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(181,206,168)">200</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(181,206,168)">200</span><span style="color:rgb(212,212,212)">)) </span></div><div><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(156,220,254)">net</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(220,220,170)">addRawNode</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(78,201,176)">NetworkNode</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(206,145,120)">"AP_9"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(206,145,120)">"AP_9"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(206,145,120)">"PoP_5"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(78,201,176)">NodeType</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(79,193,255)">ap</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(181,206,168)">120</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(181,206,168)">120</span><span style="color:rgb(212,212,212)">))</span></div><div><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(156,220,254)">net</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(220,220,170)">addRawNode</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(78,201,176)">NetworkNode</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(206,145,120)">"PoP_6"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(206,145,120)">"PoP_6"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(206,145,120)">"PoP_5"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(78,201,176)">NodeType</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(79,193,255)">site</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(181,206,168)">60</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(181,206,168)">60</span><span style="color:rgb(212,212,212)">))</span></div><div><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(156,220,254)">net</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(220,220,170)">addRawNode</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(78,201,176)">NetworkNode</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(206,145,120)">"AP_11"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(206,145,120)">"AP_11"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(206,145,120)">"PoP_6"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(78,201,176)">NodeType</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(79,193,255)">ap</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(181,206,168)">30</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(181,206,168)">30</span><span style="color:rgb(212,212,212)">))</span></div><div><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(156,220,254)">net</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(220,220,170)">addRawNode</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(78,201,176)">NetworkNode</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(206,145,120)">"PoP_1"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(206,145,120)">"PoP_1"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(206,145,120)">"Site_2"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(78,201,176)">NodeType</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(79,193,255)">site</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(181,206,168)">200</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(181,206,168)">200</span><span style="color:rgb(212,212,212)">))</span></div><div><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(156,220,254)">net</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(220,220,170)">addRawNode</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(78,201,176)">NetworkNode</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(206,145,120)">"AP_7"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(206,145,120)">"AP_7"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(206,145,120)">"PoP_1"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(78,201,176)">NodeType</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(79,193,255)">ap</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(181,206,168)">100</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(181,206,168)">100</span><span style="color:rgb(212,212,212)">))</span></div><div><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(156,220,254)">net</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(220,220,170)">addRawNode</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(78,201,176)">NetworkNode</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(206,145,120)">"AP_1"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(206,145,120)">"AP_1"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(206,145,120)">"Site_2"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(78,201,176)">NodeType</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(79,193,255)">ap</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(181,206,168)">150</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(181,206,168)">150</span><span style="color:rgb(212,212,212)">))</span></div><div><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(156,220,254)">net</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(220,220,170)">prepareTree</span><span style="color:rgb(212,212,212)">()</span></div><div><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(156,220,254)">net</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(220,220,170)">createNetworkJson</span><span style="color:rgb(212,212,212)">()</span></div></div>
</div><div><br></div><div>(The id and name fields are duplicated right now, I'm using readable names to keep me sane. The third string is the parent, and the last two numbers are bandwidth limits)</div><div>The nice, readable format being:</div><div>
<div style="color:rgb(212,212,212);background-color:rgb(30,30,30);font-family:Consolas,"Courier New",monospace;font-weight:normal;font-size:14px;line-height:19px;white-space:pre-wrap"><div><span style="color:rgb(78,201,176)">NetworkNode</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(156,220,254)">id</span><span style="color:rgb(212,212,212)">=</span><span style="color:rgb(206,145,120)">"Site_1"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(156,220,254)">displayName</span><span style="color:rgb(212,212,212)">=</span><span style="color:rgb(206,145,120)">"Site_1"</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(156,220,254)">parentId</span><span style="color:rgb(212,212,212)">=</span><span style="color:rgb(206,145,120)">""</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(156,220,254)">type</span><span style="color:rgb(212,212,212)">=</span><span style="color:rgb(78,201,176)">NodeType</span><span style="color:rgb(212,212,212)">.</span><span style="color:rgb(79,193,255)">site</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(156,220,254)">download</span><span style="color:rgb(212,212,212)">=</span><span style="color:rgb(181,206,168)">1000</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(156,220,254)">upload</span><span style="color:rgb(212,212,212)">=</span><span style="color:rgb(181,206,168)">1000</span><span style="color:rgb(212,212,212)">)</span></div></div>
</div><div><br></div><div>That in turns gives you the example network:</div><div><img src="cid:ii_l9ss9eiu1" alt="image.png" width="578" height="459"><br><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Oct 28, 2022 at 7:40 AM Herbert Wolverson <<a href="mailto:herberticus@gmail.com" target="_blank">herberticus@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Dave: I love those Gource animations! Game development is my other hobby, I could easily get lost for weeks tweaking the shaders to make the glow "just right". :-)</div><div><br></div><div>Dan: Discovery would be nice, but I don't think we're ready to look in that direction yet. I'm trying to build a "common grammar" to make it easier to express network layout from integrations; that would be another form/layer of integration and a lot easier to work with once there's a solid foundation. Preseem does some of this (admittedly over-eagerly; nothing needs to query SNMP that often!), and the SNMP route is quite remarkably convoluted. Their support turned on a few "extra" modules to deal with things like PMP450 clients that change MAC when you put them in bridge mode vs NAT mode (and report the bridge mode CPE in some places either way), Elevate CPEs that almost but not quite make sense. Robert's code has the beginnings of some of this, scanning Mikrotik routers for IPv6 allocations by MAC (this is also the hardest part for me to test, since I don't have any v6 to test, currently).</div><div><br></div><div>We tend to use UISP as the "source of truth" and treat it like a database for a ton of external tools (mostly ones we've created).<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Oct 27, 2022 at 7:27 PM dan <<a href="mailto:dandenson@gmail.com" target="_blank">dandenson@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">we're pretty similar in that we've made UISP a mess. Multiple paths to a pop. multiple pops on the network. failover between pops. Lots of 'other' devices. handing out /29 etc to customers.<br><br>Some sort of discovery would be nice. Ideally though, pulling something from SNMP or router APIs etc to build the paths, but having a 'network elements' list with each of the links described. ie, backhaul 12 has MACs ..01 and ...02 at 300x100 and then build the topology around that from discovery. <br><br>I've also thought about doing routine trace routes or watching TTLs or something like that to get some indication that topology has changed and then do another discovery and potential tree rebuild. </div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Oct 27, 2022 at 3:48 PM Robert Chacón via LibreQoS <<a href="mailto:libreqos@lists.bufferbloat.net" target="_blank">libreqos@lists.bufferbloat.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>This is awesome! Way to go here. Thank you for contributing this.</div><div>Being able to map out these complex integrations will help ISPs a ton, and I really like that it is sharing common features between the Splynx and UISP integrations.</div><div><br></div><div>Thanks,</div><div>Robert<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Oct 27, 2022 at 3:33 PM Herbert Wolverson via LibreQoS <<a href="mailto:libreqos@lists.bufferbloat.net" target="_blank">libreqos@lists.bufferbloat.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>So I've been doing some work on getting UISP integration (and integrations in general) to work a bit more smoothly.</div><div><br></div><div>I started by implementing a graph structure that mirrors both the networks and sites system. It's not done yet, but the basics are coming together nicely. You can see my progress so far at: <a href="https://github.com/thebracket/LibreQoS/tree/integration-common-graph" target="_blank">https://github.com/thebracket/LibreQoS/tree/integration-common-graph</a></div><div><br></div><div>Our UISP instance is a <i>great</i> testcase for torturing the system. I even found a case of UISP somehow auto-generating a circular portion of the tree. We have:</div><div><ul><li>Non Ubiquiti devices as "other devices"</li><li>Sections that need shaping by subnet (e.g. "all of <a href="http://192.168.1.0/24" target="_blank">192.168.1.0/24</a> shared 100 mbit")</li><li>Bridge mode devices using Option 82 to always allocate the same IP, with a "service IP" entry</li><li>Various bits of infrastructure mapped</li><li>Sites that go to client sites, which go to other client sites</li></ul><div>In other words, over the years we've unleashed a bit of a monster. Cleaning it up is a useful talk, but I wanted the integration to be able to handle pathological cases like us!</div><div><br></div><div>So I fed our network into the current graph generator, and used graphviz to spit out a directed graph:</div><div><img src="cid:ii_l9rkwmjp0" alt="image.png" width="558" height="426"><br></div><div>That doesn't include client sites! Legend:</div><div><br></div><div><ul><li>Green = the root site.</li><li>Red = a site</li><li>Blue = an access point</li><li>Magenta = a client site that has children</li></ul><div>So the part in "common" is designed heavily to reduce repetition. When it's done, you should be able to feed in sites, APs, clients, devices, etc. in a pretty flexible manner. Given how much code is shared between the UISP and Splynx integration code, I'm pretty sure both will be cut to a tiny fraction of the total code. :-)</div><div><br></div><div>I can't post the full tree, it's full of client names.<br></div></div></div></div>
_______________________________________________<br>
LibreQoS mailing list<br>
<a href="mailto:LibreQoS@lists.bufferbloat.net" target="_blank">LibreQoS@lists.bufferbloat.net</a><br>
<a href="https://lists.bufferbloat.net/listinfo/libreqos" rel="noreferrer" target="_blank">https://lists.bufferbloat.net/listinfo/libreqos</a><br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr">Robert Chacón<br>CEO | <a href="http://jackrabbitwireless.com" target="_blank">JackRabbit Wireless LLC</a><br></div></div>
_______________________________________________<br>
LibreQoS mailing list<br>
<a href="mailto:LibreQoS@lists.bufferbloat.net" target="_blank">LibreQoS@lists.bufferbloat.net</a><br>
<a href="https://lists.bufferbloat.net/listinfo/libreqos" rel="noreferrer" target="_blank">https://lists.bufferbloat.net/listinfo/libreqos</a><br>
</blockquote></div>
</blockquote></div>
</blockquote></div>
_______________________________________________<br>
LibreQoS mailing list<br>
<a href="mailto:LibreQoS@lists.bufferbloat.net" target="_blank">LibreQoS@lists.bufferbloat.net</a><br>
<a href="https://lists.bufferbloat.net/listinfo/libreqos" rel="noreferrer" target="_blank">https://lists.bufferbloat.net/listinfo/libreqos</a><br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr">Robert Chacón<br><div>CEO | <a href="http://jackrabbitwireless.com" target="_blank">JackRabbit Wireless LLC</a></div><div>Dev | <a href="http://LibreQoS.io" target="_blank">LibreQoS.io</a><br></div><br></div></div>
</blockquote></div>
_______________________________________________<br>
LibreQoS mailing list<br>
<a href="mailto:LibreQoS@lists.bufferbloat.net" target="_blank">LibreQoS@lists.bufferbloat.net</a><br>
<a href="https://lists.bufferbloat.net/listinfo/libreqos" rel="noreferrer" target="_blank">https://lists.bufferbloat.net/listinfo/libreqos</a><br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr">Robert Chacón<br><div>CEO | <a href="http://jackrabbitwireless.com" target="_blank">JackRabbit Wireless LLC</a></div><div>Dev | <a href="http://LibreQoS.io" target="_blank">LibreQoS.io</a><br></div><br></div></div>
</blockquote></div>
_______________________________________________<br>
LibreQoS mailing list<br>
<a href="mailto:LibreQoS@lists.bufferbloat.net" target="_blank">LibreQoS@lists.bufferbloat.net</a><br>
<a href="https://lists.bufferbloat.net/listinfo/libreqos" rel="noreferrer" target="_blank">https://lists.bufferbloat.net/listinfo/libreqos</a><br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr"><div>This song goes out to all the folk that thought Stadia would work:</div><div><a href="https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz" target="_blank">https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz</a></div><div>Dave Täht CEO, TekLibre, LLC <br></div></div></div></div>
</blockquote></div>