[LibreQoS] routing protocols and daemons

Thu Oct 27 09:29:22 EDT 2022

OSPF is a pretty stable protocol, so it doesn't change all that much (one
of the reasons I like it is that it is supported everywhere, and works
cross-vendor consistently). In answer to your questions:

* With OSPFv3 you can push IPv6 routes. I've only used this a bit, but it
worked fine last time we tried it.
* You have a fair amount of flexibility on how hops talk to one another
(configured on the interface). The default is still multicast, but
"point-to-point" is just a unicast broadcast (and VERY fast route
coalescing; it's our default), "point-to-multipoint" can do the same for
1:many relationships (I don't use this one, I prefer one interface to one
destination - even if the interfaces are VLANs. Makes traffic analysis a
whole lot easier). There's also NBMA, which has you type in the addresses
of adjacent neighbors (which feels like it defeats some of the point!) -
but doesn't even broadcast. There was a time that NBMA was the most
consistent over old Ubiquiti equipment, but these days the other modes work
fine.
* ECMP (Equal Cost Multipath) is awesome. If two routes to the destination
sum out at equal costs, traffic is split equally between them. Traffic is
split at layer 3 (on an address-port hash), so flows stay together. It's
rock solid, we have it sharing traffic over two backhauls in a few places -
and it handles one interface going down flawlessly (with either BFD, or
point-to-point mode which coalesces so fast).

Dijkstra's algorithm remains a very natural approach to mapping a graph
(I've waxed lyrical about it in various book/article/blog posts over the
years in the game-dev world), so I find it a very comfy way to model the
underlying network. Very easy to reason "this will go that way".

So what's bad about it?

* ECMP is equal; if you have routes with different costs it will only use
the lowest cost - it won't try and "mesh" some of the traffic in other
directions. Likewise, if you have two equal routes - and one of them is
running at a low capacity - you wind up only utilizing twice the capacity
of the slowest link. You're often better off dropping the link over the
degraded circuit (hence "carrier drop" features on various radios).
* OSPF has no idea what the capacities of various links are. It'll use the
shortest cost route, and leave the details up to the lower layers of the
stack.

There have been various attempts to integrate capacity into network design,
and I've yet to see one that holds up well on a multi-vendor network. If
you ask a Ubiquiti Rocket AC Gen2 its capacity, it'll often give you some
nice big number - but it'll stall out far before that number. The Force
400C link we just put up doesn't even try to estimate its capacity. So
multi-vendor mesh routing tends to be problematic, because the advertised
capability of "we'll utilize everywhere you have capacity" tends to get
snarled up in figuring out what the capacity is. I understand Teragraph is
supposed to do better there. MPLS traffic engineering was originally
announced as solving this one, too! I don't see a lot of  hope for
capacity-aware routing protocols taking off, but I imagine we'll get a few
new ones announced and then quietly forgotten as before.

And lastly: once your network gets really big, OSPF tables can get too
large - and you're stuck either dividing your OSPF zones and/or using some
BGP in interior mode. You can mitigate this with some careful design.

On Wed, Oct 26, 2022 at 5:35 PM dan <dandenson at gmail.com> wrote:

> I have played with batman-adv quite a bit and there are some concepts in
> it I really like.  Not being shortest path for one, and rating a link
> quality instead of hard up/down.   I also like the layer2 model so it looks
> like a big switch.  It's very clean from an operational perspective as it
> behaves essentially like an MPLS/VPLS network administratively.
>
> What I think we're missing is the integration of network attributes and
> class of service.  For instance, user to 'internet' has 3 potential paths
> with each having these end-to-end latency, upload throughput, download
> throughput, and say 'quality' or packet loss.  Then having your QoS engine
> able to tag packets for how it perceives them to need routed and then have
> the routing engine pick routes based on availability.  So you might have a
> longer path that will suffer some on latency because of the hops and link
> type but has big bandwidth 'available' (ie large capacity and underused) so
> it should ask for that flow to take the underused high capacity (and yet
> still meets other criteria) path.  Something considered realtime might
> prefer that 700Mbps licenced path that has lower and more predictable
> latency and enough available capacity for the job. By encouraging high
> throughput needs to take paths with a lot of availability and some
> mechanism to prevent occilations from reroutes you could keep lower latency
> links less busy and get load balancing by a more intelligent choice.    You
> could also have some sort of reservation number tagged onto that packet to
> ask the intermediate hops to reduce their available amount.  If you were
> going to go all out on this and have devices that spoke this everywhere,
> you could put your shapers everywhere as well, getting that desired egrees
> shapping on both sides and letting the network sort of reserve a bit of
> bandwidth for each customer based on that.
>
> Of course this means scaling issues almost inherently because those
> 'available capacity' numbers and packet loss need to be communicated.
> computationally intensive.
>
> batman-adv does this in a way with it's OGM/ELP system.  You can take a
> longer path through a batman-adv network because of a saturated link and it
> doesn't consider that saturated link 'down'.
>
> rflo was an interesting tech that did some of these once upon a time.
>
> Just thoughts.
>
> On Wed, Oct 26, 2022 at 3:38 PM Dave Taht via LibreQoS <
> libreqos at lists.bufferbloat.net> wrote:
>
>> On Wed, Oct 26, 2022 at 1:53 PM Herbert Wolverson via LibreQoS
>> <libreqos at lists.bufferbloat.net> wrote:
>> >
>> > My name is Herbert, and I'm an OSPF addict... seriously, I love OSPF.
>> Right down to stub sites, not-so-stubby sites, and isolating IP blocks
>> within a site into "stub" nets and ensuring they are aggregated properly. I
>> should probably go outside more...
>>
>> haha.
>>
>> My name is dave, and I think all routing protocols should have evolved
>> much better to elegantly meet the real world problems they were trying
>> to solve, than they have.
>>
>> To avoid burying the lede, to what extent does OSPF still rely on
>> multicast? How well can it carry ipv6 now? What extensions are common
>> in the real WISP world?
>>
>> BGP needs a few more napkins.
>>
>> RIP was a VERY good start but we drew the wrong lessons from its
>> failures, and the super-duper-trendline towards centralized
>> controllers inherent in OSPF and ISIS that happened in the 90s that
>> doesn't scale anywhere near as I'd like.
>>
>> I liked the rise of meshy 802.11 networks, I know the author of AODV
>> well (charlie perkins is arguably one of the fathers of mesh
>> networking, far too few have read his books from the 90s). And I've
>> been involved in the "battlemesh" group for many years with those
>> trying to make 'em work better on networks such as guifi,
>> wlan-slovinia, etc.
>>
>> Backstory. Back in 07, in Nicaragua, I was (stupidly) trying to get
>> ipv6 to work over nanostation m2s or m5s I forget which, and the basic
>> option was to run two copies of the ospf daemon to manage 4 and 6
>> independently. I only had 32MB of memory and it didn't fit, so I
>> started looking for alternatives, found babel, corresponded with (and
>> frankly thoroughly annoyed) the author, and starting giving it a go.
>> It transported 4 and 6 in the same packets, was tiny, was
>> distance-vector (thus, I thought, more a match for bgp), and (to me)
>> most importantly, solved the ipv4 and ipv6 routing problems in the
>> same daemon at the same time, and actually fit into less memory than
>> ospf did. It was good enough it seemed, to deploy to a few hundred
>> routers without having to play major tricks with areas and stubs and
>> so on.
>>
>> Babel is so simple that toke wrote a near complete implementation from
>> the spec, in python, during a string of extremely boring IETF
>> meetings, over the course of a week. He later took on the bird port.
>> Over the years we've wedged most (but not all) of the key features I
>> thought a meshy wireless routing protocol should have, with
>> implementations in a standalone daemon, bird, and FRR. (there was a
>> quagga port at one point too. I forget what happened to toke's python
>> version).
>>
>> https://www.rfc-editor.org/rfc/rfc8966.html babel
>> https://arxiv.org/abs/1403.0445 source specific routing
>> https://datatracker.ietf.org/doc/rfc8967/ HMAC authentication
>> https://datatracker.ietf.org/doc/html/draft-ietf-babel-rtt-extension-00
>> RTT metric
>>
>> https://datatracker.ietf.org/meeting/99/materials/slides-99-babel-unicast-hellos-00.pdf
>>  unicast hellos
>>
>> Missing is BFD support, and the slightest bit of traction outside of
>> the shrinking battlemesh communities.
>>
>> Althea is using babel and fq_codel in their blockchain routing thing
>> (I reserve comment), and I don't know where else, besides as part of
>> wireguard tunnels, babel is being used today. But I'm rather
>> interested in how OSPF evolved since I last touched it, and what use
>> cases it is good at and fails at?
>>
>>
>> > On Wed, Oct 26, 2022 at 3:29 PM Dave Taht via LibreQoS <
>> libreqos at lists.bufferbloat.net> wrote:
>> >>
>> >> OK, since I'm getting such great updates on the state of the wisp
>> >> world, far more in a few days than I've had in 10 years... and btw, no
>> >> need to leap on dr science guy research questions like mine if you
>> >> have like, towers flooding or the phone ringing off the hook....
>> >>
>> >> What routing protocols are in use nowadays? BGP, yes, and it seems
>> >> ospf is popular?
>> >>
>> >> How about ISIS?
>> >>
>> >> I figure babel has zero traction or awareness despite being mandated
>> >> by the ietf homenet working group.
>> >>
>> >> Secondly, do you rely on BGP based on the edge router or use it in
>> >> software (frr? quagga? bird?). Using RPKI? Push FIBs anywhere? (route
>> >> 666 in particular)
>> >>
>> >> Similar question related to the IGP protocol in use, where do you rely
>> >> on it, vs all the tunnels you have, on what kinds of hardware?
>> >>
>> >> I note that robert at some point, somewhere, pointed out how fq_codel
>> >> saved his bacon when there was a major routing mishap (as there is no
>> >> congestion control in ospf), and I'd like to hear more of that story.
>> >>
>> >> BATMAN has been mentioned. There's other wireless protocols I've liked
>> >> - OLSR for example...
>> >>
>> >> Nobody knows what lies underneath many consumer wireless meshes
>> >> although it looks like 802.11s is a starting point, none, so far as I
>> >> know interoperate across brands.
>> >>
>> >> --
>> >> This song goes out to all the folk that thought Stadia would work:
>> >>
>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
>> >> Rip Van Winkle COO, TekLibre, LLC
>> >> _______________________________________________
>> >> LibreQoS mailing list
>> >> LibreQoS at lists.bufferbloat.net
>> >> https://lists.bufferbloat.net/listinfo/libreqos
>> >
>> > _______________________________________________
>> > LibreQoS mailing list
>> > LibreQoS at lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/libreqos
>>
>>
>>
>> --
>> This song goes out to all the folk that thought Stadia would work:
>>
>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
>> Dave Täht CEO, TekLibre, LLC
>> _______________________________________________
>> LibreQoS mailing list
>> LibreQoS at lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/libreqos
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/libreqos/attachments/20221027/87503ea8/attachment-0001.html>