[LibreQoS] routing protocols and daemons

[LibreQoS] routing protocols and daemons

[Dave Taht]

OK, since I'm getting such great updates on the state of the wisp
world, far more in a few days than I've had in 10 years... and btw, no
need to leap on dr science guy research questions like mine if you
have like, towers flooding or the phone ringing off the hook....

What routing protocols are in use nowadays? BGP, yes, and it seems
ospf is popular?

How about ISIS?

I figure babel has zero traction or awareness despite being mandated
by the ietf homenet working group.

Secondly, do you rely on BGP based on the edge router or use it in
software (frr? quagga? bird?). Using RPKI? Push FIBs anywhere? (route
666 in particular)

Similar question related to the IGP protocol in use, where do you rely
on it, vs all the tunnels you have, on what kinds of hardware?

I note that robert at some point, somewhere, pointed out how fq_codel
saved his bacon when there was a major routing mishap (as there is no
congestion control in ospf), and I'd like to hear more of that story.

BATMAN has been mentioned. There's other wireless protocols I've liked
- OLSR for example...

Nobody knows what lies underneath many consumer wireless meshes
although it looks like 802.11s is a starting point, none, so far as I
know interoperate across brands.

-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
Rip Van Winkle COO, TekLibre, LLC

Re: [LibreQoS] routing protocols and daemons

[Herbert Wolverson]

[-- Attachment #1: Type: text/plain, Size: 2008 bytes --]

 My name is Herbert, and I'm an OSPF addict... seriously, I love OSPF.
Right down to stub sites, not-so-stubby sites, and isolating IP blocks
within a site into "stub" nets and ensuring they are aggregated properly. I
should probably go outside more...

On Wed, Oct 26, 2022 at 3:29 PM Dave Taht via LibreQoS <
libreqos@lists.bufferbloat.net> wrote:

> OK, since I'm getting such great updates on the state of the wisp
> world, far more in a few days than I've had in 10 years... and btw, no
> need to leap on dr science guy research questions like mine if you
> have like, towers flooding or the phone ringing off the hook....
>
> What routing protocols are in use nowadays? BGP, yes, and it seems
> ospf is popular?
>
> How about ISIS?
>
> I figure babel has zero traction or awareness despite being mandated
> by the ietf homenet working group.
>
> Secondly, do you rely on BGP based on the edge router or use it in
> software (frr? quagga? bird?). Using RPKI? Push FIBs anywhere? (route
> 666 in particular)
>
> Similar question related to the IGP protocol in use, where do you rely
> on it, vs all the tunnels you have, on what kinds of hardware?
>
> I note that robert at some point, somewhere, pointed out how fq_codel
> saved his bacon when there was a major routing mishap (as there is no
> congestion control in ospf), and I'd like to hear more of that story.
>
> BATMAN has been mentioned. There's other wireless protocols I've liked
> - OLSR for example...
>
> Nobody knows what lies underneath many consumer wireless meshes
> although it looks like 802.11s is a starting point, none, so far as I
> know interoperate across brands.
>
> --
> This song goes out to all the folk that thought Stadia would work:
>
> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
> Rip Van Winkle COO, TekLibre, LLC
> _______________________________________________
> LibreQoS mailing list
> LibreQoS@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/libreqos
>

[-- Attachment #2: Type: text/html, Size: 2764 bytes --]

Re: [LibreQoS] routing protocols and daemons

[Dave Taht]

On Wed, Oct 26, 2022 at 1:53 PM Herbert Wolverson via LibreQoS
<libreqos@lists.bufferbloat.net> wrote:
>
> My name is Herbert, and I'm an OSPF addict... seriously, I love OSPF. Right down to stub sites, not-so-stubby sites, and isolating IP blocks within a site into "stub" nets and ensuring they are aggregated properly. I should probably go outside more...

haha.

My name is dave, and I think all routing protocols should have evolved
much better to elegantly meet the real world problems they were trying
to solve, than they have.

To avoid burying the lede, to what extent does OSPF still rely on
multicast? How well can it carry ipv6 now? What extensions are common
in the real WISP world?

BGP needs a few more napkins.

RIP was a VERY good start but we drew the wrong lessons from its
failures, and the super-duper-trendline towards centralized
controllers inherent in OSPF and ISIS that happened in the 90s that
doesn't scale anywhere near as I'd like.

I liked the rise of meshy 802.11 networks, I know the author of AODV
well (charlie perkins is arguably one of the fathers of mesh
networking, far too few have read his books from the 90s). And I've
been involved in the "battlemesh" group for many years with those
trying to make 'em work better on networks such as guifi,
wlan-slovinia, etc.

Backstory. Back in 07, in Nicaragua, I was (stupidly) trying to get
ipv6 to work over nanostation m2s or m5s I forget which, and the basic
option was to run two copies of the ospf daemon to manage 4 and 6
independently. I only had 32MB of memory and it didn't fit, so I
started looking for alternatives, found babel, corresponded with (and
frankly thoroughly annoyed) the author, and starting giving it a go.
It transported 4 and 6 in the same packets, was tiny, was
distance-vector (thus, I thought, more a match for bgp), and (to me)
most importantly, solved the ipv4 and ipv6 routing problems in the
same daemon at the same time, and actually fit into less memory than
ospf did. It was good enough it seemed, to deploy to a few hundred
routers without having to play major tricks with areas and stubs and
so on.

Babel is so simple that toke wrote a near complete implementation from
the spec, in python, during a string of extremely boring IETF
meetings, over the course of a week. He later took on the bird port.
Over the years we've wedged most (but not all) of the key features I
thought a meshy wireless routing protocol should have, with
implementations in a standalone daemon, bird, and FRR. (there was a
quagga port at one point too. I forget what happened to toke's python
version).

https://www.rfc-editor.org/rfc/rfc8966.html babel
https://arxiv.org/abs/1403.0445 source specific routing
https://datatracker.ietf.org/doc/rfc8967/ HMAC authentication
https://datatracker.ietf.org/doc/html/draft-ietf-babel-rtt-extension-00
RTT metric
https://datatracker.ietf.org/meeting/99/materials/slides-99-babel-unicast-hellos-00.pdf
 unicast hellos

Missing is BFD support, and the slightest bit of traction outside of
the shrinking battlemesh communities.

Althea is using babel and fq_codel in their blockchain routing thing
(I reserve comment), and I don't know where else, besides as part of
wireguard tunnels, babel is being used today. But I'm rather
interested in how OSPF evolved since I last touched it, and what use
cases it is good at and fails at?


> On Wed, Oct 26, 2022 at 3:29 PM Dave Taht via LibreQoS <libreqos@lists.bufferbloat.net> wrote:
>>
>> OK, since I'm getting such great updates on the state of the wisp
>> world, far more in a few days than I've had in 10 years... and btw, no
>> need to leap on dr science guy research questions like mine if you
>> have like, towers flooding or the phone ringing off the hook....
>>
>> What routing protocols are in use nowadays? BGP, yes, and it seems
>> ospf is popular?
>>
>> How about ISIS?
>>
>> I figure babel has zero traction or awareness despite being mandated
>> by the ietf homenet working group.
>>
>> Secondly, do you rely on BGP based on the edge router or use it in
>> software (frr? quagga? bird?). Using RPKI? Push FIBs anywhere? (route
>> 666 in particular)
>>
>> Similar question related to the IGP protocol in use, where do you rely
>> on it, vs all the tunnels you have, on what kinds of hardware?
>>
>> I note that robert at some point, somewhere, pointed out how fq_codel
>> saved his bacon when there was a major routing mishap (as there is no
>> congestion control in ospf), and I'd like to hear more of that story.
>>
>> BATMAN has been mentioned. There's other wireless protocols I've liked
>> - OLSR for example...
>>
>> Nobody knows what lies underneath many consumer wireless meshes
>> although it looks like 802.11s is a starting point, none, so far as I
>> know interoperate across brands.
>>
>> --
>> This song goes out to all the folk that thought Stadia would work:
>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
>> Rip Van Winkle COO, TekLibre, LLC
>> _______________________________________________
>> LibreQoS mailing list
>> LibreQoS@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/libreqos
>
> _______________________________________________
> LibreQoS mailing list
> LibreQoS@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/libreqos



-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
Dave Täht CEO, TekLibre, LLC

Re: [LibreQoS] routing protocols and daemons

[dan]

[-- Attachment #1: Type: text/plain, Size: 8505 bytes --]

I have played with batman-adv quite a bit and there are some concepts in it
I really like.  Not being shortest path for one, and rating a link quality
instead of hard up/down.   I also like the layer2 model so it looks like a
big switch.  It's very clean from an operational perspective as it behaves
essentially like an MPLS/VPLS network administratively.

What I think we're missing is the integration of network attributes and
class of service.  For instance, user to 'internet' has 3 potential paths
with each having these end-to-end latency, upload throughput, download
throughput, and say 'quality' or packet loss.  Then having your QoS engine
able to tag packets for how it perceives them to need routed and then have
the routing engine pick routes based on availability.  So you might have a
longer path that will suffer some on latency because of the hops and link
type but has big bandwidth 'available' (ie large capacity and underused) so
it should ask for that flow to take the underused high capacity (and yet
still meets other criteria) path.  Something considered realtime might
prefer that 700Mbps licenced path that has lower and more predictable
latency and enough available capacity for the job. By encouraging high
throughput needs to take paths with a lot of availability and some
mechanism to prevent occilations from reroutes you could keep lower latency
links less busy and get load balancing by a more intelligent choice.    You
could also have some sort of reservation number tagged onto that packet to
ask the intermediate hops to reduce their available amount.  If you were
going to go all out on this and have devices that spoke this everywhere,
you could put your shapers everywhere as well, getting that desired egrees
shapping on both sides and letting the network sort of reserve a bit of
bandwidth for each customer based on that.

Of course this means scaling issues almost inherently because those
'available capacity' numbers and packet loss need to be communicated.
computationally intensive.

batman-adv does this in a way with it's OGM/ELP system.  You can take a
longer path through a batman-adv network because of a saturated link and it
doesn't consider that saturated link 'down'.

rflo was an interesting tech that did some of these once upon a time.

Just thoughts.

On Wed, Oct 26, 2022 at 3:38 PM Dave Taht via LibreQoS <
libreqos@lists.bufferbloat.net> wrote:

> On Wed, Oct 26, 2022 at 1:53 PM Herbert Wolverson via LibreQoS
> <libreqos@lists.bufferbloat.net> wrote:
> >
> > My name is Herbert, and I'm an OSPF addict... seriously, I love OSPF.
> Right down to stub sites, not-so-stubby sites, and isolating IP blocks
> within a site into "stub" nets and ensuring they are aggregated properly. I
> should probably go outside more...
>
> haha.
>
> My name is dave, and I think all routing protocols should have evolved
> much better to elegantly meet the real world problems they were trying
> to solve, than they have.
>
> To avoid burying the lede, to what extent does OSPF still rely on
> multicast? How well can it carry ipv6 now? What extensions are common
> in the real WISP world?
>
> BGP needs a few more napkins.
>
> RIP was a VERY good start but we drew the wrong lessons from its
> failures, and the super-duper-trendline towards centralized
> controllers inherent in OSPF and ISIS that happened in the 90s that
> doesn't scale anywhere near as I'd like.
>
> I liked the rise of meshy 802.11 networks, I know the author of AODV
> well (charlie perkins is arguably one of the fathers of mesh
> networking, far too few have read his books from the 90s). And I've
> been involved in the "battlemesh" group for many years with those
> trying to make 'em work better on networks such as guifi,
> wlan-slovinia, etc.
>
> Backstory. Back in 07, in Nicaragua, I was (stupidly) trying to get
> ipv6 to work over nanostation m2s or m5s I forget which, and the basic
> option was to run two copies of the ospf daemon to manage 4 and 6
> independently. I only had 32MB of memory and it didn't fit, so I
> started looking for alternatives, found babel, corresponded with (and
> frankly thoroughly annoyed) the author, and starting giving it a go.
> It transported 4 and 6 in the same packets, was tiny, was
> distance-vector (thus, I thought, more a match for bgp), and (to me)
> most importantly, solved the ipv4 and ipv6 routing problems in the
> same daemon at the same time, and actually fit into less memory than
> ospf did. It was good enough it seemed, to deploy to a few hundred
> routers without having to play major tricks with areas and stubs and
> so on.
>
> Babel is so simple that toke wrote a near complete implementation from
> the spec, in python, during a string of extremely boring IETF
> meetings, over the course of a week. He later took on the bird port.
> Over the years we've wedged most (but not all) of the key features I
> thought a meshy wireless routing protocol should have, with
> implementations in a standalone daemon, bird, and FRR. (there was a
> quagga port at one point too. I forget what happened to toke's python
> version).
>
> https://www.rfc-editor.org/rfc/rfc8966.html babel
> https://arxiv.org/abs/1403.0445 source specific routing
> https://datatracker.ietf.org/doc/rfc8967/ HMAC authentication
> https://datatracker.ietf.org/doc/html/draft-ietf-babel-rtt-extension-00
> RTT metric
>
> https://datatracker.ietf.org/meeting/99/materials/slides-99-babel-unicast-hellos-00.pdf
>  unicast hellos
>
> Missing is BFD support, and the slightest bit of traction outside of
> the shrinking battlemesh communities.
>
> Althea is using babel and fq_codel in their blockchain routing thing
> (I reserve comment), and I don't know where else, besides as part of
> wireguard tunnels, babel is being used today. But I'm rather
> interested in how OSPF evolved since I last touched it, and what use
> cases it is good at and fails at?
>
>
> > On Wed, Oct 26, 2022 at 3:29 PM Dave Taht via LibreQoS <
> libreqos@lists.bufferbloat.net> wrote:
> >>
> >> OK, since I'm getting such great updates on the state of the wisp
> >> world, far more in a few days than I've had in 10 years... and btw, no
> >> need to leap on dr science guy research questions like mine if you
> >> have like, towers flooding or the phone ringing off the hook....
> >>
> >> What routing protocols are in use nowadays? BGP, yes, and it seems
> >> ospf is popular?
> >>
> >> How about ISIS?
> >>
> >> I figure babel has zero traction or awareness despite being mandated
> >> by the ietf homenet working group.
> >>
> >> Secondly, do you rely on BGP based on the edge router or use it in
> >> software (frr? quagga? bird?). Using RPKI? Push FIBs anywhere? (route
> >> 666 in particular)
> >>
> >> Similar question related to the IGP protocol in use, where do you rely
> >> on it, vs all the tunnels you have, on what kinds of hardware?
> >>
> >> I note that robert at some point, somewhere, pointed out how fq_codel
> >> saved his bacon when there was a major routing mishap (as there is no
> >> congestion control in ospf), and I'd like to hear more of that story.
> >>
> >> BATMAN has been mentioned. There's other wireless protocols I've liked
> >> - OLSR for example...
> >>
> >> Nobody knows what lies underneath many consumer wireless meshes
> >> although it looks like 802.11s is a starting point, none, so far as I
> >> know interoperate across brands.
> >>
> >> --
> >> This song goes out to all the folk that thought Stadia would work:
> >>
> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
> >> Rip Van Winkle COO, TekLibre, LLC
> >> _______________________________________________
> >> LibreQoS mailing list
> >> LibreQoS@lists.bufferbloat.net
> >> https://lists.bufferbloat.net/listinfo/libreqos
> >
> > _______________________________________________
> > LibreQoS mailing list
> > LibreQoS@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/libreqos
>
>
>
> --
> This song goes out to all the folk that thought Stadia would work:
>
> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
> Dave Täht CEO, TekLibre, LLC
> _______________________________________________
> LibreQoS mailing list
> LibreQoS@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/libreqos
>

[-- Attachment #2: Type: text/html, Size: 10877 bytes --]

Re: [LibreQoS] routing protocols and daemons

[Herbert Wolverson]

[-- Attachment #1: Type: text/plain, Size: 12268 bytes --]

OSPF is a pretty stable protocol, so it doesn't change all that much (one
of the reasons I like it is that it is supported everywhere, and works
cross-vendor consistently). In answer to your questions:

* With OSPFv3 you can push IPv6 routes. I've only used this a bit, but it
worked fine last time we tried it.
* You have a fair amount of flexibility on how hops talk to one another
(configured on the interface). The default is still multicast, but
"point-to-point" is just a unicast broadcast (and VERY fast route
coalescing; it's our default), "point-to-multipoint" can do the same for
1:many relationships (I don't use this one, I prefer one interface to one
destination - even if the interfaces are VLANs. Makes traffic analysis a
whole lot easier). There's also NBMA, which has you type in the addresses
of adjacent neighbors (which feels like it defeats some of the point!) -
but doesn't even broadcast. There was a time that NBMA was the most
consistent over old Ubiquiti equipment, but these days the other modes work
fine.
* ECMP (Equal Cost Multipath) is awesome. If two routes to the destination
sum out at equal costs, traffic is split equally between them. Traffic is
split at layer 3 (on an address-port hash), so flows stay together. It's
rock solid, we have it sharing traffic over two backhauls in a few places -
and it handles one interface going down flawlessly (with either BFD, or
point-to-point mode which coalesces so fast).

Dijkstra's algorithm remains a very natural approach to mapping a graph
(I've waxed lyrical about it in various book/article/blog posts over the
years in the game-dev world), so I find it a very comfy way to model the
underlying network. Very easy to reason "this will go that way".

So what's bad about it?

* ECMP is equal; if you have routes with different costs it will only use
the lowest cost - it won't try and "mesh" some of the traffic in other
directions. Likewise, if you have two equal routes - and one of them is
running at a low capacity - you wind up only utilizing twice the capacity
of the slowest link. You're often better off dropping the link over the
degraded circuit (hence "carrier drop" features on various radios).
* OSPF has no idea what the capacities of various links are. It'll use the
shortest cost route, and leave the details up to the lower layers of the
stack.

There have been various attempts to integrate capacity into network design,
and I've yet to see one that holds up well on a multi-vendor network. If
you ask a Ubiquiti Rocket AC Gen2 its capacity, it'll often give you some
nice big number - but it'll stall out far before that number. The Force
400C link we just put up doesn't even try to estimate its capacity. So
multi-vendor mesh routing tends to be problematic, because the advertised
capability of "we'll utilize everywhere you have capacity" tends to get
snarled up in figuring out what the capacity is. I understand Teragraph is
supposed to do better there. MPLS traffic engineering was originally
announced as solving this one, too! I don't see a lot of  hope for
capacity-aware routing protocols taking off, but I imagine we'll get a few
new ones announced and then quietly forgotten as before.

And lastly: once your network gets really big, OSPF tables can get too
large - and you're stuck either dividing your OSPF zones and/or using some
BGP in interior mode. You can mitigate this with some careful design.

On Wed, Oct 26, 2022 at 5:35 PM dan <dandenson@gmail.com> wrote:

> I have played with batman-adv quite a bit and there are some concepts in
> it I really like.  Not being shortest path for one, and rating a link
> quality instead of hard up/down.   I also like the layer2 model so it looks
> like a big switch.  It's very clean from an operational perspective as it
> behaves essentially like an MPLS/VPLS network administratively.
>
> What I think we're missing is the integration of network attributes and
> class of service.  For instance, user to 'internet' has 3 potential paths
> with each having these end-to-end latency, upload throughput, download
> throughput, and say 'quality' or packet loss.  Then having your QoS engine
> able to tag packets for how it perceives them to need routed and then have
> the routing engine pick routes based on availability.  So you might have a
> longer path that will suffer some on latency because of the hops and link
> type but has big bandwidth 'available' (ie large capacity and underused) so
> it should ask for that flow to take the underused high capacity (and yet
> still meets other criteria) path.  Something considered realtime might
> prefer that 700Mbps licenced path that has lower and more predictable
> latency and enough available capacity for the job. By encouraging high
> throughput needs to take paths with a lot of availability and some
> mechanism to prevent occilations from reroutes you could keep lower latency
> links less busy and get load balancing by a more intelligent choice.    You
> could also have some sort of reservation number tagged onto that packet to
> ask the intermediate hops to reduce their available amount.  If you were
> going to go all out on this and have devices that spoke this everywhere,
> you could put your shapers everywhere as well, getting that desired egrees
> shapping on both sides and letting the network sort of reserve a bit of
> bandwidth for each customer based on that.
>
> Of course this means scaling issues almost inherently because those
> 'available capacity' numbers and packet loss need to be communicated.
> computationally intensive.
>
> batman-adv does this in a way with it's OGM/ELP system.  You can take a
> longer path through a batman-adv network because of a saturated link and it
> doesn't consider that saturated link 'down'.
>
> rflo was an interesting tech that did some of these once upon a time.
>
> Just thoughts.
>
> On Wed, Oct 26, 2022 at 3:38 PM Dave Taht via LibreQoS <
> libreqos@lists.bufferbloat.net> wrote:
>
>> On Wed, Oct 26, 2022 at 1:53 PM Herbert Wolverson via LibreQoS
>> <libreqos@lists.bufferbloat.net> wrote:
>> >
>> > My name is Herbert, and I'm an OSPF addict... seriously, I love OSPF.
>> Right down to stub sites, not-so-stubby sites, and isolating IP blocks
>> within a site into "stub" nets and ensuring they are aggregated properly. I
>> should probably go outside more...
>>
>> haha.
>>
>> My name is dave, and I think all routing protocols should have evolved
>> much better to elegantly meet the real world problems they were trying
>> to solve, than they have.
>>
>> To avoid burying the lede, to what extent does OSPF still rely on
>> multicast? How well can it carry ipv6 now? What extensions are common
>> in the real WISP world?
>>
>> BGP needs a few more napkins.
>>
>> RIP was a VERY good start but we drew the wrong lessons from its
>> failures, and the super-duper-trendline towards centralized
>> controllers inherent in OSPF and ISIS that happened in the 90s that
>> doesn't scale anywhere near as I'd like.
>>
>> I liked the rise of meshy 802.11 networks, I know the author of AODV
>> well (charlie perkins is arguably one of the fathers of mesh
>> networking, far too few have read his books from the 90s). And I've
>> been involved in the "battlemesh" group for many years with those
>> trying to make 'em work better on networks such as guifi,
>> wlan-slovinia, etc.
>>
>> Backstory. Back in 07, in Nicaragua, I was (stupidly) trying to get
>> ipv6 to work over nanostation m2s or m5s I forget which, and the basic
>> option was to run two copies of the ospf daemon to manage 4 and 6
>> independently. I only had 32MB of memory and it didn't fit, so I
>> started looking for alternatives, found babel, corresponded with (and
>> frankly thoroughly annoyed) the author, and starting giving it a go.
>> It transported 4 and 6 in the same packets, was tiny, was
>> distance-vector (thus, I thought, more a match for bgp), and (to me)
>> most importantly, solved the ipv4 and ipv6 routing problems in the
>> same daemon at the same time, and actually fit into less memory than
>> ospf did. It was good enough it seemed, to deploy to a few hundred
>> routers without having to play major tricks with areas and stubs and
>> so on.
>>
>> Babel is so simple that toke wrote a near complete implementation from
>> the spec, in python, during a string of extremely boring IETF
>> meetings, over the course of a week. He later took on the bird port.
>> Over the years we've wedged most (but not all) of the key features I
>> thought a meshy wireless routing protocol should have, with
>> implementations in a standalone daemon, bird, and FRR. (there was a
>> quagga port at one point too. I forget what happened to toke's python
>> version).
>>
>> https://www.rfc-editor.org/rfc/rfc8966.html babel
>> https://arxiv.org/abs/1403.0445 source specific routing
>> https://datatracker.ietf.org/doc/rfc8967/ HMAC authentication
>> https://datatracker.ietf.org/doc/html/draft-ietf-babel-rtt-extension-00
>> RTT metric
>>
>> https://datatracker.ietf.org/meeting/99/materials/slides-99-babel-unicast-hellos-00.pdf
>>  unicast hellos
>>
>> Missing is BFD support, and the slightest bit of traction outside of
>> the shrinking battlemesh communities.
>>
>> Althea is using babel and fq_codel in their blockchain routing thing
>> (I reserve comment), and I don't know where else, besides as part of
>> wireguard tunnels, babel is being used today. But I'm rather
>> interested in how OSPF evolved since I last touched it, and what use
>> cases it is good at and fails at?
>>
>>
>> > On Wed, Oct 26, 2022 at 3:29 PM Dave Taht via LibreQoS <
>> libreqos@lists.bufferbloat.net> wrote:
>> >>
>> >> OK, since I'm getting such great updates on the state of the wisp
>> >> world, far more in a few days than I've had in 10 years... and btw, no
>> >> need to leap on dr science guy research questions like mine if you
>> >> have like, towers flooding or the phone ringing off the hook....
>> >>
>> >> What routing protocols are in use nowadays? BGP, yes, and it seems
>> >> ospf is popular?
>> >>
>> >> How about ISIS?
>> >>
>> >> I figure babel has zero traction or awareness despite being mandated
>> >> by the ietf homenet working group.
>> >>
>> >> Secondly, do you rely on BGP based on the edge router or use it in
>> >> software (frr? quagga? bird?). Using RPKI? Push FIBs anywhere? (route
>> >> 666 in particular)
>> >>
>> >> Similar question related to the IGP protocol in use, where do you rely
>> >> on it, vs all the tunnels you have, on what kinds of hardware?
>> >>
>> >> I note that robert at some point, somewhere, pointed out how fq_codel
>> >> saved his bacon when there was a major routing mishap (as there is no
>> >> congestion control in ospf), and I'd like to hear more of that story.
>> >>
>> >> BATMAN has been mentioned. There's other wireless protocols I've liked
>> >> - OLSR for example...
>> >>
>> >> Nobody knows what lies underneath many consumer wireless meshes
>> >> although it looks like 802.11s is a starting point, none, so far as I
>> >> know interoperate across brands.
>> >>
>> >> --
>> >> This song goes out to all the folk that thought Stadia would work:
>> >>
>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
>> >> Rip Van Winkle COO, TekLibre, LLC
>> >> _______________________________________________
>> >> LibreQoS mailing list
>> >> LibreQoS@lists.bufferbloat.net
>> >> https://lists.bufferbloat.net/listinfo/libreqos
>> >
>> > _______________________________________________
>> > LibreQoS mailing list
>> > LibreQoS@lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/libreqos
>>
>>
>>
>> --
>> This song goes out to all the folk that thought Stadia would work:
>>
>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
>> Dave Täht CEO, TekLibre, LLC
>> _______________________________________________
>> LibreQoS mailing list
>> LibreQoS@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/libreqos
>>
>

[-- Attachment #2: Type: text/html, Size: 14996 bytes --]

Re: [LibreQoS] routing protocols and daemons

[Mark Steckel]

Herbert,

Great info!   (And just a quick shout out that this has quickly become the best signal to noise email list of any I'm currently on. Really appreciate the expertise and depth of knowledge. Thanks everyone!)

I also really like OSPF except when it goes wonky on me...

I've been struggling with a OSPF issue for awhile and can't seem to track down the source (other than generically grousing at Ubnt's firmware and software development practices). The short version of the story is that the ospfd process consumes 100% cpu and starts consuming memory which it is eventually exhausted. When this happens usually 2 or more routers are involved. My working theory is that the routers involved are caught in an OSPF update storm. Our internal router table is not horribly large.

mjs@PWTW04-NNNN-STREET-ST-RTR:~$ show ip route summary 
IP routing table name is Default-IP-Routing-Table(0)
IP routing table maximum-paths   : 8
Total number of IPv4 routes      : 861
Total number of IPv4 paths       : 861
Route Source    Networks
connected       9
ospf            852
Total           861
FIB             852


I am slowly replacing all Ubnt routers at major sites with VyOS on DC powered Supermicro servers. So far, really liking VyOS. Example: https://www.supermicro.com/en/products/system/Mini-ITX/SYS-E300-9D-8CN8TP.cfm

A bit more below.


---- On Thu, 27 Oct 2022 09:29:22 -0400 Herbert Wolverson via LibreQoS  wrote ---

 > OSPF is a pretty stable protocol, so it doesn't change all that much (one of the reasons I like it is that it is supported everywhere, and works cross-vendor consistently). In answer to your questions:
 > 
 > * With OSPFv3 you can push IPv6 routes. I've only used this a bit, but it worked fine last time we tried it.
 > * You have a fair amount of flexibility on how hops talk to one another (configured on the interface). The default is still multicast, but "point-to-point" is just a unicast broadcast (and VERY fast route coalescing; it's our default),

A few years ago I briefly looked at these config options but I never implemented anything. This morning I updated 2 routers to your "point-to-point" (I think...). Is anything more than the following needed on both routers (adjusting for ports, etc)?

set interfaces ethernet eth1 ip ospf network point-to-point



> "point-to-multipoint" can do the same for 1:many relationships (I don't use this one, I prefer one interface to one destination - even if the interfaces are VLANs. Makes traffic analysis a whole lot easier). There's also NBMA, which has you type in the addresses of adjacent neighbors (which feels like it defeats some of the point!) - but doesn't even broadcast. 

At major sites, we typically have 2 routers running in an active-active manner: router-1 has a.b.c.2, router-2 has a.b.c.3 and vrrp is used to float a.b.c.1 between both routers.

Is "point-to-multpoint" required in this situation?



> There was a time that NBMA was the most consistent over old Ubiquiti equipment, but these days the other modes work fine.
 > * ECMP (Equal Cost Multipath) is awesome. If two routes to the destination sum out at equal costs, traffic is split equally between them. Traffic is split at layer 3 (on an address-port hash), so flows stay together. It's rock solid, we have it sharing traffic over two backhauls in a few places - and it handles one interface going down flawlessly (with either BFD, or point-to-point mode which coalesces so fast).
 > 
 > Dijkstra's algorithm remains a very natural approach to mapping a graph (I've waxed lyrical about it in various book/article/blog posts over the years in the game-dev world), so I find it a very comfy way to model the underlying network. Very easy to reason "this will go that way".
 > 
 > So what's bad about it?
 > 
 > * ECMP is equal; if you have routes with different costs it will only use the lowest cost - it won't try and "mesh" some of the traffic in other directions. Likewise, if you have two equal routes - and one of them is running at a low capacity - you wind up only utilizing twice the capacity of the slowest link. You're often better off dropping the link over the degraded circuit (hence "carrier drop" features on various radios).
 > * OSPF has no idea what the capacities of various links are. It'll use the shortest cost route, and leave the details up to the lower layers of the stack.
 > 
 > There have been various attempts to integrate capacity into network design, and I've yet to see one that holds up well on a multi-vendor network. If you ask a Ubiquiti Rocket AC Gen2 its capacity, it'll often give you some nice big number - but it'll stall out far before that number. The Force 400C link we just put up doesn't even try to estimate its capacity. So multi-vendor mesh routing tends to be problematic, because the advertised capability of "we'll utilize everywhere you have capacity" tends to get snarled up in figuring out what the capacity is. I understand Teragraph is supposed to do better there. MPLS traffic engineering was originally announced as solving this one, too! I don't see a lot of  hope for capacity-aware routing protocols taking off, but I imagine we'll get a few new ones announced and then quietly forgotten as before.
 > 
 > And lastly: once your network gets really big, OSPF tables can get too large - and you're stuck either dividing your OSPF zones and/or using some BGP in interior mode. You can mitigate this with some careful design.

What is typically considered "too large" and how does "too large" type problems typically show up?

Thanks
Mark



 > 
 > On Wed, Oct 26, 2022 at 5:35 PM dan dandenson@gmail.com> wrote:
 > I have played with batman-adv quite a bit and there are some concepts in it I really like.  Not being shortest path for one, and rating a link quality instead of hard up/down.   I also like the layer2 model so it looks like a big switch.  It's very clean from an operational perspective as it behaves essentially like an MPLS/VPLS network administratively.
 > 
 > What I think we're missing is the integration of network attributes and class of service.  For instance, user to 'internet' has 3 potential paths with each having these end-to-end latency, upload throughput, download throughput, and say 'quality' or packet loss.  Then having your QoS engine able to tag packets for how it perceives them to need routed and then have the routing engine pick routes based on availability.  So you might have a longer path that will suffer some on latency because of the hops and link type but has big bandwidth 'available' (ie large capacity and underused) so it should ask for that flow to take the underused high capacity (and yet still meets other criteria) path.  Something considered realtime might prefer that 700Mbps licenced path that has lower and more predictable latency and enough available capacity for the job. By encouraging high throughput needs to take paths with a lot of availability and some mechanism to prevent occilations from reroutes you could keep lower latency links less busy and get load balancing by a more intelligent choice.    You could also have some sort of reservation number tagged onto that packet to ask the intermediate hops to reduce their available amount.  If you were going to go all out on this and have devices that spoke this everywhere, you could put your shapers everywhere as well, getting that desired egrees shapping on both sides and letting the network sort of reserve a bit of bandwidth for each customer based on that. 
 > 
 > Of course this means scaling issues almost inherently because those 'available capacity' numbers and packet loss need to be communicated.  computationally intensive.  
 > 
 > batman-adv does this in a way with it's OGM/ELP system.  You can take a longer path through a batman-adv network because of a saturated link and it doesn't consider that saturated link 'down'.  
 > 
 > rflo was an interesting tech that did some of these once upon a time.
 > 
 > Just thoughts.
 > On Wed, Oct 26, 2022 at 3:38 PM Dave Taht via LibreQoS libreqos@lists.bufferbloat.net> wrote:
 > On Wed, Oct 26, 2022 at 1:53 PM Herbert Wolverson via LibreQoS
 >  libreqos@lists.bufferbloat.net> wrote:
 >  >
 >  > My name is Herbert, and I'm an OSPF addict... seriously, I love OSPF. Right down to stub sites, not-so-stubby sites, and isolating IP blocks within a site into "stub" nets and ensuring they are aggregated properly. I should probably go outside more...
 >  
 >  haha.
 >  
 >  My name is dave, and I think all routing protocols should have evolved
 >  much better to elegantly meet the real world problems they were trying
 >  to solve, than they have.
 >  
 >  To avoid burying the lede, to what extent does OSPF still rely on
 >  multicast? How well can it carry ipv6 now? What extensions are common
 >  in the real WISP world?
 >  
 >  BGP needs a few more napkins.
 >  
 >  RIP was a VERY good start but we drew the wrong lessons from its
 >  failures, and the super-duper-trendline towards centralized
 >  controllers inherent in OSPF and ISIS that happened in the 90s that
 >  doesn't scale anywhere near as I'd like.
 >  
 >  I liked the rise of meshy 802.11 networks, I know the author of AODV
 >  well (charlie perkins is arguably one of the fathers of mesh
 >  networking, far too few have read his books from the 90s). And I've
 >  been involved in the "battlemesh" group for many years with those
 >  trying to make 'em work better on networks such as guifi,
 >  wlan-slovinia, etc.
 >  
 >  Backstory. Back in 07, in Nicaragua, I was (stupidly) trying to get
 >  ipv6 to work over nanostation m2s or m5s I forget which, and the basic
 >  option was to run two copies of the ospf daemon to manage 4 and 6
 >  independently. I only had 32MB of memory and it didn't fit, so I
 >  started looking for alternatives, found babel, corresponded with (and
 >  frankly thoroughly annoyed) the author, and starting giving it a go.
 >  It transported 4 and 6 in the same packets, was tiny, was
 >  distance-vector (thus, I thought, more a match for bgp), and (to me)
 >  most importantly, solved the ipv4 and ipv6 routing problems in the
 >  same daemon at the same time, and actually fit into less memory than
 >  ospf did. It was good enough it seemed, to deploy to a few hundred
 >  routers without having to play major tricks with areas and stubs and
 >  so on.
 >  
 >  Babel is so simple that toke wrote a near complete implementation from
 >  the spec, in python, during a string of extremely boring IETF
 >  meetings, over the course of a week. He later took on the bird port.
 >  Over the years we've wedged most (but not all) of the key features I
 >  thought a meshy wireless routing protocol should have, with
 >  implementations in a standalone daemon, bird, and FRR. (there was a
 >  quagga port at one point too. I forget what happened to toke's python
 >  version).
 >  
 >  https://www.rfc-editor.org/rfc/rfc8966.html babel
 >  https://arxiv.org/abs/1403.0445 source specific routing
 >  https://datatracker.ietf.org/doc/rfc8967/ HMAC authentication
 >  https://datatracker.ietf.org/doc/html/draft-ietf-babel-rtt-extension-00
 >  RTT metric
 >  https://datatracker.ietf.org/meeting/99/materials/slides-99-babel-unicast-hellos-00.pdf
 >   unicast hellos
 >  
 >  Missing is BFD support, and the slightest bit of traction outside of
 >  the shrinking battlemesh communities.
 >  
 >  Althea is using babel and fq_codel in their blockchain routing thing
 >  (I reserve comment), and I don't know where else, besides as part of
 >  wireguard tunnels, babel is being used today. But I'm rather
 >  interested in how OSPF evolved since I last touched it, and what use
 >  cases it is good at and fails at?
 >  
 >  
 >  > On Wed, Oct 26, 2022 at 3:29 PM Dave Taht via LibreQoS libreqos@lists.bufferbloat.net> wrote:
 >  >>
 >  >> OK, since I'm getting such great updates on the state of the wisp
 >  >> world, far more in a few days than I've had in 10 years... and btw, no
 >  >> need to leap on dr science guy research questions like mine if you
 >  >> have like, towers flooding or the phone ringing off the hook....
 >  >>
 >  >> What routing protocols are in use nowadays? BGP, yes, and it seems
 >  >> ospf is popular?
 >  >>
 >  >> How about ISIS?
 >  >>
 >  >> I figure babel has zero traction or awareness despite being mandated
 >  >> by the ietf homenet working group.
 >  >>
 >  >> Secondly, do you rely on BGP based on the edge router or use it in
 >  >> software (frr? quagga? bird?). Using RPKI? Push FIBs anywhere? (route
 >  >> 666 in particular)
 >  >>
 >  >> Similar question related to the IGP protocol in use, where do you rely
 >  >> on it, vs all the tunnels you have, on what kinds of hardware?
 >  >>
 >  >> I note that robert at some point, somewhere, pointed out how fq_codel
 >  >> saved his bacon when there was a major routing mishap (as there is no
 >  >> congestion control in ospf), and I'd like to hear more of that story.
 >  >>
 >  >> BATMAN has been mentioned. There's other wireless protocols I've liked
 >  >> - OLSR for example...
 >  >>
 >  >> Nobody knows what lies underneath many consumer wireless meshes
 >  >> although it looks like 802.11s is a starting point, none, so far as I
 >  >> know interoperate across brands.
 >  >>
 >  >> --
 >  >> This song goes out to all the folk that thought Stadia would work:
 >  >> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
 >  >> Rip Van Winkle COO, TekLibre, LLC
 >  >> _______________________________________________
 >  >> LibreQoS mailing list
 >  >> LibreQoS@lists.bufferbloat.net
 >  >> https://lists.bufferbloat.net/listinfo/libreqos
 >  >
 >  > _______________________________________________
 >  > LibreQoS mailing list
 >  > LibreQoS@lists.bufferbloat.net
 >  > https://lists.bufferbloat.net/listinfo/libreqos
 >  
 >  
 >  
 >  -- 
 >  This song goes out to all the folk that thought Stadia would work:
 >  https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
 >  Dave Täht CEO, TekLibre, LLC
 >  _______________________________________________
 >  LibreQoS mailing list
 >  LibreQoS@lists.bufferbloat.net
 >  https://lists.bufferbloat.net/listinfo/libreqos
 > _______________________________________________
 > LibreQoS mailing list
 > LibreQoS@lists.bufferbloat.net
 > https://lists.bufferbloat.net/listinfo/libreqos
 > 

Re: [LibreQoS] routing protocols and daemons

[Dave Taht]

On Thu, Oct 27, 2022 at 8:29 AM Mark Steckel via LibreQoS
<libreqos@lists.bufferbloat.net> wrote:
>
> Herbert,
>
> Great info!   (And just a quick shout out that this has quickly become the best signal to noise email list of any I'm currently on. Really appreciate the expertise and depth of knowledge. Thanks everyone!)

I have a golden contact list. Most of us, though, are getting close to
or past the age of retirement, but I will continue to cultivate this
list with the best people I can find. If there is anyone y'all know
into a culture of sharing information and problems and solutions,
please ask them to join also? So delighted in my doddering years to
have been learning from you all.

I was and remain heartly sick and tired of the top down
leader/follower trend that has taken over the internet in the last
decade, and the marketing drivel, (the front page of google now reads
like national enquirer, where before it had things like the economist
and stuff I wanted to read) and if getting off of twitter and into
email is what it takes, with a simple signup, to get rational
discourse on the internet again, well, I'm all for it!

Easy pass/fail test for someone suited for hearabouts, is if your
potential invitee digs the song I have in my .sig.

>
> I also really like OSPF except when it goes wonky on me...
>
> I've been struggling with a OSPF issue for awhile and can't seem to track down the source (other than generically grousing at Ubnt's firmware and software development practices). The short version of the story is that the ospfd process consumes 100% cpu and starts consuming memory which it is eventually exhausted. When this happens usually 2 or more routers are involved. My working theory is that the routers involved are caught in an OSPF update storm. Our internal router table is not horribly large.


Oh, boy, did I wrestle with this on multiple fronts a few years ago. I
ended up writing the "routing tables of death" tool:
https://github.com/dtaht/rtod#readme

The readme is pretty funny, I think. Enjoy. Perhaps leveraging that
tool you can isolate the problem better.

I found so many bugs in route handling (and reported most of them to
no avail), in the linux kernel, babel,
bgp, and olsr. They included

Excessive updates saturating other daemons like dnsmasq and odhcpd
listing on the route socket
Totally wiping out the wifi (16 seconds of multicast latency on one of
the devices I was testing
The dameon not prioritizing essential work so it fell way, way behind
Not filtering out events you didn't care about in general
Not failing safe (e.g. if you are way behind, at LEAST make sure you
are announcing the default routes on time)

I did a pretty thorough rewrite of babeld (buggy tho) to handle loads
like this and ultimately got it to
scale past 4096 routes on a 400mhz mips box without falling over (bird
is much better here), but in the end I concluded a rewrite in rust or
go was needed with a sharp eye towards multi-cpu and smarter event
handling and I wasn't up for that.

The linux kernel itself didn't at the time, let you do a make before
break, either - you needed to do a
del/add on a better route, rather than a add/del. Some version of the
kernel fixed that (later), and I
think bird picked up that fix also.

> mjs@PWTW04-NNNN-STREET-ST-RTR:~$ show ip route summary
> IP routing table name is Default-IP-Routing-Table(0)
> IP routing table maximum-paths   : 8
> Total number of IPv4 routes      : 861
> Total number of IPv4 paths       : 861
> Route Source    Networks
> connected       9
> ospf            852
> Total           861
> FIB             852

Well, inject a few thousand routes with rtod and make popcorn on the cpu?

>
> I am slowly replacing all Ubnt routers at major sites with VyOS on DC powered Supermicro servers. So far, really liking VyOS. Example: https://www.supermicro.com/en/products/system/Mini-ITX/SYS-E300-9D-8CN8TP.cfm

I really liked the fast for a typist interface the vyos (and cisco)
and mikrotik have. Always loved how lotus 123 did things. To heck with
clicking.

UBNT really goofed when they abandoned edgeos and laid off everyone in
san jose. The stock price tho,
hasn't suffered.


>
> A bit more below.
>
>
> ---- On Thu, 27 Oct 2022 09:29:22 -0400 Herbert Wolverson via LibreQoS  wrote ---
>
>  > OSPF is a pretty stable protocol, so it doesn't change all that much (one of the reasons I like it is that it is supported everywhere, and works cross-vendor consistently). In answer to your questions:
>  >
>  > * With OSPFv3 you can push IPv6 routes. I've only used this a bit, but it worked fine last time we tried it.
>  > * You have a fair amount of flexibility on how hops talk to one another (configured on the interface). The default is still multicast, but "point-to-point" is just a unicast broadcast (and VERY fast route coalescing; it's our default),
>
> A few years ago I briefly looked at these config options but I never implemented anything. This morning I updated 2 routers to your "point-to-point" (I think...). Is anything more than the following needed on both routers (adjusting for ports, etc)?
>
> set interfaces ethernet eth1 ip ospf network point-to-point
>
>
>
> > "point-to-multipoint" can do the same for 1:many relationships (I don't use this one, I prefer one interface to one destination - even if the interfaces are VLANs. Makes traffic analysis a whole lot easier). There's also NBMA, which has you type in the addresses of adjacent neighbors (which feels like it defeats some of the point!) - but doesn't even broadcast.
>
> At major sites, we typically have 2 routers running in an active-active manner: router-1 has a.b.c.2, router-2 has a.b.c.3 and vrrp is used to float a.b.c.1 between both routers.
>
> Is "point-to-multpoint" required in this situation?
>
>
>
> > There was a time that NBMA was the most consistent over old Ubiquiti equipment, but these days the other modes work fine.
>  > * ECMP (Equal Cost Multipath) is awesome. If two routes to the destination sum out at equal costs, traffic is split equally between them. Traffic is split at layer 3 (on an address-port hash), so flows stay together. It's rock solid, we have it sharing traffic over two backhauls in a few places - and it handles one interface going down flawlessly (with either BFD, or point-to-point mode which coalesces so fast).
>  >
>  > Dijkstra's algorithm remains a very natural approach to mapping a graph (I've waxed lyrical about it in various book/article/blog posts over the years in the game-dev world), so I find it a very comfy way to model the underlying network. Very easy to reason "this will go that way".
>  >
>  > So what's bad about it?
>  >
>  > * ECMP is equal; if you have routes with different costs it will only use the lowest cost - it won't try and "mesh" some of the traffic in other directions. Likewise, if you have two equal routes - and one of them is running at a low capacity - you wind up only utilizing twice the capacity of the slowest link. You're often better off dropping the link over the degraded circuit (hence "carrier drop" features on various radios).
>  > * OSPF has no idea what the capacities of various links are. It'll use the shortest cost route, and leave the details up to the lower layers of the stack.
>  >
>  > There have been various attempts to integrate capacity into network design, and I've yet to see one that holds up well on a multi-vendor network. If you ask a Ubiquiti Rocket AC Gen2 its capacity, it'll often give you some nice big number - but it'll stall out far before that number. The Force 400C link we just put up doesn't even try to estimate its capacity. So multi-vendor mesh routing tends to be problematic, because the advertised capability of "we'll utilize everywhere you have capacity" tends to get snarled up in figuring out what the capacity is. I understand Teragraph is supposed to do better there. MPLS traffic engineering was originally announced as solving this one, too! I don't see a lot of  hope for capacity-aware routing protocols taking off, but I imagine we'll get a few new ones announced and then quietly forgotten as before.
>  >
>  > And lastly: once your network gets really big, OSPF tables can get too large - and you're stuck either dividing your OSPF zones and/or using some BGP in interior mode. You can mitigate this with some careful design.
>
> What is typically considered "too large" and how does "too large" type problems typically show up?
>
> Thanks
> Mark
>
>
>
>  >
>  > On Wed, Oct 26, 2022 at 5:35 PM dan dandenson@gmail.com> wrote:
>  > I have played with batman-adv quite a bit and there are some concepts in it I really like.  Not being shortest path for one, and rating a link quality instead of hard up/down.   I also like the layer2 model so it looks like a big switch.  It's very clean from an operational perspective as it behaves essentially like an MPLS/VPLS network administratively.
>  >
>  > What I think we're missing is the integration of network attributes and class of service.  For instance, user to 'internet' has 3 potential paths with each having these end-to-end latency, upload throughput, download throughput, and say 'quality' or packet loss.  Then having your QoS engine able to tag packets for how it perceives them to need routed and then have the routing engine pick routes based on availability.  So you might have a longer path that will suffer some on latency because of the hops and link type but has big bandwidth 'available' (ie large capacity and underused) so it should ask for that flow to take the underused high capacity (and yet still meets other criteria) path.  Something considered realtime might prefer that 700Mbps licenced path that has lower and more predictable latency and enough available capacity for the job. By encouraging high throughput needs to take paths with a lot of availability and some mechanism to prevent occilations from reroutes you could keep lower latency links less busy and get load balancing by a more intelligent choice.    You could also have some sort of reservation number tagged onto that packet to ask the intermediate hops to reduce their available amount.  If you were going to go all out on this and have devices that spoke this everywhere, you could put your shapers everywhere as well, getting that desired egrees shapping on both sides and letting the network sort of reserve a bit of bandwidth for each customer based on that.
>  >
>  > Of course this means scaling issues almost inherently because those 'available capacity' numbers and packet loss need to be communicated.  computationally intensive.
>  >
>  > batman-adv does this in a way with it's OGM/ELP system.  You can take a longer path through a batman-adv network because of a saturated link and it doesn't consider that saturated link 'down'.
>  >
>  > rflo was an interesting tech that did some of these once upon a time.
>  >
>  > Just thoughts.
>  > On Wed, Oct 26, 2022 at 3:38 PM Dave Taht via LibreQoS libreqos@lists.bufferbloat.net> wrote:
>  > On Wed, Oct 26, 2022 at 1:53 PM Herbert Wolverson via LibreQoS
>  >  libreqos@lists.bufferbloat.net> wrote:
>  >  >
>  >  > My name is Herbert, and I'm an OSPF addict... seriously, I love OSPF. Right down to stub sites, not-so-stubby sites, and isolating IP blocks within a site into "stub" nets and ensuring they are aggregated properly. I should probably go outside more...
>  >
>  >  haha.
>  >
>  >  My name is dave, and I think all routing protocols should have evolved
>  >  much better to elegantly meet the real world problems they were trying
>  >  to solve, than they have.
>  >
>  >  To avoid burying the lede, to what extent does OSPF still rely on
>  >  multicast? How well can it carry ipv6 now? What extensions are common
>  >  in the real WISP world?
>  >
>  >  BGP needs a few more napkins.
>  >
>  >  RIP was a VERY good start but we drew the wrong lessons from its
>  >  failures, and the super-duper-trendline towards centralized
>  >  controllers inherent in OSPF and ISIS that happened in the 90s that
>  >  doesn't scale anywhere near as I'd like.
>  >
>  >  I liked the rise of meshy 802.11 networks, I know the author of AODV
>  >  well (charlie perkins is arguably one of the fathers of mesh
>  >  networking, far too few have read his books from the 90s). And I've
>  >  been involved in the "battlemesh" group for many years with those
>  >  trying to make 'em work better on networks such as guifi,
>  >  wlan-slovinia, etc.
>  >
>  >  Backstory. Back in 07, in Nicaragua, I was (stupidly) trying to get
>  >  ipv6 to work over nanostation m2s or m5s I forget which, and the basic
>  >  option was to run two copies of the ospf daemon to manage 4 and 6
>  >  independently. I only had 32MB of memory and it didn't fit, so I
>  >  started looking for alternatives, found babel, corresponded with (and
>  >  frankly thoroughly annoyed) the author, and starting giving it a go.
>  >  It transported 4 and 6 in the same packets, was tiny, was
>  >  distance-vector (thus, I thought, more a match for bgp), and (to me)
>  >  most importantly, solved the ipv4 and ipv6 routing problems in the
>  >  same daemon at the same time, and actually fit into less memory than
>  >  ospf did. It was good enough it seemed, to deploy to a few hundred
>  >  routers without having to play major tricks with areas and stubs and
>  >  so on.
>  >
>  >  Babel is so simple that toke wrote a near complete implementation from
>  >  the spec, in python, during a string of extremely boring IETF
>  >  meetings, over the course of a week. He later took on the bird port.
>  >  Over the years we've wedged most (but not all) of the key features I
>  >  thought a meshy wireless routing protocol should have, with
>  >  implementations in a standalone daemon, bird, and FRR. (there was a
>  >  quagga port at one point too. I forget what happened to toke's python
>  >  version).
>  >
>  >  https://www.rfc-editor.org/rfc/rfc8966.html babel
>  >  https://arxiv.org/abs/1403.0445 source specific routing
>  >  https://datatracker.ietf.org/doc/rfc8967/ HMAC authentication
>  >  https://datatracker.ietf.org/doc/html/draft-ietf-babel-rtt-extension-00
>  >  RTT metric
>  >  https://datatracker.ietf.org/meeting/99/materials/slides-99-babel-unicast-hellos-00.pdf
>  >   unicast hellos
>  >
>  >  Missing is BFD support, and the slightest bit of traction outside of
>  >  the shrinking battlemesh communities.
>  >
>  >  Althea is using babel and fq_codel in their blockchain routing thing
>  >  (I reserve comment), and I don't know where else, besides as part of
>  >  wireguard tunnels, babel is being used today. But I'm rather
>  >  interested in how OSPF evolved since I last touched it, and what use
>  >  cases it is good at and fails at?
>  >
>  >
>  >  > On Wed, Oct 26, 2022 at 3:29 PM Dave Taht via LibreQoS libreqos@lists.bufferbloat.net> wrote:
>  >  >>
>  >  >> OK, since I'm getting such great updates on the state of the wisp
>  >  >> world, far more in a few days than I've had in 10 years... and btw, no
>  >  >> need to leap on dr science guy research questions like mine if you
>  >  >> have like, towers flooding or the phone ringing off the hook....
>  >  >>
>  >  >> What routing protocols are in use nowadays? BGP, yes, and it seems
>  >  >> ospf is popular?
>  >  >>
>  >  >> How about ISIS?
>  >  >>
>  >  >> I figure babel has zero traction or awareness despite being mandated
>  >  >> by the ietf homenet working group.
>  >  >>
>  >  >> Secondly, do you rely on BGP based on the edge router or use it in
>  >  >> software (frr? quagga? bird?). Using RPKI? Push FIBs anywhere? (route
>  >  >> 666 in particular)
>  >  >>
>  >  >> Similar question related to the IGP protocol in use, where do you rely
>  >  >> on it, vs all the tunnels you have, on what kinds of hardware?
>  >  >>
>  >  >> I note that robert at some point, somewhere, pointed out how fq_codel
>  >  >> saved his bacon when there was a major routing mishap (as there is no
>  >  >> congestion control in ospf), and I'd like to hear more of that story.
>  >  >>
>  >  >> BATMAN has been mentioned. There's other wireless protocols I've liked
>  >  >> - OLSR for example...
>  >  >>
>  >  >> Nobody knows what lies underneath many consumer wireless meshes
>  >  >> although it looks like 802.11s is a starting point, none, so far as I
>  >  >> know interoperate across brands.
>  >  >>
>  >  >> --
>  >  >> This song goes out to all the folk that thought Stadia would work:
>  >  >> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
>  >  >> Rip Van Winkle COO, TekLibre, LLC
>  >  >> _______________________________________________
>  >  >> LibreQoS mailing list
>  >  >> LibreQoS@lists.bufferbloat.net
>  >  >> https://lists.bufferbloat.net/listinfo/libreqos
>  >  >
>  >  > _______________________________________________
>  >  > LibreQoS mailing list
>  >  > LibreQoS@lists.bufferbloat.net
>  >  > https://lists.bufferbloat.net/listinfo/libreqos
>  >
>  >
>  >
>  >  --
>  >  This song goes out to all the folk that thought Stadia would work:
>  >  https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
>  >  Dave Täht CEO, TekLibre, LLC
>  >  _______________________________________________
>  >  LibreQoS mailing list
>  >  LibreQoS@lists.bufferbloat.net
>  >  https://lists.bufferbloat.net/listinfo/libreqos
>  > _______________________________________________
>  > LibreQoS mailing list
>  > LibreQoS@lists.bufferbloat.net
>  > https://lists.bufferbloat.net/listinfo/libreqos
>  >
> _______________________________________________
> LibreQoS mailing list
> LibreQoS@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/libreqos



-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
Dave Täht CEO, TekLibre, LLC

Re: [LibreQoS] routing protocols and daemons

[Juliusz Chroboczek]

> What I think we're missing is the integration of network attributes and class
> of service.  For instance, user to 'internet' has 3 potential paths with each
> having these end-to-end latency, upload throughput, download throughput, and
> say 'quality' or packet loss.  Then having your QoS engine able to tag packets
> for how it perceives them to need routed and then have the routing engine pick
> routes based on availability.

We did implement something like that in Babel:

  https://datatracker.ietf.org/doc/html/draft-chouasne-babel-tos-specific

However, we never merged the code into babeld, due to lack of interest
from our users.  It turns out that it is seldom the case that there is
enough path diversity to make ToS-specific routing worthwile.

-- Juliusz
<

Re: [LibreQoS] routing protocols and daemons

[Juliusz Chroboczek]

> found babel, corresponded with (and frankly thoroughly annoyed) the
> author,

Being said author, I can confirm that you did thoroughly annoy me.  But
then, you also made me think.

> Babel is so simple that toke wrote a near complete implementation from
> the spec, in python, during a string of extremely boring IETF
> meetings, over the course of a week. He later took on the bird port.

This is not correct.  Babel was first reimplemented in Python during two
nights during an IETF meeting by Markus Stenberg.  As to Toke, he did the
BIRD reimplementation in C during a Battlemesh meeting, and it took him
a whole four days.  I later did a minimal implementation in C, which
compiled to 20kB of x86 code.

> I forget what happened to toke's python version).

Markus Stenberg's.  It's still available, but fairly obsolete due to
advances in babeld and BIRD.

  https://github.com/fingon/pybabel

> Althea is using babel and fq_codel in their blockchain routing thing
> (I reserve comment), and I don't know where else, besides as part of
> wireguard tunnels, babel is being used today.

Now that Babel is no longer a legitimate research project, the main user
and main source of funding for Babel are Nexedi, who use it in their
distributed cloud

  https://www.nexedi.com/

But I agree with you, Dave, Babel did not take over the world.  The main
reason, I suspect, is that OSPF is very good, and that most people are
happy enough with it.

Notwithstanding that, we're still maintaining both the standalone babeld
and Toke's BIRD module, and we've been busy extending the protocol with
source-specific routing, with v4-via-v6 routing, with MAC protection.

-- Juliusz

Re: [LibreQoS] routing protocols and daemons

[Dave Taht]

On Fri, Oct 28, 2022 at 2:45 PM Juliusz Chroboczek <jch@irif.fr> wrote:
>
> > found babel, corresponded with (and frankly thoroughly annoyed) the
> > author,
>
> Being said author, I can confirm that you did thoroughly annoy me.  But
> then, you also made me think.

I'm glad, ultimately, we also drank, and I'll always remember (somewhat
blurrily) the day/night of our toasts to the builders of Notre-Dame.

earlier on in this thread, herbert listed his complaints about OSPF,
if you have a spare moment, have you ever had any deep thoughts about
it?

--snip snip--
Dijkstra's algorithm remains a very natural approach to mapping a
graph (I've waxed lyrical about it in various book/article/blog posts
over the years in the game-dev world), so I find it a very comfy way
to model the underlying network. Very easy to reason "this will go
that way".

So what's bad about it?

* ECMP is equal; if you have routes with different costs it will only
use the lowest cost - it won't try and "mesh" some of the traffic in
other directions. Likewise, if you have two equal routes - and one of
them is running at a low capacity - you wind up only utilizing twice
the capacity of the slowest link. You're often better off dropping the
link over the degraded circuit (hence "carrier drop" features on
various radios).
* OSPF has no idea what the capacities of various links are. It'll use
the shortest cost route, and leave the details up to the lower layers
of the stack.

There have been various attempts to integrate capacity into network
design, and I've yet to see one that holds up well on a multi-vendor
network. If you ask a Ubiquiti Rocket AC Gen2 its capacity, it'll
often give you some nice big number - but it'll stall out far before
that number. The Force 400C link we just put up doesn't even try to
estimate its capacity. So multi-vendor mesh routing tends to be
problematic, because the advertised capability of "we'll utilize
everywhere you have capacity" tends to get snarled up in figuring out
what the capacity is. I understand Teragraph is supposed to do better
there. MPLS traffic engineering was originally announced as solving
this one, too! I don't see a lot of  hope for capacity-aware routing
protocols taking off, but I imagine we'll get a few new ones announced
and then quietly forgotten as before.

And lastly: once your network gets really big, OSPF tables can get too
large - and you're stuck either dividing your OSPF zones and/or using
some BGP in interior mode. You can mitigate this with some careful
design.
-- unsnip --

I don't suppose you have ever had any ideas to how to improve things?


>
> > Babel is so simple that toke wrote a near complete implementation from
> > the spec, in python, during a string of extremely boring IETF
> > meetings, over the course of a week. He later took on the bird port.
>
> This is not correct.  Babel was first reimplemented in Python during two
> nights during an IETF meeting by Markus Stenberg.  As to Toke, he did the
> BIRD reimplementation in C during a Battlemesh meeting, and it took him
> a whole four days.  I later did a minimal implementation in C, which
> compiled to 20kB of x86 code.

You're working on galene, now, primarily? How's android treating you?

>
> > I forget what happened to toke's python version).
>
> Markus Stenberg's.  It's still available, but fairly obsolete due to
> advances in babeld and BIRD.
>
>   https://github.com/fingon/pybabel

yea!

> > Althea is using babel and fq_codel in their blockchain routing thing
> > (I reserve comment), and I don't know where else, besides as part of
> > wireguard tunnels, babel is being used today.
>
> Now that Babel is no longer a legitimate research project, the main user
> and main source of funding for Babel are Nexedi, who use it in their
> distributed cloud
>
>   https://www.nexedi.com/

If you can stomach it, I have a copy of althea's most current preso,
which I can send under separate cover. They scored 15 out of a 100 on
a funding round with it, but there babel and fq_codel were, with an
rtt metric on page 16...

At least they leave breadcrumbs back to where the actual good ideas
came from. Many others in web3 haven't.

>
> But I agree with you, Dave, Babel did not take over the world.  The main
> reason, I suspect, is that OSPF is very good, and that most people are
> happy enough with it.

Everything can be improved.

> Notwithstanding that, we're still maintaining both the standalone babeld
> and Toke's BIRD module, and we've been busy extending the protocol with
> source-specific routing, with v4-via-v6 routing, with MAC protection.

I am so in love with v4 via v6. It's just having to wait 6+ years for
places like mikrotik to ship it that I can't stand.

>
> -- Juliusz


--
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
Dave Täht CEO, TekLibre, LLC

Re: [LibreQoS] routing protocols and daemons

[Dave Taht]

Taking juliusz off the cc for a sec. He's the author of a lot of cool
stuff: https://www.rfc-editor.org/rfc/rfc8965 being one of them,
but I first got to know him via the polipo web proxy... Anyone ever use that?

I also thought his work on continuation passing C (CPC
https://www.irif.fr/~jch/research/ ) was way, way ahead of its time.
One of his students wrote the then-fastest torrent server... and you
can see reflections of this work in modern frameworks in tokio and
elsewhere.

I switched to his galene.org videoconferencing server early on in the
pandemic and have never looked back.




On Fri, Oct 28, 2022 at 5:01 PM Dave Taht <dave.taht@gmail.com> wrote:
>
> On Fri, Oct 28, 2022 at 2:45 PM Juliusz Chroboczek <jch@irif.fr> wrote:
> >
> > > found babel, corresponded with (and frankly thoroughly annoyed) the
> > > author,
> >
> > Being said author, I can confirm that you did thoroughly annoy me.  But
> > then, you also made me think.
>
> I'm glad, ultimately, we also drank, and I'll always remember (somewhat
> blurrily) the day/night of our toasts to the builders of Notre-Dame.
>
> earlier on in this thread, herbert listed his complaints about OSPF,
> if you have a spare moment, have you ever had any deep thoughts about
> it?
>
> --snip snip--
> Dijkstra's algorithm remains a very natural approach to mapping a
> graph (I've waxed lyrical about it in various book/article/blog posts
> over the years in the game-dev world), so I find it a very comfy way
> to model the underlying network. Very easy to reason "this will go
> that way".
>
> So what's bad about it?
>
> * ECMP is equal; if you have routes with different costs it will only
> use the lowest cost - it won't try and "mesh" some of the traffic in
> other directions. Likewise, if you have two equal routes - and one of
> them is running at a low capacity - you wind up only utilizing twice
> the capacity of the slowest link. You're often better off dropping the
> link over the degraded circuit (hence "carrier drop" features on
> various radios).
> * OSPF has no idea what the capacities of various links are. It'll use
> the shortest cost route, and leave the details up to the lower layers
> of the stack.
>
> There have been various attempts to integrate capacity into network
> design, and I've yet to see one that holds up well on a multi-vendor
> network. If you ask a Ubiquiti Rocket AC Gen2 its capacity, it'll
> often give you some nice big number - but it'll stall out far before
> that number. The Force 400C link we just put up doesn't even try to
> estimate its capacity. So multi-vendor mesh routing tends to be
> problematic, because the advertised capability of "we'll utilize
> everywhere you have capacity" tends to get snarled up in figuring out
> what the capacity is. I understand Teragraph is supposed to do better
> there. MPLS traffic engineering was originally announced as solving
> this one, too! I don't see a lot of  hope for capacity-aware routing
> protocols taking off, but I imagine we'll get a few new ones announced
> and then quietly forgotten as before.
>
> And lastly: once your network gets really big, OSPF tables can get too
> large - and you're stuck either dividing your OSPF zones and/or using
> some BGP in interior mode. You can mitigate this with some careful
> design.
> -- unsnip --
>
> I don't suppose you have ever had any ideas to how to improve things?
>
>
> >
> > > Babel is so simple that toke wrote a near complete implementation from
> > > the spec, in python, during a string of extremely boring IETF
> > > meetings, over the course of a week. He later took on the bird port.
> >
> > This is not correct.  Babel was first reimplemented in Python during two
> > nights during an IETF meeting by Markus Stenberg.  As to Toke, he did the
> > BIRD reimplementation in C during a Battlemesh meeting, and it took him
> > a whole four days.  I later did a minimal implementation in C, which
> > compiled to 20kB of x86 code.
>
> You're working on galene, now, primarily? How's android treating you?
>
> >
> > > I forget what happened to toke's python version).
> >
> > Markus Stenberg's.  It's still available, but fairly obsolete due to
> > advances in babeld and BIRD.
> >
> >   https://github.com/fingon/pybabel
>
> yea!
>
> > > Althea is using babel and fq_codel in their blockchain routing thing
> > > (I reserve comment), and I don't know where else, besides as part of
> > > wireguard tunnels, babel is being used today.
> >
> > Now that Babel is no longer a legitimate research project, the main user
> > and main source of funding for Babel are Nexedi, who use it in their
> > distributed cloud
> >
> >   https://www.nexedi.com/
>
> If you can stomach it, I have a copy of althea's most current preso,
> which I can send under separate cover. They scored 15 out of a 100 on
> a funding round with it, but there babel and fq_codel were, with an
> rtt metric on page 16...
>
> At least they leave breadcrumbs back to where the actual good ideas
> came from. Many others in web3 haven't.
>
> >
> > But I agree with you, Dave, Babel did not take over the world.  The main
> > reason, I suspect, is that OSPF is very good, and that most people are
> > happy enough with it.
>
> Everything can be improved.
>
> > Notwithstanding that, we're still maintaining both the standalone babeld
> > and Toke's BIRD module, and we've been busy extending the protocol with
> > source-specific routing, with v4-via-v6 routing, with MAC protection.
>
> I am so in love with v4 via v6. It's just having to wait 6+ years for
> places like mikrotik to ship it that I can't stand.
>
> >
> > -- Juliusz
>
>
> --
> This song goes out to all the folk that thought Stadia would work:
> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
> Dave Täht CEO, TekLibre, LLC



-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
Dave Täht CEO, TekLibre, LLC

Re: [LibreQoS] routing protocols and daemons

[Juliusz Chroboczek]

> our toasts to the builders of Notre-Dame.

...which then burnt down :-/

> Dijkstra's algorithm remains a very natural approach to mapping a
> graph

I'm not sure what that means.  Dijkstra's is a shortest path algorithm,
it's not in the business of mapping.  I guess the author meant that
representing a graph as an adjacency list (the LSDB) is natural, which is
certainly true, but in no way specific to OSPF.

> I don't suppose you have ever had any ideas to how to improve things?

Modern OSPF and IS-IS have pretty much reached a local optimum: all the
low-hanging fruit has been picked, I doubt there's much that can still be
done to improve them without a complete redesign.  Well-implemented OSPF
and IS-IS work beautifully in a well-administered network, any other
protocol is going to converge slower and give less visibility into the
network.

On the other hand, OSPF is extremely fragile in the presence of bad
implementation.  If two routers have the same id, OSPF is going to create
routing pathologies.  If a router corrupts its LSDB (for example due to
bad RAM), OSPF will create routing pathologies which will only go away
once the faulty LSA expires (30 minutes worst case).  If a router runs out
of memory for its LSDB, it needs to stop participating in the protocol,
lest it cause routing pathologies (IS-IS has the overload bit to deal with
this case, which causes the router to become a stub router).  Compare this
with distance vector, where a corrupt routing table entry will only
interfere with the traffic to that particular destination, and where it is
perfectly correct to run with a partial routing table.

OSPF also requires a skilled administrator.  Splitting a network into
areas without causing suboptimal routing takes significant skill, route
filtering can only happen on area boundaries, and there are multiple
different ways of redistributing routes into OSPF (external LSAs).

In my opinion, you want to be running OSPF in parts of your network that
are implemented with reliable gear and are managed by a competent
administrator, but you'll prefer a modern distance-vector protocol
(somebody mentioned Babel) where the hardware is cheap and the
administator is busy with other things.  Fortunately, due to the
flexibility of route redistribution in distance-vector protocols, you can
do both: a stable backbone using OSPF, and unadministered Babel bits at
the edges.

-- Juliusz

Re: [LibreQoS] routing protocols and daemons

[Herbert Wolverson]

[-- Attachment #1: Type: text/plain, Size: 4740 bytes --]

Juliusz, it's a pleasure to meet you. I've seen your name quite
often in the async/await world. Admittedly, usually in the detailed
"how things work" part - while I tend to be on the "teaching how to
use an implementation" side of things.

> > Dijkstra's algorithm remains a very natural approach to mapping a
> > graph
> I'm not sure what that means.  Dijkstra's is a shortest path algorithm,
> it's not in the business of mapping.  I guess the author meant that
> representing a graph as an adjacency list (the LSDB) is natural, which is
> certainly true, but in no way specific to OSPF.

Absolutely. Most of my development background is in game development,
I also do a lot of GIS. In both fields, Dijkstra's algorithm - and its
adaptations
(A*, weighted flow maps, etc.) refer to mapping in the spatial sense; and
converting
a map to a node graph (whether grid, waypoint, etc.) and then working with
cost-based adjacency (not raw adjacency) is a very natural way to
resolve the issue of "how do I get from X to Y" on a map. It's in no way
specific to OSPF (although specific adjacency cost specification was
one of many reasons OSPF outperforms RIP).

OSPF is where it is now because it's "good enough (for now)" and just
about everything supports it. Sure, an implementation that spits out bad
LSAs is going to break everything - you're going to get some pretty nasty
results from sending out broken destination-distance-vector data, too.
Garbage-in, garbage-out is one of the few truly universal rules! I agree,
though - I wouldn't hand out large-scale OSPF administration to the new
guy (although "here's the standard router config, plug in the numbers for
the locally attached networks here" does work).

I'd love to see good support for dynamic capacity analysis, unequal
cost multipath and similar. Babel looks very promising, but the chicken-egg
problem is very real; I can't put it to use until it's widely available, but
it won't become widely available until enough people put it to use. (It
seems like wireless vendors are busy trying to reinvent it at layer 2 with
proprietary meshing that doesn't talk to other proprietary meshing; ugh)




On Sat, Oct 29, 2022 at 4:15 AM Juliusz Chroboczek <jch@irif.fr> wrote:

> > our toasts to the builders of Notre-Dame.
>
> ...which then burnt down :-/
>
> > Dijkstra's algorithm remains a very natural approach to mapping a
> > graph
>
> I'm not sure what that means.  Dijkstra's is a shortest path algorithm,
> it's not in the business of mapping.  I guess the author meant that
> representing a graph as an adjacency list (the LSDB) is natural, which is
> certainly true, but in no way specific to OSPF.
>
> > I don't suppose you have ever had any ideas to how to improve things?
>
> Modern OSPF and IS-IS have pretty much reached a local optimum: all the
> low-hanging fruit has been picked, I doubt there's much that can still be
> done to improve them without a complete redesign.  Well-implemented OSPF
> and IS-IS work beautifully in a well-administered network, any other
> protocol is going to converge slower and give less visibility into the
> network.
>
> On the other hand, OSPF is extremely fragile in the presence of bad
> implementation.  If two routers have the same id, OSPF is going to create
> routing pathologies.  If a router corrupts its LSDB (for example due to
> bad RAM), OSPF will create routing pathologies which will only go away
> once the faulty LSA expires (30 minutes worst case).  If a router runs out
> of memory for its LSDB, it needs to stop participating in the protocol,
> lest it cause routing pathologies (IS-IS has the overload bit to deal with
> this case, which causes the router to become a stub router).  Compare this
> with distance vector, where a corrupt routing table entry will only
> interfere with the traffic to that particular destination, and where it is
> perfectly correct to run with a partial routing table.
>
> OSPF also requires a skilled administrator.  Splitting a network into
> areas without causing suboptimal routing takes significant skill, route
> filtering can only happen on area boundaries, and there are multiple
> different ways of redistributing routes into OSPF (external LSAs).
>
> In my opinion, you want to be running OSPF in parts of your network that
> are implemented with reliable gear and are managed by a competent
> administrator, but you'll prefer a modern distance-vector protocol
> (somebody mentioned Babel) where the hardware is cheap and the
> administator is busy with other things.  Fortunately, due to the
> flexibility of route redistribution in distance-vector protocols, you can
> do both: a stable backbone using OSPF, and unadministered Babel bits at
> the edges.
>
> -- Juliusz
>

[-- Attachment #2: Type: text/html, Size: 6229 bytes --]

Re: [LibreQoS] routing protocols and daemons

[dan]

[-- Attachment #1: Type: text/plain, Size: 6576 bytes --]

Keep in mind that often (wisps:  mikrotik, ubiquiti) run gear that often
have mediocre implementations of OSPF (and other protocols) and the
products are made to be as cheap as possible so hardware issues are more
likely than an enterprise vendor.

I field a lot of OSPF questions in support forums and they usually come
down to automatic IDs being set incorrectly because admin didn't set
them.... or the big ticket item being MTU.   MTU on the router's interface,
MTU issues on the radio links, or radios that say one MTU and aren't
passing it for some reason.  Also attempts to use broadcast and that being
unreliable over wireless networks.  Those things kinda make OSPF 'finicky'.

I would very much prefer ISIS for a number of reasons, mainly around
simplicity and reduction of finicky issues and dedicated PTP subnets and
configs etc.  ISIS is not available on mikrotik or ubiquiti products so
there you go.

Ultimately though, these are just shortest hop path builders and need some
other kit on top of them to do any sort of traffic engineering or load
balancing.  ECMP doesn't work for me, and doesn't work for a lot of wisps
so much so that after they've spent the time and effort to implement, they
undo it because different link characteristics make ECMP wreck their
customer's experience.  The 'E' for equal also is a PITA, requiring you to
start stacking up VLANs for example to get a ratio of 'ECMP' across
disimmilar links.

On Sat, Oct 29, 2022 at 7:48 AM Herbert Wolverson via LibreQoS <
libreqos@lists.bufferbloat.net> wrote:

> Juliusz, it's a pleasure to meet you. I've seen your name quite
> often in the async/await world. Admittedly, usually in the detailed
> "how things work" part - while I tend to be on the "teaching how to
> use an implementation" side of things.
>
> > > Dijkstra's algorithm remains a very natural approach to mapping a
> > > graph
> > I'm not sure what that means.  Dijkstra's is a shortest path algorithm,
> > it's not in the business of mapping.  I guess the author meant that
> > representing a graph as an adjacency list (the LSDB) is natural, which is
> > certainly true, but in no way specific to OSPF.
>
> Absolutely. Most of my development background is in game development,
> I also do a lot of GIS. In both fields, Dijkstra's algorithm - and its
> adaptations
> (A*, weighted flow maps, etc.) refer to mapping in the spatial sense; and
> converting
> a map to a node graph (whether grid, waypoint, etc.) and then working with
> cost-based adjacency (not raw adjacency) is a very natural way to
> resolve the issue of "how do I get from X to Y" on a map. It's in no way
> specific to OSPF (although specific adjacency cost specification was
> one of many reasons OSPF outperforms RIP).
>
> OSPF is where it is now because it's "good enough (for now)" and just
> about everything supports it. Sure, an implementation that spits out bad
> LSAs is going to break everything - you're going to get some pretty nasty
> results from sending out broken destination-distance-vector data, too.
> Garbage-in, garbage-out is one of the few truly universal rules! I agree,
> though - I wouldn't hand out large-scale OSPF administration to the new
> guy (although "here's the standard router config, plug in the numbers for
> the locally attached networks here" does work).
>
> I'd love to see good support for dynamic capacity analysis, unequal
> cost multipath and similar. Babel looks very promising, but the chicken-egg
> problem is very real; I can't put it to use until it's widely available,
> but
> it won't become widely available until enough people put it to use. (It
> seems like wireless vendors are busy trying to reinvent it at layer 2 with
> proprietary meshing that doesn't talk to other proprietary meshing; ugh)
>
>
>
>
> On Sat, Oct 29, 2022 at 4:15 AM Juliusz Chroboczek <jch@irif.fr> wrote:
>
>> > our toasts to the builders of Notre-Dame.
>>
>> ...which then burnt down :-/
>>
>> > Dijkstra's algorithm remains a very natural approach to mapping a
>> > graph
>>
>> I'm not sure what that means.  Dijkstra's is a shortest path algorithm,
>> it's not in the business of mapping.  I guess the author meant that
>> representing a graph as an adjacency list (the LSDB) is natural, which is
>> certainly true, but in no way specific to OSPF.
>>
>> > I don't suppose you have ever had any ideas to how to improve things?
>>
>> Modern OSPF and IS-IS have pretty much reached a local optimum: all the
>> low-hanging fruit has been picked, I doubt there's much that can still be
>> done to improve them without a complete redesign.  Well-implemented OSPF
>> and IS-IS work beautifully in a well-administered network, any other
>> protocol is going to converge slower and give less visibility into the
>> network.
>>
>> On the other hand, OSPF is extremely fragile in the presence of bad
>> implementation.  If two routers have the same id, OSPF is going to create
>> routing pathologies.  If a router corrupts its LSDB (for example due to
>> bad RAM), OSPF will create routing pathologies which will only go away
>> once the faulty LSA expires (30 minutes worst case).  If a router runs out
>> of memory for its LSDB, it needs to stop participating in the protocol,
>> lest it cause routing pathologies (IS-IS has the overload bit to deal with
>> this case, which causes the router to become a stub router).  Compare this
>> with distance vector, where a corrupt routing table entry will only
>> interfere with the traffic to that particular destination, and where it is
>> perfectly correct to run with a partial routing table.
>>
>> OSPF also requires a skilled administrator.  Splitting a network into
>> areas without causing suboptimal routing takes significant skill, route
>> filtering can only happen on area boundaries, and there are multiple
>> different ways of redistributing routes into OSPF (external LSAs).
>>
>> In my opinion, you want to be running OSPF in parts of your network that
>> are implemented with reliable gear and are managed by a competent
>> administrator, but you'll prefer a modern distance-vector protocol
>> (somebody mentioned Babel) where the hardware is cheap and the
>> administator is busy with other things.  Fortunately, due to the
>> flexibility of route redistribution in distance-vector protocols, you can
>> do both: a stable backbone using OSPF, and unadministered Babel bits at
>> the edges.
>>
>> -- Juliusz
>>
> _______________________________________________
> LibreQoS mailing list
> LibreQoS@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/libreqos
>

[-- Attachment #2: Type: text/html, Size: 8257 bytes --]

Re: [LibreQoS] routing protocols and daemons

[Juliusz Chroboczek]

> Ultimately though, these are just shortest hop path builders and need some
> other kit on top of them to do any sort of traffic engineering or load
> balancing.  ECMP doesn't work for me, and doesn't work for a lot of wisps

Perhaps one of you could explain what kind of traffic engineering and load
balancing you'd like to have?  Please don't try to be realistic, try to
describe the pony you wish you had.

-- Juliusz

Re: [LibreQoS] routing protocols and daemons

[dan]

[-- Attachment #1: Type: text/plain, Size: 4024 bytes --]

I wish I had a fancy pony with:

Routing based on diffserv.

'Hitless' adjustments to path diffserv cost.  ie, if a link has a cost
change, propagate that out without dumping routes.

link cost with metrics matching diffserv model.
ie, if diffserv 4 then a link might have:
a cost for the hop
a cost of it's expected latency + it's measured latency
a cost for it's fixed/set throughput with some method to modify this
programmatically (say radio modulation changes, admins can run scripts to
adjust this)
Above link cost, have a router cost.  So if a router is considered to have
a throughput limit of 1.5Gbps aggregate, then it might have a cost that
varies based on throughput or cpu use or something like that.  So we can
route around saturated sites.  This might also make the network be more
ad-hoc, lots of sites might try to jam data through a central hub site but
when it get's saturated, other paths might start looking good.
type of service cost.  This might be an IP map or something, but for
operators to classify business vs residential vs enterprise services.
and a reservation cost which may be a bit hard to describe, but something
along the lines of 'if a service is being used up by a higher class of
service (enterprise...) then other routers should know to try to route
around that side.

obviously, some method to slow adjustments and ramp up/down on various
factors.  Maybe some method to do some reservations with rolling updates to
keep things from oscillating or routes from flapping.  If a path gets
hammered so latency spikes, back it off and then creep up slowly.
 Probably configurable on each link.  An AF60LR for example has some more
or less fixed modulation levels so when it starts bucking, it's going to
modulate down super fast.  Maybe the cost for throughput has some steps
built in so it falls back to pre-defined levels quickly.  Whilte AF5XHD is
going to be more a noise issue, new noise might reduce modulations but a
bit so the fixed numbers put in might need backed down.

Then on the client side or the client's PoP it should discover it's paths
for each diffserv to gateways.  This could be pre-mapping 1st, 2nd, 3rd
next-hops for each diffserv to each gateway for example, so failover for
link downs is pretty quick.

BFD capabilities, preferably built in vs separate BFD process.

'Route Oracles'.  Routers that advertise that they'll store the router's
cost tables for all other routers to subscribe to.  Would allow routers to
evaluate which other routers are even remotely in their path and then watch
for cost changes.  Also give operators access to routing maps to use for
crazy shit like LibreQoS tree building.   or show links that are
underperforming vs stated fixed values etc.

Finally, before I make this wish list too long, REDUNDANT packets.  For
instance, maybe we want to tag, for a specific customer or customer class,
that anything that looks realtime should have a FEC type packet sent and
the most different but usable path possible to be re-assembled on the other
side.  This might even be a link that is described as a 'dont use this for
anything but FEC except last resort route, and still only use it for real
time packets' so you could strap in an LTE or Starlink type link.  The FEC
packets are smaller, say 5-10% adjustable, so you don't need to entirely
match the capacity, just to catch packets that get lost in a re-route due
to link packet loss.

ok, that's enough for now.  You said I could have a pony, I asked for a
clydesdale.

On Sat, Oct 29, 2022 at 4:18 PM Juliusz Chroboczek <jch@irif.fr> wrote:

> > Ultimately though, these are just shortest hop path builders and need
> some
> > other kit on top of them to do any sort of traffic engineering or load
> > balancing.  ECMP doesn't work for me, and doesn't work for a lot of wisps
>
> Perhaps one of you could explain what kind of traffic engineering and load
> balancing you'd like to have?  Please don't try to be realistic, try to
> describe the pony you wish you had.
>
> -- Juliusz
>

[-- Attachment #2: Type: text/html, Size: 4534 bytes --]

Re: [LibreQoS] routing protocols and daemons

[Juliusz Chroboczek]

> I wish I had a fancy pony with:

That's not a pony.  That's a whole herd.

> Routing based on diffserv.

That was originally supported by OSPFv2, but was removed from later
versions of the specification, to be replaced by multi-topology routing
(RFC 4915).  It's a little costly if you only have a small number of
ToS-specific routes, but I'm told it works well.

In Babel, we defined and implemented an extension for ToS-specific
routing, but found out there was no interest, so we dropped it.

  https://datatracker.ietf.org/doc/html/draft-chouasne-babel-tos-specific

> 'Hitless' adjustments to path diffserv cost.  ie, if a link has a cost change,
> propagate that out without dumping routes.

I'm not sure what you mean.

> link cost with metrics matching diffserv model.

[...]

> with some method to modify this programmatically

It's not sound engineering to implement the whole complexity of DiffServ
within the routing daemon.  On the other hand, one could envision, as you
suggest, a DiffServ daemon that dynamically tweaks route metrics.

Babeld has a simple IPC interface that allows tweaking the configuration
of the daemon at runtime, and I'd be interested in working with someone
who implements such a control daemon.

> but something along the lines of 'if a service is being used up by
> a higher class of service (enterprise...) then other routers should know
> to try to route around that side.

Again, that's not something that's likely to make it into routing
software, but it's something that could conceivably be implemented in
a separate daemon.

> Then on the client side or the client's PoP it should discover it's paths for
> each diffserv to gateways.  This could be pre-mapping 1st, 2nd, 3rd next-hops
> for each diffserv to each gateway for example, so failover for link downs is
> pretty quick.

Modern distance vector protocols do that.  For example, EIGRP maintains
three next-hops for each destination, while Babel maintains all next hops
to a given destination, and only discards them when it runs low on memory
(which never happens in my experience).

> BFD capabilities, preferably built in vs separate BFD process.

I'd have no objection to implementing BFD if there's user demand.
However, since Babel can do Hellos up to 100 times per second, our users
are not exactly clamouring for BFD.

(BFD is useful with OSPF, where Hellos are overloaded and therefore huge
data structures.  Not as much with Babel, where Hellos are tiny.  See

  https://www.rfc-editor.org/rfc/rfc8966.html#name-neighbour-acquisition

for details.)

> 'Route Oracles'.  Routers that advertise that they'll store the router's cost
> tables for all other routers to subscribe to.

I'm not sure what you mean.

> Finally, before I make this wish list too long, REDUNDANT packets.

That's a data plane feature, and therefore has little to do with the
routing protocol.  A fun little project, to be sure, the difficulty lies
in determining the right amount of FEC, which requires very accurate loss
statistics (you need to distinguish between random loss and burst loss,
and tune the amount of FEC for the former while carefully ignoring the
latter).  Hence, I suspect that it's better to do that at the application
layer, e.g. in the videoconferencing software.

-- Juliusz

Re: [LibreQoS] routing protocols and daemons

[Juliusz Chroboczek]

> OSPF is where it is now because it's "good enough (for now)"

It is very good.

> Sure, an implementation that spits out bad LSAs is going to break
> everything - you're going to get some pretty nasty results from sending
> out broken destination-distance-vector data, too.

I claim that in DV incorrect data does not have as much of an effect as in
LS, and that it gets cleared faster.  I'm not able to formally quantify
the effect.

> (It seems like wireless vendors are busy trying to reinvent it at layer
> 2 with proprietary meshing that doesn't talk to other proprietary
> meshing; ugh)

Uh-huh.

-- Juliusz