[Bloat] speedtest-cli on multihomed gateway

[Kenneth Porter]

On 2/3/2023 4:54 AM, Michael Richardson via Bloat wrote:
> A new network namespace would certainly work, but it may be unmanageable
> overkill.
>
> What you probably need are policy-based routes, which you can establish
> statically and then --source ought to work.

That's exactly what turned out to be the issue. I'd already planned to 
implement it to make some clients use the alternate ISP, but didn't 
realize it applies to packets with source address set to that interface. 
I'd read through a few tutorials and tested that and it worked!

> I put these into "up" statements into my /etc/network/interfaes, but you say
> you are running RHEL... I'm sure that there is a netplan way.
> This also means that if you have a monitoring system elsewhere (smokeping or
> something), and you ping each interface, then it will reply on that
> interface.

That turned out to be harder to figure out. RHEL 8 is a transition from 
traditional network scripts in /etc/sysconfig/network-scripts to 
NetworkManager connections. So the RH NM has to be able to understand 
the old files. I'm used to editing files with a text editor to make 
changes, but the RHEL docs don't expose the NM files and require one to 
use a manager program (nmcli) to make changes. I have no idea where my 
rules went when I changed them with nmcli. I do see the new 
per-interface route tables in the old scripts location. RHEL 9 moves 
over completely to NM, so I'll need to figure out how to migrate the 
config when I upgrade.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/configuring-policy-based-routing-to-define-alternative-routes_configuring-and-managing-networking

BTW, I noticed that the numeric routing table IDs are 32-bit, but the 
reserved IDs are from 0-255. Some online tutorials specify that custom 
tables need to be in the 8-bit range. I suspect that the earliest 
implementation used a single byte for the table ID and it widened in 
later kernels. The example number used in the above documentation is 
5000. So I'm using 2000 and 4000 for eno2 and eno4. (eno3 is the default 
link and eno1 is the LAN with an explicit global rule.)