[Bloat] speedtest-cli on multihomed gateway

[Bloat] speedtest-cli on multihomed gateway

[Kenneth Porter]

I'm trying to get my head around how to run speedtest-cli on a Rocky 8
box (RHEL8 respin) with LAN and three WAN connections. I want to run
speedtest-cli on all the WAN links to test each, and run the tests
from a timer to watch the speeds throughout the day (eg. a symlink to
a test script in /etc/cron.hourly or a systemd timer unit).
speedtest-cli accepts a --source option but it wants to always route
through the WAN link set to the default route when I specify the
addresses of the other two. From googling around it looks like I want
to run it in a "network namespace" but I haven't figured out how to
make that work.

I found this stackexchange answer on the namespace basics but I think
I need a few more commands to actually make an app usable in the
namespace by ifup'ing the interface, routing, and DNS.

https://unix.stackexchange.com/questions/234583/routing-on-per-application-basis

Hardware setup:

eno1: LAN, 10.96.0.64, default route for LAN
eno2: WAN1, 172.24.96.xxx, to ATT fiber gateway at 172.24.96.1
eno3: WAN2, 172.24.69.xxx, to Xfinity gateway at 172.24.69.1
eno4: WAN3, 172.24.0.xxx, to Xfinity gateway at 172.24.0.1

What I'm trying:

# ip netns add comcast-1
# ip link set eno4 netns comcast-1
# ip netns exec comcast-1 speedtest-cli
Retrieving speedtest.net configuration...
Cannot retrieve speedtest configuration
ERROR: <urlopen error [Errno -2] Name or service not known>

At this point I'm not sure what I need to do to make the network
namespace usable.

Re: [Bloat] speedtest-cli on multihomed gateway

[Paul Tagliamonte]

I wasn't going to reply, since I figured others would get here first
with more constructive notes; but since I don't see any, here's some
pointers, but alas, not anything concrete; a lot is still left as an
exercise to the reader. Sorry about that.

Sorry this is a bit long, i'm going to try to make this as helpful as I
can.

On Wed, Feb 01, 2023 at 12:20:56PM -0800, Kenneth Porter via Bloat wrote:
> # ip netns add comcast-1
> # ip link set eno4 netns comcast-1
> # ip netns exec comcast-1 speedtest-cli

Network Namespaces work like the other Linux namespaces (ok fine, not
*all* of them, but most of them) -- when you create a new one, you're in
an entirely different universe that is not connected to your existing
world. This world doesn't talk to other namespaces, unless you use
something like a `ip link add link-name0 type veth peer link-name1`, and
move one end of the veth "wormhole" into the network namespace, leave
the other out of it, and use it to bridge. This is fundementally how
things like Docker work.

All this to say, by moving eno4 to netns comcast-1, the host won't be
able to meaningfully use it anymore. This is likely not what you want
(an outage during the speedtest), so my guess is you'd want a network
namespace, veth pair with one end on the host, one end in the network
namespace, and a bridge to join the veth0 to comcast-1.

Let's create two network veth interfaces (it's like a pipe if you've
never used one directly before), on-host0 which will live on my host's
network namespace, and `in-ns0` which will be moved into the network
namespace once we set it up. These are bad names I'm picking to make
this super explicit.

prompts like `host$` are done via bash on my host, perhaps with sudo.
Prompts like `ns$` are done via bash inside a network namespace. Also,
perhaps with sudo.

```
host$ ip link add on-host0 type veth peer in-ns0
```

Note `ip link`. It'll show `on-host0` and `in-ns0` in the host
namespace, and also down.

Let's add a network namespace.

```
host$ sudo ip netns add bloat
host$ sudo ip netns exec bloat $(which bash)
ns$ ip link
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
ns$
```

```
host$ ip link set in-ns0 netns bloat
host$ 
```

```
ns$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
17: in-ns0@if18: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether fe:a8:45:16:c5:5a brd ff:ff:ff:ff:ff:ff link-netnsid 0
```

You'll note that I currently have no interfaces, *none* of the host
interfaces show -- and even `lo` is down. Let's fix that.

```
ns$ ip link set lo up
ns$ ip link set in-ns0 up
ns$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
17: in-ns0@if18: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default qlen 1000
    link/ether fe:a8:45:16:c5:5a brd ff:ff:ff:ff:ff:ff link-netnsid 0
```

You'll note `in-ns0` is still LAYERDOWN, this is because the host veth
end is still down.

The last big thing is because this is its own network namespace, you'll
need to add an IP address to the veth device depending on configuration
(e.g., you could NAT from the veth on-host0 -> comcast-1, bridge it, or
whatever your setup and upstream will allow), and then you can set up
your routes as required.

```
ns$ ip route
ns$ 
```

From here on out, I suspect you've got it, given you're juggling 4 WAN
ports, I guess you can fill in the rest of the blanks here, it's just a
few routes/interface configurations on the host and inside your netns.

FWIW, at some point this becomes almost exactly like a Docker container
(or podman or what have you) without the nicities -- Docker and other
container daemons/launchers are usually capable of automating this
bridge+veth+netns dance for you. If that's not an option due to the
platform, doing it by hand is doable, but requires a bit of work to
debug.

> At this point I'm not sure what I need to do to make the network
> namespace usable.

Best of luck,
   paultag

-- 
:wq

Re: [Bloat] speedtest-cli on multihomed gateway

[Paul Tagliamonte]

On Thu, Feb 02, 2023 at 10:15:23AM -0500, Paul Tagliamonte wrote:
> [... stuff ...]

Additional note: the nonfree / closed source speedtest cli binary downloadable
at no cost at https://www.speedtest.net/apps/cli contains a `-i` flag for
binding to a specific interface, which means you don't have to faff
around with network namespacing :)

   paultag

-- 
:wq

Re: [Bloat] speedtest-cli on multihomed gateway

[Michael Richardson]

[-- Attachment #1: Type: text/plain, Size: 1126 bytes --]


Kenneth Porter via Bloat <bloat@lists.bufferbloat.net> wrote:
    > script in /etc/cron.hourly or a systemd timer unit).  speedtest-cli
    > accepts a --source option but it wants to always route through the WAN
    > link set to the default route when I specify the addresses of the other
    > two. From googling around it looks like I want to run it in a "network
    > namespace" but I haven't figured out how to make that work.

A new network namespace would certainly work, but it may be unmanageable
overkill.

What you probably need are policy-based routes, which you can establish
statically and then --source ought to work.

ip route now has a "from" option, but it doesn't always work in my
experience.  The old way is to do:

  ip rule add from A.B.C.D/32    table 123
  ip route add default via 192.0.1.2 table 123

I put these into "up" statements into my /etc/network/interfaes, but you say
you are running RHEL... I'm sure that there is a netplan way.
This also means that if you have a monitoring system elsewhere (smokeping or
something), and you ping each interface, then it will reply on that
interface.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [Bloat] speedtest-cli on multihomed gateway

[Kenneth Porter]

On 2/3/2023 4:54 AM, Michael Richardson via Bloat wrote:
> A new network namespace would certainly work, but it may be unmanageable
> overkill.
>
> What you probably need are policy-based routes, which you can establish
> statically and then --source ought to work.

That's exactly what turned out to be the issue. I'd already planned to 
implement it to make some clients use the alternate ISP, but didn't 
realize it applies to packets with source address set to that interface. 
I'd read through a few tutorials and tested that and it worked!

> I put these into "up" statements into my /etc/network/interfaes, but you say
> you are running RHEL... I'm sure that there is a netplan way.
> This also means that if you have a monitoring system elsewhere (smokeping or
> something), and you ping each interface, then it will reply on that
> interface.

That turned out to be harder to figure out. RHEL 8 is a transition from 
traditional network scripts in /etc/sysconfig/network-scripts to 
NetworkManager connections. So the RH NM has to be able to understand 
the old files. I'm used to editing files with a text editor to make 
changes, but the RHEL docs don't expose the NM files and require one to 
use a manager program (nmcli) to make changes. I have no idea where my 
rules went when I changed them with nmcli. I do see the new 
per-interface route tables in the old scripts location. RHEL 9 moves 
over completely to NM, so I'll need to figure out how to migrate the 
config when I upgrade.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/configuring-policy-based-routing-to-define-alternative-routes_configuring-and-managing-networking

BTW, I noticed that the numeric routing table IDs are 32-bit, but the 
reserved IDs are from 0-255. Some online tutorials specify that custom 
tables need to be in the 8-bit range. I suspect that the earliest 
implementation used a single byte for the table ID and it widened in 
later kernels. The example number used in the above documentation is 
5000. So I'm using 2000 and 4000 for eno2 and eno4. (eno3 is the default 
link and eno1 is the LAN with an explicit global rule.)