[Cake] CAKE host isolation modes with NAT - two routers

[Cake] CAKE host isolation modes with NAT - two routers

[Nils Andreas Svee]

Hi folks

Currently my setup looks something like this: LAN <-> EdgeRouter <->
WireGuard <-> VPS <-> Internet.

CAKE for upstream is running on the EdgeRouter and downstream on the
VPS.

The public IPs are all on the VPS per today, so that the host isolation
can do its job with NAT enabled.

Ideally I'd like to route the public IPs to each endpoint and handle
NAT-ing there, but then I'd obviously lose the ability to do proper
host isolation.

Now, I've been toying with the idea of using an userspace application
to extract conntrack information, to let the VPS know which host hash
it should use.

I might be way of here, but I'm thinking of using NFQUEUE to mark new
flows based on information from the EdgeRouter, and let tc filters set
the host hash based on that mark. For performance purposes only send
unmarked flows to NFQUEUE.

I realise this is kinda overkill, but it might we a fun weekend
project.

-- 
Best Regards,
Nils

Re: [Cake] CAKE host isolation modes with NAT - two routers

[John Sager]

[-- Attachment #1: Type: text/plain, Size: 1696 bytes --]

I did something similar some years ago in an attempt to divine video servers (eg YouTube) from their TLS certificates in Https connections to mark the connection appropriately. The nfqueue stuff worked beautifully, the cert stuff less so, so I abandoned it. With the latest TLS version the cert stuff is no longer visible anyway.

There is a Python binding to libnetfilter_queue  which might make it easier to play quickly.

regards,
John


On 20 May 2021 17:07:43 BST, Nils Andreas Svee <me@lochnair.net> wrote:
>Hi folks
>
>Currently my setup looks something like this: LAN <-> EdgeRouter <->
>WireGuard <-> VPS <-> Internet.
>
>CAKE for upstream is running on the EdgeRouter and downstream on the
>VPS.
>
>The public IPs are all on the VPS per today, so that the host isolation
>can do its job with NAT enabled.
>
>Ideally I'd like to route the public IPs to each endpoint and handle
>NAT-ing there, but then I'd obviously lose the ability to do proper
>host isolation.
>
>Now, I've been toying with the idea of using an userspace application
>to extract conntrack information, to let the VPS know which host hash
>it should use.
>
>I might be way of here, but I'm thinking of using NFQUEUE to mark new
>flows based on information from the EdgeRouter, and let tc filters set
>the host hash based on that mark. For performance purposes only send
>unmarked flows to NFQUEUE.
>
>I realise this is kinda overkill, but it might we a fun weekend
>project.
>
>-- 
>Best Regards,
>Nils
>
>_______________________________________________
>Cake mailing list
>Cake@lists.bufferbloat.net
>https://lists.bufferbloat.net/listinfo/cake

-- 
Sent from the Aether.

[-- Attachment #2: Type: text/html, Size: 1830 bytes --]

Re: [Cake] CAKE host isolation modes with NAT - two routers

[Nils Andreas Svee]

For the time being you _can_ still see what domain a users connecting
to over TLS 1.3 (assuming ESNI isn't used). I wrote a iptables module
doing just that a few years ago [1]. I also toyed with a nfqueue
version a couple years back written in Go [2]. Of course, whenever ESNI
becomes the norm they're both useless.

On-topic: So far I'm thinking I'll have to add one tc filter per host
to get proper isolation. Not sure if there's a big enough performance
impact by adding a filter per host at boot time that I should add these
dynamically when new hosts show up.

I don't know tc all that well, but I imagine this'll do it:
> tc filter add dev eth0 parent 1: handle <fwmark> fw classid
<fwmark>:0

[1]: https://github.com/Lochnair/xt_tls
[2]: https://github.com/Lochnair/nfq-tls

-- 
Best Regards,
Nils

On Fri, 2021-05-21 at 16:51 +0100, John Sager wrote:
> I did something similar some years ago in an attempt to divine video
> servers (eg YouTube) from their TLS certificates in Https connections
> to mark the connection appropriately. The nfqueue stuff worked
> beautifully, the cert stuff less so, so I abandoned it. With the latest
> TLS version the cert stuff is no longer visible anyway.
> 
> There is a Python binding to libnetfilter_queue which might make it
> easier to play quickly.
> 
> regards,
> John
> 
> 
> On 20 May 2021 17:07:43 BST, Nils Andreas Svee <me@lochnair.net> wrote:
> > Hi folks
> > 
> > Currently my setup looks something like this: LAN <-> EdgeRouter <->
> > WireGuard <-> VPS <-> Internet.
> > 
> > CAKE for upstream is running on the EdgeRouter and downstream on the
> > VPS.
> > 
> > The public IPs are all on the VPS per today, so that the host
> > isolation
> > can do its job with NAT enabled.
> > 
> > Ideally I'd like to route the public IPs to each endpoint and handle
> > NAT-ing there, but then I'd obviously lose the ability to do proper
> > host isolation.
> > 
> > Now, I've been toying with the idea of using an userspace application
> > to extract conntrack information, to let the VPS know which host hash
> > it should use.
> > 
> > I might be way of here, but I'm thinking of using NFQUEUE to mark new
> > flows based on information from the EdgeRouter, and let tc filters
> > set
> > the host hash based on that mark. For performance purposes only send
> > unmarked flows to NFQUEUE.
> > 
> > I realise this is kinda overkill, but it might we a fun weekend
> > project.
> 
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake