[Cake] Advantages to tightly tuning latency

[Maxime Bizon]

On Thursday 23 Apr 2020 à 18:42:11 (+0200), Toke Høiland-Jørgensen wrote:

> Didn't make it in until 5.5, unfortunately... :(
> 
> I can try to produce a patch that you can manually apply on top of 5.4
> if you're interested?

I could do it, but the thing I'm more worried about is the lack of
test coverage from everyone else.

> Anyhow, my larger point was that we really do want to enable such use
> cases for XDP; but we are lacking the details of what exactly is missing
> before we can get to something that's useful / deployable. So any
> details you could share about what feature set you are supporting in
> your own 'fast path' implementation would be really helpful. As would
> details about the hardware platform you are using. You can send them
> off-list if you don't want to make it public, of course :)

there is no hardware specific feature used, it's all software

imagine this "simple" setup, pretty much what anyone's home router is
doing:

<br0> with <eth0> + <wlan0> inside, private IPv4 address
<wan0.vlan> with IPv6, vlan interface over <wan0>
<map0> with IPv4, MAP-E tunnel over <wan0.vlan>

then:
  - IPv6 routing between <br0> and <wan0.vlan>
  - IPv4 routing + NAT between <br0> and <map0>

iptables would be filled with usual rules, per interface ALLOW rules
in FORWARD chain, DNAT rules in PREROUTING to access LAN from WAN...

and then you want this to be fast :)

What we do is build a "flow" table on top of conntrack, so with a
single lookup we find the flow, the destination interface, and what
modifications to apply to the packet (L3 address to change, encap to
add/remove, etc etc)

Then we do this lookup more or less early in RX path, on our oldest
platform we even had to do this from the ethernet driver, and do TX
from there too, skipping qdisc layer and allowing cache maintenance
hacks (partial invalidation and wback)


nftable with flowtables seems to be have developped something that
could replace our flow cache, but I'm not sure if it can handle our
tunneling scheme yet. It even has a notion of offloaded flow for
hardware that can support it.

If you add an XDP offload to it, with an option to do the
lookup/modification/tx at the layer you want, depending on the
performance you need, whether you want qdisc.. that you'd give you
pretty much the same thing we use today, but with a cleaner design.


> Depends on the TCP stack (I think). 

I guess Linux deals with OFO better, but unfortunately that's not the
main OS used by our subscribers...

> Steam is perhaps a bad example as that is doing something very much like
> bittorrent AFAIK; but point taken, people do occasionally run
> single-stream downloads and want them to be fast. I'm just annoyed that
> this becomes the *one* benchmark people run, to the exclusion of
> everything else that has a much larger impact on the overall user
> experience :/

that one is easy

convince ookla to add some kind of "latency under load" metric, and
have them report it as a big red flag when too high, and even better
add scary messages like "this connection is not suitable for online
gaming".

subscribers will bug telco, then telco will bug SOCs vendors

-- 
Maxime