[Cake] Advantages to tightly tuning latency

[Sebastian Moeller]

Hi Maxime,

if I might ask you a tangential question, did you also look at MAP-T (translation) and if so, what made you choose MAP-E (encapsulation)?

Best Regards
	Sebastian


> On Apr 23, 2020, at 19:31, Maxime Bizon <mbizon at freebox.fr> wrote:
> 
> 
> On Thursday 23 Apr 2020 à 18:42:11 (+0200), Toke Høiland-Jørgensen wrote:
> 
>> Didn't make it in until 5.5, unfortunately... :(
>> 
>> I can try to produce a patch that you can manually apply on top of 5.4
>> if you're interested?
> 
> I could do it, but the thing I'm more worried about is the lack of
> test coverage from everyone else.
> 
>> Anyhow, my larger point was that we really do want to enable such use
>> cases for XDP; but we are lacking the details of what exactly is missing
>> before we can get to something that's useful / deployable. So any
>> details you could share about what feature set you are supporting in
>> your own 'fast path' implementation would be really helpful. As would
>> details about the hardware platform you are using. You can send them
>> off-list if you don't want to make it public, of course :)
> 
> there is no hardware specific feature used, it's all software
> 
> imagine this "simple" setup, pretty much what anyone's home router is
> doing:
> 
> <br0> with <eth0> + <wlan0> inside, private IPv4 address
> <wan0.vlan> with IPv6, vlan interface over <wan0>
> <map0> with IPv4, MAP-E tunnel over <wan0.vlan>
> 
> then:
>  - IPv6 routing between <br0> and <wan0.vlan>
>  - IPv4 routing + NAT between <br0> and <map0>
> 
> iptables would be filled with usual rules, per interface ALLOW rules
> in FORWARD chain, DNAT rules in PREROUTING to access LAN from WAN...
> 
> and then you want this to be fast :)
> 
> What we do is build a "flow" table on top of conntrack, so with a
> single lookup we find the flow, the destination interface, and what
> modifications to apply to the packet (L3 address to change, encap to
> add/remove, etc etc)
> 
> Then we do this lookup more or less early in RX path, on our oldest
> platform we even had to do this from the ethernet driver, and do TX
> from there too, skipping qdisc layer and allowing cache maintenance
> hacks (partial invalidation and wback)
> 
> 
> nftable with flowtables seems to be have developped something that
> could replace our flow cache, but I'm not sure if it can handle our
> tunneling scheme yet. It even has a notion of offloaded flow for
> hardware that can support it.
> 
> If you add an XDP offload to it, with an option to do the
> lookup/modification/tx at the layer you want, depending on the
> performance you need, whether you want qdisc.. that you'd give you
> pretty much the same thing we use today, but with a cleaner design.
> 
> 
>> Depends on the TCP stack (I think). 
> 
> I guess Linux deals with OFO better, but unfortunately that's not the
> main OS used by our subscribers...
> 
>> Steam is perhaps a bad example as that is doing something very much like
>> bittorrent AFAIK; but point taken, people do occasionally run
>> single-stream downloads and want them to be fast. I'm just annoyed that
>> this becomes the *one* benchmark people run, to the exclusion of
>> everything else that has a much larger impact on the overall user
>> experience :/
> 
> that one is easy
> 
> convince ookla to add some kind of "latency under load" metric, and
> have them report it as a big red flag when too high, and even better
> add scary messages like "this connection is not suitable for online
> gaming".
> 
> subscribers will bug telco, then telco will bug SOCs vendors
> 
> -- 
> Maxime
> _______________________________________________
> Cake mailing list
> Cake at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake