[Bloat] finally got the proper cake experience (was: Re: openwrt e1000e (was: Re: cake + ipv6))

[Daniel Sterling <sterling.daniel@gmail.com>]

On Tue, Nov 17, 2020 at 7:00 PM Daniel Sterling
<sterling.daniel@gmail.com> wrote:
> Wanted to give an update on this. All my issues (odd latency, slow
> throughput, etc) went away when I switched from the in-kernel e1000e
> driver to Intel's NAPI driver.

OK y'all, after switching to this driver, I configured my openwrt
snapshot setup as follows:

** e1000e loaded with InterruptThrottleRate=0:
rmmod r8169 ; modprobe r8169
rmmod e1000e ; insmod e1000e InterruptThrottleRate=0

** All ethtool -K offloading turned off:
rx tx sg tso gso gro all off

** flow control / pause frames turned off:
autoneg off rx off tx off

** ring buffers set to smallest setting:
rx 64 tx 64

** byte queue limits set to 3000
echo 3000 > /sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/net/eth1/queues/tx-0/byte_queue_limits/limit_max
echo 3000 > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/net/eth0/queues/tx-0/byte_queue_limits/limit_max

cake configured with besteffort and no bandwidth limit:
/usr/sbin/tc qdisc add dev $WAN handle 1: root cake besteffort
internet nat egress ack-filter dual-srchost ethernet
/usr/sbin/tc qdisc add dev $LAN handle 1: root cake besteffort
internet ingress dual-dsthost ethernet


AND ALL MY LATENCY IS GONE.

I've finally experienced the Zen of low latency and high throughput of
the Promised Cake.

This is amazing.

Previously, I was able to get low latency by setting the cake
"bandwidth" setting to about 40-50mbps out of the LAN interface.
Higher than that and I'd see latency.

But now, I can run as fast as the wifi / wired connections allow with
no latency anywhere on the network. This is truly wild.

Why did no one tell me I didn't need that bandwidth setting?? I
finally pushed and pushed and messed with things until I got it
working this way. If this is the intended final configuration, it
needs to be proselytized! Truly wire-line speed and no latency. This
is just, there are no words. The dream is real

Enormous thank you to everyone who worked on cake!

Thanks,
Dan