[Bloat] excellent result with OpenWrt 18.06.1 + CAKE on FTTH

[Dave Taht]

On Mon, Nov 12, 2018 at 7:38 AM Mikael Abrahamsson <swmike at swm.pp.se> wrote:
>
> On Mon, 12 Nov 2018, Dave Taht wrote:
>
> > tc -s qdisc show dev your_device?
> > tc -s qdisc show dev your_ifbdevice?
>
> I haven't restarted in 40 days and I don't remember restarting cake, so
> this should be several weeks of data.
>
> qdisc cake 8031: dev eth1.2 root refcnt 2 bandwidth 100Mbit besteffort
> triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms raw

Pretty nice. There is not much need for it, but it would be interesting to know
how much cpu overhead a full set of uploads with and without the
ack-filter enabled
at 100Mbit with this arch takes.

> overhead 0
>   Sent 70822286277 bytes 202513660 pkt (dropped 13984, overlimits 25350421
> requeues 0)
>   backlog 0b 0p requeues 0
>   memory used: 5156288b of 5000000b
>   capacity estimate: 100Mbit
>   min/max network layer size:           42 /    1514
>   min/max overhead-adjusted size:       42 /    1514
>   average network hdr offset:           14
>
>                    Tin 0
>    thresh        100Mbit
>    target          5.0ms
>    interval      100.0ms
>    pk_delay          4us
>    av_delay          1us
>    sp_delay          1us
>    backlog            0b
>    pkts        202527644
>    bytes     70842325936

Don't use this connection much, do you? :)

>    way_inds      4939006
>    way_miss     11834545
>    way_cols            0
>    drops           13984

It's nice to know AQM is still needed at these speeds.

>    marks             512

and that you have at least one device with ecn enabled. Would this be
OSX or IOS perhaps?

>    ack_drop            0

And how many ack-drops you get.
>    sp_flows            2
>    bk_flows            1
>    un_flows            0
>    max_len         28766

I really hate GRO. At 100Mbit, without splitting, that's 2.5ms of
jitter. I've had a long held dream of being able to do the "jamophone"
with less than 2.5ms latency end-2-end across town, with multichannel
48khz 24 bit audio... at 130us, that leaves some room for the encoder.

I don't suppose you have someone else "across town" you could run some
benchmarks against?

I did a extremely successful test of sonic's fiber network in SF where
I got well under 2.5ms jitter for an 8 channel stream... then I left
SF. :(

I was kind of expecting 64k, though, here.

>    quantum          1514
>
> qdisc ingress ffff: dev eth1.2 parent ffff:fff1 ----------------
>   Sent 807912654344 bytes 631652827 pkt (dropped 0, overlimits 0 requeues
> 0)
>   backlog 0b 0p requeues 0
> qdisc cake 8032: dev ifb4eth1.2 root refcnt 2 bandwidth 250Mbit besteffort
> triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms raw
> overhead 0

Similarly, a cpu number under load. I note here, that splitting GSO
has a big cost, (primarily in routing table lookup) and you can at
these speeds, probably disable it.

ack filtering will not help.

>   Sent 829571211610 bytes 631641016 pkt (dropped 11811, overlimits
> 790055554 requeues 0)
>   backlog 0b 0p requeues 0
>   memory used: 4540528b of 12500000b
>   capacity estimate: 250Mbit
>   min/max network layer size:           60 /    1514
>   min/max overhead-adjusted size:       60 /    1514
>   average network hdr offset:           14
>
>                    Tin 0
>    thresh        250Mbit
>    target          5.0ms
>    interval      100.0ms
>    pk_delay        1.2ms
>    av_delay        559us
>    sp_delay          1us
>    backlog            0b
>    pkts        631652827
>    bytes    829588333230
>    way_inds     12061686
>    way_miss     12913211
>    way_cols            1
>    drops           11811
>    marks            3589
>    ack_drop            0
>    sp_flows            1
>    bk_flows            1
>    un_flows            0
>    max_len         38444

I was also expecting 64k here. I imagine you are using modern linuxes
that don't overuse TSO anymore,
and osx and windows never got into it to the extreme that linux did.

>    quantum          1514
>
>
>
>
> --
> Mikael Abrahamsson    email: swmike at swm.pp.se
> _______________________________________________
> Bloat mailing list
> Bloat at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740