From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qk1-x734.google.com (mail-qk1-x734.google.com [IPv6:2607:f8b0:4864:20::734]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id CA93B3B2A4 for ; Mon, 12 Nov 2018 10:55:47 -0500 (EST) Received: by mail-qk1-x734.google.com with SMTP id y16so14079010qki.7 for ; Mon, 12 Nov 2018 07:55:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=PwEHqLOvuGlyipgHzFngG8KQSpKf7GiQfa0RFSWxnik=; b=eDeXLNA+L5OllHOIIngKqrnC7vp9GRUf2XEXDXmFMpUwFfUNXj87apksfvZcW9j9jl bmKmhl2A1u3rGT4XNA5ZdNYtJ+bY0rKgO/DhsSuN7PLVy3GSCYYiThXb/HuDM6PN9035 DRqreuLZ+WrjjNSLD5OLg1fbt3MvTQxPrqTcjhTXysPRgvHciaO5qnERnudRc1tsyclr ixDGQ7baCFPGd+1U6aXdUfw5+kq/VtsgLNfjvwZjS8H16ySNiErkczKZ/yMBf2529F05 gDOZOV6UheZRDsg1mtdlwYxwuuT6IcD5NuA/hClITW558viXtcH68qFsnqy+8wHYFNR4 b/hg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=PwEHqLOvuGlyipgHzFngG8KQSpKf7GiQfa0RFSWxnik=; b=NPeXbam6bAmXD1jADdbzVR0WOWRPUrCrG4Ua5m3Rv31VhZwxF4oIidvUj91hYDE8Tr tfWiB8bIljD0OduYpeYdFXcPjO2LaljPYK49Q/4zLKkPsnhVI0R7NipG/POkpGgMJ0uR a0sSi6992svxGJ2jbh1Z3Gm0LuTyOKUoh8iUkquKYn0ARC6nsPTV0MLKr7s3RX1QQViJ jTGM236Cy3B5eF+GEHodvVL4uwF6bvc4l5QDrdkZIsJlzdjQ8rnWom/c0nivCkAHSzWj X3qfotwsuGpdBS7vuFmvZGDnMXQY1W3vf3dapepSUMBrpgdVOGVs1rtvpHJk2Tha+mEQ 8h5g== X-Gm-Message-State: AGRZ1gKUQQVFz+/iix5BPzXReAicKvZEyZAyG0Rxr/Heftw8V+nwdu6l ssN5+bwPtDJR5dEztMHzM9ztpfUFuHyx0E7UZApNYFHY X-Google-Smtp-Source: AJdET5fxL1+G8vw2Zq+IeDXUOCgPiPO3YCdffPTT1lF1JPMIoi+fh261ijfI79A/nkJXoSACUHxzuADhTxcu6tQPZ5o= X-Received: by 2002:ac8:6606:: with SMTP id c6mr1409935qtp.376.1542038147055; Mon, 12 Nov 2018 07:55:47 -0800 (PST) MIME-Version: 1.0 References: <87va529v0l.fsf@taht.net> In-Reply-To: From: Dave Taht Date: Mon, 12 Nov 2018 07:55:35 -0800 Message-ID: To: Mikael Abrahamsson Cc: =?UTF-8?Q?Dave_T=C3=A4ht?= , bloat Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Bloat] excellent result with OpenWrt 18.06.1 + CAKE on FTTH X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Nov 2018 15:55:47 -0000 On Mon, Nov 12, 2018 at 7:38 AM Mikael Abrahamsson wrote= : > > On Mon, 12 Nov 2018, Dave Taht wrote: > > > tc -s qdisc show dev your_device? > > tc -s qdisc show dev your_ifbdevice? > > I haven't restarted in 40 days and I don't remember restarting cake, so > this should be several weeks of data. > > qdisc cake 8031: dev eth1.2 root refcnt 2 bandwidth 100Mbit besteffort > triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms raw Pretty nice. There is not much need for it, but it would be interesting to = know how much cpu overhead a full set of uploads with and without the ack-filter enabled at 100Mbit with this arch takes. > overhead 0 > Sent 70822286277 bytes 202513660 pkt (dropped 13984, overlimits 2535042= 1 > requeues 0) > backlog 0b 0p requeues 0 > memory used: 5156288b of 5000000b > capacity estimate: 100Mbit > min/max network layer size: 42 / 1514 > min/max overhead-adjusted size: 42 / 1514 > average network hdr offset: 14 > > Tin 0 > thresh 100Mbit > target 5.0ms > interval 100.0ms > pk_delay 4us > av_delay 1us > sp_delay 1us > backlog 0b > pkts 202527644 > bytes 70842325936 Don't use this connection much, do you? :) > way_inds 4939006 > way_miss 11834545 > way_cols 0 > drops 13984 It's nice to know AQM is still needed at these speeds. > marks 512 and that you have at least one device with ecn enabled. Would this be OSX or IOS perhaps? > ack_drop 0 And how many ack-drops you get. > sp_flows 2 > bk_flows 1 > un_flows 0 > max_len 28766 I really hate GRO. At 100Mbit, without splitting, that's 2.5ms of jitter. I've had a long held dream of being able to do the "jamophone" with less than 2.5ms latency end-2-end across town, with multichannel 48khz 24 bit audio... at 130us, that leaves some room for the encoder. I don't suppose you have someone else "across town" you could run some benchmarks against? I did a extremely successful test of sonic's fiber network in SF where I got well under 2.5ms jitter for an 8 channel stream... then I left SF. :( I was kind of expecting 64k, though, here. > quantum 1514 > > qdisc ingress ffff: dev eth1.2 parent ffff:fff1 ---------------- > Sent 807912654344 bytes 631652827 pkt (dropped 0, overlimits 0 requeues > 0) > backlog 0b 0p requeues 0 > qdisc cake 8032: dev ifb4eth1.2 root refcnt 2 bandwidth 250Mbit besteffor= t > triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms raw > overhead 0 Similarly, a cpu number under load. I note here, that splitting GSO has a big cost, (primarily in routing table lookup) and you can at these speeds, probably disable it. ack filtering will not help. > Sent 829571211610 bytes 631641016 pkt (dropped 11811, overlimits > 790055554 requeues 0) > backlog 0b 0p requeues 0 > memory used: 4540528b of 12500000b > capacity estimate: 250Mbit > min/max network layer size: 60 / 1514 > min/max overhead-adjusted size: 60 / 1514 > average network hdr offset: 14 > > Tin 0 > thresh 250Mbit > target 5.0ms > interval 100.0ms > pk_delay 1.2ms > av_delay 559us > sp_delay 1us > backlog 0b > pkts 631652827 > bytes 829588333230 > way_inds 12061686 > way_miss 12913211 > way_cols 1 > drops 11811 > marks 3589 > ack_drop 0 > sp_flows 1 > bk_flows 1 > un_flows 0 > max_len 38444 I was also expecting 64k here. I imagine you are using modern linuxes that don't overuse TSO anymore, and osx and windows never got into it to the extreme that linux did. > quantum 1514 > > > > > -- > Mikael Abrahamsson email: swmike@swm.pp.se > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat --=20 Dave T=C3=A4ht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740