From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io1-xd2d.google.com (mail-io1-xd2d.google.com [IPv6:2607:f8b0:4864:20::d2d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 609153B2A4 for ; Thu, 3 Oct 2019 14:41:46 -0400 (EDT) Received: by mail-io1-xd2d.google.com with SMTP id a1so7947763ioc.6 for ; Thu, 03 Oct 2019 11:41:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=eXPjNldGqUd5Eitl2i+NGXE2MOOiEsx39UxURz2qoe0=; b=mezeiYVkOqGPjmMHjgRZgrAl4Q5hn+TENsVFzxjf7fMCsa4p/BU9U9/hUWKnPHPF0l xjumUyzKcQCRcBrX0CKt6mEo6W3T6Zv/9aEz4aZHWesoNsn4F9B2b0aOXk/4VxrR10JA 6zaPuaslinuia8ToLrtAHNJ7WXXveBtkdghM90AEJprF8Aj77mnaBdAML+/9diOP9gFp 3zq0e1nWg3g6Fzxh5LUHqySNDZ1f+XwaclRibc/QkrhgvFA+p07OtwRRL16PJcYxrYcX XCGtmCZTlNpVZt11OJujlDVfn68IXuQby7cVpk7S3lDXHtbpFG2ztS7gdR5Cktuj9tLm Hovw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=eXPjNldGqUd5Eitl2i+NGXE2MOOiEsx39UxURz2qoe0=; b=oXkI2UrfzqSCCU6fKC/bfBvAAyxTRXQMWSXhVxjwZUKx+7eRbOjlrlZChYA3xxfdV7 4YLmmG9lavrhqql2t6iLGOZowebavYknbC2TUgCyYqc06bSnp+ZYh16yV71fhzmrJY0E JUT6BtA45dNJI0uT3/OGlzLvz72XqOIwwrDfV8Vi9EJRDsV3oQBbewYw90Spq4H5x4UM y6LRnTpjVhNmVyaVSKbcC0xIZ1rBCdwXmzFbx8ZfkzRTE9PbwCZ44d5xNYwfdGScuyR5 02rzvy1dFJB45qBYRidOg1cEBDe+x0sPJCk4N25vm1jhkdZqc8w/5YX6Y+Kqz9X5mM5P 4vDA== X-Gm-Message-State: APjAAAUXxSRNMzc/u5OVKw4fRgd8ZNuHZf0+KdT1ZuWxjyB/fGva9c1E L+JuD23ocgjBNyxo/wV24tp/MFyQTYat0Skn2PYyMMY3 X-Google-Smtp-Source: APXvYqynpGyCO6Ou7wnAg/a/shQgYmotEBkkg9F03ywXgc2Rso9kfdHvoukWgVSckhxLLEMluEbG89o8z86Cl52gZe4= X-Received: by 2002:a5d:8911:: with SMTP id b17mr9554613ion.287.1570128105704; Thu, 03 Oct 2019 11:41:45 -0700 (PDT) MIME-Version: 1.0 References: <2825CE14-2109-4580-A086-9701F4D3ADF0@gmail.com> <18b1c174-b88d-4664-9aa8-9c42925fc14c@www.fastmail.com> <9a90111b-2389-4dc6-8409-18c40f895540@www.fastmail.com> <43F02160-E691-4393-A0C0-8AB4AD962700@gmail.com> In-Reply-To: From: Dave Taht Date: Thu, 3 Oct 2019 11:41:33 -0700 Message-ID: To: Justin Kilpatrick Cc: Jonathan Morton , Cake List Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Cake] Fighting bloat in the face of uncertinty X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Oct 2019 18:41:46 -0000 Heh. We need a t-shirt... from TunnelManager::from_registry().do_send(GotBloat { .. GotBloat() ? more_fq_codel : fq_codel; On Thu, Oct 3, 2019 at 10:52 AM Justin Kilpatrick wrote= : > > I've developed a rough version of this and put it into production Monday.= After a few tweaks we're seeing a ~10x reduction in the magnitude of laten= cy spikes at high usage times. > > https://github.com/althea-net/althea_rs/blob/master/rita/src/rita_common/= network_monitor/mod.rs#L288 > > The average and standard deviation of latency to a given neighbor is scra= ped from Babel and when the standard deviation exceeds 10x the average we = reduce the throughput of the connection by 20%. > > It's not theoretically sound yet because I still need to expose single di= rection latency in Babel rather than only round trip. Bloat caused by the o= ther side of the link currently causes connections to be reduced all the wa= y down to the throughput minimum unnecessarily. > > It would also be advantageous to observe what throughput we've recorded f= or the last 5 seconds and put a threshold there. Rather than doing any prob= ing ourselves we can just observe if the user was saturating the connection= or if it was a transient radio problem. > > If anyone else is interested in using this I can split it off from our ap= plication and into a stand alone (if somewhat bulky) binary without much tr= ouble. > > -- > Justin Kilpatrick > justin@althea.net > > On Sun, Sep 8, 2019, at 1:27 PM, Jonathan Morton wrote: > > >> You could also set it back to 'internet' and progressively reduce th= e > > >> bandwidth parameter, making the Cake shaper into the actual bottlene= ck. > > >> This is the correct fix for the problem, and you should notice an > > >> instant improvement as soon as the bandwidth parameter is correct. > > > > > > Hand tuning this one link is not a problem. I'm searching for a set o= f settings that will provide generally good performance across a wide range= of devices, links, and situations. > > > > > > From what you've indicated so far there's nothing as effective as a c= orrect bandwidth estimation if we consider the antenna (link) a black box. = Expecting the user to input expected throughput for every link and then man= aging that information is essentially a non-starter. > > > > > > Radio tuning provides some improvement, but until ubiquiti starts shi= pping with Codel on non-router devices I don't think there's a good solutio= n here. > > > > > > Any way to have the receiving device detect bloat and insert an ECN? > > > > That's what the qdisc itself is supposed to do. > > > > > I don't think the time spent in the intermediate device is detectable= at the kernel level but we keep track of latency for routing decisions and= could detect bloat with some accuracy, the problem is how to respond. > > > > As long as you can detect which link the bloat is on (and in which > > direction), you can respond by reducing the bandwidth parameter on that > > half-link by a small amount. Since you have a cooperating network, > > maintaining a time standard on each node sufficient to observe one-way > > delays seems feasible, as is establishing a normal baseline latency for > > each link. > > > > The characteristics of the bandwidth parameter being too high are easy > > to observe. Not only will the one-way delay go up, but the received > > throughput in the same direction at the same time will be lower than > > configured. You might use the latter as a hint as to how far you need > > to reduce the shaped bandwidth. > > > > Deciding when and by how much to *increase* bandwidth, which is > > presumably desirable when link conditions improve, is a more difficult > > problem when the link hardware doesn't cooperate by informing you of > > its status. (This is something you could reasonably ask Ubiquiti to > > address.) > > > > I would assume that link characteristics will change slowly, and run an > > occasional explicit bandwidth probe to see if spare bandwidth is > > available. If that probe comes through without exhibiting bloat, *and* > > the link is otherwise loaded to capacity, then increase the shaper by > > an amount within the probe's capacity of measurement - and schedule a > > repeat. > > > > A suitable probe might be 100x 1500b packets paced out over a second, > > bypassing the shaper. This will occupy just over 1Mbps of bandwidth, > > and can be expected to induce 10ms of delay if injected into a > > saturated 100Mbps link. Observe the delay experienced by each packet > > *and* the quantity of other traffic that appears between them. Only if > > both are favourable can you safely open the shaper, by 1Mbps. > > > > Since wireless links can be expected to change their capacity over > > time, due to eg. weather and tree growth, this seems to be more > > generally useful than a static guess. You could deploy a new link with > > a conservative "guess" of say 10Mbps, and just probe from there. > > > > - Jonathan Morton > _______________________________________________ > Cake mailing list > Cake@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake --=20 Dave T=C3=A4ht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740