From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-il1-x134.google.com (mail-il1-x134.google.com [IPv6:2607:f8b0:4864:20::134]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 6C8483B2A4; Sun, 28 Nov 2021 13:24:25 -0500 (EST) Received: by mail-il1-x134.google.com with SMTP id i6so14768373ila.0; Sun, 28 Nov 2021 10:24:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=eGb+3hGJ23wCDnmf8SOHCggNTgEO2DiSaYONBgC9sdo=; b=fcoaZz3aSd/s+pcnzTwWmiCwx8oidiCuUwpZIjN5VwS2QdwXcnZkWGk7a81qBauddG dM+RgWDFJU3seOu0OV0fdjSyDXITG/4GlzELuR6tPKGK97YqLyYf23IoKuAinFBHaXoM tHOGb5VFNWt/mJOSzM3BJ46To9hkk1Is75i2cbrfv3eHD87UcyfTDyVdim5aZx0xpmmp XOm8PQHWcJWxaEsm90tvT+l8/Odu/vXveQQAAyMtHf7RocZRsvj0ZxDUmPnUKCJoHV9Q RKTsMJuZLNkJEtHN00DSmyIXC34EDYSasgDWaTZsfqjou0MF/gXqR4OpaFcXXeoc3Lnh i3RQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=eGb+3hGJ23wCDnmf8SOHCggNTgEO2DiSaYONBgC9sdo=; b=qFpULcFTMIN9hMoprBGMqzImDBuI7bFuiBzFLQjwCAi/ySX62d49q2aa4Vbvl8eX88 MpfdGSiinplVWRczTVkLQTqmm+P4TrfF3q9bQEpFuXtaAEFE3jKIYPgosem63Md4zwVZ T+ZHo5dwdJn+K0SPWp5IJNe7yJwW+xgDiReWxerlF+LmcLM77hx8pzyVdENAY866/kBV fUxT0PKvwSScLdJZRyhku5p+2RiWLLAjC0Z2h0t+YA2rX3NBSpYHDXsxk5tCEqC+mVgr iNCAMyr6o9tzn+pSy62Rrun4qg+qNighl2nJI3RRCEv0VLRSaVxtJgfBuMrH0EmEbl8f 4c7A== X-Gm-Message-State: AOAM533PLRWxZusFS9a9ZPhyjXVfutMTj1dZmdevn2QVQvoMoZbyXupF 8y2xyd6SK+3WARrmD6W2e/Mg9TC6Ervm0zPFKhuHB1Aiyk8= X-Google-Smtp-Source: ABdhPJwFpyanL/C+e+/b9Bgs2qze68grrqScO41q9mHUpeknUit0svHgqrvAez3k2zPZGeSB+gQEUJmp8N+KtvTsc/A= X-Received: by 2002:a05:6e02:20ea:: with SMTP id q10mr42263025ilv.283.1638123864759; Sun, 28 Nov 2021 10:24:24 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Dave Taht Date: Sun, 28 Nov 2021 10:24:12 -0800 Message-ID: To: Neal Cardwell Cc: bloat , Rpm Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Rpm] [Bloat] *under*bloated networks X-BeenThere: rpm@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: revolutions per minute - a new metric for measuring responsiveness List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Nov 2021 18:24:25 -0000 On Sun, Nov 28, 2021 at 7:42 AM Neal Cardwell wrote: > > > > On Mon, Nov 22, 2021 at 10:35 PM Dave Taht wrote: >> >> In the last two weeks I have found two dramatically underbuffered Gbit >> fiber networks. >> >> This one appears to have about a 400 full size packet uplink buffer (5ms= )[1] >> >> https://imgur.com/a/Bm9hdNf >> >> It was pretty remarkable to see how well multiple tcp flows still >> achieved close to the full rate with such a small fixed size queue, >> eventually. And baseline rtts not increasing more than 5ms before seeing a drop with cu= bic. https://imgur.com/a/S5v2EF3 >> >> A single bbr flow can't crack 150mbits: https://imgur.com/a/DpydL5K. > > > Thanks, Dave. The single-flow BBR upload case is interesting. > > I took a look at the packet traces in the later thread: > https://www.reddit.com/r/eero/comments/qxbkcl/66_is_out/hltlep0/ > > For the single-flow BBR case it seems that.... > > (1) the BBR(v1) flow is running into a 300 Mbps bottleneck rate (visible = in the slope of the green ACK line in the zoomed-in trace, attached). This particular bottleneck is capable of a gbit, according to the 8 flow tests. A single cubic flow peaks at 300Mbit, in earlier tests. > (2) the BBR(v1) flow is achieving an average rate a bit above 150 Mbps be= cause it repeatedly runs into receive window limits (the yellow line in the= zoomed-out trace, attached). The frequent receive window limits mean that = the flow spends a lot of time unable to send anything, thus leading to lowe= r average throughput. In all honesty, I haven't looked at BBR, + long rtts, and rates > 100Mbit often enough. (I actually have never had more than a 35Mbit uplink to the internet in my whole life.) What I see here is a big sack block, and that pause, and yes, the window getting smaller... And universally, this fellah's link kicking flows out of slow start at around 100Mbit... but what does it mean? this server is running 5.11.0-40-generic #44-Ubuntu SMP, if there is something I can tune. Most speedtest sites use 8+ flows on the upload portion of the test... > > cheers, > neal > --=20 I tried to build a better future, a few times: https://wayforward.archive.org/?site=3Dhttps%3A%2F%2Fwww.icei.org Dave T=C3=A4ht CEO, TekLibre, LLC