From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-vs1-xe34.google.com (mail-vs1-xe34.google.com [IPv6:2607:f8b0:4864:20::e34]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 316C23B29D for ; Fri, 20 Nov 2020 18:34:21 -0500 (EST) Received: by mail-vs1-xe34.google.com with SMTP id v8so5447093vso.2 for ; Fri, 20 Nov 2020 15:34:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=WtPbgzRStTqDR/E7koZZE6cX+DQg5ZgLtA3JBo5x8dk=; b=Jm6FBQsLlpHrtBCLOop74etsAsmmRwGMgJG5P2CZ6SViiDNF8zULsgPt/HtgRsJNjr jJJfz5JDG5D66FRJKtSsz2Xgv98tzbIhfI8DTwwVbfsMgmqdpnErVYmvvKOzvfExGBmY 1rSITLnRQVAy7PnRwqqdHhBPadcy0K7k5MXtpDIGSFCViysW3hwAS/34CRtsGgZozhgs SjPrnfpAQmZrukoo5RpMdOeD7ihwA4+e4UYkA6e3dbQ9G4vEZFVoMA2xApHb/leN/c2s a5MBr9PM+n2K5IxVxmS+K0CS8fV6FoxM1yseXpUNS/rToS0fj+p4fcexPqzPpsOJVEC2 TSUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=WtPbgzRStTqDR/E7koZZE6cX+DQg5ZgLtA3JBo5x8dk=; b=POz+kJJi8LFugikz21oabgvF4XXZ6GMo2K+8+3pMDLtpYPOmSdiXH+Ehnv6VqngxAI 52wy38nDlyrpp6fbSzUpAToNj1sURZZ9FDhJA7LsIGEXOWC28OfH62zFQKSQX3x6eFU2 TGAhPuVG+T7byWVvlxEf3M8WUuVaLrOeIC6uB+hiUSKC+7coEZVb1TChd0EAaA37cjYD JwnszbwIHZ/oFPI40mJ5TJJcw55oIMWT6zYCZ+fLOlrzgfbPScgQF/jQA1fvsY9Az92s rERusRkghPfNWorTjw6lMj82qFQm60MneOk01O0wLnf2GuApMFOsbXRIXSRpE3gWXAf9 3q8A== X-Gm-Message-State: AOAM530bHmzrOqmHC1QSYtv6ippUrMLPzUSJidpIjtm69pO0nFZ+h2Kb DdT6JrfhZWLhEEsjy/mwooWuqp4Zq/O3tIhrJXwOsQ== X-Google-Smtp-Source: ABdhPJwXMJl12o+cdWai3fj8Dy+AldPF37NYml+yz0+PuTDZqhlPO/v8dh/KS0ApCOnqFUTqceqVTXZZ9TBPx7DGCqI= X-Received: by 2002:a67:ff01:: with SMTP id v1mr15503121vsp.16.1605915260485; Fri, 20 Nov 2020 15:34:20 -0800 (PST) MIME-Version: 1.0 References: <1605540351240.98589@telenor.com> <1605607524611.20916@telenor.com> <20201117160744.395f108e@carbon> In-Reply-To: <20201117160744.395f108e@carbon> From: Neal Cardwell Date: Fri, 20 Nov 2020 18:34:04 -0500 Message-ID: To: Jesper Dangaard Brouer Cc: erik.taraldsen@telenor.com, Priyaranjan Jha , bloat Content-Type: text/plain; charset="UTF-8" Subject: Re: [Bloat] BBR implementations, knobs to turn? X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Nov 2020 23:34:21 -0000 On Tue, Nov 17, 2020 at 10:08 AM Jesper Dangaard Brouer wrote: > > On Tue, 17 Nov 2020 10:05:24 +0000 wrote: > > > Thank you for the response Neal > > Yes. And it is impressive how many highly qualified people are on the > bufferbloat list. > > > old_hw # uname -r > > 5.3.0-64-generic > > (Ubuntu 19.10 on xenon workstation, integrated network card, 1Gbit > > GPON access. Used as proof of concept from the lab at work) > > > > > > new_hw # uname -r > > 4.18.0-193.19.1.el8_2.x86_64 > > (Centos 8.2 on xenon rack server, discrete 10Gbit network card, > > 40Gbit server farm link (low utilization on link), intended as fully > > supported and run service. Not possible to have newer kernel and > > still get service agreement in my organization) > > Let me help out here. The CentOS/RHEL8 kernels have a huge amount of > backports. I've attached a patch/diff of net/ipv4/tcp_bbr.c changes > missing in RHEL8. > > It looks like these patches are missing in CentOS/RHEL8: > [1] https://git.kernel.org/torvalds/c/78dc70ebaa38aa3 > [2] https://git.kernel.org/torvalds/c/a87c83d5ee25cf7 > > Could missing patch [1] result in the issue Erik is seeing? > (It explicitly mentions improvements for WiFi...) Thanks, Erik, for the detailed information. This is super-useful. And thanks, Jesper, for the patch analysis. Yes, I agree that missing patch [1] is likely the cause of the lower BBR throughout in the "new_hw" case. Since the "new_hw" is running an older kernel that's missing this important patch, it would be expected to have lower throughput in a workload like this. It's unfortunate that it's not possible to have a newer kernel on the newer hardware; it does seem in this case that this would probably do the trick. best, neal