From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 21C003CB37 for ; Fri, 20 Nov 2020 06:10:41 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605870640; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nGc5KkzxOwBgv3dgTaLO0Co1Ydv4WKDbL2rJ9mpr1to=; b=fe0r7QqIKy356hfD0yoU9U1m76Tb4ORZOSLscrST6oxCnn46nQg+g4EBtxKX+LjoXDSNtr PKwutsrbVn5LOfRQJ12vPKIuk84Aau0GnS1Ta3KEnf7SQ7eyTjFQ0xfaVV39LrPYVqxJyn cw6aNMTTkaTtcrpjXoAJjEaWkVsN3ro= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-211-pTibtBdHMBObZjTwqeNRmA-1; Fri, 20 Nov 2020 06:10:36 -0500 X-MC-Unique: pTibtBdHMBObZjTwqeNRmA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5A8111005D60; Fri, 20 Nov 2020 11:10:34 +0000 (UTC) Received: from carbon (unknown [10.36.110.8]) by smtp.corp.redhat.com (Postfix) with ESMTP id E37D95D9C6; Fri, 20 Nov 2020 11:10:28 +0000 (UTC) Date: Fri, 20 Nov 2020 12:10:27 +0100 From: Jesper Dangaard Brouer To: Cc: , , , , brouer@redhat.com, Toke =?UTF-8?B?SMO4aWxhbmQtSsO4cmdlbnNlbg==?= , Marek Majkowski , Carlo Carraro Message-ID: <20201120121027.2a61b319@carbon> In-Reply-To: <1605796526806.72220@telenor.com> References: <1605540351240.98589@telenor.com> <1605607524611.20916@telenor.com> <20201117160744.395f108e@carbon> <1605772623976.41134@telenor.com> <1605796526806.72220@telenor.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=brouer@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Subject: Re: [Bloat] BBR implementations, knobs to turn? X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Nov 2020 11:10:41 -0000 Hi Erik, I really appreciate that you are reaching out to the bufferbloat community for this real-life 5G mobile testing. Lets all help out Erik. >From your graphs, it does look like you are measuring latency under-load, e.g. while the curl download/upload is running. This is great as this is the first rule of bufferbloat measuring :-) (and Luca hinted to this) The Huawei policer/shaper sounds scary. And 1000 packets deep queue also sound like a recipe for bufferbloat. I would of-cause like to re-write the Huawei policer/shaper with the knowledge and techniques we know from our bufferbloat work in the Linux Kernel. (If only I knew someone that coded on 5G solutions that could implement this on their hardware solution, and provide a better product Cc. Carlo) Are you familiar with Toke's (cc) work/PhD on handling bufferbloat on wireless networks? (Hint: Airtime fairness) Solving bufferbloat in wireless networks require more than applying fq_codel on the bottleneck queue, it requires Airtime fairness. Doing scheduling based Clients use of Radio-time and transmit-opportunities (TXOP), instead of shaping based on bytes. (This is why it can (if you are very careful) make sense to "holding back packets a bit" to generate a packet aggregate that only consumes one TXOP). The culprit is that each Client/MobilePhone will be sending at different rates, and scheduling based on bytes, will cause a Client with a low rate to consume a too large part of the shared radio airtime. That basically sums up Toke's PhD ;-) --=20 Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer Cc. Marek due to his twitter post[1] and link to 5G-BBR blogpost[2]: [1] https://twitter.com/majek04/status/1329708548297732097 [2] https://blog.acolyer.org/2020/10/05/understanding-operational-5g/ On Thu, 19 Nov 2020 14:35:27 +0000 wrote: > Hello Luca >=20 >=20 > The current PGW is a policer. What the next version will be, I'm not su= re. >=20 >=20 > However on parts of the Huawei RAN the policing rate is set to be a > shaper speed on the eNodeB (radio antenna). 1000 packets deep. And > it not only shapes down to 30Mb, but tries to aggregate packets to > keep a level speed whenever using the radio interface. Meaning > holding back packets a bit to try and get to 30Mbit when sending in > bulk in case of less than 30Mbit user traffic. 30Mbit being an > example subscription speed. >=20 >=20 > We are rolling out a fix to turn of that Huawei shaper, but it is not > done nation wide yet. >=20 > The test device is in a lab area, using close to, but not entirely > the same as production 5G setup from Ericsson. Here there should not > be any shapers involved in the downstream path here. There is > however a bloated buffer on the upstream path which we are working on > correcting. >=20 >=20 > The curl graphs are "time to complete a curl download of x file > size", using a apache webserver running bbr. >=20 >=20 > -Erik >=20 >=20 > ________________________________ > Fra: Luca Muscariello > Sendt: 19. november 2020 14:32 > Til: Taraldsen Erik > Kopi: Jesper Dangaard Brouer; priyarjha@google.com; bloat; Luca Muscariel= lo > Emne: Re: [Bloat] BBR implementations, knobs to turn? >=20 > Hi Erick, >=20 > one question about the PGW: is it a policer or a shaper that you have ins= talled? > Also, have you tried to run a ping session before and in parallel to the = curl sessions? >=20 > Luca >=20 >=20 >=20 > On Thu, Nov 19, 2020 at 2:15 PM > wrote: > Update: > The 5G router was connected to a new base station. Now the limiting fact= or of throughput is the policer on the PGW in mobile core, not the radio li= nk itself. The SIM card used is limited to 30Mbit/s. This scenario favour= s the new server. I have attached graphs comparing radio link limited vs P= GW policer results, and a zoomed in graph of the policer >=20 >=20 > We have Huawei RAN and Ericsson RAN, rate limited and not rate limited su= bscriptions, 4G and 5G access, and we are migrating to a new core with new = PGW (policer). Starting to be a bit of a matrix to set up tests for. >=20 >=20 > -Erik >=20 >=20 > ________________________________________ > Fra: Jesper Dangaard Brouer > > Sendt: 17. november 2020 16:07 > Til: Taraldsen Erik; Priyaranjan Jha > Kopi: brouer@redhat.com; ncardwell@google.com; bloat@lists.bufferbloat.net > Emne: Re: [Bloat] BBR implementations, knobs to turn? >=20 > On Tue, 17 Nov 2020 10:05:24 +0000 > wrote: >=20 > > Thank you for the response Neal =20 >=20 > Yes. And it is impressive how many highly qualified people are on the > bufferbloat list. >=20 > > old_hw # uname -r > > 5.3.0-64-generic > > (Ubuntu 19.10 on xenon workstation, integrated network card, 1Gbit > > GPON access. Used as proof of concept from the lab at work) > > > > > > new_hw # uname -r > > 4.18.0-193.19.1.el8_2.x86_64 > > (Centos 8.2 on xenon rack server, discrete 10Gbit network card, > > 40Gbit server farm link (low utilization on link), intended as fully > > supported and run service. Not possible to have newer kernel and > > still get service agreement in my organization) =20 >=20 > Let me help out here. The CentOS/RHEL8 kernels have a huge amount of > backports. I've attached a patch/diff of net/ipv4/tcp_bbr.c changes > missing in RHEL8. >=20 > It looks like these patches are missing in CentOS/RHEL8: > [1] https://git.kernel.org/torvalds/c/78dc70ebaa38aa3 > [2] https://git.kernel.org/torvalds/c/a87c83d5ee25cf7 >=20 > Could missing patch [1] result in the issue Erik is seeing? > (It explicitly mentions improvements for WiFi...) >=20 > -- > Best regards, > Jesper Dangaard Brouer > MSc.CS, Principal Kernel Engineer at Red Hat > LinkedIn: http://www.linkedin.com/in/brouer > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat