[Bloat] BBR implementations, knobs to turn?

Thu Nov 19 09:35:27 EST 2020

Hello Luca

The current PGW is a policer.   What the next version will be, I'm not sure.

However on parts of the Huawei RAN the policing rate is set to be a shaper speed on the eNodeB (radio antenna).  1000 packets deep. And it not only shapes down to 30Mb, but tries to aggregate packets to keep a level speed whenever using the radio interface.  Meaning holding back packets a bit to try and get to 30Mbit when sending in bulk in case of less than 30Mbit user traffic.  30Mbit being an example subscription speed.

We are rolling out a fix to turn of that Huawei shaper, but it is not done nation wide yet.

The test device is in a lab area, using close to, but not entirely the same as production 5G setup from Ericsson.  Here there should not be any shapers involved in the downstream path here.  There is however a bloated buffer on the upstream path which we are working on correcting.

The curl graphs are "time to complete a curl download of x file size", using a apache webserver running bbr.

-Erik

________________________________
Fra: Luca Muscariello <muscariello at ieee.org>
Sendt: 19. november 2020 14:32
Til: Taraldsen Erik
Kopi: Jesper Dangaard Brouer; priyarjha at google.com; bloat; Luca Muscariello
Emne: Re: [Bloat] BBR implementations, knobs to turn?

Hi Erick,

one question about the PGW: is it a policer or a shaper that you have installed?
Also, have you tried to run a ping session before and in parallel to the curl sessions?

Luca

On Thu, Nov 19, 2020 at 2:15 PM <erik.taraldsen at telenor.com<mailto:erik.taraldsen at telenor.com>> wrote:
Update:
The 5G router was connected to a new base station.  Now the limiting factor of throughput is the policer on the PGW in mobile core, not the radio link itself.  The SIM card used is limited to 30Mbit/s.  This scenario favours the new server.  I have attached graphs comparing radio link limited vs PGW policer results, and a zoomed in graph of the policer

We have Huawei RAN and Ericsson RAN, rate limited and not rate limited subscriptions, 4G and 5G access, and we are migrating to a new core with new PGW (policer).  Starting to be a bit of a matrix to set up tests for.

-Erik

________________________________________
Fra: Jesper Dangaard Brouer <brouer at redhat.com<mailto:brouer at redhat.com>>
Sendt: 17. november 2020 16:07
Til: Taraldsen Erik; Priyaranjan Jha
Kopi: brouer at redhat.com<mailto:brouer at redhat.com>; ncardwell at google.com<mailto:ncardwell at google.com>; bloat at lists.bufferbloat.net<mailto:bloat at lists.bufferbloat.net>
Emne: Re: [Bloat] BBR implementations, knobs to turn?

On Tue, 17 Nov 2020 10:05:24 +0000 <erik.taraldsen at telenor.com<mailto:erik.taraldsen at telenor.com>> wrote:

> Thank you for the response Neal

Yes. And it is impressive how many highly qualified people are on the
bufferbloat list.

> old_hw # uname -r
> 5.3.0-64-generic
> (Ubuntu 19.10 on xenon workstation, integrated network card, 1Gbit
> GPON access.  Used as proof of concept from the lab at work)
>
>
> new_hw # uname -r
> 4.18.0-193.19.1.el8_2.x86_64
> (Centos 8.2 on xenon rack server, discrete 10Gbit network card,
> 40Gbit server farm link (low utilization on link), intended as fully
> supported and run service.  Not possible to have newer kernel and
> still get service agreement in my organization)

Let me help out here.  The CentOS/RHEL8 kernels have a huge amount of
backports.  I've attached a patch/diff of net/ipv4/tcp_bbr.c changes
missing in RHEL8.

It looks like these patches are missing in CentOS/RHEL8:
 [1] https://git.kernel.org/torvalds/c/78dc70ebaa38aa3
 [2] https://git.kernel.org/torvalds/c/a87c83d5ee25cf7

Could missing patch [1] result in the issue Erik is seeing?
(It explicitly mentions improvements for WiFi...)

--
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer
_______________________________________________
Bloat mailing list
Bloat at lists.bufferbloat.net<mailto:Bloat at lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/bloat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/bloat/attachments/20201119/21894119/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: througput latency and pageload.png
Type: image/png
Size: 138050 bytes
Desc: througput latency and pageload.png
URL: <https://lists.bufferbloat.net/pipermail/bloat/attachments/20201119/21894119/attachment-0001.png>