Re: [Bloat] BBR implementations, knobs to turn?

[<erik.taraldsen@telenor.com>]

[-- Attachment #1.1: Type: text/plain, Size: 4125 bytes --]

Hello Luca


The current PGW is a policer.   What the next version will be, I'm not sure.


However on parts of the Huawei RAN the policing rate is set to be a shaper speed on the eNodeB (radio antenna).  1000 packets deep. And it not only shapes down to 30Mb, but tries to aggregate packets to keep a level speed whenever using the radio interface.  Meaning holding back packets a bit to try and get to 30Mbit when sending in bulk in case of less than 30Mbit user traffic.  30Mbit being an example subscription speed.


We are rolling out a fix to turn of that Huawei shaper, but it is not done nation wide yet.


The test device is in a lab area, using close to, but not entirely the same as production 5G setup from Ericsson.  Here there should not be any shapers involved in the downstream path here.  There is however a bloated buffer on the upstream path which we are working on correcting.


The curl graphs are "time to complete a curl download of x file size", using a apache webserver running bbr.



-Erik


________________________________
Fra: Luca Muscariello <muscariello@ieee.org>
Sendt: 19. november 2020 14:32
Til: Taraldsen Erik
Kopi: Jesper Dangaard Brouer; priyarjha@google.com; bloat; Luca Muscariello
Emne: Re: [Bloat] BBR implementations, knobs to turn?

Hi Erick,

one question about the PGW: is it a policer or a shaper that you have installed?
Also, have you tried to run a ping session before and in parallel to the curl sessions?

Luca



On Thu, Nov 19, 2020 at 2:15 PM <erik.taraldsen@telenor.com<mailto:erik.taraldsen@telenor.com>> wrote:
Update:
The 5G router was connected to a new base station.  Now the limiting factor of throughput is the policer on the PGW in mobile core, not the radio link itself.  The SIM card used is limited to 30Mbit/s.  This scenario favours the new server.  I have attached graphs comparing radio link limited vs PGW policer results, and a zoomed in graph of the policer


We have Huawei RAN and Ericsson RAN, rate limited and not rate limited subscriptions, 4G and 5G access, and we are migrating to a new core with new PGW (policer).  Starting to be a bit of a matrix to set up tests for.


-Erik


________________________________________
Fra: Jesper Dangaard Brouer <brouer@redhat.com<mailto:brouer@redhat.com>>
Sendt: 17. november 2020 16:07
Til: Taraldsen Erik; Priyaranjan Jha
Kopi: brouer@redhat.com<mailto:brouer@redhat.com>; ncardwell@google.com<mailto:ncardwell@google.com>; bloat@lists.bufferbloat.net<mailto:bloat@lists.bufferbloat.net>
Emne: Re: [Bloat] BBR implementations, knobs to turn?

On Tue, 17 Nov 2020 10:05:24 +0000 <erik.taraldsen@telenor.com<mailto:erik.taraldsen@telenor.com>> wrote:

> Thank you for the response Neal

Yes. And it is impressive how many highly qualified people are on the
bufferbloat list.

> old_hw # uname -r
> 5.3.0-64-generic
> (Ubuntu 19.10 on xenon workstation, integrated network card, 1Gbit
> GPON access.  Used as proof of concept from the lab at work)
>
>
> new_hw # uname -r
> 4.18.0-193.19.1.el8_2.x86_64
> (Centos 8.2 on xenon rack server, discrete 10Gbit network card,
> 40Gbit server farm link (low utilization on link), intended as fully
> supported and run service.  Not possible to have newer kernel and
> still get service agreement in my organization)

Let me help out here.  The CentOS/RHEL8 kernels have a huge amount of
backports.  I've attached a patch/diff of net/ipv4/tcp_bbr.c changes
missing in RHEL8.

It looks like these patches are missing in CentOS/RHEL8:
 [1] https://git.kernel.org/torvalds/c/78dc70ebaa38aa3
 [2] https://git.kernel.org/torvalds/c/a87c83d5ee25cf7

Could missing patch [1] result in the issue Erik is seeing?
(It explicitly mentions improvements for WiFi...)

--
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer
_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net<mailto:Bloat@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/bloat

[-- Attachment #1.2: Type: text/html, Size: 6742 bytes --]

[-- Attachment #2: througput latency and pageload.png --]
[-- Type: image/png, Size: 138050 bytes --]