[Bloat] BBR implementations, knobs to turn?

Fri Nov 20 06:10:27 EST 2020

Hi Erik,

I really appreciate that you are reaching out to the bufferbloat community
for this real-life 5G mobile testing.  Lets all help out Erik.

>From your graphs, it does look like you are measuring latency
under-load, e.g. while the curl download/upload is running.  This is
great as this is the first rule of bufferbloat measuring :-)  (and Luca
hinted to this)

The Huawei policer/shaper sounds scary.  And 1000 packets deep queue
also sound like a recipe for bufferbloat.  I would of-cause like to
re-write the Huawei policer/shaper with the knowledge and techniques we
know from our bufferbloat work in the Linux Kernel.  (If only I knew
someone that coded on 5G solutions that could implement this on their
hardware solution, and provide a better product Cc. Carlo)

Are you familiar with Toke's (cc) work/PhD on handling bufferbloat on
wireless networks?  (Hint: Airtime fairness)

Solving bufferbloat in wireless networks require more than applying
fq_codel on the bottleneck queue, it requires Airtime fairness.  Doing
scheduling based Clients use of Radio-time and transmit-opportunities
(TXOP), instead of shaping based on bytes. (This is why it can (if you
are very careful) make sense to "holding back packets a bit" to
generate a packet aggregate that only consumes one TXOP).

The culprit is that each Client/MobilePhone will be sending at
different rates, and scheduling based on bytes, will cause a Client with
a low rate to consume a too large part of the shared radio airtime.
That basically sums up Toke's PhD ;-)

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

Cc. Marek due to his twitter post[1] and link to 5G-BBR blogpost[2]:
 [1] https://twitter.com/majek04/status/1329708548297732097
 [2] https://blog.acolyer.org/2020/10/05/understanding-operational-5g/

On Thu, 19 Nov 2020 14:35:27 +0000 <erik.taraldsen at telenor.com> wrote:

> Hello Luca
> 
> 
> The current PGW is a policer.   What the next version will be, I'm not sure.
> 
> 
> However on parts of the Huawei RAN the policing rate is set to be a
> shaper speed on the eNodeB (radio antenna).  1000 packets deep. And
> it not only shapes down to 30Mb, but tries to aggregate packets to
> keep a level speed whenever using the radio interface.  Meaning
> holding back packets a bit to try and get to 30Mbit when sending in
> bulk in case of less than 30Mbit user traffic.  30Mbit being an
> example subscription speed.
> 
> 
> We are rolling out a fix to turn of that Huawei shaper, but it is not
> done nation wide yet.
> 
> The test device is in a lab area, using close to, but not entirely
> the same as production 5G setup from Ericsson.  Here there should not
> be any shapers involved in the downstream path here.  There is
> however a bloated buffer on the upstream path which we are working on
> correcting.
> 
> 
> The curl graphs are "time to complete a curl download of x file
> size", using a apache webserver running bbr.
> 
> 
> -Erik
> 
> 
> ________________________________
> Fra: Luca Muscariello <muscariello at ieee.org>
> Sendt: 19. november 2020 14:32
> Til: Taraldsen Erik
> Kopi: Jesper Dangaard Brouer; priyarjha at google.com; bloat; Luca Muscariello
> Emne: Re: [Bloat] BBR implementations, knobs to turn?
> 
> Hi Erick,
> 
> one question about the PGW: is it a policer or a shaper that you have installed?
> Also, have you tried to run a ping session before and in parallel to the curl sessions?
> 
> Luca
> 
> 
> 
> On Thu, Nov 19, 2020 at 2:15 PM <erik.taraldsen at telenor.com<mailto:erik.taraldsen at telenor.com>> wrote:
> Update:
> The 5G router was connected to a new base station.  Now the limiting factor of throughput is the policer on the PGW in mobile core, not the radio link itself.  The SIM card used is limited to 30Mbit/s.  This scenario favours the new server.  I have attached graphs comparing radio link limited vs PGW policer results, and a zoomed in graph of the policer
> 
> 
> We have Huawei RAN and Ericsson RAN, rate limited and not rate limited subscriptions, 4G and 5G access, and we are migrating to a new core with new PGW (policer).  Starting to be a bit of a matrix to set up tests for.
> 
> 
> -Erik
> 
> 
> ________________________________________
> Fra: Jesper Dangaard Brouer <brouer at redhat.com<mailto:brouer at redhat.com>>
> Sendt: 17. november 2020 16:07
> Til: Taraldsen Erik; Priyaranjan Jha
> Kopi: brouer at redhat.com<mailto:brouer at redhat.com>; ncardwell at google.com<mailto:ncardwell at google.com>; bloat at lists.bufferbloat.net<mailto:bloat at lists.bufferbloat.net>
> Emne: Re: [Bloat] BBR implementations, knobs to turn?
> 
> On Tue, 17 Nov 2020 10:05:24 +0000 <erik.taraldsen at telenor.com<mailto:erik.taraldsen at telenor.com>> wrote:
> 
> > Thank you for the response Neal  
> 
> Yes. And it is impressive how many highly qualified people are on the
> bufferbloat list.
> 
> > old_hw # uname -r
> > 5.3.0-64-generic
> > (Ubuntu 19.10 on xenon workstation, integrated network card, 1Gbit
> > GPON access.  Used as proof of concept from the lab at work)
> >
> >
> > new_hw # uname -r
> > 4.18.0-193.19.1.el8_2.x86_64
> > (Centos 8.2 on xenon rack server, discrete 10Gbit network card,
> > 40Gbit server farm link (low utilization on link), intended as fully
> > supported and run service.  Not possible to have newer kernel and
> > still get service agreement in my organization)  
> 
> Let me help out here.  The CentOS/RHEL8 kernels have a huge amount of
> backports.  I've attached a patch/diff of net/ipv4/tcp_bbr.c changes
> missing in RHEL8.
> 
> It looks like these patches are missing in CentOS/RHEL8:
>  [1] https://git.kernel.org/torvalds/c/78dc70ebaa38aa3
>  [2] https://git.kernel.org/torvalds/c/a87c83d5ee25cf7
> 
> Could missing patch [1] result in the issue Erik is seeing?
> (It explicitly mentions improvements for WiFi...)
> 
> --
> Best regards,
>   Jesper Dangaard Brouer
>   MSc.CS, Principal Kernel Engineer at Red Hat
>   LinkedIn: http://www.linkedin.com/in/brouer
> _______________________________________________
> Bloat mailing list
> Bloat at lists.bufferbloat.net<mailto:Bloat at lists.bufferbloat.net>
> https://lists.bufferbloat.net/listinfo/bloat