[Bloat] Bufferbloat on 4G Connexion

[Bloat] Bufferbloat on 4G Connexion

[Guillaume ROBIER]

[-- Attachment #1: Type: text/plain, Size: 539 bytes --]

Hello,



I am new to this mailing list and I discovered the bufferbloat in December 2018. I work on 4G routers and the bufferbloat is very present on this type of link (4G). I contact you today to find out if people have experimented with solutions on this type of link or have configuration suggestions, because the classic fq_codel or piece_of_cake and pie do not allow to fix the bufferbloat.



My architecture: ARM/MIPS with dd-wrt and OpenWRT  kernel  3.10.14+



Thank you in advance.



Sincerely,

Guillaume


[-- Attachment #2: Type: text/html, Size: 2685 bytes --]

Re: [Bloat] Bufferbloat on 4G Connexion

[Jonathan Morton]

> On 11 Oct, 2019, at 5:56 pm, Guillaume ROBIER <grobier@icow-systems.com> wrote:
> 
> I am new to this mailing list and I discovered the bufferbloat in December 2018. I work on 4G routers and the bufferbloat is very present on this type of link (4G). I contact you today to find out if people have experimented with solutions on this type of link or have configuration suggestions, because the classic fq_codel or piece_of_cake and pie do not allow to fix the bufferbloat.

This is actually my own situation at home.  My solution is to insert Cake shapers on both upstream *and* downstream directions, and adjust their bandwidth settings according to variations in available 4G speed.  To do this I use an IQrouter, which is basically a TP-Link Archer C7 with custom firmware.

One of the difficulties with 4G in particular is that the link capacity varies a great deal according to both radio propagation conditions (weather, obstructions) and local usage by other subscribers.  That means the right bandwidth setting for the small hours of the night, when nobody is awake, will leave you with a lot of bloat in the evening, when everyone is both awake and home from school/work.  You will need to measure these trends and set up a bandwidth schedule accordingly.

 - Jonathan Morton

Re: [Bloat] Bufferbloat on 4G Connexion

[erik.taraldsen]

If you could influence the 4G vendors to de-bloat their equipment, would you recommend BQL, L4S or codel/cake?


-Erik


________________________________________
Fra: Bloat <bloat-bounces@lists.bufferbloat.net> på vegne av Jonathan Morton <chromatix99@gmail.com>
Sendt: 22. oktober 2019 23:02
Til: Guillaume ROBIER
Kopi: bloat@lists.bufferbloat.net
Emne: Re: [Bloat] Bufferbloat on 4G Connexion

> On 11 Oct, 2019, at 5:56 pm, Guillaume ROBIER <grobier@icow-systems.com> wrote:
>
> I am new to this mailing list and I discovered the bufferbloat in December 2018. I work on 4G routers and the bufferbloat is very present on this type of link (4G). I contact you today to find out if people have experimented with solutions on this type of link or have configuration suggestions, because the classic fq_codel or piece_of_cake and pie do not allow to fix the bufferbloat.

This is actually my own situation at home.  My solution is to insert Cake shapers on both upstream *and* downstream directions, and adjust their bandwidth settings according to variations in available 4G speed.  To do this I use an IQrouter, which is basically a TP-Link Archer C7 with custom firmware.

One of the difficulties with 4G in particular is that the link capacity varies a great deal according to both radio propagation conditions (weather, obstructions) and local usage by other subscribers.  That means the right bandwidth setting for the small hours of the night, when nobody is awake, will leave you with a lot of bloat in the evening, when everyone is both awake and home from school/work.  You will need to measure these trends and set up a bandwidth schedule accordingly.

 - Jonathan Morton
_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Bufferbloat on 4G Connexion

[David Lang]

[-- Attachment #1: Type: text/plain, Size: 3268 bytes --]

As I understand it, BQL is pretty much a prereq for anything better, if you 
treat a 64 byte packet as if it was a 1500 byte packet, everything else is going 
to fail.

fq_codel is the next stage, it keeps large queues from forming, which makes 
everything work much better.

Cake layers on additional 'fairness' optimizations, but for optimal results, you 
want to have it know the available bandwidth (which is dynamic and changing).

there isn't a single 'do this' right answer, there are a series of 
optimizations. Each additional layer gives better results when tuned, but can
require more tuning, and in many cases, can give worse results than a simpler 
layer if the more complex one is badly tuned.

The most complex part of things is when you try to have one side control the 
queues on the other. If you can say that you will always have active queue 
management on both sides of the link, things get much simpler.

David Lang


On Wed, 23 Oct 2019, erik.taraldsen@telenor.com wrote:

> Date: Wed, 23 Oct 2019 07:28:05 +0000
> From: erik.taraldsen@telenor.com
> To: chromatix99@gmail.com, grobier@icow-systems.com
> Cc: bloat@lists.bufferbloat.net
> Subject: Re: [Bloat] Bufferbloat on 4G Connexion
> 
> If you could influence the 4G vendors to de-bloat their equipment, would you recommend BQL, L4S or codel/cake?
>
>
> -Erik
>
>
> ________________________________________
> Fra: Bloat <bloat-bounces@lists.bufferbloat.net> på vegne av Jonathan Morton <chromatix99@gmail.com>
> Sendt: 22. oktober 2019 23:02
> Til: Guillaume ROBIER
> Kopi: bloat@lists.bufferbloat.net
> Emne: Re: [Bloat] Bufferbloat on 4G Connexion
>
>> On 11 Oct, 2019, at 5:56 pm, Guillaume ROBIER <grobier@icow-systems.com> wrote:
>>
>> I am new to this mailing list and I discovered the bufferbloat in December 
>> 2018. I work on 4G routers and the bufferbloat is very present on this type 
>> of link (4G). I contact you today to find out if people have experimented 
>> with solutions on this type of link or have configuration suggestions, 
>> because the classic fq_codel or piece_of_cake and pie do not allow to fix the 
>> bufferbloat.
>
> This is actually my own situation at home.  My solution is to insert Cake 
> shapers on both upstream *and* downstream directions, and adjust their 
> bandwidth settings according to variations in available 4G speed.  To do this 
> I use an IQrouter, which is basically a TP-Link Archer C7 with custom 
> firmware.
>
> One of the difficulties with 4G in particular is that the link capacity varies 
> a great deal according to both radio propagation conditions (weather, 
> obstructions) and local usage by other subscribers.  That means the right 
> bandwidth setting for the small hours of the night, when nobody is awake, will 
> leave you with a lot of bloat in the evening, when everyone is both awake and 
> home from school/work.  You will need to measure these trends and set up a 
> bandwidth schedule accordingly.
>
> - Jonathan Morton _______________________________________________ Bloat 
> mailing list Bloat@lists.bufferbloat.net 
> https://lists.bufferbloat.net/listinfo/bloat 
> _______________________________________________ Bloat mailing list 
> Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Bufferbloat on 4G Connexion

[Sebastian Moeller]

> On Oct 23, 2019, at 09:28, <erik.taraldsen@telenor.com> <erik.taraldsen@telenor.com> wrote:
> 
> If you could influence the 4G vendors to de-bloat their equipment, would you recommend BQL, L4S or codel/cake?

IMHO, something like BQL would be great, as would codel/cake in the devices. 
L4S however, has not been thoroughly tested and hence should be considered research-grade at best. To my knowledge none of the L4S papers/thesis so far have been performed bi-directional saturation of the network, even the PhD thesis titled Destruction Testing: Ultra-Low Delay using Dual Queue Coupled Active Queue Management).
The IMHO best set of L4S tests at https://github.com/heistp/sce-l4s-bakeoff indicates a number of "rough" edges in L4S that should be addressed before considering a roll-out
Personally I doubt that all of these can be fixed within the restraint solution space the L4S project forced upon itself (avoid flow-queueing at any cost).
So as it stands both codel and cake (and especially the fq_codel variant) have proven their usefulness in the real world, one of the big issues with both is that unless the transport interface implements some thing similar to BQL both require a computationally costly traffic shaper (cake has its own shaper built in codel/fq_codel need an additional shaper). It is not necessarily sheer number of CPU cycles, but that they seem to require low latency access to the CPU when ever deadlines approach, to try to put things to simplistically.



Best Regards
	Sebastian


> 
> 
> -Erik
> 
> 
> ________________________________________
> Fra: Bloat <bloat-bounces@lists.bufferbloat.net> på vegne av Jonathan Morton <chromatix99@gmail.com>
> Sendt: 22. oktober 2019 23:02
> Til: Guillaume ROBIER
> Kopi: bloat@lists.bufferbloat.net
> Emne: Re: [Bloat] Bufferbloat on 4G Connexion
> 
>> On 11 Oct, 2019, at 5:56 pm, Guillaume ROBIER <grobier@icow-systems.com> wrote:
>> 
>> I am new to this mailing list and I discovered the bufferbloat in December 2018. I work on 4G routers and the bufferbloat is very present on this type of link (4G). I contact you today to find out if people have experimented with solutions on this type of link or have configuration suggestions, because the classic fq_codel or piece_of_cake and pie do not allow to fix the bufferbloat.
> 
> This is actually my own situation at home.  My solution is to insert Cake shapers on both upstream *and* downstream directions, and adjust their bandwidth settings according to variations in available 4G speed.  To do this I use an IQrouter, which is basically a TP-Link Archer C7 with custom firmware.
> 
> One of the difficulties with 4G in particular is that the link capacity varies a great deal according to both radio propagation conditions (weather, obstructions) and local usage by other subscribers.  That means the right bandwidth setting for the small hours of the night, when nobody is awake, will leave you with a lot of bloat in the evening, when everyone is both awake and home from school/work.  You will need to measure these trends and set up a bandwidth schedule accordingly.
> 
> - Jonathan Morton
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Bufferbloat on 4G Connexion

[Jonathan Morton]

> On 23 Oct, 2019, at 10:28 am, <erik.taraldsen@telenor.com> <erik.taraldsen@telenor.com> wrote:
> 
> If you could influence the 4G vendors to de-bloat their equipment, would you recommend BQL, L4S or codel/cake?

BQL is a good start, but needs to go with something else to actually fix the problem.  All it does is move (most of) the queue out of hardware and into software.  The software side then needs to capitalise on having control of the bottleneck queue, by implementing FQ, AQM, or both.

L4S is a complete waste of time.  It requires basically replacing the entire Internet, including endpoints, to work properly.  If the endpoints are not replaced, then the middleboxes behave like a conventional AQM (see below).

Codel would be a HUGE help.  It's a well-tested example of an AQM that's specifically designed to work with TCP, and should work equally well with other protocols designed to be TCP friendly.  There are other AQMs, such as PIE, BLUE, and even the old standby WRED, that would also be major improvements over the status quo, though they don't perform as well on common TCP traffic as Codel.  The basic requirement here is to bring the maximum *standing* queue down from hundreds or thousands of milliseconds to tens or singles; Codel achieves 5ms with its default Internet-tuned settings.

I use a slight variant of Codel in Cake, which I call COBALT.  The main algorithm is just a refactoring of Codel, but I've also addressed two blind spots in Codel's design.  One is the behaviour upon reactivating very shortly after the standing queue was initially brought under control; COBALT keeps better track of previous state in that case.  The other is addressing heavy load from non-TCP-friendly traffic, which requires a probabilistic drop function rather than a scheduled drop or mark.  To take care of the latter, I've overlaid a version of the BLUE algorithm.

I believe Codel and COBALT should be reasonably straightforward to implement in hardware.  I could help to explain the algorithmic details and rationale to hardware engineers if need be, and help them determine the right design tradeoff.

FQ's main benefit is allowing "sparse" flows of traffic, which tend to be more sensitive to latency, to bypass queues of "bulk" traffic which tends to be more throughput sensitive.  This amounts to reducing "inter-flow induced latency", where AQM focuses on reducing "intra-flow induced latency".  The main difficulty is allegedly in identifying flows and maintaining the per-flow state and individual queues; these should not be insurmountable obstacles, but I will admit the complexity of implementing this in hardware is higher than for a plain AQM.

I would note, however, that 4G already incurs some of this complexity through the need to individually queue, aggregate, schedule, and transmit a stream of traffic to each mobile station.  It should be relatively easy to add an AQM instance per station, thereby ensuring that congestion signals incurred by one subscriber saturating their own capacity don't spill over into other subscribers' traffic.

Combining AQM with FQ is the gold standard, minimising both inter-flow and intra-flow induced latency.  That's what fq_codel and Cake do.  If that was deployed in the right places in a 4G network, you could consider the problem solved.  For an estimate of implementation complexity, take an FQ implementation and add an AQM for each flow's queue.

As a middle ground I could suggest one of my latest projects, CNQ (Cheap Nasty Queuing).  This aims to provide some of the benefits of FQ+AQM, but with only slightly higher implementation complexity than a plain AQM, at the cost of inferior flow-isolating performance than true FQ.  It should therefore be more suitable for high-volume nodes than an FQ-based algorithm.  A software implementation for demo purposes is presently in the works, and we should have some initial performance data fairly soon.  I would seriously consider deploying CNQ on a per-station basis in 4G.

 - Jonathan Morton

Re: [Bloat] Bufferbloat on 4G Connexion

[Rich Brown]

[-- Attachment #1: Type: text/plain, Size: 954 bytes --]


> On Oct 23, 2019, at 5:54 AM,<erik.taraldsen@telenor.com <mailto:erik.taraldsen@telenor.com>> wrote:
> 
> If you could influence the 4G vendors to de-bloat their equipment, would you recommend BQL, L4S or codel/cake?

I've been enjoying this discussion and wonder whether the work going on in the make-wifi-fast (https://lists.bufferbloat.net/pipermail/make-wifi-fast/) is relevant.

I only have a 30,000 foot understanding of this work, but it seems the use of AQL (Airtime Queue Limit) maps better onto the vagaries of 4G/5G radio transmissions than BQL. Specifically, having a measurement of the actual time it takes to transmit a packet might give additional information about the current link speed, with the potential for adjusting the codel target, etc.

Separately, I also wonder whether the Air Time Fairness algorithm might provide a benefit if the cellphone tower station manufacturers chose to get into the game.

Thanks.

Rich

[-- Attachment #2: Type: text/html, Size: 2300 bytes --]

Re: [Bloat] Bufferbloat on 4G Connexion

[Toke Høiland-Jørgensen]

Rich Brown <richb.hanover@gmail.com> writes:

>> On Oct 23, 2019, at 5:54 AM,<erik.taraldsen@telenor.com <mailto:erik.taraldsen@telenor.com>> wrote:
>> 
>> If you could influence the 4G vendors to de-bloat their equipment,
>> would you recommend BQL, L4S or codel/cake?
>
> I've been enjoying this discussion and wonder whether the work going
> on in the make-wifi-fast
> (https://lists.bufferbloat.net/pipermail/make-wifi-fast/) is relevant.
>
> I only have a 30,000 foot understanding of this work, but it seems the
> use of AQL (Airtime Queue Limit) maps better onto the vagaries of
> 4G/5G radio transmissions than BQL. Specifically, having a measurement
> of the actual time it takes to transmit a packet might give additional
> information about the current link speed, with the potential for
> adjusting the codel target, etc.

Indeed, I suspect something like AQL would work for LTE as well. At the
right level; think this might need to be in the firmware (which in turn
could push back on the host).

> Separately, I also wonder whether the Air Time Fairness algorithm
> might provide a benefit if the cellphone tower station manufacturers
> chose to get into the game.

LTE base stations already does TDMA scheduling (which they can do easily
because they are centralised and own the license band); airtime fairness
is about getting the same benefits into WiFi that LTE has been enjoying
from the get-go :)

-Toke

Re: [Bloat] Bufferbloat on 4G Connexion

[Luca Muscariello]

[-- Attachment #1: Type: text/plain, Size: 3042 bytes --]

On Wed, Oct 23, 2019 at 2:27 PM Toke Høiland-Jørgensen <toke@redhat.com>
wrote:

> Rich Brown <richb.hanover@gmail.com> writes:
>
> >> On Oct 23, 2019, at 5:54 AM,<erik.taraldsen@telenor.com <mailto:
> erik.taraldsen@telenor.com>> wrote:
> >>
> >> If you could influence the 4G vendors to de-bloat their equipment,
> >> would you recommend BQL, L4S or codel/cake?
> >
> > I've been enjoying this discussion and wonder whether the work going
> > on in the make-wifi-fast
> > (https://lists.bufferbloat.net/pipermail/make-wifi-fast/) is relevant.
> >
> > I only have a 30,000 foot understanding of this work, but it seems the
> > use of AQL (Airtime Queue Limit) maps better onto the vagaries of
> > 4G/5G radio transmissions than BQL. Specifically, having a measurement
> > of the actual time it takes to transmit a packet might give additional
> > information about the current link speed, with the potential for
> > adjusting the codel target, etc.
>
> Indeed, I suspect something like AQL would work for LTE as well. At the
> right level; think this might need to be in the firmware (which in turn
> could push back on the host).
>
> > Separately, I also wonder whether the Air Time Fairness algorithm
> > might provide a benefit if the cellphone tower station manufacturers
> > chose to get into the game.
>
> LTE base stations already does TDMA scheduling (which they can do easily
> because they are centralised and own the license band); airtime fairness
> is about getting the same benefits into WiFi that LTE has been enjoying
> from the get-go :)
>

There is one main difference between ATF and the kind of TDMA
realized by an LTE scheduler (but also HSDPA/HSUPA).
Toke correct me if I'm wrong.

The current ATF scheduler for WiFi does airtime-DRR based on the current
PHY rates,
is that right? Side question, how do you measure current?

In LTE TDMA makes use of what is called multi-user diversity gain
by scheduling users when they are at their relative best radio condition.
Typically the user with the best current radio condition NORMALIZED
over the average radio conditions. The average can be based on a
moving average or a sliding window. This is the case of the widely used
David Tse's proportional fair scheduler.

This means that TDMA is still in place to share air-time fairly but the
scheduler will tend to avoid bad radio conditions.

From a theoretical point of view if you do that the total capacity
of the AP can increase with the number of stations (I think logarithmically)
as the scheduler surfs across radio quality peaks and not the average radio
quality. Very smart.

In LTE this is doable as the scheduling time slot is 1ms and the feedback
channel
is as fast. Not all TDMAs are equal.
Maybe the current scheduler in WiFi can be improved to do that. Maybe.

Luca




>
> -Toke
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>

[-- Attachment #2: Type: text/html, Size: 6267 bytes --]

Re: [Bloat] Bufferbloat on 4G Connexion

[Toke Høiland-Jørgensen]

Luca Muscariello <muscariello@ieee.org> writes:

> On Wed, Oct 23, 2019 at 2:27 PM Toke Høiland-Jørgensen <toke@redhat.com>
> wrote:
>
>> Rich Brown <richb.hanover@gmail.com> writes:
>>
>> >> On Oct 23, 2019, at 5:54 AM,<erik.taraldsen@telenor.com <mailto:
>> erik.taraldsen@telenor.com>> wrote:
>> >>
>> >> If you could influence the 4G vendors to de-bloat their equipment,
>> >> would you recommend BQL, L4S or codel/cake?
>> >
>> > I've been enjoying this discussion and wonder whether the work going
>> > on in the make-wifi-fast
>> > (https://lists.bufferbloat.net/pipermail/make-wifi-fast/) is relevant.
>> >
>> > I only have a 30,000 foot understanding of this work, but it seems the
>> > use of AQL (Airtime Queue Limit) maps better onto the vagaries of
>> > 4G/5G radio transmissions than BQL. Specifically, having a measurement
>> > of the actual time it takes to transmit a packet might give additional
>> > information about the current link speed, with the potential for
>> > adjusting the codel target, etc.
>>
>> Indeed, I suspect something like AQL would work for LTE as well. At the
>> right level; think this might need to be in the firmware (which in turn
>> could push back on the host).
>>
>> > Separately, I also wonder whether the Air Time Fairness algorithm
>> > might provide a benefit if the cellphone tower station manufacturers
>> > chose to get into the game.
>>
>> LTE base stations already does TDMA scheduling (which they can do easily
>> because they are centralised and own the license band); airtime fairness
>> is about getting the same benefits into WiFi that LTE has been enjoying
>> from the get-go :)
>>
>
> There is one main difference between ATF and the kind of TDMA
> realized by an LTE scheduler (but also HSDPA/HSUPA).
> Toke correct me if I'm wrong.
>
> The current ATF scheduler for WiFi does airtime-DRR based on the
> current PHY rates, is that right? Side question, how do you measure
> current?

s/current/last/. The ATF scheduler does everything after-the-fact, by
accounting the actual TX time of a transmission after it has completed.
So no fancy scheduling or prediction tricks are needed; with the
tradeoff being coarser granularity of the fairness achieved (i.e., there
can be unfairness on short timescales).

In the airtime queue limit work that's ongoing, we do ahead-of-time
airtime estimation to limit queueing in firmware. But this still just
uses the last TX rate recorded for the given station to calculate the
estimate.

> In LTE TDMA makes use of what is called multi-user diversity gain
> by scheduling users when they are at their relative best radio condition.
> Typically the user with the best current radio condition NORMALIZED
> over the average radio conditions. The average can be based on a
> moving average or a sliding window. This is the case of the widely used
> David Tse's proportional fair scheduler.
>
> This means that TDMA is still in place to share air-time fairly but the
> scheduler will tend to avoid bad radio conditions.
>
> From a theoretical point of view if you do that the total capacity
> of the AP can increase with the number of stations (I think logarithmically)
> as the scheduler surfs across radio quality peaks and not the average radio
> quality. Very smart.
>
> In LTE this is doable as the scheduling time slot is 1ms and the
> feedback channel is as fast. Not all TDMAs are equal.

Yeah, the LTE MAC is pretty cool. Just a shame that the equipment is so
expensive :(

> Maybe the current scheduler in WiFi can be improved to do that. Maybe.

I think 802.11ax is going in that direction. Nothing nearly as advanced,
but at least there's the possibility of a dedicated back channel...

-Toke

Re: [Bloat] Bufferbloat on 4G Connexion

[Luca Muscariello]

[-- Attachment #1: Type: text/plain, Size: 4243 bytes --]

On Thu, Oct 24, 2019 at 11:34 AM Toke Høiland-Jørgensen <toke@redhat.com>
wrote:

> Luca Muscariello <muscariello@ieee.org> writes:
>
> > On Wed, Oct 23, 2019 at 2:27 PM Toke Høiland-Jørgensen <toke@redhat.com>
> > wrote:
> >
> >> Rich Brown <richb.hanover@gmail.com> writes:
> >>
> >> >> On Oct 23, 2019, at 5:54 AM,<erik.taraldsen@telenor.com <mailto:
> >> erik.taraldsen@telenor.com>> wrote:
> >> >>
> >> >> If you could influence the 4G vendors to de-bloat their equipment,
> >> >> would you recommend BQL, L4S or codel/cake?
> >> >
> >> > I've been enjoying this discussion and wonder whether the work going
> >> > on in the make-wifi-fast
> >> > (https://lists.bufferbloat.net/pipermail/make-wifi-fast/) is
> relevant.
> >> >
> >> > I only have a 30,000 foot understanding of this work, but it seems the
> >> > use of AQL (Airtime Queue Limit) maps better onto the vagaries of
> >> > 4G/5G radio transmissions than BQL. Specifically, having a measurement
> >> > of the actual time it takes to transmit a packet might give additional
> >> > information about the current link speed, with the potential for
> >> > adjusting the codel target, etc.
> >>
> >> Indeed, I suspect something like AQL would work for LTE as well. At the
> >> right level; think this might need to be in the firmware (which in turn
> >> could push back on the host).
> >>
> >> > Separately, I also wonder whether the Air Time Fairness algorithm
> >> > might provide a benefit if the cellphone tower station manufacturers
> >> > chose to get into the game.
> >>
> >> LTE base stations already does TDMA scheduling (which they can do easily
> >> because they are centralised and own the license band); airtime fairness
> >> is about getting the same benefits into WiFi that LTE has been enjoying
> >> from the get-go :)
> >>
> >
> > There is one main difference between ATF and the kind of TDMA
> > realized by an LTE scheduler (but also HSDPA/HSUPA).
> > Toke correct me if I'm wrong.
> >
> > The current ATF scheduler for WiFi does airtime-DRR based on the
> > current PHY rates, is that right? Side question, how do you measure
> > current?
>
> s/current/last/. The ATF scheduler does everything after-the-fact, by
> accounting the actual TX time of a transmission after it has completed.
> So no fancy scheduling or prediction tricks are needed; with the
> tradeoff being coarser granularity of the fairness achieved (i.e., there
> can be unfairness on short timescales).
>
> In the airtime queue limit work that's ongoing, we do ahead-of-time
> airtime estimation to limit queueing in firmware. But this still just
> uses the last TX rate recorded for the given station to calculate the
> estimate.
>
> > In LTE TDMA makes use of what is called multi-user diversity gain
> > by scheduling users when they are at their relative best radio condition.
> > Typically the user with the best current radio condition NORMALIZED
> > over the average radio conditions. The average can be based on a
> > moving average or a sliding window. This is the case of the widely used
> > David Tse's proportional fair scheduler.
> >
> > This means that TDMA is still in place to share air-time fairly but the
> > scheduler will tend to avoid bad radio conditions.
> >
> > From a theoretical point of view if you do that the total capacity
> > of the AP can increase with the number of stations (I think
> logarithmically)
> > as the scheduler surfs across radio quality peaks and not the average
> radio
> > quality. Very smart.
> >
> > In LTE this is doable as the scheduling time slot is 1ms and the
> > feedback channel is as fast. Not all TDMAs are equal.
>
> Yeah, the LTE MAC is pretty cool. Just a shame that the equipment is so
> expensive :(
>

It looks like there is a positive correlation between the size
of the specifications and the cost to build the associated product :)




>
> > Maybe the current scheduler in WiFi can be improved to do that. Maybe.
>
> I think 802.11ax is going in that direction. Nothing nearly as advanced,
> but at least there's the possibility of a dedicated back channel...
>

That's right. ax does much better.



>
> -Toke
>
>

[-- Attachment #2: Type: text/html, Size: 6166 bytes --]

Re: [Bloat] Bufferbloat on 4G Connexion

[Toke Høiland-Jørgensen]

Luca Muscariello <muscariello@ieee.org> writes:

> On Thu, Oct 24, 2019 at 11:34 AM Toke Høiland-Jørgensen <toke@redhat.com>
> wrote:
>
>> Luca Muscariello <muscariello@ieee.org> writes:
>>
>> > On Wed, Oct 23, 2019 at 2:27 PM Toke Høiland-Jørgensen <toke@redhat.com>
>> > wrote:
>> >
>> >> Rich Brown <richb.hanover@gmail.com> writes:
>> >>
>> >> >> On Oct 23, 2019, at 5:54 AM,<erik.taraldsen@telenor.com <mailto:
>> >> erik.taraldsen@telenor.com>> wrote:
>> >> >>
>> >> >> If you could influence the 4G vendors to de-bloat their equipment,
>> >> >> would you recommend BQL, L4S or codel/cake?
>> >> >
>> >> > I've been enjoying this discussion and wonder whether the work going
>> >> > on in the make-wifi-fast
>> >> > (https://lists.bufferbloat.net/pipermail/make-wifi-fast/) is
>> relevant.
>> >> >
>> >> > I only have a 30,000 foot understanding of this work, but it seems the
>> >> > use of AQL (Airtime Queue Limit) maps better onto the vagaries of
>> >> > 4G/5G radio transmissions than BQL. Specifically, having a measurement
>> >> > of the actual time it takes to transmit a packet might give additional
>> >> > information about the current link speed, with the potential for
>> >> > adjusting the codel target, etc.
>> >>
>> >> Indeed, I suspect something like AQL would work for LTE as well. At the
>> >> right level; think this might need to be in the firmware (which in turn
>> >> could push back on the host).
>> >>
>> >> > Separately, I also wonder whether the Air Time Fairness algorithm
>> >> > might provide a benefit if the cellphone tower station manufacturers
>> >> > chose to get into the game.
>> >>
>> >> LTE base stations already does TDMA scheduling (which they can do easily
>> >> because they are centralised and own the license band); airtime fairness
>> >> is about getting the same benefits into WiFi that LTE has been enjoying
>> >> from the get-go :)
>> >>
>> >
>> > There is one main difference between ATF and the kind of TDMA
>> > realized by an LTE scheduler (but also HSDPA/HSUPA).
>> > Toke correct me if I'm wrong.
>> >
>> > The current ATF scheduler for WiFi does airtime-DRR based on the
>> > current PHY rates, is that right? Side question, how do you measure
>> > current?
>>
>> s/current/last/. The ATF scheduler does everything after-the-fact, by
>> accounting the actual TX time of a transmission after it has completed.
>> So no fancy scheduling or prediction tricks are needed; with the
>> tradeoff being coarser granularity of the fairness achieved (i.e., there
>> can be unfairness on short timescales).
>>
>> In the airtime queue limit work that's ongoing, we do ahead-of-time
>> airtime estimation to limit queueing in firmware. But this still just
>> uses the last TX rate recorded for the given station to calculate the
>> estimate.
>>
>> > In LTE TDMA makes use of what is called multi-user diversity gain
>> > by scheduling users when they are at their relative best radio condition.
>> > Typically the user with the best current radio condition NORMALIZED
>> > over the average radio conditions. The average can be based on a
>> > moving average or a sliding window. This is the case of the widely used
>> > David Tse's proportional fair scheduler.
>> >
>> > This means that TDMA is still in place to share air-time fairly but the
>> > scheduler will tend to avoid bad radio conditions.
>> >
>> > From a theoretical point of view if you do that the total capacity
>> > of the AP can increase with the number of stations (I think
>> logarithmically)
>> > as the scheduler surfs across radio quality peaks and not the average
>> radio
>> > quality. Very smart.
>> >
>> > In LTE this is doable as the scheduling time slot is 1ms and the
>> > feedback channel is as fast. Not all TDMAs are equal.
>>
>> Yeah, the LTE MAC is pretty cool. Just a shame that the equipment is so
>> expensive :(
>>
>
> It looks like there is a positive correlation between the size
> of the specifications and the cost to build the associated product :)

Hehe, yeah, funny how that works.

IMO the LTE people went a little bit overboard on the complexity,
though... ;)

-Toke

Re: [Bloat] Bufferbloat on 4G Connexion

[David Lang]

On Wed, 23 Oct 2019, Rich Brown wrote:

> I only have a 30,000 foot understanding of this work, but it seems the use of AQL (Airtime Queue Limit) maps better onto the vagaries of 4G/5G radio transmissions than BQL. Specifically, having a measurement of the actual time it takes to transmit a packet might give additional information about the current link speed, with the potential for adjusting the codel target, etc.

The key thing in both BQL and AQL is that instead of treating all packets the 
same, you account for how long it will take to transmit them.

In Wifi, your transmission rate can vary by a factor of 1000x (1Mb to over 1Gb 
per second), so just counting bytes as BQL does is no longer good enough.

how much does the actual over-the-air transmission rate vary in 4G? I'd bet that 
it varies less than wifi does because wifi must retain compatibility with the 
earliest equipment while 4G no longer supports 2G protocols.

is it just a matter of each phone getting a smaller slice of the airtime when 
things get busy? or does the actual data rate for that slice of airtime 
decrease as well?

David Lang

Re: [Bloat] Bufferbloat on 4G Connexion

[erik.taraldsen]

> The key thing in both BQL and AQL is that instead of treating all packets the
> same, you account for how long it will take to transmit them.
>
> In Wifi, your transmission rate can vary by a factor of 1000x (1Mb to over 1Gb
> per second), so just counting bytes as BQL does is no longer good enough.
>
> how much does the actual over-the-air transmission rate vary in 4G? I'd bet that
> it varies less than wifi does because wifi must retain compatibility with the
> earliest equipment while 4G no longer supports 2G protocols.

So in a scenario with a fixed (wall mounted outdoor unit) LTE device BQL could work 
quite well as the radio will not fluctuate too much.  

But for a mobil "pocket router" being carried around an AQL approach would be needed?  


-Erik

Re: [Bloat] Bufferbloat on 4G Connexion

[Toke Høiland-Jørgensen]

<erik.taraldsen@telenor.com> writes:

>> The key thing in both BQL and AQL is that instead of treating all packets the
>> same, you account for how long it will take to transmit them.
>>
>> In Wifi, your transmission rate can vary by a factor of 1000x (1Mb to over 1Gb
>> per second), so just counting bytes as BQL does is no longer good enough.
>>
>> how much does the actual over-the-air transmission rate vary in 4G? I'd bet that
>> it varies less than wifi does because wifi must retain compatibility with the
>> earliest equipment while 4G no longer supports 2G protocols.
>
> So in a scenario with a fixed (wall mounted outdoor unit) LTE device
> BQL could work quite well as the radio will not fluctuate too much.
>
> But for a mobil "pocket router" being carried around an AQL approach
> would be needed?

Having setup such an outdoor-mounted unit, I'd say that it's probably a
bit too optimistic to think that it doesn't fluctuate much. Cell
scheduling parameters (including activity from other stations), as well
as weather, can lead to quite significant fluctuations even in a
physically static setup.

So ideally, you'd want to be able to react to changes in either case.
The firmware should certainly have enough information to do that, but if
you can't get that changed, it may be possible to do something BQL-like
(either AQL, or simply a more dynamic variant of BQL) on the host.

-Toke

Re: [Bloat] Bufferbloat on 4G Connexion

[Jonathan Morton]

> On 24 Oct, 2019, at 2:51 pm, <erik.taraldsen@telenor.com> <erik.taraldsen@telenor.com> wrote:
> 
>> The key thing in both BQL and AQL is that instead of treating all packets the
>> same, you account for how long it will take to transmit them.
>> 
>> In Wifi, your transmission rate can vary by a factor of 1000x (1Mb to over 1Gb
>> per second), so just counting bytes as BQL does is no longer good enough.
>> 
>> how much does the actual over-the-air transmission rate vary in 4G? I'd bet that
>> it varies less than wifi does because wifi must retain compatibility with the
>> earliest equipment while 4G no longer supports 2G protocols.
> 
> So in a scenario with a fixed (wall mounted outdoor unit) LTE device BQL could work 
> quite well as the radio will not fluctuate too much.  
> 
> But for a mobil "pocket router" being carried around an AQL approach would be needed?  

Even in a fixed application, there's a lot of variation in where the unit is installed - from practically underneath the cell site tower to several miles away behind some trees - and service outages at the primary cell site could cause large, temporary changes of link bandwidth as the unit switches to a more distant cell.  Also, over the course of a day I personally observe tenfold variations in available bandwidth due to congestion at peak times, though I'm not quite certain whether that's due to radio contention on this particular cell site, or on my ISP's backhaul.

AQL would correct for these variations automatically.  All it needs in addition to BQL is some indication of how many bytes are appropriate to queue in the hardware at any given moment.

 - Jonathan Morton