[Bloat] Question about fq_codel vs modem buffers

[Bloat] Question about fq_codel vs modem buffers

[Rich Brown]

I posted a message about using SQM & OpenWrt on Tom's Hardware, and got a response from someone who's somewhat knowledgeable. I'm not sure of the proper response, so I wanted to ask here first. Here's the question that has me stumped.

http://www.tomshardware.com/answers/id-2615979/high-latency-person-internet.html#15786081

My thoughts:

I know fq_codel takes control of the bottleneck by being set to a couple percent below the fastest link speeds observed in each direction.

The author of the rebuttal says that the (DSL or Cable) modem buffers will fill up if the "upload drops from 3 to 2mbps". I can think of two ways this can happen:
	- Actual link bit rate drops
	- Congestion/oversubscription in the head end causes effective data rate to drop

But I don't know enough about the physical characteristics of cable/dsl links to understand how they actually work, nor how fq_codel can (or can't) accommodate degradation.

Could someone help me shape a response? Thanks.

Rich

Re: [Bloat] Question about fq_codel vs modem buffers

[Sebastian Moeller]

Hi Rich,

On May 2, 2015, at 04:49 , Rich Brown <richb.hanover@gmail.com> wrote:

> I posted a message about using SQM & OpenWrt on Tom's Hardware, and got a response from someone who's somewhat knowledgeable. I'm not sure of the proper response, so I wanted to ask here first. Here's the question that has me stumped.
> 
> http://www.tomshardware.com/answers/id-2615979/high-latency-person-internet.html#15786081
> 
> My thoughts:
> 
> I know fq_codel takes control of the bottleneck by being set to a couple percent below the fastest link speeds observed in each direction.
> 
> The author of the rebuttal says that the (DSL or Cable) modem buffers will fill up if the "upload drops from 3 to 2mbps". I can think of two ways this can happen:
> 	- Actual link bit rate drops

	Yes that can happen, e.g. in oversubscribed cable systems; currently sqm-scripts and or qos-scripts can NOT deal with this, gargoyle arguable can with its active controller (which is ping based and will allow to control reasonably slow bandwidth fluctuations on the order of seconds). This should be quite uncommon for DSL-links (caveat: seamless rate adaptation would make this les rare, but I yet have to see proof f SRA deployment in the field ;) ).

> 	- Congestion/oversubscription in the head end causes effective data rate to drop

	This should not fill up the modem’s buffer, but the DSLAM’s/MMTS's; but no matter where as long as the buffers are badly controlled buffer bloat will rear its ugly head again ;) In my opinion, the poster is right in theory the shaper needs to be set at <= the effective max transmit rate. Now oversubscription of the DSLAM’s/CMTS’s uplink is somewhat tricky to handle as it it will result in wildly fluctuating rates that are hard to track from the CPE side, also I think there is no guarantee that each CPE gets the same % of the uplink, so shaping in the CPE to reduce buffer bloat might result in no latency under load improvement while still sacrificing bandwidth. One really hopes that the head-end manufacturers get fq_codel into those devices (and potentially not in a flow fair, but in a prefix fair way so that all CPEs/customers are treated equally).
> 
> But I don't know enough about the physical characteristics of cable/dsl links to understand how they actually work, nor how fq_codel can (or can't) accommodate degradation.

	So currently we relay on HTB (or in the near future cake) to shape the rate so that we make sure the bottleneck link stays under our control. Neither HTB nor cake have enough insight in the upstream network topology/traffic to adjust their shaper to counter degradation of effective bandwidth. I really should look into gargoyle’s controller to figure out whether we can borrow this easily and include this as an add-on to sqm-scripts (it should not be rocket science, just configure a target-IP for continuous ping probes and react to latency increases above a threshold with rate reductions; the devil will be in the details, like when to open up the bandwidth again, and how much to actually change the throttle, but these should be open to empiric research).

Best Regards
	Sebastian

> 
> Could someone help me shape a response? Thanks.
> 
> Rich
> 
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Question about fq_codel vs modem buffers

[Jan Ceuleers]

On 02/05/15 04:49, Rich Brown wrote:
> But I don't know enough about the physical characteristics of
cable/dsl links to understand how they actually work, nor how fq_codel
can (or can't) accommodate degradation.

I know a little about the DSL PHY behaviour.

It is possible for line conditions to degrade, forcing the modems (i.e.
the one in the DSLAM and the one in the CPE) to retrain. It is also
possible that the newly-negotiated line rate is lower than the
previous-one, so the end user would indeed see a reduction in the
available bandwidth.

However, DSL operators (the good-ones anyway) measure, log and manage
the headroom between signal and noise on a per-line basis, so as to
avoid this kind of instability. This is the case particularly for
operators who also provide video services over DSL because retrains
(which take quite a few seconds) kill the user experience.

Take my line for example. I don't exactly know the length of my line to
the DSLAM but it must be at least 1200m. Up until a few months ago I had
25.5Mbit/s down and 2Mbit/s up using VDSL2 without vectoring. But I know
that my service provider, being in stiff competition with cable, has
been silently upping the bandwidth of people's lines by first gathering
stats on each individual line for a few months, and then, using a
feature called dynamic line management that allows them to model the
expected impact of increasing the rate of one line on the noise
(crosstalk) experienced by the other lines in the same cable bundle,
determine how far they can stretch each line.

I now have 30M down and 2M up, still without vectoring, and with a
training margin as reported by the modem of 9.6dB. (Before the speed
upgrade I vaguely remember it being something like 14dB).

So anyway: retrains are supposed to be very exceptional and indicative
of something being wrong.

By the way: another way for DSL operators to improve stability is to use
interleaving. Specifically: interleaving helps to improve stability in
the face of impulse noise at the expense of increased latency (which is
why I mention it here).

Jan

Re: [Bloat] Question about fq_codel vs modem buffers

[Alan Jenkins]

On 02/05/15 03:49, Rich Brown wrote:
> I posted a message about using SQM & OpenWrt on Tom's Hardware, and got a response from someone who's somewhat knowledgeable. I'm not sure of the proper response, so I wanted to ask here first. Here's the question that has me stumped.
>
> http://www.tomshardware.com/answers/id-2615979/high-latency-person-internet.html#15786081
>
> My thoughts:
>
> I know fq_codel takes control of the bottleneck by being set to a couple percent below the fastest link speeds observed in each direction.
>
> The author of the rebuttal says that the (DSL or Cable) modem buffers will fill up if the "upload drops from 3 to 2mbps". I can think of two ways this can happen:
> 	- Actual link bit rate drops
> 	- Congestion/oversubscription in the head end causes effective data rate to drop
>
> But I don't know enough about the physical characteristics of cable/dsl links to understand how they actually work, nor how fq_codel can (or can't) accommodate degradation.
>
> Could someone help me shape a response? Thanks.
>
> Rich

Basically don't worry too much about this.  The criticism was correct. 
Your critic is not spreading damagingly incorrect technical information 
and is on-cause AFAICT.  I don't think there's anything left to add to 
that thread now, just accept the criticism (post an agreement if you 
like, or not).

You have to "throttle" to under the link speed in openwrt.  If the link 
speed drops below that, your bloated modem will become the slowest link 
again ("bottleneck" - where it narrows and flows slowest, get it?), and 
a queue will build there.

OP has a variable (rural) link so it sucks.  But if you're noticing 
1000-4000ms bufferbloat that sucks more in many cases, if you can avoid 
it by throttling below your slowest usable link speed, you'll likely 
benefit from that.

Particularly if you're above 5Mbps anyway like this guy; allegedly 
that's the cutoff where throughput stops mattering so much for web browsing.

https://www.igvita.com/2012/07/19/latency-the-new-web-performance-bottleneck/
(un-cited, graph, sorry, I wish I knew where it came from).

As he says, the correct technical solution is to fix the modem to run 
fq_codel.  Then fq_codel will limit the queue that builds up at the 
bottleneck.  It avoids the need for throttling and will work with 
variable line rates.  But the modems aren't open-source, so we're 
demonstrating the fix with external routers.  Eventually ISPs will wise 
up and procure modems with fixes.  Like the PIE queuing discipline which 
is being standardized for next-generation cable modems.  (It's a start 
anyway, even if it's not as good as fq_codel).

Alan

Re: [Bloat] Question about fq_codel vs modem buffers

[Dave Taht]

On Sat, May 2, 2015 at 4:45 AM, Alan Jenkins
<alan.christopher.jenkins@gmail.com> wrote:
> On 02/05/15 03:49, Rich Brown wrote:
>>
>> I posted a message about using SQM & OpenWrt on Tom's Hardware, and got a
>> response from someone who's somewhat knowledgeable. I'm not sure of the
>> proper response, so I wanted to ask here first. Here's the question that has
>> me stumped.
>>
>>
>> http://www.tomshardware.com/answers/id-2615979/high-latency-person-internet.html#15786081
>>
>> My thoughts:
>>
>> I know fq_codel takes control of the bottleneck by being set to a couple
>> percent below the fastest link speeds observed in each direction.

in software rate limiting *HTB* takes control of the bottleneck.
fq_codel then manages the queue length and packet scheduling.

IF the fq_codel algorithm is instead located close to the mac layer of
the hardware and can "see" the latency, then it can respond to changes
in line rate.

This is how BQL works. It works fairly poorly (today) on wifi as there
are too many buffers between fq_codel and the wifi device (and a few
other reasons)

>> The author of the rebuttal says that the (DSL or Cable) modem buffers will
>> fill up if the "upload drops from 3 to 2mbps". I can think of two ways this
>> can happen:
>>         - Actual link bit rate drops
>>         - Congestion/oversubscription in the head end causes effective
>> data rate to drop

I tried to answer on the post, couldn't get it to take my answer. I wrote:

@kewix25:

You are correct:

A) if your line rate is not fixed, then software shaping htb +
fq_codel to a rate below that observed on one test will misbehave on
the times that the line rate drops below that. It is generally unknown
to what extent ISP line rates vary in this way, although line quality
and contention ARE two figures that most ISPs monitor closely. In all
my testing I have only found one cable plant (a large apartment
building) that regularly showed major variance in the actual line rate
achieved.

B) Bufferbloat.net's push has always been for the ISPs and CPE makers
to fix their gear in either hardware or software.

If in software:

It would help that - after getting a sqm-scripts install that worked -
for users (and their ISPs!) be monitoring the line with smokeping, to
see the cases where the line rate might drop and latencies skyrocket.

Other ongoing measurements are possible (see gargoyle-router project
for one implementation) - but the right answer has always been to get
the hardware right.

C) The promise in latency sensing AQM algorithms like pie and codel is
indeed that they can dynamically adjust to changes of rate... when the
algorithms are close enough to the hardware to measure it accurately.
Next up for bufferbloat.net's researchers is trying to tackle wifi,
which is far, far far more dynamic than dsl and cable links.

There is some discussion of these questions on the bufferbloat.net bloat list.
>> But I don't know enough about the physical characteristics of cable/dsl
>> links to understand how they actually work, nor how fq_codel can (or can't)
>> accommodate degradation.
>>
>> Could someone help me shape a response? Thanks.
>>
>> Rich
>
>
> Basically don't worry too much about this.  The criticism was correct. Your
> critic is not spreading damagingly incorrect technical information and is
> on-cause AFAICT.  I don't think there's anything left to add to that thread
> now, just accept the criticism (post an agreement if you like, or not).
>
> You have to "throttle" to under the link speed in openwrt.  If the link
> speed drops below that, your bloated modem will become the slowest link
> again ("bottleneck" - where it narrows and flows slowest, get it?), and a
> queue will build there.
>
> OP has a variable (rural) link so it sucks.  But if you're noticing
> 1000-4000ms bufferbloat that sucks more in many cases, if you can avoid it
> by throttling below your slowest usable link speed, you'll likely benefit
> from that.
>
> Particularly if you're above 5Mbps anyway like this guy; allegedly that's
> the cutoff where throughput stops mattering so much for web browsing.
>
> https://www.igvita.com/2012/07/19/latency-the-new-web-performance-bottleneck/
> (un-cited, graph, sorry, I wish I knew where it came from).
>
> As he says, the correct technical solution is to fix the modem to run
> fq_codel.  Then fq_codel will limit the queue that builds up at the
> bottleneck.  It avoids the need for throttling and will work with variable
> line rates.  But the modems aren't open-source, so we're demonstrating the
> fix with external routers.  Eventually ISPs will wise up and procure modems
> with fixes.  Like the PIE queuing discipline which is being standardized for
> next-generation cable modems.  (It's a start anyway, even if it's not as
> good as fq_codel).
>
> Alan
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
Dave Täht
Open Networking needs **Open Source Hardware**

https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67