[Bloat] RED against bufferbloat

[Bloat] RED against bufferbloat

[sahil grover]

[-- Attachment #1: Type: text/plain, Size: 431 bytes --]

(i) First of all,i want to know whether RED was implemented or not?
if not then what were the reasons(major) ?
anyone please tell me in simple words here only,because i don't want to
read any paper like "RED in a different light".

(ii)Second, as we all know RED controls the  average queue size from
growing.
So it also controls delay in a way or  we can say  is a solution to
bufferbloat problem. Then why it was not considered.

[-- Attachment #2: Type: text/html, Size: 654 bytes --]

Re: [Bloat] RED against bufferbloat

[Matt Mathis]

[-- Attachment #1: Type: text/plain, Size: 1262 bytes --]

RED doesn't work so well, by hindsight it has the wrong estimators and
control functions.   However every algorithm on the table today is in some
sense derived from that ground breaking work.

"Red in a different light" and others are the stepping stones between then
and now.

Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

Privacy matters!  We know from recent events that people are using our
services to speak in defiance of unjust governments.   We treat privacy and
security as matters of life and death, because for some users, they are.

On Tue, Feb 24, 2015 at 7:43 AM, sahil grover <sahilgrover013@gmail.com>
wrote:

> (i) First of all,i want to know whether RED was implemented or not?
> if not then what were the reasons(major) ?
> anyone please tell me in simple words here only,because i don't want to
> read any paper like "RED in a different light".
>
> (ii)Second, as we all know RED controls the  average queue size from
> growing.
> So it also controls delay in a way or  we can say  is a solution to
> bufferbloat problem. Then why it was not considered.
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
>

[-- Attachment #2: Type: text/html, Size: 2089 bytes --]

Re: [Bloat] RED against bufferbloat

[Kathleen Nichols]

Good, point. I like to think of the original RED as more 1) noting that all
those full buffers weren't doing anyone any good and 2) noting that one
could perhaps be aware of creeping congestion and "gently" prevent it.

Those two items are still important it's just that making 2) work has more
"ahas" in it than was apparent at first blush. I think it is good for us as
a community to figure out what doesn't work and what directions we should
explore.

For example, as has been mentioned, creating better mixes of traffic shows
the issues of acks mixed with data real fast. Separate queues fixes those
issues very nicely. But we have to be standing on a stepping stone along the
path to see this stuff.

	Kathie

On 2/24/15 8:13 AM, Matt Mathis wrote:
> RED doesn't work so well, by hindsight it has the wrong estimators and
> control functions.   However every algorithm on the table today is in
> some sense derived from that ground breaking work.
> 
> "Red in a different light" and others are the stepping stones between
> then and now.
> 
> Thanks,
> --MM--
> The best way to predict the future is to create it.  - Alan Kay
> 
> Privacy matters!  We know from recent events that people are using our
> services to speak in defiance of unjust governments.   We treat privacy
> and security as matters of life and death, because for some users, they are.
> 
> On Tue, Feb 24, 2015 at 7:43 AM, sahil grover <sahilgrover013@gmail.com
> <mailto:sahilgrover013@gmail.com>> wrote:
> 
>     (i) First of all,i want to know whether RED was implemented or not? 
>     if not then what were the reasons(major) ?
>     anyone please tell me in simple words here only,because i don't want
>     to read any paper like "RED in a different light".
> 
>     (ii)Second, as we all know RED controls the  average queue size from
>     growing.
>     So it also controls delay in a way or  we can say  is a solution to
>     bufferbloat problem. Then why it was not considered.
> 
>     _______________________________________________
>     Bloat mailing list
>     Bloat@lists.bufferbloat.net <mailto:Bloat@lists.bufferbloat.net>
>     https://lists.bufferbloat.net/listinfo/bloat
> 
> 
> 
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
> 

Re: [Bloat] RED against bufferbloat

[Mikael Abrahamsson]

On Tue, 24 Feb 2015, sahil grover wrote:

> (i) First of all,i want to know whether RED was implemented or not?
> if not then what were the reasons(major) ?

RED has been available on most platforms, but it was generally not turned 
on. It also needs configuration from an operator, and it's hard to know 
how to configure.

> anyone please tell me in simple words here only,because i don't want to 
> read any paper like "RED in a different light".

These issues are not simple. There are several presentations/talks 
available on Youtube on the issues if you want it in presentation form. 
Search for "Dave Taht", "Jim Gettys", "bufferbloat" and other such topics 
and you'll find excellent presentations from different forums.

> (ii)Second, as we all know RED controls the  average queue size from
> growing.
> So it also controls delay in a way or  we can say  is a solution to
> bufferbloat problem. Then why it was not considered.

It was designed to fix "bufferbloat" long before the bufferbloat word was 
even invented. It's just that in practice, it doesn't work very well. RED 
is configured with a drop probability slope at certain buffer depths, and 
that's it. It doesn't react or change depending on conditions. You have to 
guess at configure-time.

What we need are mechanisms that work better in real life and that are 
adaptive.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] RED against bufferbloat

[David Lang]

On Wed, 25 Feb 2015, Mikael Abrahamsson wrote:

> On Tue, 24 Feb 2015, sahil grover wrote:
>
>> (i) First of all,i want to know whether RED was implemented or not?
>> if not then what were the reasons(major) ?
>
> RED has been available on most platforms, but it was generally not turned on. 
> It also needs configuration from an operator, and it's hard to know how to 
> configure.
>
>> anyone please tell me in simple words here only,because i don't want to 
>> read any paper like "RED in a different light".
>
> These issues are not simple. There are several presentations/talks available 
> on Youtube on the issues if you want it in presentation form. Search for 
> "Dave Taht", "Jim Gettys", "bufferbloat" and other such topics and you'll 
> find excellent presentations from different forums.
>
>> (ii)Second, as we all know RED controls the  average queue size from
>> growing.
>> So it also controls delay in a way or  we can say  is a solution to
>> bufferbloat problem. Then why it was not considered.
>
> It was designed to fix "bufferbloat" long before the bufferbloat word was 
> even invented. It's just that in practice, it doesn't work very well. RED is 
> configured with a drop probability slope at certain buffer depths, and that's 
> it. It doesn't react or change depending on conditions. You have to guess at 
> configure-time.

more importantly (as I understand it), if you use RED while the rest of the 
users on the network stick with stock systems, you will keep yielding to them 
and only get to use a fraction of the available bandwidth.

David Lang

Re: [Bloat] RED against bufferbloat

[Mikael Abrahamsson]

On Tue, 24 Feb 2015, David Lang wrote:

> more importantly (as I understand it), if you use RED while the rest of 
> the users on the network stick with stock systems, you will keep 
> yielding to them and only get to use a fraction of the available 
> bandwidth.

I have a hard time imagining a setup where this would be the case. Could 
you please elaborate?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] RED against bufferbloat

[Alex Elsayed]

David Lang wrote:

> On Wed, 25 Feb 2015, Mikael Abrahamsson wrote:
> 
>> On Tue, 24 Feb 2015, sahil grover wrote:
>>
>>> (i) First of all,i want to know whether RED was implemented or not?
>>> if not then what were the reasons(major) ?
>>
>> RED has been available on most platforms, but it was generally not turned
>> on. It also needs configuration from an operator, and it's hard to know
>> how to configure.
>>
>>> anyone please tell me in simple words here only,because i don't want to
>>> read any paper like "RED in a different light".
>>
>> These issues are not simple. There are several presentations/talks
>> available on Youtube on the issues if you want it in presentation form.
>> Search for "Dave Taht", "Jim Gettys", "bufferbloat" and other such topics
>> and you'll find excellent presentations from different forums.
>>
>>> (ii)Second, as we all know RED controls the  average queue size from
>>> growing.
>>> So it also controls delay in a way or  we can say  is a solution to
>>> bufferbloat problem. Then why it was not considered.
>>
>> It was designed to fix "bufferbloat" long before the bufferbloat word was
>> even invented. It's just that in practice, it doesn't work very well. RED
>> is configured with a drop probability slope at certain buffer depths, and
>> that's it. It doesn't react or change depending on conditions. You have
>> to guess at configure-time.
> 
> more importantly (as I understand it), if you use RED while the rest of
> the users on the network stick with stock systems, you will keep yielding
> to them and only get to use a fraction of the available bandwidth.

I suspect you're conflating RED with delay-based congestion-control 
algorithms such as Vegas.

Re: [Bloat] RED against bufferbloat

[Bob Briscoe]

Sahil,

At 06:46 25/02/2015, Mikael Abrahamsson wrote:
>On Tue, 24 Feb 2015, sahil grover wrote:
>
>>(i) First of all,i want to know whether RED was implemented or not?
>>if not then what were the reasons(major) ?
>
>RED has been available on most platforms, but it was generally not 
>turned on. It also needs configuration from an operator, and it's 
>hard to know how to configure.

About a decade ago my company (BT) widely deployed RED in the 
upstream 'head-end' of our global MPLS network, i.e. the likely 
bottleneck in the customer edge router where the customer's LAN 
traffic enters their access link. We deployed it as WRED, i.e. 
different configurations of RED across the various diffserv classes, 
in order to minimise queuing latency in all the classes, including 
the lowest priority class. A configuration calculator was developed 
to help the engineers during set up. We still use this setup 
successfuly today, including for all our particularly latency 
sensitive customers in the finance sector.

We did not deploy RED on our broadband platform (ie public Internet), 
altho in retrospect we should have done, because any AQM is much 
better than none. We're fixing that now.

>>(ii)Second, as we all know RED controls the  average queue size from
>>growing.
>>So it also controls delay in a way or  we can say  is a solution to
>>bufferbloat problem. Then why it was not considered.
>
>It was designed to fix "bufferbloat" long before the bufferbloat 
>word was even invented. It's just that in practice, it doesn't work 
>very well. RED is configured with a drop probability slope at 
>certain buffer depths, and that's it. It doesn't react or change 
>depending on conditions. You have to guess at configure-time.
>
>What we need are mechanisms that work better in real life and that 
>are adaptive.

If you were prepared to read a paper, I would have suggested:
"The New AQM Kids on the Block: An Experimental Evaluation of CoDel and PIE"
<http://infocom2014.ieee-infocom.org/GI14-slides/GI14-s2-3.pdf>

This compares CoDel and PIE against Adaptive RED, which was a variant 
of RED proposed by Sally Floyd & co-authors in 2001 and available 
since Linux kernel version 3.3. ARED addressed the configuration 
sensitivity problem of RED by adapting the parameters to link rate 
and load conditions.

The paper convinced me that ARED is good enough (in the paper's 
simulations it was often better than PIE or CoDel), at least for 
links with fixed rate (or only occasionally varying rate like DSL).* 
This is important for us because it means we can consider deploying 
AQM by adding soft controls on top of the RED implementations we 
already have in existing equipment. This could reduce deployment 
completion time from decades to a few months.

* I'm not sure ARED would be able to cope with the rapidly changing 
rate of a wireless link tho.

HTH


Bob



________________________________________________________________
Bob Briscoe,                                                  BT 

Re: [Bloat] RED against bufferbloat

[Alex Elsayed]

Bob Briscoe wrote:

> Sahil,
> 
> At 06:46 25/02/2015, Mikael Abrahamsson wrote:
>>On Tue, 24 Feb 2015, sahil grover wrote:
>>
>>>(i) First of all,i want to know whether RED was implemented or not?
>>>if not then what were the reasons(major) ?
>>
>>RED has been available on most platforms, but it was generally not
>>turned on. It also needs configuration from an operator, and it's
>>hard to know how to configure.
> 
> About a decade ago my company (BT) widely deployed RED in the
> upstream 'head-end' of our global MPLS network, i.e. the likely
> bottleneck in the customer edge router where the customer's LAN
> traffic enters their access link. We deployed it as WRED, i.e.
> different configurations of RED across the various diffserv classes,
> in order to minimise queuing latency in all the classes, including
> the lowest priority class. A configuration calculator was developed
> to help the engineers during set up. We still use this setup
> successfuly today, including for all our particularly latency
> sensitive customers in the finance sector.
> 
> We did not deploy RED on our broadband platform (ie public Internet),
> altho in retrospect we should have done, because any AQM is much
> better than none. We're fixing that now.
> 
>>>(ii)Second, as we all know RED controls the  average queue size from
>>>growing.
>>>So it also controls delay in a way or  we can say  is a solution to
>>>bufferbloat problem. Then why it was not considered.
>>
>>It was designed to fix "bufferbloat" long before the bufferbloat
>>word was even invented. It's just that in practice, it doesn't work
>>very well. RED is configured with a drop probability slope at
>>certain buffer depths, and that's it. It doesn't react or change
>>depending on conditions. You have to guess at configure-time.
>>
>>What we need are mechanisms that work better in real life and that
>>are adaptive.
> 
> If you were prepared to read a paper, I would have suggested:
> "The New AQM Kids on the Block: An Experimental Evaluation of CoDel and
> PIE" <http://infocom2014.ieee-infocom.org/GI14-slides/GI14-s2-3.pdf>
> 
> This compares CoDel and PIE against Adaptive RED, which was a variant
> of RED proposed by Sally Floyd & co-authors in 2001 and available
> since Linux kernel version 3.3. ARED addressed the configuration
> sensitivity problem of RED by adapting the parameters to link rate
> and load conditions.
> 
> The paper convinced me that ARED is good enough (in the paper's
> simulations it was often better than PIE or CoDel), at least for
> links with fixed rate (or only occasionally varying rate like DSL).*
> This is important for us because it means we can consider deploying
> AQM by adding soft controls on top of the RED implementations we
> already have in existing equipment. This could reduce deployment
> completion time from decades to a few months.
> 
> * I'm not sure ARED would be able to cope with the rapidly changing
> rate of a wireless link tho.

One thing that was brought up on the CoDel list (which Sahil's original 
question was cross-posted to) by Dave Taht is that much of this testing 
utterly fails to account for two crucial factors:

1.) Asymmetric paths. When the uplink is considerably smaller than the 
downlink, he's seen significant behavioral differences - and that's 
_exactly_ the case of DSL.

2.) Elephants, mice and ants - response of mixed (and latency-sensitive) 
traffic under load. The RRUL (Realtime Response Under Load) toolkit he 
created is explicitly designed to test this case... which is a close match 
to common use cases like watching a Youtube video, but still needing things 
like DNS to be responsive. Or the bursty traffic of web browsing while a 
VoIP call is occurring.

The former is completely ignored by the presentation you linked to, and the 
latter is a one-line mention under "future work":

"More realistic traffic types (here, only bulk TCP traffic) including
bursty traffic"

Considering those, that slide deck convinces me of very, very little indeed.

Re: [Bloat] RED against bufferbloat

[Michael Welzl]

[-- Attachment #1: Type: text/plain, Size: 6296 bytes --]

Two points,

below...

On 25 Feb 2015, at 09:42, Alex Elsayed <eternaleye@gmail.com<mailto:eternaleye@gmail.com>> wrote:

Bob Briscoe wrote:

Sahil,

At 06:46 25/02/2015, Mikael Abrahamsson wrote:
On Tue, 24 Feb 2015, sahil grover wrote:

(i) First of all,i want to know whether RED was implemented or not?
if not then what were the reasons(major) ?

RED has been available on most platforms, but it was generally not
turned on. It also needs configuration from an operator, and it's
hard to know how to configure.

About a decade ago my company (BT) widely deployed RED in the
upstream 'head-end' of our global MPLS network, i.e. the likely
bottleneck in the customer edge router where the customer's LAN
traffic enters their access link. We deployed it as WRED, i.e.
different configurations of RED across the various diffserv classes,
in order to minimise queuing latency in all the classes, including
the lowest priority class. A configuration calculator was developed
to help the engineers during set up. We still use this setup
successfuly today, including for all our particularly latency
sensitive customers in the finance sector.

We did not deploy RED on our broadband platform (ie public Internet),
altho in retrospect we should have done, because any AQM is much
better than none. We're fixing that now.

(ii)Second, as we all know RED controls the  average queue size from
growing.
So it also controls delay in a way or  we can say  is a solution to
bufferbloat problem. Then why it was not considered.

It was designed to fix "bufferbloat" long before the bufferbloat
word was even invented. It's just that in practice, it doesn't work
very well. RED is configured with a drop probability slope at
certain buffer depths, and that's it. It doesn't react or change
depending on conditions. You have to guess at configure-time.

What we need are mechanisms that work better in real life and that
are adaptive.

If you were prepared to read a paper, I would have suggested:
"The New AQM Kids on the Block: An Experimental Evaluation of CoDel and
PIE" <http://infocom2014.ieee-infocom.org/GI14-slides/GI14-s2-3.pdf>

This compares CoDel and PIE against Adaptive RED, which was a variant
of RED proposed by Sally Floyd & co-authors in 2001 and available
since Linux kernel version 3.3. ARED addressed the configuration
sensitivity problem of RED by adapting the parameters to link rate
and load conditions.

The paper convinced me that ARED is good enough (in the paper's
simulations it was often better than PIE or CoDel), at least for
links with fixed rate (or only occasionally varying rate like DSL).*
This is important for us because it means we can consider deploying
AQM by adding soft controls on top of the RED implementations we
already have in existing equipment. This could reduce deployment
completion time from decades to a few months.

* I'm not sure ARED would be able to cope with the rapidly changing
rate of a wireless link tho.

One thing that was brought up on the CoDel list (which Sahil's original
question was cross-posted to) by Dave Taht is that much of this testing
utterly fails to account for two crucial factors:

1.) Asymmetric paths. When the uplink is considerably smaller than the
downlink, he's seen significant behavioral differences - and that's
_exactly_ the case of DSL.

2.) Elephants, mice and ants - response of mixed (and latency-sensitive)
traffic under load. The RRUL (Realtime Response Under Load) toolkit he
created is explicitly designed to test this case... which is a close match
to common use cases like watching a Youtube video, but still needing things
like DNS to be responsive. Or the bursty traffic of web browsing while a
VoIP call is occurring.

The former is completely ignored by the presentation you linked to,

Why exactly did you think we should have looked at asymmetric paths? To study what?
( I'm not debating that asymmetric paths play out different in behavior. I'm just saying that one needs to be clear about what exactly is being investigated, and why.)


and the
latter is a one-line mention under "future work":

"More realistic traffic types (here, only bulk TCP traffic) including
bursty traffic"

Considering those, that slide deck convinces me of very, very little indeed.

- it's not just a slide deck, it's a paper, in case you're interested. The longer version is freely available:
https://www.duo.uio.no/bitstream/handle/10852/37381/khademi-AQM_Kids_TR434.pdf?sequence=5

Why no other traffic types? Because we felt that looking at just bulk TCP is enough to make the key point that we wanted to get across: maybe it's worth taking a look at the vast amount of work that exists on AQMs, as even a stone old mechanism like ARED does not seem to generally be significantly worse than CoDel and PIE.

We didn't really want to sell ARED as it is as a much better solution under all conditions and say that it should now be used instead of CoDel and PIE (though we do conclude that it creating an ARED++ type mechanism seems worth considering). To make that point, I'd agree that one would have to see other traffic types too. A paper saying "We propose ARED++ as a replacement for CoDel or PIE" should have that.

My point is: when developing something new, compare against the state of the art. RED is *not* the state of the art, it's very old. I have seen arguments like needing parameterless operation because that was the failure of RED. Well, Sally Floyd addressed that with ARED ages ago. Most papers cited in this survey:
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6329367
begin with "RED didn't work because parameters had to be tuned. We propose a mechanism that doesn't require such tuning..."

What I've seen so far:
- CoDel compared with RED and BLUE
- PIE compared with CoDel
- ARED compared with CoDel and PIE

... it would seem reasonable to take one of the many, many mechanisms that exist - one that was already shown to be better than RED and many others - , make it use delay as input, and test CoDel and PIE against that. Then I'd say we have a comparison against the state of the art. Now we don't really have that, there's a gap here.

Cheers,
Michael


[-- Attachment #2: Type: text/html, Size: 25055 bytes --]

Re: [Bloat] RED against bufferbloat

[Sebastian Moeller]

Hi Michael,

On Feb 25, 2015, at 10:18 , Michael Welzl <michawe@ifi.uio.no> wrote:

> Two points,
> 
> below...
> 
> [...]
> Why exactly did you think we should have looked at asymmetric paths? To study what?
> ( I'm not debating that asymmetric paths play out different in behavior. I'm just saying that one needs to be clear about what exactly is being investigated, and why.)
> 
> [...]

	I have an opinion on that: because a) asymmetric is the general case and symmetric the special case b) a large percentage of end-nodes sit behind asymmetric paths that often are the bottlenecks that cause the queues for AQM to act on, so any AQM better work well in that situation to avoid spending its live in the ivory tower ;). So to put it in different words, as reality has an "asymmetry bias” ;) 


Best Regards

Re: [Bloat] RED against bufferbloat

[Michael Welzl]

> On 25 Feb 2015, at 10:29, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> Hi Michael,
> 
> On Feb 25, 2015, at 10:18 , Michael Welzl <michawe@ifi.uio.no> wrote:
> 
>> Two points,
>> 
>> below...
>> 
>> [...]
>> Why exactly did you think we should have looked at asymmetric paths? To study what?
>> ( I'm not debating that asymmetric paths play out different in behavior. I'm just saying that one needs to be clear about what exactly is being investigated, and why.)
>> 
>> [...]
> 
> 	I have an opinion on that: because a) asymmetric is the general case and symmetric the special case b) a large percentage of end-nodes sit behind asymmetric paths that often are the bottlenecks that cause the queues for AQM to act on, so any AQM better work well in that situation to avoid spending its live in the ivory tower ;). So to put it in different words, as reality has an "asymmetry bias” ;) 

So this is the type of argument I object against (this "realistic rather than ivory tower" argument is also what I saw coming from Dave Taht). The reason is simple: we want to understand what's going on. The more realistic we make it, the more factors play into the behavior, and the harder it becomes to understand.

If you want to get realistic, sure, create a wireless connection with interference and other hosts, background traffic that resembles reality, realistic load on the server, and in the end you have no clue why you see the results you see.

This is why we often use simplified scenarios such as dumbbell links etc. You want to start as simple as possible to get an understanding, and build from that.

Now, asymmetry by itself doesn't necessarily have a huge effect - it very much depends on the traffic type studied. If you're looking at downloads, then asymmetry lets you study the effect of losing ACKs. Then you're likely doing an investigation into the behaviour of delayed ACK vs. non-delayed ACK, and you'll get different results with a Linux receiver than with a FreeBSD receiver because of the no-delaying-during-slow-start thing that Linux is doing (sorry, I think it's got a name, just forgot it now).

That's all fine, but did you notice that in the previous paragraph I'm not talking about AQM?

To me, it's all about understanding what's going on, which means to minimize cross-effects of things that are not under investigation. If we want to study the effect of CoDel on ACKs, fine - but let's not look at a download with CoDel and make the link asymmetric just to be "more realistic" when this CAN have an influence on the result and then we don't know what's going on.

All this being said: ACKs are not usually a huge problem (thanks to delaying them), and I would be surprised if asymmetry would have changed the results in our paper significantly in one way or another. I would be *hugely* surprised if they would have consistently rendered one AQM mechanism better than another.

I suspect that some of you folks like realistic experiments so much because they let you see the power of Fair Queuing (certainly, FQ_whatever can give you consistently less delay than any AQM mechanism when ever you have multiple flows, which is maybe even more pronounced when combined with ACKs on asymmetric links). Well great, this is fine!  :-)   but that's FQ (or FQ_CoDel's changed FQ variant), much more than the AQM mechanism in use (as we have also seen presented by Toke at the last ICCRG meeting). But this discussion is about AQM mechanisms, not (changed)FQ.

Cheers,
Michael

Re: [Bloat] RED against bufferbloat

[Toke Høiland-Jørgensen]

Michael Welzl <michawe@ifi.uio.no> writes:

> but that's FQ (or FQ_CoDel's changed FQ variant), much more than the
> AQM mechanism in use (as we have also seen presented by Toke at the
> last ICCRG meeting).

Well, actually, that presentation did also include an evaluation of the
AQMs in an asymmetrical scenario. And that shows that while generally
ARED does perform fairly well, it tends to be a bit on the aggressive
side. In the asymmetrical case this results in too many drops on the
slow side of the asymmetrical link (typically upload), hurting throughput
in the other direction due to lost ACKs.

There's also some other issues in there, with PIE and CoDel in
particular, most notably their reactions when conditions change: it can
take tens of seconds for the algorithms to get queueing latency under
control in this case.

Slides for the IETF presentation available here:
http://www.ietf.org/proceedings/91/slides/slides-91-iccrg-4.pdf

There's also a longer version of the talk (from the Stanford Netseminar)
available on Youtube: https://www.youtube.com/watch?v=kePhqfKA3SM

> But this discussion is about AQM mechanisms, not (changed)FQ.

While the academic side of me enjoys studying AQMs (and I'm still far
from anything resembling a thorough understanding of them), the
practical "I just want my network to work" side of me has become
increasingly convinced (in part by doing the experiments in the above
mentioned talk) that FQ is more important than AQM in many scenarios. As
such, I think that excluding FQ from the conversation is mostly of, well,
academic interest ;)

-Toke

Re: [Bloat] RED against bufferbloat

[Mikael Abrahamsson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2080 bytes --]

On Wed, 25 Feb 2015, Toke Høiland-Jørgensen wrote:

> While the academic side of me enjoys studying AQMs (and I'm still far 
> from anything resembling a thorough understanding of them), the 
> practical "I just want my network to work" side of me has become 
> increasingly convinced (in part by doing the experiments in the above 
> mentioned talk) that FQ is more important than AQM in many scenarios. As 
> such, I think that excluding FQ from the conversation is mostly of, 
> well, academic interest ;)

To me, FQ is important especially for lower speeds. This also maps well 
into the CPU based nature of FQ (that it's hard to do in "silicon").

We're not going to see FQ_CODEL on a 200 interface large router designed 
to push 100 gigabit/s of traffic, at least not in any interesting price 
point.

So one question that I would be interested in understanding is this 
scenario:

INTERNET----AR---CPE--HOST

Let's now say the AR has a stupid token bucket based policer (which 
doesn't queue, just drops packets) rate-limiting the traffic input/output 
towards the CPE to, let's say 120 megabit/s averaged with 1 second worth 
of burst bytes (because the ISP wants to make sure the customer can 
speed-test to 100 megabit/s of TCP throughput).

Can we do bidirectional traffic FQ_CODEL on the CPE to try to achieve 
basically zero loss in the AR policer for any normal kind of use scenario 
using TCP. I have not seen any simulations looking at this.

What about 12 megabit/s policer to a 10 megabit/s service? What about a 
12 megabit/s FIFO shaper (that would actually delay packets, but because 
it's FIFO we just rarely want to see buffering there).

Because my belief is that even though we might say we love FQ here, we're 
not going to see high speed FQ on higher end ISP aggregation routers. So 
if we want FQ, we're going to have to do it in the CPEs.

I am using terminology that is explained in this article:
http://www.cisco.com/c/en/us/support/docs/quality-of-service-qos/qos-policing/19645-policevsshape.html

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] RED against bufferbloat

[Toke Høiland-Jørgensen]

Mikael Abrahamsson <swmike@swm.pp.se> writes:

> To me, FQ is important especially for lower speeds. This also maps
> well into the CPU based nature of FQ (that it's hard to do in
> "silicon").

Well my research has been mainly focused on the consumer / edge case.
I.e. things a Linux box can drive in hardware, so up to 100Mbit to a
Gbit depending on the size of the box. In these cases, fq_codel adds no
measurable overhead compared to the CPU load of just pushing the
packets.

> We're not going to see FQ_CODEL on a 200 interface large router
> designed to push 100 gigabit/s of traffic, at least not in any
> interesting price point.

Maybe not; but note that recent work in the Linux kernel makes it quite
capable of pushing 40GigE, and I believe sched_fq (which does 'real'
fairness queueing with a queue per flow, not hash buckets as fq_codel
does) has been shown to scale to millions of concurrent flows.

> Can we do bidirectional traffic FQ_CODEL on the CPE to try to achieve
> basically zero loss in the AR policer for any normal kind of use
> scenario using TCP. I have not seen any simulations looking at this.

Well this is basically what we're doing with the SQM scripts in CeroWRT:
put a software rate limiter in the CPE and configure it to a slightly
lower value than whatever the cable/DSL modem runs at. This works quite
well, but the software rate limiter is fairly CPU hungry, so small boxes
struggle. I had to substitute a full x86 box for my Netgear router to
run at 100Mbit for my home connection.

> Because my belief is that even though we might say we love FQ here,
> we're not going to see high speed FQ on higher end ISP aggregation
> routers. So if we want FQ, we're going to have to do it in the CPEs.

Well OpenWRT turns on fq_codel by default now. :)

And yeah, I'm not holding by breath for big iron vendors to include
fq_codel in their products. But that doesn't mean it can't go into CPEs.
And end hosts and access points. And inside tunnels (VPN daemons for
instance).

-Toke

Re: [Bloat] RED against bufferbloat

[Bill Ver Steeg (versteb)]

Regarding the statement
> We're not going to see FQ_CODEL on a 200 interface large router 
> designed to push 100 gigabit/s of traffic, at least not in any 
> interesting price point.

There are two points under the covers here. I would say a 2 more accurate statements would be -

1- "We are unlikely to see any AQM scheme that includes many queues in a core router." This includes FQ_CODEL, FQ_PIE, and any other scheme that uses a statistical hashing of flows into a large number of queues. The required silicon to support so many queues is simply too expensive. There may be new silicon designs that make this more feasible, but this is non-trivial.
2- "There are several AQMs under consideration for use in a core router. There are advantages to <doing work> at enqueue time rather than dequeue time, and this may drive some designs." In other words, AQM algorithms that are similar to PIE are likely to be in large aggregation devices (like a CMTS) and core routers. You could probably design new silicon that did work at de-queue time, but that is simply not how the current designs operate. 

The good news is that as we debate the relative merits of the new AQM schemes, they are all MUCH better than the legacy algorithms. Putting FQ_XXX (where XXX== CODEL today, and perhaps other algorithms in the future) into devices that do queue management in software is a big win. Putting CODEL/PIE (using a handful of QOS mapped queues, using microcode running on current generation silicon) into the big iron is also a big win.

So, let's get modern AQM pushed into all of the devices. One design will not fit all devices........


Bill Ver Steeg
DISTINGUISHED ENGINEER 
versteb@cisco.com












-----Original Message-----
From: bloat-bounces@lists.bufferbloat.net [mailto:bloat-bounces@lists.bufferbloat.net] On Behalf Of Toke Høiland-Jørgensen
Sent: Wednesday, February 25, 2015 6:05 AM
To: Mikael Abrahamsson
Cc: bloat@lists.bufferbloat.net
Subject: Re: [Bloat] RED against bufferbloat

Mikael Abrahamsson <swmike@swm.pp.se> writes:

> To me, FQ is important especially for lower speeds. This also maps 
> well into the CPU based nature of FQ (that it's hard to do in 
> "silicon").

Well my research has been mainly focused on the consumer / edge case.
I.e. things a Linux box can drive in hardware, so up to 100Mbit to a Gbit depending on the size of the box. In these cases, fq_codel adds no measurable overhead compared to the CPU load of just pushing the packets.

> We're not going to see FQ_CODEL on a 200 interface large router 
> designed to push 100 gigabit/s of traffic, at least not in any 
> interesting price point.

Maybe not; but note that recent work in the Linux kernel makes it quite capable of pushing 40GigE, and I believe sched_fq (which does 'real'
fairness queueing with a queue per flow, not hash buckets as fq_codel
does) has been shown to scale to millions of concurrent flows.

> Can we do bidirectional traffic FQ_CODEL on the CPE to try to achieve 
> basically zero loss in the AR policer for any normal kind of use 
> scenario using TCP. I have not seen any simulations looking at this.

Well this is basically what we're doing with the SQM scripts in CeroWRT:
put a software rate limiter in the CPE and configure it to a slightly lower value than whatever the cable/DSL modem runs at. This works quite well, but the software rate limiter is fairly CPU hungry, so small boxes struggle. I had to substitute a full x86 box for my Netgear router to run at 100Mbit for my home connection.

> Because my belief is that even though we might say we love FQ here, 
> we're not going to see high speed FQ on higher end ISP aggregation 
> routers. So if we want FQ, we're going to have to do it in the CPEs.

Well OpenWRT turns on fq_codel by default now. :)

And yeah, I'm not holding by breath for big iron vendors to include fq_codel in their products. But that doesn't mean it can't go into CPEs.
And end hosts and access points. And inside tunnels (VPN daemons for instance).

-Toke
_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] RED against bufferbloat

[MUSCARIELLO Luca IMT/OLN]

On 02/25/2015 07:39 PM, Bill Ver Steeg (versteb) wrote:
> Regarding the statement
>> We're not going to see FQ_CODEL on a 200 interface large router
>> designed to push 100 gigabit/s of traffic, at least not in any
>> interesting price point.
> There are two points under the covers here. I would say a 2 more accurate statements would be -
>
> 1- "We are unlikely to see any AQM scheme that includes many queues in a core router." This includes FQ_CODEL, FQ_PIE, and any other scheme that uses a statistical hashing of flows into a large number of queues. The required silicon to support so many queues is simply too expensive. There may be new silicon designs that make this more feasible, but this is non-trivial.

I agree and at the same time it is not the core the segment where this 
would be useful.
If we have the same understanding of core.

> 2- "There are several AQMs under consideration for use in a core router. There are advantages to <doing work> at enqueue time rather than dequeue time, and this may drive some designs." In other words, AQM algorithms that are similar to PIE are likely to be in large aggregation devices (like a CMTS) and core routers. You could probably design new silicon that did work at de-queue time, but that is simply not how the current designs operate.

I am not familiar with CMTS, but I guess it is not much larger than an OLT.
I do not know the maximum rates neither in cable modems but these
equipments might be incapable to guarantee very high rates and proper QoS
when dealing with 1Gbps to the customer.
If the target is up to 100Mbps the story is different but honestly at 
this rate
this can be done in software today for such a small fanout.

I wonder when we'll have software routers in residential networks to 
innovate a little
faster than what happens today just like how already happens  in some 
data centers.

Re: [Bloat] RED against bufferbloat

[Mikael Abrahamsson]

On Thu, 26 Feb 2015, MUSCARIELLO Luca IMT/OLN wrote:

> I wonder when we'll have software routers in residential networks to 
> innovate a little faster than what happens today just like how already 
> happens in some data centers.

Well, it would help if operators tried to buy CPEs taht were not only 
bare-minimum for what they need today.

Right now I hear more about "virtualised CPE" to save cost (ie move part 
of the CPE implementation to the data center) than to spend more money on 
features in the CPE.

My hope is still on the mobile phone SoCs trickling down to the home 
gateway space so we get more CPU power there at decent price point.

Also possibly that the whole SDN movement means hardware will get 
standardized APIs so in case there is hardware acceleration on the CPE, 
this can be handled by any kernel and not just the kernel that the vendor 
has modified to work with their special hardware.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] RED against bufferbloat

[Toke Høiland-Jørgensen]

Mikael Abrahamsson <swmike@swm.pp.se> writes:

> Right now I hear more about "virtualised CPE" to save cost (ie move
> part of the CPE implementation to the data center) than to spend more
> money on features in the CPE.

Erm, how does that work? What part of the functionality can reasonably
be moved to the data center? The configuration web interface?

-Toke

Re: [Bloat] RED against bufferbloat

[Mikael Abrahamsson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 652 bytes --]

On Thu, 26 Feb 2015, Toke Høiland-Jørgensen wrote:

> Erm, how does that work? What part of the functionality can reasonably 
> be moved to the data center? The configuration web interface?

Basically you extend the home L2 domain via some kind of tunnel or vlan to 
a server, and run the CPE there.

The only thing left in the home is basically L2 bridging between WAN, LAN 
and Wifi and the possibiliy of configuring settings such as Wifi SSID, 
crypto etc.

http://www.juniper.net/techpubs/en_US/junos14.1/topics/concept/ccpe-overview.html

http://blogs.cisco.com/sp/home-gateway-cpe-virtualization

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] RED against bufferbloat

[Toke Høiland-Jørgensen]

Mikael Abrahamsson <swmike@swm.pp.se> writes:

> Basically you extend the home L2 domain via some kind of tunnel or
> vlan to a server, and run the CPE there.
>
> The only thing left in the home is basically L2 bridging between WAN,
> LAN and Wifi and the possibiliy of configuring settings such as Wifi
> SSID, crypto etc.

Ah, I see. Yeah, that does seem to be the wrong direction... :(

-Toke

Re: [Bloat] RED against bufferbloat

[Sebastian Moeller]

On Feb 26, 2015, at 11:44 , Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Thu, 26 Feb 2015, Toke Høiland-Jørgensen wrote:
> 
>> Erm, how does that work? What part of the functionality can reasonably be moved to the data center? The configuration web interface?
> 
> Basically you extend the home L2 domain via some kind of tunnel or vlan to a server, and run the CPE there.

	We would need a new extension for CPE, for sure, "customer premise” will no longer describe it well. Also egress shaping will need to stay at the customers end of the bottleneck link (or better all uplinks increased to either 10, 100 or 1000 Mbps so that ethernet link speeds can be used to avoid the link to become the bottleneck). Also the typical switch and AP will be somewhat hard to virtualize… And then we basically need the same hardware at the customer end as we have right now, simply for the modem switch and AP parts...

Best Regards




> 
> The only thing left in the home is basically L2 bridging between WAN, LAN and Wifi and the possibiliy of configuring settings such as Wifi SSID, crypto etc.
> 
> http://www.juniper.net/techpubs/en_US/junos14.1/topics/concept/ccpe-overview.html

	… Deep Inspection/Monitoring (lovely, I am not sure whether Juniper actually want to market to ISPs or the NSA here)

> 
> http://blogs.cisco.com/sp/home-gateway-cpe-virtualization

	• Security and firewall features
	• Parental control services: 

I could be wrong, but I fail to see the business case for an ISP to switch from traditional CPE to basically traditional CPE plus “virtual CPE” in the data center (to badly  paraphrase J. Zawinski “now you have two problems");

> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se_______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] RED against bufferbloat

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 401 bytes --]

On 26 Feb 2015 12:45, "Mikael Abrahamsson" <swmike@swm.pp.se> wrote:
> The only thing left in the home is basically L2 bridging between WAN, LAN
and Wifi and the possibiliy of configuring settings such as Wifi SSID,
crypto etc.

Funny, I thought that was what CPE was supposed to do in the first place.
The only thing left out of that picture is the firewalling effect of IPv4
NAT.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 512 bytes --]

Re: [Bloat] RED against bufferbloat

[Dave Taht]

On Thu, Feb 26, 2015 at 2:44 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Thu, 26 Feb 2015, Toke Høiland-Jørgensen wrote:
>
>> Erm, how does that work? What part of the functionality can reasonably be
>> moved to the data center? The configuration web interface?
>
>
> Basically you extend the home L2 domain via some kind of tunnel or vlan to a
> server, and run the CPE there.
>
> The only thing left in the home is basically L2 bridging between WAN, LAN
> and Wifi and the possibiliy of configuring settings such as Wifi SSID,
> crypto etc.

> http://www.juniper.net/techpubs/en_US/junos14.1/topics/concept/ccpe-overview.html

"Some functions that are delivered today by other types of CPE devices
(for example, security appliances, storage servers, WiFi hotspots, and
so forth) may also shift partially or totally to the cloud CPE
architecture."

that's terrifying. They want me to move *MY firewall** into *their
cloud*? What could possibly go wrong?

> http://blogs.cisco.com/sp/home-gateway-cpe-virtualization
>
>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>



-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

Re: [Bloat] RED against bufferbloat

[Sebastian Moeller]

Hi Mikael,


On Feb 26, 2015, at 11:39 , Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Thu, 26 Feb 2015, MUSCARIELLO Luca IMT/OLN wrote:
> 
>> I wonder when we'll have software routers in residential networks to innovate a little faster than what happens today just like how already happens in some data centers.
> 
> Well, it would help if operators tried to buy CPEs taht were not only bare-minimum for what they need today.

	What is quite ironic is that many operators now switched to charge rental fees for their under-powered, and not timely upgraded devices, rent that could easily be used to purchase far more capable devices (that would actually justify the rental fee).

> 
> Right now I hear more about "virtualised CPE" to save cost (ie move part of the CPE implementation to the data center) than to spend more money on features in the CPE.

	Well, the ingress traffic shaping definitely should be virtualized upstream of the actual CPE!

> 
> My hope is still on the mobile phone SoCs trickling down to the home gateway space so we get more CPU power there at decent price point.

	But do we know whether mobile phone SOCs actually make good routers?

Best Regards
	Sebastian

> 
> Also possibly that the whole SDN movement means hardware will get standardized APIs so in case there is hardware acceleration on the CPE, this can be handled by any kernel and not just the kernel that the vendor has modified to work with their special hardware.
> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] RED against bufferbloat

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 1056 bytes --]

> But do we know whether mobile phone SOCs actually make good routers?

Many of them probably don't, due to limited off chip I/O bandwidth, but
their CPU cores would be useful in an SoC which did have such bandwidth.

The BCM2435/6 used in the Raspberry Pi/2 is a classic example of such a
phone SoC; its I/O is very flexible and its GPU remarkably powerful (really
it is a GPU with a vestigial CPU hanging off the side), but the only high
bandwidth off-chip interfaces are HDMI and USB 2, so it can just about
handle Fast Ethernet but doesn't have a hope of reliably sustaining low
latency at full line rate.

However it does demonstrate that a quad Cortex A7 CPU cluster is now very
cheap to implement. One of those integrated with a couple of GMACs and a
couple of PCIe lanes for Wi-Fi would make a good router SoC.

While I'm on the subject, I think the whole concept of web configurators is
wrong-headed. Give the box a little LCD and let the user plug a USB
keyboard in. Security problems solved (well, reduced a lot) at a stroke.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 1181 bytes --]

Re: [Bloat] RED against bufferbloat

[Mikael Abrahamsson]

On Thu, 26 Feb 2015, Jonathan Morton wrote:

> However it does demonstrate that a quad Cortex A7 CPU cluster is now 
> very cheap to implement. One of those integrated with a couple of GMACs 
> and a couple of PCIe lanes for Wi-Fi would make a good router SoC.

I for instance think this looks promising:

http://www.marvell.com/embedded-processors/armada-38x/

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] RED against bufferbloat

[MUSCARIELLO Luca IMT/OLN]

On 02/26/2015 11:39 AM, Mikael Abrahamsson wrote:
> On Thu, 26 Feb 2015, MUSCARIELLO Luca IMT/OLN wrote:
>
>> I wonder when we'll have software routers in residential networks to 
>> innovate a little faster than what happens today just like how 
>> already happens in some data centers.
>
> Well, it would help if operators tried to buy CPEs taht were not only 
> bare-minimum for what they need today.

I think the problem is a bit more complex. For what concerns queuing  
the chipcos
do a lot of offloading in silicon. This is an entire industry choice and 
that silicon
is programmable but opaque and not open.
If chipcos made that choice there is a reason.


>
> Right now I hear more about "virtualised CPE" to save cost (ie move 
> part of the CPE implementation to the data center) than to spend more 
> money on features in the CPE.

That won't solve the problem discussed in this list. I like the approach 
though.

>
> My hope is still on the mobile phone SoCs trickling down to the home 
> gateway space so we get more CPU power there at decent price point.
>
> Also possibly that the whole SDN movement means hardware will get 
> standardized APIs so in case there is hardware acceleration on the 
> CPE, this can be handled by any kernel and not just the kernel that 
> the vendor has modified to work with their special hardware.

I don't  how SDN would help for queuing.
DPDK, netmap instead would help but we are far from having that kind of 
support in CPEs.

Re: [Bloat] RED against bufferbloat

[Mikael Abrahamsson]

On Thu, 26 Feb 2015, MUSCARIELLO Luca IMT/OLN wrote:

>> Also possibly that the whole SDN movement means hardware will get 
>> standardized APIs so in case there is hardware acceleration on the CPE, 
>> this can be handled by any kernel and not just the kernel that the vendor 
>> has modified to work with their special hardware.
>
> I don't  how SDN would help for queuing.

I never said it would. I said that the SDN movement might give 
standardized APIs into what the forwarding hardware does, including 
packet treatment such as queuing behaviour. It might also mean that if 
someone implemented a certain queuing behaviour and this ended up defined 
by the API description, more vendors might implement the same thing.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] RED against bufferbloat

[Sebastian Moeller]

Hallo Mikael,

On Feb 25, 2015, at 11:47 , Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Wed, 25 Feb 2015, Toke Høiland-Jørgensen wrote:
> 
>> While the academic side of me enjoys studying AQMs (and I'm still far from anything resembling a thorough understanding of them), the practical "I just want my network to work" side of me has become increasingly convinced (in part by doing the experiments in the above mentioned talk) that FQ is more important than AQM in many scenarios. As such, I think that excluding FQ from the conversation is mostly of, well, academic interest ;)
> 
> To me, FQ is important especially for lower speeds. This also maps well into the CPU based nature of FQ (that it's hard to do in "silicon").
> 
> We're not going to see FQ_CODEL on a 200 interface large router designed to push 100 gigabit/s of traffic, at least not in any interesting price point.
> 
> So one question that I would be interested in understanding is this scenario:
> 
> INTERNET----AR---CPE--HOST
> 
> Let's now say the AR has a stupid token bucket based policer (which doesn't queue, just drops packets) rate-limiting the traffic input/output towards the CPE to, let's say 120 megabit/s averaged with 1 second worth of burst bytes (because the ISP wants to make sure the customer can speed-test to 100 megabit/s of TCP throughput).
> 
> Can we do bidirectional traffic FQ_CODEL on the CPE to try to achieve basically zero loss in the AR policer for any normal kind of use scenario using TCP. I have not seen any simulations looking at this.

	Yes, we can. Openwrt’s qos-scripts and cerowrt’s sqm-scripts allow exactly that. But we really could need some help from the station on the other end of the bottleneck-link, so either a cmts or a dslam (what ever these are called today). The shaper for the ingress link to the CPE is best positioned in front of the boot;neck instead of behind it in the CPE (in the CPE we need to sacrifice more bandwidth and in case of a sudden inrush of packets ill still see more AR policer action than we would like).
 	The only argument for ingress shaping on the CPE is that this allows the end user to define her own QOS criteria independent of the ISPs wishes. Best of both worlds would be user configurable QOS-shaping on the slam/bras/whatever…

Best Regards
	Sebastian

> 
> What about 12 megabit/s policer to a 10 megabit/s service? What about a 12 megabit/s FIFO shaper (that would actually delay packets, but because it's FIFO we just rarely want to see buffering there).
> 
> Because my belief is that even though we might say we love FQ here, we're not going to see high speed FQ on higher end ISP aggregation routers. So if we want FQ, we're going to have to do it in the CPEs.
> 
> I am using terminology that is explained in this article:
> http://www.cisco.com/c/en/us/support/docs/quality-of-service-qos/qos-policing/19645-policevsshape.html
> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se_______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] RED against bufferbloat

[Mikael Abrahamsson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 734 bytes --]

On Wed, 25 Feb 2015, Sebastian Moeller wrote:

> 	The only argument for ingress shaping on the CPE is that this 
> allows the end user to define her own QOS criteria independent of the 
> ISPs wishes. Best of both worlds would be user configurable QOS-shaping 
> on the slam/bras/whatever…

As I said before, doing FQ_CODEL in the AR is an expensive proposition for 
medium and high speed access. So if this could successfully be pushed to 
the CPE it would mean it would be more widely deployed.

I am very much aware that this is being done (I have done it myself), but 
my question was if someone had actually done this in a lab and found out 
how well it works in RRUL tests etc.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] RED against bufferbloat

[Toke Høiland-Jørgensen]

Mikael Abrahamsson <swmike@swm.pp.se> writes:

> I am very much aware that this is being done (I have done it myself),
> but my question was if someone had actually done this in a lab and
> found out how well it works in RRUL tests etc.

So you mean comparing the scenario where the AQM runs on both sides of
the bottleneck to one where it runs as an ingress shaper on one side
only?

-Toke

Re: [Bloat] RED against bufferbloat

[Mikael Abrahamsson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 988 bytes --]

On Wed, 25 Feb 2015, Toke Høiland-Jørgensen wrote:

> So you mean comparing the scenario where the AQM runs on both sides of 
> the bottleneck to one where it runs as an ingress shaper on one side 
> only?

Yes. How much lower speed/rate would the CPE ingress (internet->CPE) AQM 
have to run at to have a reasonable low probability of traffic being 
delayed by that shaper (or dropped by the policer).

I realise this would be different depending on speed of the access, but 
some data points would be interesting.

For instance, some ISPs I've heard will have a policer and 2 seconds worth 
of burst, before they drop packets. Some others might have lower burst 
values. Some will have a pretty decent FIFO shaper. There are some 
different scenarios, how much lower speed do I need to run my AQM at in 
order to try to avoid this policer or shaper to affect my traffic? Gut 
feeling would be 10-20% should be enough, but I don't know.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] RED against bufferbloat

[Bill Ver Steeg (versteb)]

I have seen some (to date unpublished) studies that show a ~5% "guard band" below the link rate can be enough to provide both good performance and bloat resistance. Take that with a grain of salt, though.

The real-world problem is that the anti-bloat proxy does not really know the bottleneck link rate. At 2 in the morning on an SP network (aka HFC, DSL or FTTx) , you are quite likely to get your SLA rate. At 7PM on a Friday afternoon, the link rate will probably be degraded. If you shape to the SLA rate when the network is busy, you are very likely to not have the desired behavior.

Throw in some "turbo boost" (where the network operator allows a large spike of traffic if the subscriber has been idle for a while) and the problem gets even harder. Turbo boost has very favorable benefits for bursty HTTP browsing traffic, and you do not want to miss out on that benefit by having a bump-in-the-wire doing rate shaping.

IMHO, the correct answer is to run a modern AQM algorithm (take your pick, they are all better than tail drop) on all devices. If the bottleneck link has a variable link speed, it is in a position to manage the buffer effectively.  Until we get to that point, there may be some benefit in proxy solutions, but there are dragons here.


Bill Ver Steeg
DISTINGUISHED ENGINEER 
versteb@cisco.com












-----Original Message-----
From: bloat-bounces@lists.bufferbloat.net [mailto:bloat-bounces@lists.bufferbloat.net] On Behalf Of Mikael Abrahamsson
Sent: Wednesday, February 25, 2015 9:06 AM
To: Toke Høiland-Jørgensen
Cc: bloat@lists.bufferbloat.net
Subject: Re: [Bloat] RED against bufferbloat

On Wed, 25 Feb 2015, Toke Høiland-Jørgensen wrote:

> So you mean comparing the scenario where the AQM runs on both sides of 
> the bottleneck to one where it runs as an ingress shaper on one side 
> only?

Yes. How much lower speed/rate would the CPE ingress (internet->CPE) AQM have to run at to have a reasonable low probability of traffic being delayed by that shaper (or dropped by the policer).

I realise this would be different depending on speed of the access, but some data points would be interesting.

For instance, some ISPs I've heard will have a policer and 2 seconds worth of burst, before they drop packets. Some others might have lower burst values. Some will have a pretty decent FIFO shaper. There are some different scenarios, how much lower speed do I need to run my AQM at in order to try to avoid this policer or shaper to affect my traffic? Gut feeling would be 10-20% should be enough, but I don't know.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] RED against bufferbloat

[MUSCARIELLO Luca IMT/OLN]

On 02/25/2015 02:36 PM, Mikael Abrahamsson wrote:
>>
>
> As I said before, doing FQ_CODEL in the AR is an expensive proposition 
> for medium and high speed access. So if this could successfully be 
> pushed to the CPE it would mean it would be more widely deployed.
I do not agree. Well, it depends on what you mean with expensive.

Doing FQ in silicon is easy. It must be very easy as I did myself in a 
MIPS Ikanos Vx185 chipset
and I am not an hardware expert. This was for a CPE with a 5X1Gbps ports.

If you go a little deeper in the network and you pick an OLT you won't 
find much intelligence.
A little deeper and in a local aggregation router (all vendors) you'll 
find what you would
need to implement FQ. A card here is already managing several ports at 
10Gbps and the
challenge is more or less the same as in the CPE down to the customer 
(with downsized rates/fanouts
and resources: cpu and memory).
The hardware resources (CPU and memory)
available in current  equipment are sized to do shaping/policy and some 
AQMs.
No guarantees it works as you would expect but technically FQ is not 
expensive if
properly implemented.

If you mean that it is expensive in terms of time (and probably money) 
to educate equipment
vendors that we need much more than what you have now, than yes I do agree.

At the same time I acknowledge, as Bob wrote somewhere in this thread, 
that doing
nothing is worse than doing something, even if suboptimal.

Some "positive" view: access with Gbps (up to 1Gbps) with a range of RTT 
(1ms to 100ms)
will need smarter mechanisms in the equipment as inefficiencies will be 
crystal clear
and business consequences will be substantially different.

  Luca

Re: [Bloat] RED against bufferbloat

[Mikael Abrahamsson]

On Wed, 25 Feb 2015, MUSCARIELLO Luca IMT/OLN wrote:

> Doing FQ in silicon is easy. It must be very easy as I did myself in a 
> MIPS Ikanos Vx185 chipset and I am not an hardware expert. This was for 
> a CPE with a 5X1Gbps ports.

I guess we have different definitions on what "silicon" is. Mine is 
"ASIC". If it has a MIPS CPU (and the CPU does forwarding), then it's not 
"silicon", then it's done in CPU.

I keep hearing from vendors that queues are expensive. The smallest FQ 
implementation that seems to be reasonable has 32 queues. Let's say we 
have 5k customers per port on a BNG, that equates to 160k queues.

Compare that to an efficient active ethernet p-t-p ETTH deployment without 
BNG at all, where the access L2 switch is the one doing policing. What 
would be the cost to equip this device with FQ per port?

> If you go a little deeper in the network and you pick an OLT you won't

I don't do PON.

> find much intelligence. A little deeper and in a local aggregation 
> router (all vendors) you'll find what you would need to implement FQ. A

If it's a BNG yes. These linecards with hundreds of thousands of queues 
are a lot more expensive than a linecard without these queues. Usually 
today there are 3-4 queues per customer, now we want to expand this to 32 
or more. What is the cost increase for this?

If you put a BNG in there with lots of intelligence then the cost of 
adding the FQ machinery might not be too bad. If you don't have the BNG at 
all, what is the cost then? I still believe it will be cheaper to try to 
do this in the CPE.

> Some "positive" view: access with Gbps (up to 1Gbps) with a range of RTT 
> (1ms to 100ms) will need smarter mechanisms in the equipment as 
> inefficiencies will be crystal clear and business consequences will be 
> substantially different.

Please elaborate. I'd say FIFO is less harmful at these speeds because of 
TCP inefficiencies meaning most end systems won't come up in high enough 
transfer rates to saturate that part of the network. Now the bottle neck 
will move elsewhere.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] RED against bufferbloat

[MUSCARIELLO Luca IMT/OLN]

On 02/25/2015 05:09 PM, Mikael Abrahamsson wrote:
> On Wed, 25 Feb 2015, MUSCARIELLO Luca IMT/OLN wrote:
>
>> Doing FQ in silicon is easy. It must be very easy as I did myself in 
>> a MIPS Ikanos Vx185 chipset and I am not an hardware expert. This was 
>> for a CPE with a 5X1Gbps ports.
>
> I guess we have different definitions on what "silicon" is. Mine is 
> "ASIC". If it has a MIPS CPU (and the CPU does forwarding), then it's 
> not "silicon", then it's done in CPU.

MIPS based board with a central unit and several programmable asics as 
accelerators.
Fast path is not managed by the central unit.

>
> I keep hearing from vendors that queues are expensive. The smallest FQ 
> implementation that seems to be reasonable has 32 queues. Let's say we 
> have 5k customers per port on a BNG, that equates to 160k queues.
>
> Compare that to an efficient active ethernet p-t-p ETTH deployment 
> without BNG at all, where the access L2 switch is the one doing 
> policing. What would be the cost to equip this device with FQ per port?
>
>> If you go a little deeper in the network and you pick an OLT you won't
>
> I don't do PON.
>
>> find much intelligence. A little deeper and in a local aggregation 
>> router (all vendors) you'll find what you would need to implement FQ. A
>
> If it's a BNG yes. These linecards with hundreds of thousands of 
> queues are a lot more expensive than a linecard without these queues. 
> Usually today there are 3-4 queues per customer, now we want to expand 
> this to 32 or more. What is the cost increase for this?

Yes I have BNG in mind. Downstream equipment has not enough resources on 
a per user basis.
However I do not think that static per-flow queues is the right 
implementation for FQ.
That requires a lot of resources just for instantiation and if the 
hardware polls the
queues the cost can be very high for the user fanout and the rates we 
are considering here.

A single FQ instantiation has to consume no more memory than a single 
FIFO instantiation.
Per-flow queues should be managed as virtual queues. On a per customer 
basis.

>
> If you put a BNG in there with lots of intelligence then the cost of 
> adding the FQ machinery might not be too bad. If you don't have the 
> BNG at all, what is the cost then? I still believe it will be cheaper 
> to try to do this in the CPE.

I think this should also be done in the CPE but not in ingress for the 
downlink. If that's what you meant.
Only in egress for the uplink.

>
>> Some "positive" view: access with Gbps (up to 1Gbps) with a range of 
>> RTT (1ms to 100ms) will need smarter mechanisms in the equipment as 
>> inefficiencies will be crystal clear and business consequences will 
>> be substantially different.
>
> Please elaborate. I'd say FIFO is less harmful at these speeds because 
> of TCP inefficiencies meaning most end systems won't come up in high 
> enough transfer rates to saturate that part of the network. Now the 
> bottle neck will move elsewhere.

I assume you want TCP to be able to do 1Gbps if this is what you sell to 
the customers.

Also if you sell 1Gbps, you might want to do shaping instead of policing 
in the BNG
because the BDP of a 1Gbps flow can be pretty high depending on the RTT,
and downstream equipment (OLT, switch, DSLAM...) can have much smaller
queues than what you have in a BNG.

That is where inefficiencies come from. If in addition you have a mix of 
applications
in this customer queue than you'll need FQ because the traffic mix can 
be very heterogeneous
and flow isolation bec necessary.
omes

Re: [Bloat] RED against bufferbloat

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 409 bytes --]

> A single FQ instantiation has to consume no more memory than a single
FIFO instantiation.

That's easy enough. You can fit an awful lot of linked list pointers into
the space of a single IP packet. Even if you're only assigning 64KB per
subscriber, you can store 43 full packets and still have pointers to spare.
A properly functioning AQM should mostly keep the queue smaller than that.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 479 bytes --]

Re: [Bloat] RED against bufferbloat

[Mikael Abrahamsson]

On Wed, 25 Feb 2015, MUSCARIELLO Luca IMT/OLN wrote:

> MIPS based board with a central unit and several programmable asics as 
> accelerators. Fast path is not managed by the central unit.

And you made the programmable asics to FQ? I'd love to be able to pitch 
this to vendors as an example what should be done.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] RED against bufferbloat

[MUSCARIELLO Luca IMT/OLN]

On 02/26/2015 01:54 PM, Mikael Abrahamsson wrote:
> On Wed, 25 Feb 2015, MUSCARIELLO Luca IMT/OLN wrote:
>
>> MIPS based board with a central unit and several programmable asics 
>> as accelerators. Fast path is not managed by the central unit.
>
> And you made the programmable asics to FQ? I'd love to be able to 
> pitch this to vendors as an example what should be done.
>

Done with the vendor itself with related NDA etc. It takes longer to set 
the agreement than coding the system.
The problem is that this process is not ok. An ISP cannot maintain 
someone else product if it is closed.

Re: [Bloat] RED against bufferbloat

[Mikael Abrahamsson]

On Thu, 26 Feb 2015, MUSCARIELLO Luca IMT/OLN wrote:

> Done with the vendor itself with related NDA etc. It takes longer to set 
> the agreement than coding the system. The problem is that this process 
> is not ok. An ISP cannot maintain someone else product if it is closed.

Do you have a requirement document that makes sense to the people 
programming these ASICs for vendors? When I try to explain what needs to 
be done I usually run into very frustrating discussions.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] RED against bufferbloat

[MUSCARIELLO Luca IMT/OLN]

[-- Attachment #1: Type: text/plain, Size: 1455 bytes --]

On 02/26/2015 03:18 PM, Mikael Abrahamsson wrote:
> On Thu, 26 Feb 2015, MUSCARIELLO Luca IMT/OLN wrote:
>
>> Done with the vendor itself with related NDA etc. It takes longer to 
>> set the agreement than coding the system. The problem is that this 
>> process is not ok. An ISP cannot maintain someone else product if it 
>> is closed.
>
> Do you have a requirement document that makes sense to the people 
> programming these ASICs for vendors? When I try to explain what needs 
> to be done I usually run into very frustrating discussions.
>

I think there are people in this list that should be able to answer to 
this question better than me.

AFAIK the process is complex because even vendors use network processors 
they don't build and
traffic management is developed by the chipco in the chip. Especially 
for the segment we are considering here.
In the end the dequeue process is always managed  by someone else and 
mechanisms and their implementations opaque.
You can do testing on the equipment and do some reverse engineering. 
What a waste of time...

This is why single queue AQM is preferred by vendors, because it does 
not affect current product lines
and the enqueue is easier to code. FQ requires to recode the dequeue or 
to shadow the hardware dequeue.

My experience is not based on providing a requirement document, well we 
tried that first,
but on joint coding with the chipco because you need to see a lot of 
chip internals.

[-- Attachment #2: Type: text/html, Size: 2136 bytes --]

Re: [Bloat] RED against bufferbloat

[Dave Taht]

On Thu, Feb 26, 2015 at 7:18 AM, MUSCARIELLO Luca IMT/OLN
<luca.muscariello@orange.com> wrote:
> On 02/26/2015 03:18 PM, Mikael Abrahamsson wrote:
>
> On Thu, 26 Feb 2015, MUSCARIELLO Luca IMT/OLN wrote:
>
> Done with the vendor itself with related NDA etc. It takes longer to set the
> agreement than coding the system. The problem is that this process is not
> ok. An ISP cannot maintain someone else product if it is closed.
>
>
> Do you have a requirement document that makes sense to the people
> programming these ASICs for vendors? When I try to explain what needs to be
> done I usually run into very frustrating discussions.
>
>
> I think there are people in this list that should be able to answer to this
> question better than me.
>
> AFAIK the process is complex because even vendors use network processors
> they don't build and
> traffic management is developed by the chipco in the chip. Especially for
> the segment we are considering here.
> In the end the dequeue process is always managed  by someone else and
> mechanisms and their implementations opaque.
> You can do testing on the equipment and do some reverse engineering. What a
> waste of time...
>
> This is why single queue AQM is preferred by vendors, because it does not
> affect current product lines
> and the enqueue is easier to code. FQ requires to recode the dequeue or to
> shadow the hardware dequeue.

OK, I need to dispel a few misconceptions.

First, everyone saying that fq_codel can't be done in hardware is
*wrong* See my last point far below, I know I write over-long
emails...

YES, at the lowest level of the hardware, where packets turn into
light or fuzzy electrons, or need to be tied up in a an aggregate (as
in cable) and shipped onto the wire, you can't do it. BUT, as BQL has
shown, you CAN stick in enough buffering on the final single queue to
*keep the device busy* - which on fiber might be a few us, on cable
modems 2ms - and then do smarter things above that portion of the
device. I am perfectly willing to lose those 2ms when you can cut off
hundreds elsewhere.

Everybody saying that it cant be done doesnt understand how BQL works.
Demonstrably. On two+ dozen devices. Already. For 3 years now.

On the higher end than CPE, people that keep whinging about have
thousands of queues are not thinking about it correctly. This is
merely a change in memory access pattern, not anything physical at
all, and the overall reduction in queue lengths lead to much better
cache behavior, and they should, um, at least try this stuff on a
modern intel processor - and just go deploy that. 10GigE was totally
feasible on day one, ongoing work is getting up to 40GigE (but running
into trouble elsewhere on the rx path which jesper can talk to in
painful detail)

Again, if you do it right, on any architecture, all the smarts happen
long before you hit the wire. You are not - quite - dropping from the
head of the queue, but we never have, and I really don't get why
people don't grok this.

EVERY benchmark I ever publish can show the intrinsic latency in the
system as hovering around Xms due to the context switch overhead of
the processor and OS - and although I don't mind shaving that figure,
compared to the gains of seconds elsewhere that can come from using
these algorithms - I find getting those down more satisfying.
(admittedly I have spent tons of time trying to shave off a few
hundred us at that level too, as has jonathon and many others in the
linux community)

I also don't give a hoot about core routers, mostly the edge. I *do
care deeply* about FQ/AQMing interconnects between providers,
particularly in event of a disaster like earthquake or tsunami when
suddenly 2/3s the interconnects get drowned. What will happen in that
case, if an earthquake hit california the same size as the one that
hit japan? It worries me. I live 500 meters from the intersection of
two major fault lines.

Secondly,  I need to clarify a statement above:

"This is why single queue AQM is preferred by vendors *participating
publicly on the aqm mailing list*, because it does not affect current
product lines, and the enqueue is easier to code. "

When fq_codel landed, the next morning, I said to myself, "ok, is it
time to walk down sand hill road? We need this in switch chips and
given the two monopolistic vendors left, it is ripe for disruption".

After wrestling with myself for a few weeks I decided it would be
simpler and easier if I tried to pursuade the chipmakers making packet
co-processing engines (like octeon, intel, tilera) that this algorithm
and htb-like rate control would be a valuable addition to their
products. Note - in *none* of their cases did it have to reduce to
gates, they have a specialized cpu co-processor that struggles to work
at line rate (with features like nat offloads, etc) - with specialized
firmware that they write and hold proprietary, and there were *no*
hardware mods needed - Their struggle at line rate was not the point,
I wanted something that could work at an ISPs set rate, which is very
easy to do...

I talked to all these chipmakers (and a few more startup-like ones in
particularly the high speed trading market).

The told me there was no demand. So I went talked to their customers...

and I am certain that more than one of the companies I have talked to
in the last 3 years is actually doing FQ now, and certain that codel
is also implemented - but I cannot reveal which ones, and for all I
know the ones that are not talking to me (anymore) are off doing. And
at least one of the companies doing it in their packet co-processor,
was getting it wrong, until I straightened 'em out, and for all I know
they didn't listen.

I figured whichever vendor shipped products first would have a market
advantage, and then everybody else would pile on, and that if I
focused on creating demand for the algorithm (as I did all over the
world, and with ubnt in particular I went to the trouble of
backporting it to their edgerouter personally), demand would be
created for better firmware, from the chipcos and products would
arrive.

and they have. Every 802.11ac router now has some sort of "better" QoS
system in it. (and of course, openwrt and derivatives). There is a ton
of stuff in the pipeline.

The streamboost folk were pretty effective in spreading their meme,
but I am mad at them about quite a few things about their
implementation and test regimes, so I'll save what they have done
wrong for another day when I have more venom stored up, and have
acquired stuff I can say publicly about their implementation via a bit
more inspection of their GPL drops and testing the related products.

...

"FQ requires to recode the dequeue or to shadow the hardware dequeue."

Well this statement is not correct.

*Lastly*: IF you want to reduce things to gates, rather than use a
packet co-processor:

1) DRR in hardware is entirely doable. How do I know this? - because
it was done for the netfpga.org project *7* years ago. Here is the
project, paper, and *verilog*:
https://github.com/NetFPGA/netfpga/wiki/DRRNetFPGA

It is a single define to synthesize a configurable number of queues,
and it worked on top of the GigE Xilinx virtex-2 pro FPGA, which is
like so low-end now as I am not even sure if they are made anymore.
http://netfpga.org/2014/#/systems/4netfpga-1g/details/

They never got around to writing a five-tuple packet inspector/hasher
but that is straightforward.

2) Rate management in hardware is entirely doable, also: Here is the
project, paper, and verilog: https://github.com/gengyl08/SENIC

3) I long ago figured out how to make something fq_codel-like work (in
theory) in hardware with enough parallization (and a bit of BQL).
Sticking points were a complete re-org of the ethernet device and
device driver, and a whole lot of Xilinx IP I wanted to dump, and I am
really too busy to do any of the work, but:

Since I am fed up with the debate, I am backing this kickstarter
project. I have had several discussions with the people doing it -
they are using all the same hardware I chose for my mental design -
and I urge others here to do so.

https://www.kickstarter.com/projects/onetswitch/onetswitch-open-source-hardware-for-networking

I am not entirely broke for a change, and plan to throw in 1k or so.
Need to find 25k from other people for them to make their initial
targets.

That board meets all the needs for fixing wifi also. They already have
shipping, more complex products that might be more right for the job,
as working out the number of gates needed is something that needs
actual work and simulation.

But I did like this:

https://github.com/MeshSr/wiki/wiki/ONetSwitch45

I will write a bit more about this (as negotiations continue) in a
less long, more specialized mail in the coming weeks, and perhaps, as
so often happens around here (I am thinking of renaming this the
"bufferbloat/stone soup project"), some EEs will show up eager to do
something truly new, and amazing, as a summer project. If you got any
spare students, well, go to town.

I really, really like chisel in particular,
https://chisel.eecs.berkeley.edu/ and the openrisc folk could use a
better ethernet device.

> My experience is not based on providing a requirement document, well we
> tried that first,
> but on joint coding with the chipco because you need to see a lot of chip
> internals.
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>



-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

Re: [Bloat] RED against bufferbloat

[Dave Taht]

On Thu, Feb 26, 2015 at 9:04 AM, Dave Taht <dave.taht@gmail.com> wrote:
> On Thu, Feb 26, 2015 at 7:18 AM, MUSCARIELLO Luca IMT/OLN
> <luca.muscariello@orange.com> wrote:
>> On 02/26/2015 03:18 PM, Mikael Abrahamsson wrote:
>>
>> On Thu, 26 Feb 2015, MUSCARIELLO Luca IMT/OLN wrote:
>>
>> Done with the vendor itself with related NDA etc. It takes longer to set the
>> agreement than coding the system. The problem is that this process is not
>> ok. An ISP cannot maintain someone else product if it is closed.
>>
>>
>> Do you have a requirement document that makes sense to the people
>> programming these ASICs for vendors? When I try to explain what needs to be
>> done I usually run into very frustrating discussions.
>>
>>
>> I think there are people in this list that should be able to answer to this
>> question better than me.
>>
>> AFAIK the process is complex because even vendors use network processors
>> they don't build and
>> traffic management is developed by the chipco in the chip. Especially for
>> the segment we are considering here.
>> In the end the dequeue process is always managed  by someone else and
>> mechanisms and their implementations opaque.
>> You can do testing on the equipment and do some reverse engineering. What a
>> waste of time...
>>
>> This is why single queue AQM is preferred by vendors, because it does not
>> affect current product lines
>> and the enqueue is easier to code. FQ requires to recode the dequeue or to
>> shadow the hardware dequeue.
>
> OK, I need to dispel a few misconceptions.
>
> First, everyone saying that fq_codel can't be done in hardware is
> *wrong* See my last point far below, I know I write over-long
> emails...
>
> YES, at the lowest level of the hardware, where packets turn into
> light or fuzzy electrons, or need to be tied up in a an aggregate (as
> in cable) and shipped onto the wire, you can't do it. BUT, as BQL has
> shown, you CAN stick in enough buffering on the final single queue to
> *keep the device busy* - which on fiber might be a few us, on cable
> modems 2ms - and then do smarter things above that portion of the
> device. I am perfectly willing to lose those 2ms when you can cut off
> hundreds elsewhere.
>
> Everybody saying that it cant be done doesnt understand how BQL works.
> Demonstrably. On two+ dozen devices. Already. For 3 years now.

Or hasn't taken apart free.fr's revolution 6 product. *Shipping*, with
DRR for classification (similar to the sqm's simple.qos model) +
fq_codel to manage the three queues - since *august 2011*.

It has, in particular, a really tightly written DSL driver which makes
everything else "just work".

Other DSL firmware makers have hopefully taken note as the only way
their market will survive against the onslought of fiber is to make
better devices. More people should clue up their DSL makers about it,
though. The recent paper from alcatel-lucent was really terrible and
missed most of the points.

> On the higher end than CPE, people that keep whinging about have
> thousands of queues are not thinking about it correctly. This is
> merely a change in memory access pattern, not anything physical at
> all, and the overall reduction in queue lengths lead to much better
> cache behavior, and they should, um, at least try this stuff on a
> modern intel processor - and just go deploy that. 10GigE was totally
> feasible on day one, ongoing work is getting up to 40GigE (but running
> into trouble elsewhere on the rx path which jesper can talk to in
> painful detail)
>
> Again, if you do it right, on any architecture, all the smarts happen
> long before you hit the wire. You are not - quite - dropping from the
> head of the queue, but we never have, and I really don't get why
> people don't grok this.
>
> EVERY benchmark I ever publish can show the intrinsic latency in the
> system as hovering around Xms due to the context switch overhead of
> the processor and OS - and although I don't mind shaving that figure,

Clarification: "The context switch overhead of processor and OS being
covered up by BQL's slight added buffering to keep the device busy".

I should really publish more 100Mbit and gbit benchmarks to show that
actual overhead. It is not a lot.

> compared to the gains of seconds elsewhere that can come from using
> these algorithms - I find getting those down more satisfying.
> (admittedly I have spent tons of time trying to shave off a few
> hundred us at that level too, as has jonathon and many others in the
> linux community)

The overhead of other packet processing now dominates the runtime. In
addition to the xmit_more work last quarter, recently a whole bunch of
*lovely* patches arrived for reducing overhead in the Linux FIB lookup
table. They are *GREAT*.

https://www.netdev01.org/docs/duyck-fib-trie.pdf

I don't have any data on how much the fib lookup used to cost compared
to fq_codel at these speeds, but I think it was *at least* 7 times
more expensive than running fq_codel.

And Dave Millers keynote at netconf01 a few weeks back was all about
smashing latencies all through the linux networking stack, and in
particular, integrating offloads well into the kernel.

https://www.netdev01.org/docs/miller-Ottawa2015-Keynote.pdf

SDN is on the rise. In fact there was so much demonstrable progress
and interest shown at that conference on issues that I care about, in
subsystems and product that are shipping or will ship widely, that I
felt like we were going to win the war against network latency much
sooner than I ever dreamed.

https://www.netdev01.org/downloads

See the rocker switch, hemmingers preso on dpdk, the hardware
accellerating talk, really great stuff, I am sorry I was too sick to
attend.

>
> I also don't give a hoot about core routers, mostly the edge. I *do
> care deeply* about FQ/AQMing interconnects between providers,
> particularly in event of a disaster like earthquake or tsunami when
> suddenly 2/3s the interconnects get drowned. What will happen in that
> case, if an earthquake hit california the same size as the one that
> hit japan? It worries me. I live 500 meters from the intersection of
> two major fault lines.
>
> Secondly,  I need to clarify a statement above:
>
> "This is why single queue AQM is preferred by vendors *participating
> publicly on the aqm mailing list*, because it does not affect current
> product lines, and the enqueue is easier to code. "
>
> When fq_codel landed, the next morning, I said to myself, "ok, is it
> time to walk down sand hill road? We need this in switch chips and
> given the two monopolistic vendors left, it is ripe for disruption".
>
> After wrestling with myself for a few weeks I decided it would be
> simpler and easier if I tried to pursuade the chipmakers making packet
> co-processing engines (like octeon, intel, tilera) that this algorithm
> and htb-like rate control would be a valuable addition to their
> products. Note - in *none* of their cases did it have to reduce to
> gates, they have a specialized cpu co-processor that struggles to work
> at line rate (with features like nat offloads, etc) - with specialized
> firmware that they write and hold proprietary, and there were *no*
> hardware mods needed - Their struggle at line rate was not the point,
> I wanted something that could work at an ISPs set rate, which is very
> easy to do...
>
> I talked to all these chipmakers (and a few more startup-like ones in
> particularly the high speed trading market).
>
> The told me there was no demand. So I went talked to their customers...
>
> and I am certain that more than one of the companies I have talked to
> in the last 3 years is actually doing FQ now, and certain that codel
> is also implemented - but I cannot reveal which ones, and for all I
> know the ones that are not talking to me (anymore) are off doing. And
> at least one of the companies doing it in their packet co-processor,
> was getting it wrong, until I straightened 'em out, and for all I know
> they didn't listen.
>
> I figured whichever vendor shipped products first would have a market
> advantage, and then everybody else would pile on, and that if I
> focused on creating demand for the algorithm (as I did all over the
> world, and with ubnt in particular I went to the trouble of
> backporting it to their edgerouter personally), demand would be
> created for better firmware, from the chipcos and products would
> arrive.
>
> and they have. Every 802.11ac router now has some sort of "better" QoS
> system in it. (and of course, openwrt and derivatives). There is a ton
> of stuff in the pipeline.
>
> The streamboost folk were pretty effective in spreading their meme,
> but I am mad at them about quite a few things about their
> implementation and test regimes, so I'll save what they have done
> wrong for another day when I have more venom stored up, and have
> acquired stuff I can say publicly about their implementation via a bit
> more inspection of their GPL drops and testing the related products.
>
> ...
>
> "FQ requires to recode the dequeue or to shadow the hardware dequeue."
>
> Well this statement is not correct.
>
> *Lastly*: IF you want to reduce things to gates, rather than use a
> packet co-processor:
>
> 1) DRR in hardware is entirely doable. How do I know this? - because
> it was done for the netfpga.org project *7* years ago. Here is the
> project, paper, and *verilog*:
> https://github.com/NetFPGA/netfpga/wiki/DRRNetFPGA
>
> It is a single define to synthesize a configurable number of queues,
> and it worked on top of the GigE Xilinx virtex-2 pro FPGA, which is
> like so low-end now as I am not even sure if they are made anymore.
> http://netfpga.org/2014/#/systems/4netfpga-1g/details/
>
> They never got around to writing a five-tuple packet inspector/hasher
> but that is straightforward.
>
> 2) Rate management in hardware is entirely doable, also: Here is the
> project, paper, and verilog: https://github.com/gengyl08/SENIC
>
> 3) I long ago figured out how to make something fq_codel-like work (in
> theory) in hardware with enough parallization (and a bit of BQL).
> Sticking points were a complete re-org of the ethernet device and
> device driver, and a whole lot of Xilinx IP I wanted to dump, and I am
> really too busy to do any of the work, but:
>
> Since I am fed up with the debate, I am backing this kickstarter
> project. I have had several discussions with the people doing it -
> they are using all the same hardware I chose for my mental design -
> and I urge others here to do so.
>
> https://www.kickstarter.com/projects/onetswitch/onetswitch-open-source-hardware-for-networking
>
> I am not entirely broke for a change, and plan to throw in 1k or so.
> Need to find 25k from other people for them to make their initial
> targets.
>
> That board meets all the needs for fixing wifi also. They already have
> shipping, more complex products that might be more right for the job,
> as working out the number of gates needed is something that needs
> actual work and simulation.
>
> But I did like this:
>
> https://github.com/MeshSr/wiki/wiki/ONetSwitch45
>
> I will write a bit more about this (as negotiations continue) in a
> less long, more specialized mail in the coming weeks, and perhaps, as
> so often happens around here (I am thinking of renaming this the
> "bufferbloat/stone soup project"), some EEs will show up eager to do
> something truly new, and amazing, as a summer project. If you got any
> spare students, well, go to town.
>
> I really, really like chisel in particular,
> https://chisel.eecs.berkeley.edu/ and the openrisc folk could use a
> better ethernet device.
>
>> My experience is not based on providing a requirement document, well we
>> tried that first,
>> but on joint coding with the chipco because you need to see a lot of chip
>> internals.
>>
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>>
>
>
>
> --
> Dave Täht
> Let's make wifi fast, less jittery and reliable again!
>
> https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb



-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

[Bloat] RE : RED against bufferbloat

[luca.muscariello]

[-- Attachment #1: Type: text/plain, Size: 11397 bytes --]

Sure.  I am waiting to see the data sheet  of that chip.

Feasibility is of course not a problem.  Availability is my problem.  HTB is already in asic so no reason why FQ shouldn't.

DSL  is not an issue at all.  It works in software.  Fiber is my problem.

BNG maybe in ten years.


-------- Message d'origine --------
De : Dave Taht
Date :2015/02/26 18:04 (GMT+01:00)
À : MUSCARIELLO Luca IMT/OLN
Cc : Mikael Abrahamsson , bloat@lists.bufferbloat.net
Objet : Re: [Bloat] RED against bufferbloat

On Thu, Feb 26, 2015 at 7:18 AM, MUSCARIELLO Luca IMT/OLN
<luca.muscariello@orange.com> wrote:
> On 02/26/2015 03:18 PM, Mikael Abrahamsson wrote:
>
> On Thu, 26 Feb 2015, MUSCARIELLO Luca IMT/OLN wrote:
>
> Done with the vendor itself with related NDA etc. It takes longer to set the
> agreement than coding the system. The problem is that this process is not
> ok. An ISP cannot maintain someone else product if it is closed.
>
>
> Do you have a requirement document that makes sense to the people
> programming these ASICs for vendors? When I try to explain what needs to be
> done I usually run into very frustrating discussions.
>
>
> I think there are people in this list that should be able to answer to this
> question better than me.
>
> AFAIK the process is complex because even vendors use network processors
> they don't build and
> traffic management is developed by the chipco in the chip. Especially for
> the segment we are considering here.
> In the end the dequeue process is always managed  by someone else and
> mechanisms and their implementations opaque.
> You can do testing on the equipment and do some reverse engineering. What a
> waste of time...
>
> This is why single queue AQM is preferred by vendors, because it does not
> affect current product lines
> and the enqueue is easier to code. FQ requires to recode the dequeue or to
> shadow the hardware dequeue.

OK, I need to dispel a few misconceptions.

First, everyone saying that fq_codel can't be done in hardware is
*wrong* See my last point far below, I know I write over-long
emails...

YES, at the lowest level of the hardware, where packets turn into
light or fuzzy electrons, or need to be tied up in a an aggregate (as
in cable) and shipped onto the wire, you can't do it. BUT, as BQL has
shown, you CAN stick in enough buffering on the final single queue to
*keep the device busy* - which on fiber might be a few us, on cable
modems 2ms - and then do smarter things above that portion of the
device. I am perfectly willing to lose those 2ms when you can cut off
hundreds elsewhere.

Everybody saying that it cant be done doesnt understand how BQL works.
Demonstrably. On two+ dozen devices. Already. For 3 years now.

On the higher end than CPE, people that keep whinging about have
thousands of queues are not thinking about it correctly. This is
merely a change in memory access pattern, not anything physical at
all, and the overall reduction in queue lengths lead to much better
cache behavior, and they should, um, at least try this stuff on a
modern intel processor - and just go deploy that. 10GigE was totally
feasible on day one, ongoing work is getting up to 40GigE (but running
into trouble elsewhere on the rx path which jesper can talk to in
painful detail)

Again, if you do it right, on any architecture, all the smarts happen
long before you hit the wire. You are not - quite - dropping from the
head of the queue, but we never have, and I really don't get why
people don't grok this.

EVERY benchmark I ever publish can show the intrinsic latency in the
system as hovering around Xms due to the context switch overhead of
the processor and OS - and although I don't mind shaving that figure,
compared to the gains of seconds elsewhere that can come from using
these algorithms - I find getting those down more satisfying.
(admittedly I have spent tons of time trying to shave off a few
hundred us at that level too, as has jonathon and many others in the
linux community)

I also don't give a hoot about core routers, mostly the edge. I *do
care deeply* about FQ/AQMing interconnects between providers,
particularly in event of a disaster like earthquake or tsunami when
suddenly 2/3s the interconnects get drowned. What will happen in that
case, if an earthquake hit california the same size as the one that
hit japan? It worries me. I live 500 meters from the intersection of
two major fault lines.

Secondly,  I need to clarify a statement above:

"This is why single queue AQM is preferred by vendors *participating
publicly on the aqm mailing list*, because it does not affect current
product lines, and the enqueue is easier to code. "

When fq_codel landed, the next morning, I said to myself, "ok, is it
time to walk down sand hill road? We need this in switch chips and
given the two monopolistic vendors left, it is ripe for disruption".

After wrestling with myself for a few weeks I decided it would be
simpler and easier if I tried to pursuade the chipmakers making packet
co-processing engines (like octeon, intel, tilera) that this algorithm
and htb-like rate control would be a valuable addition to their
products. Note - in *none* of their cases did it have to reduce to
gates, they have a specialized cpu co-processor that struggles to work
at line rate (with features like nat offloads, etc) - with specialized
firmware that they write and hold proprietary, and there were *no*
hardware mods needed - Their struggle at line rate was not the point,
I wanted something that could work at an ISPs set rate, which is very
easy to do...

I talked to all these chipmakers (and a few more startup-like ones in
particularly the high speed trading market).

The told me there was no demand. So I went talked to their customers...

and I am certain that more than one of the companies I have talked to
in the last 3 years is actually doing FQ now, and certain that codel
is also implemented - but I cannot reveal which ones, and for all I
know the ones that are not talking to me (anymore) are off doing. And
at least one of the companies doing it in their packet co-processor,
was getting it wrong, until I straightened 'em out, and for all I know
they didn't listen.

I figured whichever vendor shipped products first would have a market
advantage, and then everybody else would pile on, and that if I
focused on creating demand for the algorithm (as I did all over the
world, and with ubnt in particular I went to the trouble of
backporting it to their edgerouter personally), demand would be
created for better firmware, from the chipcos and products would
arrive.

and they have. Every 802.11ac router now has some sort of "better" QoS
system in it. (and of course, openwrt and derivatives). There is a ton
of stuff in the pipeline.

The streamboost folk were pretty effective in spreading their meme,
but I am mad at them about quite a few things about their
implementation and test regimes, so I'll save what they have done
wrong for another day when I have more venom stored up, and have
acquired stuff I can say publicly about their implementation via a bit
more inspection of their GPL drops and testing the related products.

...

"FQ requires to recode the dequeue or to shadow the hardware dequeue."

Well this statement is not correct.

*Lastly*: IF you want to reduce things to gates, rather than use a
packet co-processor:

1) DRR in hardware is entirely doable. How do I know this? - because
it was done for the netfpga.org project *7* years ago. Here is the
project, paper, and *verilog*:
https://github.com/NetFPGA/netfpga/wiki/DRRNetFPGA

It is a single define to synthesize a configurable number of queues,
and it worked on top of the GigE Xilinx virtex-2 pro FPGA, which is
like so low-end now as I am not even sure if they are made anymore.
http://netfpga.org/2014/#/systems/4netfpga-1g/details/

They never got around to writing a five-tuple packet inspector/hasher
but that is straightforward.

2) Rate management in hardware is entirely doable, also: Here is the
project, paper, and verilog: https://github.com/gengyl08/SENIC

3) I long ago figured out how to make something fq_codel-like work (in
theory) in hardware with enough parallization (and a bit of BQL).
Sticking points were a complete re-org of the ethernet device and
device driver, and a whole lot of Xilinx IP I wanted to dump, and I am
really too busy to do any of the work, but:

Since I am fed up with the debate, I am backing this kickstarter
project. I have had several discussions with the people doing it -
they are using all the same hardware I chose for my mental design -
and I urge others here to do so.

https://www.kickstarter.com/projects/onetswitch/onetswitch-open-source-hardware-for-networking

I am not entirely broke for a change, and plan to throw in 1k or so.
Need to find 25k from other people for them to make their initial
targets.

That board meets all the needs for fixing wifi also. They already have
shipping, more complex products that might be more right for the job,
as working out the number of gates needed is something that needs
actual work and simulation.

But I did like this:

https://github.com/MeshSr/wiki/wiki/ONetSwitch45

I will write a bit more about this (as negotiations continue) in a
less long, more specialized mail in the coming weeks, and perhaps, as
so often happens around here (I am thinking of renaming this the
"bufferbloat/stone soup project"), some EEs will show up eager to do
something truly new, and amazing, as a summer project. If you got any
spare students, well, go to town.

I really, really like chisel in particular,
https://chisel.eecs.berkeley.edu/ and the openrisc folk could use a
better ethernet device.

> My experience is not based on providing a requirement document, well we
> tried that first,
> but on joint coding with the chipco because you need to see a lot of chip
> internals.
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>



--
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.


[-- Attachment #2: Type: text/html, Size: 13682 bytes --]

Re: [Bloat] RED against bufferbloat

[Mikael Abrahamsson]

On Thu, 26 Feb 2015, Dave Taht wrote:

> First, everyone saying that fq_codel can't be done in hardware is 
> *wrong* See my last point far below, I know I write over-long emails...

What do you define as "hardware"?

> YES, at the lowest level of the hardware, where packets turn into light 
> or fuzzy electrons, or need to be tied up in a an aggregate (as in 
> cable) and shipped onto the wire, you can't do it. BUT, as BQL has

I am talking about doing this in the NPUs that are available on the 
market today and in the future.

We're talking about these kinds of devices:

http://www.tilera.com/News/PressRelease/?ezchip=97

I'm afraid that your experience with CPU driven devices isn't a great 
match here, and we're talking past each other.

I would love to be able to use my employers purchasing power to drive 
demand on AQM, but I need to know what the requirements are, and that 
they're feasable to do, preferrably with an engineering effort for the 
vendor in the range of few man years. I can't produce this document 
myself, I have already tried pointing them in the direction of 
presentations etc, but I need "hard" documents, outlining exactly what 
they need to do.

Do we have this currently? In that case, whereto should I point them?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] RED against bufferbloat

[Bill Ver Steeg (versteb)]

Guys-

If you want to drive AQM requirements into router/switch specification, count me in. We need to be careful to specify behaviors, as opposed to implementations. If we start to specify implementation details, the process will get bogged down in intractable differences. There is a lot of variability in what the existing hardware/software queueing components do, and we need to allow some leeway to "do the right thing" on the various platforms.

Given a good set of requirements, the community can both 1) work on next generation chipset/software/firmware/memory/board/architecture  that do AQM in an optimum manner and 2) make (hopefully simple and well-bounded) changes to the existing software/firmware.

Bvs
 


-----Original Message-----
From: bloat-bounces@lists.bufferbloat.net [mailto:bloat-bounces@lists.bufferbloat.net] On Behalf Of Mikael Abrahamsson
Sent: Thursday, February 26, 2015 1:59 PM
To: Dave Taht
Cc: bloat@lists.bufferbloat.net
Subject: Re: [Bloat] RED against bufferbloat

On Thu, 26 Feb 2015, Dave Taht wrote:

> First, everyone saying that fq_codel can't be done in hardware is
> *wrong* See my last point far below, I know I write over-long emails...

What do you define as "hardware"?

> YES, at the lowest level of the hardware, where packets turn into 
> light or fuzzy electrons, or need to be tied up in a an aggregate (as 
> in
> cable) and shipped onto the wire, you can't do it. BUT, as BQL has

I am talking about doing this in the NPUs that are available on the market today and in the future.

We're talking about these kinds of devices:

http://www.tilera.com/News/PressRelease/?ezchip=97

I'm afraid that your experience with CPU driven devices isn't a great match here, and we're talking past each other.

I would love to be able to use my employers purchasing power to drive demand on AQM, but I need to know what the requirements are, and that they're feasable to do, preferrably with an engineering effort for the vendor in the range of few man years. I can't produce this document myself, I have already tried pointing them in the direction of presentations etc, but I need "hard" documents, outlining exactly what they need to do.

Do we have this currently? In that case, whereto should I point them?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se
_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] RED against bufferbloat

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 573 bytes --]

> We need to be careful to specify behaviors, as opposed to
implementations. If we start to specify implementation details, the process
will get bogged down in intractable differences.

I understand these sorts of requirements. Since I'm waiting for one of my
computers to do something substantial, I'll have a quick stab at the
problem.

I suspect a modular approach might work best. Different modules can be
assigned to separate design teams, and a diagram can show how they fit
together. So a shaper module, a Codel queue module, a fair queue module.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 655 bytes --]

Re: [Bloat] RED against bufferbloat

[Dave Taht]

On Thu, Feb 26, 2015 at 10:59 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Thu, 26 Feb 2015, Dave Taht wrote:
>
>> First, everyone saying that fq_codel can't be done in hardware is *wrong*
>> See my last point far below, I know I write over-long emails...
>
>
> What do you define as "hardware"?
>
>> YES, at the lowest level of the hardware, where packets turn into light or
>> fuzzy electrons, or need to be tied up in a an aggregate (as in cable) and
>> shipped onto the wire, you can't do it. BUT, as BQL has
>
>
> I am talking about doing this in the NPUs that are available on the market
> today and in the future.
>
> We're talking about these kinds of devices:
>
> http://www.tilera.com/News/PressRelease/?ezchip=97

Wow. That is one impressive piece of hardware.

> I'm afraid that your experience with CPU driven devices isn't a great match
> here, and we're talking past each other.

To a certain extent we were. Thank you for dropping the above link
into the conversation.

1) I mostly care about fixing the end user devices. Which can easily
have enough horsepower to cope with whatever cheap crap the ISP wants
on their side. Sure, there are a lot of trendlines that are negative
here, the world I wanted to live in had a standard for jack you
plugged into a the wall, which you could then do anything you wanted
with, from gear you could buy from any retailer.

2) I realize that you (at least) want to do the best job possible with
the best gear available on the market, and I respectively submit that
the best thing that you can do is to avoid vendor lock.

3) If that chipmaker is unable to run a simple benchmark of their
shaper + qos system, vs the sqm-scripts with fq_codel, and not realize
that that they could, indeed, do much better, I wouldn't do business
with them.

> I would love to be able to use my employers purchasing power to drive demand
> on AQM, but I need to know what the requirements are, and that they're
> feasable to do, preferrably with an engineering effort for the vendor in the
> range of few man years. I can't produce this document myself, I have already
> tried pointing them in the direction of presentations etc, but I need "hard"
> documents, outlining exactly what they need to do.

Sucks to be you.

I am glad a few other people here care enough to help create the
needed paperwork.

As for me: I am *done* with doing big company's essential R&D for them
for free. I spent my entire retirement savings on fixing bufferbloat
in the first year alone, and have been living a hand to mouth
existence ever since. I just went through a 4 month month period with
no revenue at all. I live in a yurt, the cheapest housing I could find
in california, just so I could maintain sufficient independence to
kick the entire industry in the ass, and never have to filter a single
word, or commit through, a patent attorney, ever again.

Given that so much great stuff is now happening - I am thinking I can
finally quit. Everybody gets it now, and I am really tired of it (and
a lot of people are tired of me being such a nag about it!), and I
want to go off and do something else - anything else! But it would
have been nice somewhere along the way to have landed a position that
kept a better roof over my head, and eliminated some of my stressors
and uncertainty, and still let me keep kicking the industry in the ass
on a regular basis. Maybe, I dunno, I will quit this and go for a PHD
and get a chance to write a ton more stuff down, at least, and do
something genuinely new...

But - as much as I might want to quit - I can't quit yet.

I have one *last* thing left on bufferbloat that I care about, and
that is fixing wifi, for the 2.8+ billion current users and the
billions of new devices left to come. While I am glad we have at least
theoretically fixed the edge, wifi was the actual problem that I
started with, and there had accumulated so many fundamental problems
since I had last paid attention to it, that it has taken 2+ years
since clearly identifying what the problems there were, to having come
up with code-able fixes, to get ready to even start.

There is still quite a bit of theory work left to be done to get it
right, and a ton of rearchitectural work in the kernel and the device
firmwares themselves left to do.

Solving *wifi problems* is what has kept me awake at night - and
despite stumping around the world for the last two years to try to
raise sufficient awareness and funding to tackle the job, have still
only raised 1/10th the funds I thought would be required to do it
right.

And I am going to go start doing it anyway, starting last week, with
some of the fiercely talented volunteers we have here, pitching in to
help - and some of the the few clued companies also pitching in - and
we are going to get it done - and THEN I can quit this f-ing
self-assigned job that I took on as a favor to jim and vint and esr,
and go work on some VC's concept for the next pets.com and watch the
money start rolling back into my 401k.

If anyone here works at a company that cares that wifi gets genuinely
better, for everybody, please contribute a resource, or ante up:

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

I know I am not the only person that cares deeply about wifi, but
blows my mind how few seem to understand that it doesn't need to suck
- given what we now understand about fixing bufferbloat.

> Do we have this currently? In that case, whereto should I point them?

Does the ID not help?

>
>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se



-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

Re: [Bloat] RED against bufferbloat

[Sebastian Moeller]

Hi Mikael,


On Feb 25, 2015, at 14:36 , Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Wed, 25 Feb 2015, Sebastian Moeller wrote:
> 
>> 	The only argument for ingress shaping on the CPE is that this allows the end user to define her own QOS criteria independent of the ISPs wishes. Best of both worlds would be user configurable QOS-shaping on the slam/bras/whatever…
> 
> As I said before, doing FQ_CODEL in the AR is an expensive proposition for medium and high speed access.

	Well a vectoring DSLAM is not too wimpy and needs to do plenty of processing per line, so fq_codel on there should be more finically sane than on a device with more concurrent users, I would guess...

> So if this could successfully be pushed to the CPE it would mean it would be more widely deployed.

	But there we face the same problem, the wimpy CPEs that ISPs like to distribute do not have enough pomp for shaping a reasonably fast lane, the saving grace might be that end customers can upgrade on their own cost. And I notice a number of specialized home routers appearing on the market targeting people wiling to spend $$ for better behavior under load.

> 
> I am very much aware that this is being done (I have done it myself), but my question was if someone had actually done this in a lab and found out how well it works in RRUL tests etc.

	Not in the lab, no; I have no lab, but I used RRUL iteratively to figure out empirically what shaping percentage I need on my line to keep latency under load in bounds I consider reasonable. In my case DTAG vdsl50 this turned out to be 90% of downlink sync and 95% of uplink sync (but I since learned that DTAG has a BRAS policer that has a lower rate than the VDSL-line, so I guess I was closer to 95% and 99% percent of the BRAS policer, but heaven knows which encapsulation the BRAS accounts for…)
	So I would guess the collection of cerowrt users should be able to cough up a number of empirical shaping percentages for different link speeds and technologies.

Here is my data, the empirically derived shaping values puzzled me until I learned about the BRAS policer. I had expected that the uplink shaper could run almost at 100%, which turned out to be correct if referenced to the BRAS policer and not the vdlx sync. Anyway here is my data, maybe others want to add their’s:

Tech				downlink_kbps							uplink_kbps							CPE_shaper
				sync		ISP_policed	CPE_shaped		sync		ISP_policed	CPE_shaped		overhead_B	linklayer
VDSL2.vectoring	51390	45559		46178			10047	9460		9500			16			ethernet


Best Regards
	Sebastian

> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] RED against bufferbloat

[Michael Welzl]

> On 25 Feb 2015, at 11:24, toke@toke.dk wrote:
> 
> Michael Welzl <michawe@ifi.uio.no> writes:
> 
>> but that's FQ (or FQ_CoDel's changed FQ variant), much more than the
>> AQM mechanism in use (as we have also seen presented by Toke at the
>> last ICCRG meeting).
> 
> Well, actually, that presentation did also include an evaluation of the
> AQMs in an asymmetrical scenario. And that shows that while generally
> ARED does perform fairly well, it tends to be a bit on the aggressive
> side. In the asymmetrical case this results in too many drops on the
> slow side of the asymmetrical link (typically upload), hurting throughput
> in the other direction due to lost ACKs.
> 
> There's also some other issues in there, with PIE and CoDel in
> particular, most notably their reactions when conditions change: it can
> take tens of seconds for the algorithms to get queueing latency under
> control in this case.
> 
> Slides for the IETF presentation available here:
> http://www.ietf.org/proceedings/91/slides/slides-91-iccrg-4.pdf
> 
> There's also a longer version of the talk (from the Stanford Netseminar)
> available on Youtube: https://www.youtube.com/watch?v=kePhqfKA3SM
> 
>> But this discussion is about AQM mechanisms, not (changed)FQ.
> 
> While the academic side of me enjoys studying AQMs (and I'm still far
> from anything resembling a thorough understanding of them), the
> practical "I just want my network to work" side of me has become
> increasingly convinced (in part by doing the experiments in the above
> mentioned talk) that FQ is more important than AQM in many scenarios.

+1, certainly it has a big influence. This has been well known for many years though, and documented broadly, perhaps most notably by Jim Roberts.


> As
> such, I think that excluding FQ from the conversation is mostly of, well,
> academic interest ;)

Here I disagree, for two reasons:
1) The AQM part kicks in per flow. So, whenever you have one flow, the behavior of FQ_AQM and AQM will be the same. Investigating what an AQM mechanism does to one flow is then worthwhile.
2) Not everyone will always want FQ everywhere. There are potential disadvantanges (e.g. the often mentioned with-a-VPN-I'm-only-1-flow problem). What's necessary is to quantify them - to see how the effect of FQ (or FQ_CoDel's changed FQ) plays out, and you've done a great start there in my opinion.

Cheers,
Michael

Re: [Bloat] RED against bufferbloat

[Toke Høiland-Jørgensen]

Michael Welzl <michawe@ifi.uio.no> writes:

> +1, certainly it has a big influence. This has been well known for
> many years though, and documented broadly, perhaps most notably by Jim
> Roberts.

Being "well known for many years" in the academic community
unfortunately doesn't translate into deployment. Which is why I think
there should be a place for both approaches: the "let's simplify and get
a thorough base understanding" and the "let's run this on real stuff and
see what happens".

> Here I disagree, for two reasons:
>
> 1) The AQM part kicks in per flow. So, whenever you have one flow, the
> behavior of FQ_AQM and AQM will be the same. Investigating what an AQM
> mechanism does to one flow is then worthwhile.
>
> 2) Not everyone will always want FQ everywhere. There are potential
> disadvantanges (e.g. the often mentioned with-a-VPN-I'm-only-1-flow problem).
> What's necessary is to quantify them - to see how the effect of FQ (or
> FQ_CoDel's changed FQ) plays out, and you've done a great start there in my
> opinion.

I didn't mean that investigating AQM behaviour is not worthwhile, far
from it. But I believe that FQ needs to play a much larger role than it
does currently in solving the larger problem of network (latency)
behaviour under load. And completely separating the two is academic; not
in the derogatory sense (as in "a problem not worth discussing"), but in
the literal sense (as in "the thing we do in academia").

-Toke

Re: [Bloat] RED against bufferbloat

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 1931 bytes --]

To add my tuppence to this discussion:

I don't believe real world topologies and workloads are incompatible with
academic experimentation. All you have to do is replicate the same workload
and other conditions across each of the qdiscs you're testing, ie to change
only the qdisc between test runs. That's the basis of the scientific method.

Using a simplified topology and workload may well give you clearer and more
understandable results from which you can more easily draw a conclusion for
your paper. But that conclusion becomes correspondingly less relevant to
practical applications and bleeding edge research, as others have explained.

In the real world, I have a 3G connection shaped to half a megabit at the
provider, occasionally limited to EDGE speeds at the tower depending on
propagation conditions, typical idle latency 100ms, potential loaded
latency the best part of a MINUTE. I'm not even sure what the upload
bandwidth rate or topology is supposed to be. Packet loss is usually
acceptably low, yet the horrible loaded latency means that I can't do more
than one thing at a time on my phone; it even logs me out of Steam chat if
I load a big web page, and it also takes time to drain the traffic out of
the queue after I cancel something to do something else, or to do it a
different way.

Just now I wanted to watch an NTSB media briefing; Twitter tried to load a
preview of the video just before I clicked the link to launch the proper
YouTube app (which can do fullscreen); with the buffer thus filled, YouTube
decided the network was broken and refused to load the video. In fact it
left the navigation settings on the previous video I watched yesterday, so
that when I hit retry, it loaded that video rather than the one I'd just
clicked the link for. That's a real world workload, and a real world
failure of epic proportions, and it didn't even need bidirectional traffic
to trigger.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 2073 bytes --]

Re: [Bloat] RED against bufferbloat

[David Lang]

On Wed, 25 Feb 2015, Michael Welzl wrote:

> 2) Not everyone will always want FQ everywhere. There are potential 
> disadvantanges (e.g. the often mentioned with-a-VPN-I'm-only-1-flow problem). 
> What's necessary is to quantify them - to see how the effect of FQ (or 
> FQ_CoDel's changed FQ) plays out, and you've done a great start there in my 
> opinion.

If you only have one flow, then FQ should be just fine as you aren't competing 
against anything.

When your one flow starts mixing with other people's traffic, you will suffer a 
bit if they use multiple flows, but how could anything possibly tell that you 
are doing many things through that one flow rather than doing a single massive 
thing that should be limited for fairness with others?

The areas of the network that could have this knowledge don't have the competing 
flows, by the time you get to the point where you do have competing flows, you 
don't have any way of getting the knowledge (you can't trust whatever the user 
tells you as they could be just gaming you to get an unfair share of bandwidth)

David Lang

Re: [Bloat] RED against bufferbloat

[Michael Welzl]

> On 25. feb. 2015, at 20.04, David Lang <david@lang.hm> wrote:
> 
> On Wed, 25 Feb 2015, Michael Welzl wrote:
> 
>> 2) Not everyone will always want FQ everywhere. There are potential disadvantanges (e.g. the often mentioned with-a-VPN-I'm-only-1-flow problem). What's necessary is to quantify them - to see how the effect of FQ (or FQ_CoDel's changed FQ) plays out, and you've done a great start there in my opinion.
> 
> If you only have one flow, then FQ should be just fine as you aren't competing against anything.
> 
> When your one flow starts mixing with other people's traffic, you will suffer a bit if they use multiple flows,

this is of course the situation I meant -


> but how could anything possibly tell that you are doing many things through that one flow rather than doing a single massive thing that should be limited for fairness with others?
> 
> The areas of the network that could have this knowledge don't have the competing flows, by the time you get to the point where you do have competing flows, you don't have any way of getting the knowledge (you can't trust whatever the user tells you as they could be just gaming you to get an unfair share of bandwidth)

Not enforcing anything will let things play out like before. FQ is the opposite end, by enforcing per-flow fairness. I'm not saying one is better than the other, but recently, quite often I hear that yes, FQ is always better  :-)

Not always clear and worth investigating is all I say.

Cheers,
Michael

Re: [Bloat] RED against bufferbloat

[Alex Elsayed]

Michael Welzl wrote:

> Two points,
> 
> below...
> 
> On 25 Feb 2015, at 09:42, Alex Elsayed
> <eternaleye@gmail.com<mailto:eternaleye-
Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>>
> wrote:
<snip>
> Why exactly did you think we should have looked at asymmetric paths? To
> study what? ( I'm not debating that asymmetric paths play out different in
> behavior. I'm just saying that one needs to be clear about what exactly is
> being investigated, and why.)

It was less a criticism of your work itself, and more pointing out that Bob 
Briscoe was applying research on symmetric paths to asymmetric paths without 
questioning the applicability of its conclusions.

> and the
> latter is a one-line mention under "future work":
> 
> "More realistic traffic types (here, only bulk TCP traffic) including
> bursty traffic"
> 
> Considering those, that slide deck convinces me of very, very little
> indeed.
> 
> - it's not just a slide deck, it's a paper, in case you're interested. The
> longer version is freely available:
> https://www.duo.uio.no/bitstream/handle/10852/37381/khademi-AQM_Kids_TR434.pdf?sequence=5

Thanks for the link! I hadn't seen the full paper, reading it now.

> Why no other traffic types? Because we felt that looking at just bulk TCP
> is enough to make the key point that we wanted to get across: maybe it's
> worth taking a look at the vast amount of work that exists on AQMs, as
> even a stone old mechanism like ARED does not seem to generally be
> significantly worse than CoDel and PIE.

My objection to that is twofold:

1.) The traffic pattern it's testing is the pattern RED-like AQM was 
designed for - largely predictable both in terms of the transport's 
capabilities and the traffic traversing it. I find it relatively 
unsurprising ARED does well with that.

2.) It doesn't model what's actually seen in the real world. In designing a 
solution, an important pitfall is defining the problem out of existence.

> We didn't really want to sell ARED as it is as a much better solution
> under all conditions and say that it should now be used instead of CoDel
> and PIE (though we do conclude that it creating an ARED++ type mechanism
> seems worth considering). To make that point, I'd agree that one would
> have to see other traffic types too. A paper saying "We propose ARED++ as
> a replacement for CoDel or PIE" should have that.

Sure, but the sad fact of the matter is that the fine print of "We only 
tested it on a very narrow set of traffic patterns" tends to get lost behind 
"Look, this graph shows ARED beating CoDel and PIE!" in many cases. See the 
message I was replying to - that research was taken as relatively solid 
evidence that ARED - not necessarily even ARED++ - would be sufficient for 
internet-scale deployment, on an asymmetric transport, with real-world 
variable loads.

> My point is: when developing something new, compare against the state of
> the art. RED is *not* the state of the art, it's very old. I have seen
> arguments like needing parameterless operation because that was the
> failure of RED. Well, Sally Floyd addressed that with ARED ages ago. Most
> papers cited in this survey:
> http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6329367 begin
> with "RED didn't work because parameters had to be tuned. We propose a
> mechanism that doesn't require such tuning..."

I fully agree; I just think there's also danger in how people read a lot 
more into narrow comparisons than is justified.

> What I've seen so far:
> - CoDel compared with RED and BLUE
> - PIE compared with CoDel
> - ARED compared with CoDel and PIE
> 
> ... it would seem reasonable to take one of the many, many mechanisms that
> exist - one that was already shown to be better than RED and many others -
> , make it use delay as input, and test CoDel and PIE against that. Then
> I'd say we have a comparison against the state of the art. Now we don't
> really have that, there's a gap here.

Definitely agreed. I think we're on the same page in terms of _goals_ - 
really figuring out how the various algorithms compare in terms of their 
fitness in the solution space. There's just the fiddly bits of phrasing that 
scare me.

Re: [Bloat] RED against bufferbloat

[Michael Welzl]

> On 25 Feb 2015, at 10:31, Alex Elsayed <eternaleye@gmail.com> wrote:
> 
> Michael Welzl wrote:
> 
>> Two points,
>> 
>> below...
>> 
>> On 25 Feb 2015, at 09:42, Alex Elsayed
>> <eternaleye@gmail.com<mailto:eternaleye-
> Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>>
>> wrote:
> <snip>
>> Why exactly did you think we should have looked at asymmetric paths? To
>> study what? ( I'm not debating that asymmetric paths play out different in
>> behavior. I'm just saying that one needs to be clear about what exactly is
>> being investigated, and why.)
> 
> It was less a criticism of your work itself, and more pointing out that Bob 
> Briscoe was applying research on symmetric paths to asymmetric paths without 
> questioning the applicability of its conclusions.

No worries about criticism, I don't mind that - I asked because I'd only use an asymmetric link if I understand why, and what I'm investigating. To understand the basic behaviour of an AQM mechanism, asymmetry doesn't seem necessary to me.


>> and the
>> latter is a one-line mention under "future work":
>> 
>> "More realistic traffic types (here, only bulk TCP traffic) including
>> bursty traffic"
>> 
>> Considering those, that slide deck convinces me of very, very little
>> indeed.
>> 
>> - it's not just a slide deck, it's a paper, in case you're interested. The
>> longer version is freely available:
>> https://www.duo.uio.no/bitstream/handle/10852/37381/khademi-AQM_Kids_TR434.pdf?sequence=5
> 
> Thanks for the link! I hadn't seen the full paper, reading it now.

Thanks for reading!  AFAIK there's now much more work out there that examines the various active queue management mechanisms. 

> 
>> Why no other traffic types? Because we felt that looking at just bulk TCP
>> is enough to make the key point that we wanted to get across: maybe it's
>> worth taking a look at the vast amount of work that exists on AQMs, as
>> even a stone old mechanism like ARED does not seem to generally be
>> significantly worse than CoDel and PIE.
> 
> My objection to that is twofold:
> 
> 1.) The traffic pattern it's testing is the pattern RED-like AQM was 
> designed for - largely predictable both in terms of the transport's 
> capabilities and the traffic traversing it. I find it relatively 
> unsurprising ARED does well with that.

Same for CoDel. CoDel is very clearly designed for bulk TCP transfers. Or do we have results showing something else?
(note: CoDel, not FQ_CoDel).


> 2.) It doesn't model what's actually seen in the real world. In designing a 
> solution, an important pitfall is defining the problem out of existence.

Trying to model the real world doesn't help understanding what's going on. Understanding should be prioritized higher than realism. Once we know what happens, we can make the model gradually more real. See also my other email (answer to Sebastian Moeller).


>> We didn't really want to sell ARED as it is as a much better solution
>> under all conditions and say that it should now be used instead of CoDel
>> and PIE (though we do conclude that it creating an ARED++ type mechanism
>> seems worth considering). To make that point, I'd agree that one would
>> have to see other traffic types too. A paper saying "We propose ARED++ as
>> a replacement for CoDel or PIE" should have that.
> 
> Sure, but the sad fact of the matter is that the fine print of "We only 
> tested it on a very narrow set of traffic patterns" tends to get lost behind 
> "Look, this graph shows ARED beating CoDel and PIE!" in many cases. See the 
> message I was replying to - that research was taken as relatively solid 
> evidence that ARED - not necessarily even ARED++ - would be sufficient for 
> internet-scale deployment, on an asymmetric transport, with real-world 
> variable loads.

Well, I also haven't seen results showing that CoDel or PIE work particularly well with other traffic types. So unless such results exist, the position towards such an evaluation scenario is neutral. If ("and only if", I'd say) there are such results published somewhere, pre-dating our paper, then that's really a bad omission and you have a valid point.

Cheers,
Michael

Re: [Bloat] RED against bufferbloat

[Alex Elsayed]

Michael Welzl wrote:

> 
>> On 25 Feb 2015, at 10:31, Alex Elsayed
>> <eternaleye@gmail.com> wrote:
>> 
>> Michael Welzl wrote:
>> 
>>> Two points,
>>> 
>>> below...
>>> 
>>> On 25 Feb 2015, at 09:42, Alex Elsayed
>>> <eternaleye@gmail.com<mailto:eternaleye-
>> Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>>
>>> wrote:
>> <snip>
>>> Why exactly did you think we should have looked at asymmetric paths? To
>>> study what? ( I'm not debating that asymmetric paths play out different
>>> in behavior. I'm just saying that one needs to be clear about what
>>> exactly is being investigated, and why.)
>> 
>> It was less a criticism of your work itself, and more pointing out that
>> Bob Briscoe was applying research on symmetric paths to asymmetric paths
>> without questioning the applicability of its conclusions.
> 
> No worries about criticism, I don't mind that - I asked because I'd only
> use an asymmetric link if I understand why, and what I'm investigating. To
> understand the basic behaviour of an AQM mechanism, asymmetry doesn't seem
> necessary to me.
> 
> 
>>> and the
>>> latter is a one-line mention under "future work":
>>> 
>>> "More realistic traffic types (here, only bulk TCP traffic) including
>>> bursty traffic"
>>> 
>>> Considering those, that slide deck convinces me of very, very little
>>> indeed.
>>> 
>>> - it's not just a slide deck, it's a paper, in case you're interested.
>>> The longer version is freely available:
>>> https://www.duo.uio.no/bitstream/handle/10852/37381/khademi-AQM_Kids_TR434.pdf?sequence=5
>> 
>> Thanks for the link! I hadn't seen the full paper, reading it now.
> 
> Thanks for reading!  AFAIK there's now much more work out there that
> examines the various active queue management mechanisms.
> 
>> 
>>> Why no other traffic types? Because we felt that looking at just bulk
>>> TCP is enough to make the key point that we wanted to get across: maybe
>>> it's worth taking a look at the vast amount of work that exists on AQMs,
>>> as even a stone old mechanism like ARED does not seem to generally be
>>> significantly worse than CoDel and PIE.
>> 
>> My objection to that is twofold:
>> 
>> 1.) The traffic pattern it's testing is the pattern RED-like AQM was
>> designed for - largely predictable both in terms of the transport's
>> capabilities and the traffic traversing it. I find it relatively
>> unsurprising ARED does well with that.
> 
> Same for CoDel. CoDel is very clearly designed for bulk TCP transfers. Or
> do we have results showing something else? (note: CoDel, not FQ_CoDel).
> 
> 
>> 2.) It doesn't model what's actually seen in the real world. In designing
>> a solution, an important pitfall is defining the problem out of
>> existence.
> 
> Trying to model the real world doesn't help understanding what's going on.
> Understanding should be prioritized higher than realism. Once we know what
> happens, we can make the model gradually more real. See also my other
> email (answer to Sebastian Moeller).
> 
> 
>>> We didn't really want to sell ARED as it is as a much better solution
>>> under all conditions and say that it should now be used instead of CoDel
>>> and PIE (though we do conclude that it creating an ARED++ type mechanism
>>> seems worth considering). To make that point, I'd agree that one would
>>> have to see other traffic types too. A paper saying "We propose ARED++
>>> as a replacement for CoDel or PIE" should have that.
>> 
>> Sure, but the sad fact of the matter is that the fine print of "We only
>> tested it on a very narrow set of traffic patterns" tends to get lost
>> behind "Look, this graph shows ARED beating CoDel and PIE!" in many
>> cases. See the message I was replying to - that research was taken as
>> relatively solid evidence that ARED - not necessarily even ARED++ - would
>> be sufficient for internet-scale deployment, on an asymmetric transport,
>> with real-world variable loads.
> 
> Well, I also haven't seen results showing that CoDel or PIE work
> particularly well with other traffic types. So unless such results exist,
> the position towards such an evaluation scenario is neutral. If ("and only
> if", I'd say) there are such results published somewhere, pre-dating our
> paper, then that's really a bad omission and you have a valid point.

My main point is about the issues where the practical meets the academic.

In the academic, you are exactly right - but what I'm seeing in this very 
thread is that "ARED beats CoDel and PIE in this one specific measure on a 
symmetric transport, which indicates that we should study this more deeply" 
is treated in the _practical_ domain as strong evidence that nothing more 
than ARED is needed.

In the practical domain, FQ is hugely relevant. In the practical domain, 
asymmetry is hugely relevant. In the practical domain, ARED++ and ARED are 
sufficiently different beasts that conflating them may be quite harmful.

And seen by someone from the practical domain who wasn't aware of the 
setting in which you were operating, your results were read as a statement 
that ARED (without FQ, and seemingly not ARED++) was very likely sufficient 
for an asymmetric, variable-load, cross-traffic-heavy deployment at internet 
scale.

My original post really was very much more disputing Bob Briscoe's 
extrapolations from your results than the validity or importance of the 
results themselves.

Re: [Bloat] RED against bufferbloat

[Bob Briscoe]

Alex,

At 09:31 25/02/2015, Alex Elsayed wrote:
>It was less a criticism of your work itself, and more pointing out that Bob
>Briscoe was applying research on symmetric paths to asymmetric paths without
>questioning the applicability of its conclusions.

Mea culpa.
Just one ambiguous inference and the whole list explodes!

When I said "The paper convinced me that ARED is good enough (in the 
paper's simulations it was often better than PIE or CoDel),"

I didn't mean 'good enough to go ahead and deploy'. Don't worry we're 
testing out ARED. I meant good enough to make it the centre of my 
attention. (I did say "consider deploying" later in the sentence).

Our ARED testing is focusing on whether there are any pathologies, 
rather than whether it is slightly better or worse than the perfect 
solution X that will takes a decade to make any difference to the majority.

It's interesting that no-one picked up on the sentence "This could 
reduce deployment completion time from decades to a few months."
I take that as a symptom that the bufferbloat list is mainly 
populated by implementers. If there's a nail that can't be hit with 
the implementation hammer, it seems it's not an interesting nail, 
even if it's an extremely important nail.



Bob


________________________________________________________________
Bob Briscoe,                                                  BT 

Re: [Bloat] RED against bufferbloat

[Dave Taht]

On Wed, Feb 25, 2015 at 9:28 AM, Bob Briscoe <bob.briscoe@bt.com> wrote:
> Alex,
>
> At 09:31 25/02/2015, Alex Elsayed wrote:
>>
>> It was less a criticism of your work itself, and more pointing out that
>> Bob
>> Briscoe was applying research on symmetric paths to asymmetric paths
>> without
>> questioning the applicability of its conclusions.
>
>
> Mea culpa.
> Just one ambiguous inference and the whole list explodes!
>
> When I said "The paper convinced me that ARED is good enough (in the paper's
> simulations it was often better than PIE or CoDel),"
>
> I didn't mean 'good enough to go ahead and deploy'. Don't worry we're
> testing out ARED. I meant good enough to make it the centre of my attention.
> (I did say "consider deploying" later in the sentence).
>
> Our ARED testing is focusing on whether there are any pathologies, rather
> than whether it is slightly better or worse than the perfect solution X that
> will takes a decade to make any difference to the majority.
>
> It's interesting that no-one picked up on the sentence "This could reduce
> deployment completion time from decades to a few months."
> I take that as a symptom that the bufferbloat list is mainly populated by
> implementers. If there's a nail that can't be hit with the implementation
> hammer, it seems it's not an interesting nail, even if it's an extremely
> important nail.

I have ALWAYS said that if you can figure out how to deploy a red
based solution, do so,
and PLEASE share you you do it. Yes, if tools were generally available
to get it right, for those that do have RED, you can fix the problem
in months...

That said, all the new high end 802.11ac home routers have way better
qos systems, (though two I have tested recently were very buggy)
and it is all over openwrt and derivatives, in all of linux, all over
the world, and the only real problem remaining to fix it on CPE is to
get rate management into more hardware offload engines. Which is
happening. Sure, I would like to see the CMTSes and BRASes fixed so
customers did not have to do it themselves and more default-as-shipped
cpe the isp do the right things - but enormous progress has been made.

It is time to go fix wifi, IMHO, and let the market sort out the ISP edge.
>
> Bob
>
>
> ________________________________________________________________
> Bob Briscoe,                                                  BT
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

Re: [Bloat] RED against bufferbloat

[Sebastian Moeller]

Hi Bob,


On Feb 25, 2015, at 18:28 , Bob Briscoe <bob.briscoe@bt.com> wrote:

> Alex,
> 
> At 09:31 25/02/2015, Alex Elsayed wrote:
>> It was less a criticism of your work itself, and more pointing out that Bob
>> Briscoe was applying research on symmetric paths to asymmetric paths without
>> questioning the applicability of its conclusions.
> 
> Mea culpa.
> Just one ambiguous inference and the whole list explodes!
> 
> When I said "The paper convinced me that ARED is good enough (in the paper's simulations it was often better than PIE or CoDel),"
> 
> I didn't mean 'good enough to go ahead and deploy'. Don't worry we're testing out ARED. I meant good enough to make it the centre of my attention. (I did say "consider deploying" later in the sentence).
> 
> Our ARED testing is focusing on whether there are any pathologies, rather than whether it is slightly better or worse than the perfect solution X that will takes a decade to make any difference to the majority.
> 
> It's interesting that no-one picked up on the sentence "This could reduce deployment completion time from decades to a few months."
> I take that as a symptom that the bufferbloat list is mainly populated by implementers. If there's a nail that can't be hit with the implementation hammer, it seems it's not an interesting nail, even if it's an extremely important nail.

	My understanding was that your idea “any AQM is better than none” is the consensus; the promise of actual deployment on existing hardware is really interesting. Especially when one considers how much real world data this could produce to compare other implementations against. Also I would not be amazed if a deployment the size of BT might do, can move manufacturers to increase their interest in better AQM solutions to differentiate their products (but I guess that is dreaming ;) )

Best Regards
	Sebastian

> 
> 
> 
> Bob
> 
> 
> ________________________________________________________________
> Bob Briscoe,                                                  BT 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] RED against bufferbloat

[Dave Taht]

On Wed, Feb 25, 2015 at 12:06 AM, Bob Briscoe <bob.briscoe@bt.com> wrote:
> Sahil,
>
> At 06:46 25/02/2015, Mikael Abrahamsson wrote:
>>
>> On Tue, 24 Feb 2015, sahil grover wrote:
>>
>>> (i) First of all,i want to know whether RED was implemented or not?
>>> if not then what were the reasons(major) ?
>>
>>
>> RED has been available on most platforms, but it was generally not turned
>> on. It also needs configuration from an operator, and it's hard to know how
>> to configure.
>
>
> About a decade ago my company (BT) widely deployed RED in the upstream
> 'head-end' of our global MPLS network, i.e. the likely bottleneck in the
> customer edge router where the customer's LAN traffic enters their access
> link. We deployed it as WRED, i.e. different configurations of RED across
> the various diffserv classes, in order to minimise queuing latency in all
> the classes, including the lowest priority class. A configuration calculator
> was developed to help the engineers during set up. We still use this setup
> successfuly today, including for all our particularly latency sensitive
> customers in the finance sector.

Thank you for more public detail on this than you made available before.

I note note that free.fr deployed SFQ also about a decade ago. And
DRR+fq_codel 3 years ago. And I am told that SQF deployed also in
various places. I remain bugged I have not managed to see a study that
actually looked for FQ´s  characteristic signatures on data at scale.

In polling various ISPs at various conferences through the world - I
realize this is unscientific - uknof had 3? 4? RED users - and it was
about 30% HFSC + SFQ for downstream shaping among the rest of the
room, and about 120 in the room.

nznog (about 150 in the room) - 0 RED, 1/3 "some form of FQ".
Certainly in their world of highly mixed RTTs on and off the island I
can see why FQ was preferred. I also found from poking on their
networks that they had an unbelievable amount of bad policers
configured, with one test showing it kicking in with about 3ms of
buffering. That was what decided me that I should probably get around
to finishing the kinder/gentler "bobbie" policer - not that I have any
hope it would deploy where needed in under 5 years.

> We did not deploy RED on our broadband platform (ie public Internet), altho
> in retrospect we should have done, because any AQM is much better than none.
> We're fixing that now.

Applause. I have always said, that if you can figure out how to deploy
RED, do so, and I believe I did pester the universe for better tools
to figure out how to better automagically configure it.

>>> (ii)Second, as we all know RED controls the  average queue size from
>>> growing.
>>> So it also controls delay in a way or  we can say  is a solution to
>>> bufferbloat problem. Then why it was not considered.
>>
>>
>> It was designed to fix "bufferbloat" long before the bufferbloat word was
>> even invented. It's just that in practice, it doesn't work very well. RED is
>> configured with a drop probability slope at certain buffer depths, and
>> that's it. It doesn't react or change depending on conditions. You have to
>> guess at configure-time.
>>
>> What we need are mechanisms that work better in real life and that are
>> adaptive.
>
>
> If you were prepared to read a paper, I would have suggested:
> "The New AQM Kids on the Block: An Experimental Evaluation of CoDel and PIE"
> <http://infocom2014.ieee-infocom.org/GI14-slides/GI14-s2-3.pdf>
>
> This compares CoDel and PIE against Adaptive RED, which was a variant of RED
> proposed by Sally Floyd & co-authors in 2001 and available since Linux
> kernel version 3.3. ARED addressed the configuration sensitivity problem of
> RED by adapting the parameters to link rate and load conditions.
>
> The paper convinced me that ARED is good enough (in the paper's simulations
> it was often better than PIE or CoDel), at least for links with fixed rate
> (or only occasionally varying rate like DSL).* This is important for us
> because it means we can consider deploying AQM by adding soft controls on
> top of the RED implementations we already have in existing equipment. This
> could reduce deployment completion time from decades to a few months.

ARED is easier to configure, but I gotta admit, after having lunch and
multiple conversations with the authors 2 years back that I had
expected the follow-on work addressing my concerns to come out long
before now.

I felt that paper was massively flawed on multiple accounts and while
I see it has now been pimped into multiple publications... (which I
now understand takes a lot of work, far more than doing the
experiments themselves...)

Nobody seems to have noticed that all it tested was:

"More realistic traffic types (here, only bulk TCP traffic) including
bursty traffic"

And that doing sane analysis of more latency sensitive traffic - like
web, webrtc, voip, etc - under load in either direction - what I am
perpetually harping on - was "reserved for future work. " I am still
waiting for competent papers tackling that , and increasingly bugged
by the stream of papers that still don´t address the problem or repeat
the experiments that kicked off the whole bufferbloat effort in the
first place. (and thus I have two rants on the codel list from
yesterday). Thankfully - I have had a chance to review a few papers
prepublication in the last few months and rejected them for a variety
of truly egregious flaws... so nobody else has to suffer through them.

I like toke´s stuff FQ vs AQM work much better in comparison - which
does tackle those subjects. FQ wins over AQM alone, FQ+AQM reduces the
impact of hash collisions. He has an enormous backing set of data that
wont fit into less than 5 papers, and sadly, the first core  paper -
needed for all the follow-ons covering things like ECN, or
improvements to TCP - hasn't found publication outside the ietf as yet
- although he did get a nice tour of stanford, caltech, google, and
ucsb out of it, and got the stanford and google talks filmed - and I
think got a couple changes to TCP made out of it... and there is still
a pending improvement or two to codel... so even unpublished it has
made more impact than most of the published stuff, combined.

At caltech, Kleinrock was there (toke: "what a nice guy!") and
kleinrock listened to the preso... went back to his office, pulled out
a tattered blue mimeographed sheet from 1962 or so talking about
generalized processor sharing, and said: "Yup. Only took 50 years..."

> * I'm not sure ARED would be able to cope with the rapidly changing rate of
> a wireless link tho.

It isn´t. Neither are codel or pie (unmodified), as they work on
packets not aggregates and TXOPs, nor are sensitive to retries, and
other forms of jitter on a wifi link, like powersave.

We have made a LOT of progress on fixing wifi lately on a variety of
fronts and have most of the infrastructure architecturally in place to
massively improve things there on the only two chipsets we can fix
(mt72, ath9k). (please note there is so much else wrong in wifi that
bufferbloat is only a piece of the overall list of things we plan to
fix) The first bit of code for that has landed for review in
linux-wireless - It is called minstrel-blues which does "coupled rate
control and tx power management" - and it is *wonderful*.

http://www.linuxplumbersconf.net/2014/ocw//system/presentations/2439/original/Minstrel-Blues%20@LinuxPlumbers%202014.pdf

There is no need to transmit at the highest power if your device is
right next to the AP.
We got some pretty good test results back from it last weekend.

There is another fix to minstrel andrew just did that improves tcp
performance by - well, still measuring - it looks GREAT.

There are about 6 more things coming up for wifi over the next few
months... which includes a modified version of a fq_codel like
algorithm - so long as more funding lands... more people show up to
help... and I stop paying attention to email.  I am done talking, I am
off doing.

> HTH
>
>
> Bob
>
>
>
> ________________________________________________________________
> Bob Briscoe,                                                  BT
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

Re: [Bloat] RED against bufferbloat

[Hal Murray]

> That's easy enough. You can fit an awful lot of linked list pointers into
> the space of a single IP packet. Even if you're only assigning 64KB per
> subscriber, you can store 43 full packets and still have pointers to spare.
> A properly functioning AQM should mostly keep the queue smaller than that. 

Is a slot for the head-of-list pointer all the information you need to 
remember for an empty queue?  Or do you need some memory of recent traffic?


-- 
These are my opinions.  I hate spam.

Re: [Bloat] RED against bufferbloat

[Jonathan Morton]

> On 25 Feb, 2015, at 21:25, Hal Murray <hmurray@megapathdsl.net> wrote:
> 
>> That's easy enough. You can fit an awful lot of linked list pointers into
>> the space of a single IP packet. Even if you're only assigning 64KB per
>> subscriber, you can store 43 full packets and still have pointers to spare.
>> A properly functioning AQM should mostly keep the queue smaller than that. 
> 
> Is a slot for the head-of-list pointer all the information you need to 
> remember for an empty queue?  Or do you need some memory of recent traffic?

The Linux implementation of fq_codel uses DRR for the FQ bit.

For that, you need a head pointer (for fast dequeue), a tail pointer (for fast enqueue), and a deficit counter (to implement the fairness) per flow.  With some granularity in the pointers, or else with a fixed queue arena of some tens of megabytes maximum, you can squeeze all that into 8 bytes per flow quite comfortably.

You also need a set of codel variables per flow (these are not large), a free-list of spaces to put new packets, and a ‘next’ pointer per enqueued packet.  The latter two might already be present in some form, depending on the FIFO implementation - that is, if it’s not just a ring buffer.

With these taken into account, I think we’re still looking at 40+ full-size packets in a 64KB arena, or a larger number if smaller packets are mixed in.  DRR will tend to service flows with smaller packets more often, leading to bandwidth fairness.

 - Jonathan Morton