Re: [Cake] Recomended HW to run cake and fq_codel?

Re: [Cake] Recomended HW to run cake and fq_codel?

[Lochnair]

Hi Erik

I can't speak for what's "recommended" hardware for running CAKE on, but
from experience I can say that CAKE runs quite well on the EdgeRouters.
The EdgeRouter platform does not have CAKE support built-in, so I
maintain pre-compiled binaries here:
https://community.ubnt.com/topic/239619/cake-and-fq-pie-compiled-for-the-edgerouter-devices/1

In terms of expected performance, I've seen tests on the ER-Lite come
out at about ~80Mb/20Mb on a 200Mb/20Mb link. I've not been able to do
any "perf" runs on it myself, as my only one is in production. I don't
have an ER-X, but one of the guys on the UBNT forum claimed to get
~120-140 Mbit symmetric on the ER-X model.

Best Regards
Nils

On Fri, Apr 28, 2017, at 15:33, Erik Taraldsen via Cake wrote:
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
> Email had 1 attachment:
> + Recomended HW to run cake and fq_codel?
>   3k (message/rfc822)

Re: [Cake] Recomended HW to run cake and fq_codel?

[erik.taraldsen]

I'm actually most interested in how this works on low bandwidth accesses.  Typicaly, what can we as an ISP do to make ADSL and VDSL less sucky for our customers.  So an Edge Router PoE-5  1) or the X sfp 2) would be a good platform for this?  (Don't need the PoE or sfp, but it's the easiest accessible version here in Norway).

Nils, very good of you to keep such packages precompiled! That will save me a lot of time.


1) https://www.komplett.no/product/867933/datautstyr/nettverk/routere/ubiquiti-edgerouter-5-port-poe
2) https://www.komplett.no/product/898813/datautstyr/nettverk/routere/ubiquiti-edgerouter-x-sfp#

-Erik

________________________________________
Fra: Lochnair <me@lochnair.net>
Sendt: 28. april 2017 18:39
Til: Taraldsen Erik
Kopi: cake@lists.bufferbloat.net
Emne: Re: [Cake] Recomended HW to run cake and fq_codel?

Hi Erik

I can't speak for what's "recommended" hardware for running CAKE on, but
from experience I can say that CAKE runs quite well on the EdgeRouters.
The EdgeRouter platform does not have CAKE support built-in, so I
maintain pre-compiled binaries here:
https://community.ubnt.com/topic/239619/cake-and-fq-pie-compiled-for-the-edgerouter-devices/1

In terms of expected performance, I've seen tests on the ER-Lite come
out at about ~80Mb/20Mb on a 200Mb/20Mb link. I've not been able to do
any "perf" runs on it myself, as my only one is in production. I don't
have an ER-X, but one of the guys on the UBNT forum claimed to get
~120-140 Mbit symmetric on the ER-X model.

Best Regards
Nils

Re: [Cake] Recomended HW to run cake and fq_codel?

[Nils Andreas Svee]

Kinda surprising that the plain ER-X isn't readily available. I know
Dustin used to have them, but they're out of stock. Both of them will do
just fine, but I'd probably pick the ER-X-SFP for the beefier CPU, if
only to get some extra headroom. Mind the ER-X only have 256 MB RAM and
256 MB flash, if that matters to you.

DSL tends to suck pretty (read: very) bad without proper shaping, I
know. On that note, are you planning to run an AQM on both ends of the
bottleneck, or shape ingress traffic via a IFB device? CAKE helps a lot
when running on ingress, but it can't come close to running on both
ends.

Best Regards
Nils

On Tue, May 2, 2017, at 12:34, erik.taraldsen@telenor.com wrote:
> I'm actually most interested in how this works on low bandwidth accesses.
>  Typicaly, what can we as an ISP do to make ADSL and VDSL less sucky for
> our customers.  So an Edge Router PoE-5  1) or the X sfp 2) would be a
> good platform for this?  (Don't need the PoE or sfp, but it's the easiest
> accessible version here in Norway).
> 
> Nils, very good of you to keep such packages precompiled! That will save
> me a lot of time.
> 
> 
> 1)
> https://www.komplett.no/product/867933/datautstyr/nettverk/routere/ubiquiti-edgerouter-5-port-poe
> 2)
> https://www.komplett.no/product/898813/datautstyr/nettverk/routere/ubiquiti-edgerouter-x-sfp#
> 
> -Erik
> 
> ________________________________________
> Fra: Lochnair <me@lochnair.net>
> Sendt: 28. april 2017 18:39
> Til: Taraldsen Erik
> Kopi: cake@lists.bufferbloat.net
> Emne: Re: [Cake] Recomended HW to run cake and fq_codel?
> 
> Hi Erik
> 
> I can't speak for what's "recommended" hardware for running CAKE on, but
> from experience I can say that CAKE runs quite well on the EdgeRouters.
> The EdgeRouter platform does not have CAKE support built-in, so I
> maintain pre-compiled binaries here:
> https://community.ubnt.com/topic/239619/cake-and-fq-pie-compiled-for-the-edgerouter-devices/1
> 
> In terms of expected performance, I've seen tests on the ER-Lite come
> out at about ~80Mb/20Mb on a 200Mb/20Mb link. I've not been able to do
> any "perf" runs on it myself, as my only one is in production. I don't
> have an ER-X, but one of the guys on the UBNT forum claimed to get
> ~120-140 Mbit symmetric on the ER-X model.
> 
> Best Regards
> Nils
> 

Re: [Cake] Recomended HW to run cake and fq_codel?

[David Lang]

not having to have cake try to control the ingress also drastically reduces the 
cpu load.

so if you can put AQM on your end of the link (just about any aqm, even simple 
fq_codel), there is a huge win.

David Lang

On Tue, 2 May 2017, Nils Andreas Svee wrote:

> Kinda surprising that the plain ER-X isn't readily available. I know
> Dustin used to have them, but they're out of stock. Both of them will do
> just fine, but I'd probably pick the ER-X-SFP for the beefier CPU, if
> only to get some extra headroom. Mind the ER-X only have 256 MB RAM and
> 256 MB flash, if that matters to you.
>
> DSL tends to suck pretty (read: very) bad without proper shaping, I
> know. On that note, are you planning to run an AQM on both ends of the
> bottleneck, or shape ingress traffic via a IFB device? CAKE helps a lot
> when running on ingress, but it can't come close to running on both
> ends.
>
> Best Regards
> Nils
>
> On Tue, May 2, 2017, at 12:34, erik.taraldsen@telenor.com wrote:
>> I'm actually most interested in how this works on low bandwidth accesses.
>>  Typicaly, what can we as an ISP do to make ADSL and VDSL less sucky for
>> our customers.  So an Edge Router PoE-5  1) or the X sfp 2) would be a
>> good platform for this?  (Don't need the PoE or sfp, but it's the easiest
>> accessible version here in Norway).
>> 
>> Nils, very good of you to keep such packages precompiled! That will save
>> me a lot of time.
>> 
>> 
>> 1)
>> https://www.komplett.no/product/867933/datautstyr/nettverk/routere/ubiquiti-edgerouter-5-port-poe
>> 2)
>> https://www.komplett.no/product/898813/datautstyr/nettverk/routere/ubiquiti-edgerouter-x-sfp#
>> 
>> -Erik
>> 
>> ________________________________________
>> Fra: Lochnair <me@lochnair.net>
>> Sendt: 28. april 2017 18:39
>> Til: Taraldsen Erik
>> Kopi: cake@lists.bufferbloat.net
>> Emne: Re: [Cake] Recomended HW to run cake and fq_codel?
>> 
>> Hi Erik
>> 
>> I can't speak for what's "recommended" hardware for running CAKE on, but
>> from experience I can say that CAKE runs quite well on the EdgeRouters.
>> The EdgeRouter platform does not have CAKE support built-in, so I
>> maintain pre-compiled binaries here:
>> https://community.ubnt.com/topic/239619/cake-and-fq-pie-compiled-for-the-edgerouter-devices/1
>> 
>> In terms of expected performance, I've seen tests on the ER-Lite come
>> out at about ~80Mb/20Mb on a 200Mb/20Mb link. I've not been able to do
>> any "perf" runs on it myself, as my only one is in production. I don't
>> have an ER-X, but one of the guys on the UBNT forum claimed to get
>> ~120-140 Mbit symmetric on the ER-X model.
>> 
>> Best Regards
>> Nils
>> 
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Recomended HW to run cake and fq_codel?

[erik.taraldsen]

> Fra: Nils Andreas Svee <me@lochnair.net>
>
> just fine, but I'd probably pick the ER-X-SFP for the beefier CPU, if
> only to get some extra headroom. 

Then ER-X-SFP it is. 


>DSL tends to suck pretty (read: very) bad without proper shaping, I
> know. On that note, are you planning to run an AQM on both ends of the
> bottleneck, or shape ingress traffic via a IFB device? CAKE helps a lot
> when running on ingress, but it can't come close to running on both
> ends.

I intended to only shape on egress for this experiement.  Let downstream be handled by the BRAS's policies (Juniper ERX in our case).  Most of the customer "speed" complaints come from not throughput but latency.  And that latency is mostly ADSL/VDSL customers with large uploads to cloud services.  So I think that we will by handling the upstream better make a large improvement.  But, hey, I call it an experiemnt because it is an experiemnt.  If we see significant improvements by using IFB for downstream as well.  Then we will try to see what we can do to implement this.


-Erik

Re: [Cake] Recomended HW to run cake and fq_codel?

[Sebastian Moeller]

> On May 3, 2017, at 07:36, <erik.taraldsen@telenor.com> <erik.taraldsen@telenor.com> wrote:
> 
>> Fra: Nils Andreas Svee <me@lochnair.net>
>> 
>> just fine, but I'd probably pick the ER-X-SFP for the beefier CPU, if
>> only to get some extra headroom. 
> 
> Then ER-X-SFP it is. 
> 
> 
>> DSL tends to suck pretty (read: very) bad without proper shaping, I
>> know. On that note, are you planning to run an AQM on both ends of the
>> bottleneck, or shape ingress traffic via a IFB device? CAKE helps a lot
>> when running on ingress, but it can't come close to running on both
>> ends.
> 
> I intended to only shape on egress for this experiement.  

	Okay...

> Let downstream be handled by the BRAS's policies (Juniper ERX in our case).  

	So my ISP, (DTAG) does ingress shaping at the BRAS/BNG level and it is not terrible (see https://www.dslreports.com/speedtest/6796065 ) it still is a far cry from performance with proper ingress shaping (see: https://www.dslreports.com/speedtest/9507819 ). My interpretation of that is that the BRAS/BNG ingress shaper is too relaxed and the link behaves noticeably better if that shaper is exercised only rarely. If seen as a last line of defense against extreme latency under load increases it still has some merits though.
	Question: as an ISP what is your rationale to implement a shaper at the BRAS? Simply the fact that DSLAMs/MSANs are not capable to do it, or do you also need this to make sure there is always room for your own VoIP packets?


> Most of the customer "speed" complaints come from not throughput but latency.  And that latency is mostly ADSL/VDSL customers with large uploads to cloud services.  So I think that we will by handling the upstream better make a large improvement.  

	Well any shaping is going to help and egress shaping can be implemented at literally no bandwidth sacrifice so more power to you! I would humbly like to recommend a few issues regarding ADSL links:
0) get rid of all non-essential encapsulations, use DHCP option 82 instead of PPPoE, rethink the need for VLAN tags,
1) make sure to properly account for all the quirks of ATM’s AAL5 encapsulation (see cake’s atm keyword or “man tc-stab”)
2) preferably hoist your ADSL customers into the present and get your device manufacturers to implement PTM for adsl modems making 1) above much less involved ;)

3) as proposed by others, if you can get your CPE manufacturers to implement BQL for the xDSL interface than you might not even need to run a shaper on egress. Even though cake offers so much goodness with its different isolation modes that allow to configure per-internal-host fairness that even if the modem has BQL it still might be a customer friendly idea to run cake on the modem. I note that many lede/openwrt users seem to be quite happy with per-internal host fairness (but typically also want this on ingress, and for that I believe you need to have an ingress shaper on the CPE where it can peek into the NAT tables to figure out the internal IPs)

> But, hey, I call it an experiemnt because it is an experiemnt.  If we see significant improvements by using IFB for downstream as well.  

My prediction is that per-internal-host fairness probably makes it desirable to have an ingress shaper as well… (If you include a dedicated bit-torrent host into your test matrix this might become obvious ;) )

Best Regards
	Sebastian


> Then we will try to see what we can do to implement this.
> 
> 
> -Erik
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Recomended HW to run cake and fq_codel?

[erik.taraldsen]

> Fra: Sebastian Moeller <moeller0@gmx.de>
>
>        Question: as an ISP what is your rationale to implement a shaper at the BRAS? Simply the fact that 
> DSLAMs/MSANs are not capable to do it, or do you also need this to make sure there is always room for > your own VoIP packets?

At least the DSLAMs we use are basically switches.  Extremly limited QoS/shaping.  For a time we actually did not shape at the BRAS level and let the DSLAM's just drop whatever it could not push throug.  That was not a success.  It more or less behaved like a strictly policed access, which is not something you want.  So we went back to shaping at the BRAS.



> 0) get rid of all non-essential encapsulations, use DHCP option 82 instead of PPPoE, rethink the need 
> for VLAN tags,

Done, with the exeption of some legacy business usages.


> 1) make sure to properly account for all the quirks of ATM’s AAL5 encapsulation (see cake’s 
> atm keyword or “man tc-stab”)

Noted.

> 2) preferably hoist your ADSL customers into the present and get your device manufacturers 
> to implement PTM for adsl modems making 1) above much less involved ;)

To much legacy, so  more likely to migrate to VDSL across the board.

Regarding BQL, we need the chipset vendors to do this.  In particular Broadcom.  We do try and influence them to do this, but we simply are a to small to get traction.


-Erik

Re: [Cake] Recomended HW to run cake and fq_codel?

[Sebastian Moeller]

Hi Erik,

> On May 3, 2017, at 09:27, <erik.taraldsen@telenor.com> <erik.taraldsen@telenor.com> wrote:
> 
>> Fra: Sebastian Moeller <moeller0@gmx.de>
>> 
>>       Question: as an ISP what is your rationale to implement a shaper at the BRAS? Simply the fact that 
>> DSLAMs/MSANs are not capable to do it, or do you also need this to make sure there is always room for > your own VoIP packets?
> 
> At least the DSLAMs we use are basically switches.  Extremly limited QoS/shaping.  For a time we actually did not shape at the BRAS level and let the DSLAM's just drop whatever it could not push throug.  That was not a success.  It more or less behaved like a strictly policed access, which is not something you want.  So we went back to shaping at the BRAS.

	Ah, thanks for the insight, I had always assumed the rationale to be less obvious (like making VoIP more robust, keeping (D)DOS traffic out of the aggregation network; I had never assumed that dslams might simply be bad at traffic regulation ;) )

> 
> 
> 
>> 0) get rid of all non-essential encapsulations, use DHCP option 82 instead of PPPoE, rethink the need 
>> for VLAN tags,
> 
> Done, with the exeption of some legacy business usages.

	This is quite enlightened, great!

> 
> 
>> 1) make sure to properly account for all the quirks of ATM’s AAL5 encapsulation (see cake’s 
>> atm keyword or “man tc-stab”)
> 
> Noted.

	All I ever learned about this topic should be summarized in https://github.com/moeller0/ATM_overhead_detector/wiki .

> 
>> 2) preferably hoist your ADSL customers into the present and get your device manufacturers 
>> to implement PTM for adsl modems making 1) above much less involved ;)
> 
> To much legacy, so  more likely to migrate to VDSL across the board.

	Which effectively is the same for customers (except for unlucky ones at the end of very long lines, I believe); ATM deserves to retire ;)


> 
> Regarding BQL, we need the chipset vendors to do this.  In particular Broadcom.  We do try and influence them to do this, but we simply are a to small to get traction.

	Well, compared to most on this list you have a huge impact on the chipset vendors, so I hope for the best. Is there anybode besides Broadcom and Intel actually producing VDSL chipsets or is that the set of vendors that need to be convinced?

Best Regards
	Sebastian

> 
> 
> -Erik

Re: [Cake] Recomended HW to run cake and fq_codel?

[erik.taraldsen]

> Fra: Sebastian Moeller <moeller0@gmx.de>
>        Well, compared to most on this list you have a huge impact on the chipset vendors, so I hope for the best. 
> Is there anybode besides Broadcom and Intel actually producing VDSL chipsets or is that the set of vendors 
> that need to be convinced?

There are other chipset vendors, but they have very little precense in the west.  They typically aim for super low-cost volum in Asia.  Pure bridge units which in extreme cases can't push packets trough at sync speed.  So for all intents and purposes I think the focus for DSL has to be Broadcom and Intel.


-Erik

Re: [Cake] Recomended HW to run cake and fq_codel?

[Tristan Seligmann]

[-- Attachment #1: Type: text/plain, Size: 614 bytes --]

On Fri, 28 Apr 2017 at 18:39 Lochnair via Cake <cake@lists.bufferbloat.net>
wrote:

> In terms of expected performance, I've seen tests on the ER-Lite come
> out at about ~80Mb/20Mb on a 200Mb/20Mb link. I've not been able to do
> any "perf" runs on it myself, as my only one is in production. I don't
> have an ER-X, but one of the guys on the UBNT forum claimed to get
> ~120-140 Mbit symmetric on the ER-X model.
>

My connection is only 100/100 Mbps but I have cake installed in both egress
and ingress directions on my ER-X and it seems like I have CPU to spare
even when loading the link in both directions.

[-- Attachment #2: Type: text/html, Size: 917 bytes --]

Re: [Cake] Recomended HW to run cake and fq_codel?

[Pete Heist]

> Message: 1
> Date: Tue, 2 May 2017 10:34:45 +0000
> From: <erik.taraldsen@telenor.com>
> To: <me@lochnair.net>
> Cc: <cake@lists.bufferbloat.net>
> Subject: Re: [Cake] Recomended HW to run cake and fq_codel?
> Message-ID: <1493721285271.28909@telenor.com>
> Content-Type: text/plain; charset="iso-8859-1"
> 
> I'm actually most interested in how this works on low bandwidth accesses.  Typicaly, what can we as an ISP do to make ADSL and VDSL less sucky for our customers.  So an Edge Router PoE-5  1) or the X sfp 2) would be a good platform for this?  (Don't need the PoE or sfp, but it's the easiest accessible version here in Norway).
> 
> Nils, very good of you to keep such packages precompiled! That will save me a lot of time.

> ------------------------------
> 
> Message: 2
> Date: Tue, 02 May 2017 14:11:20 +0200
> From: Nils Andreas Svee <me@lochnair.net>
> To: erik.taraldsen@telenor.com
> Cc: cake@lists.bufferbloat.net
> Subject: Re: [Cake] Recomended HW to run cake and fq_codel?
> Message-ID:
> 	<1493727080.1510042.962956680.40220FCB@webmail.messagingengine.com>
> Content-Type: text/plain; charset="utf-8"
> 
> Kinda surprising that the plain ER-X isn't readily available. I know
> Dustin used to have them, but they're out of stock. Both of them will do
> just fine, but I'd probably pick the ER-X-SFP for the beefier CPU, if
> only to get some extra headroom. Mind the ER-X only have 256 MB RAM and
> 256 MB flash, if that matters to you.
> 
> DSL tends to suck pretty (read: very) bad without proper shaping, I
> know. On that note, are you planning to run an AQM on both ends of the
> bottleneck, or shape ingress traffic via a IFB device? CAKE helps a lot
> when running on ingress, but it can't come close to running on both
> ends.
> 
> Best Regards
> Nils

Just sharing some experience/thoughts from a few angles:

- As for low bandwidth, in my experience AQM works great on low bandwidth ADSL. A few years ago I used fq_codel at a campground to shape a 0.5 / 5 Mbps ADSL connection. With up to 130 people in the camp, it was a disaster before fq_codel, where one person saturating either the up or downstream could easily cause 600+ ms of induced latency. fq_codel could keep that to 40-50 ms under load, enough to make it usable for web browsing, at least, and Cake does better.

- I did the ER-X testing referred to in the UBNT forums (pgage). I’ve since learned much more about AQM by testing point-to-point WiFi setups, so I should really repeat those ER-X tests some time to make sure my results were accurate, but afaik ~120 Mbps using Cake is possible with the ER-X. I’m now using Cake in production on the ER-X at rates around 40 Mbps with Lochnair’s builds (on EdgeOS 1.9.1) and it does a great job.

- It’s an interesting question: what can be done as an ISP? Essentially it boils down to the fundamentals of deploying AQM- finding where the queues are forming and placing fq_codel or Cake at the bottleneck links, preferring “hardware” queue management like BQL or in the case of WiFi the ath9k’s driver in LEDE, over soft rate limiting, where possible. When soft rate limiting, the rate limiting strategy and chosen rate are the most CPU intensive and finicky parts of deploying AQM.

- I don't understand why ADSL modem vendors don’t just bake BQL-like functionality right into their devices so they can ship AQM without the need for soft rate limiting. AQM is so effective on ADSL's upstream that it seems it would just make a lot of sense. For that matter, why not on the DSLAM as well to shape the customer’s downstream, if that’s also a bottleneck?

- There seems to be a bit of upheaval now with BBR. If every host had BBR deployed, that would theoretically mitigate the need for AQM, but it’s a) it’s going to be years before that happens and b) I’m not sure all the BBR corner cases have been found yet. There are far more knowledgeable people than me on this and already more detailed discussions about it.

Re: [Cake] Recomended HW to run cake and fq_codel?

[erik.taraldsen]

> Fra: Pete Heist <peteheist@gmail.com>
> - As for low bandwidth, in my experience AQM works great on low bandwidth ADSL. A few years ago I
> used fq_codel at a campground to shape a 0.5 / 5 Mbps ADSL connection. With up to 130 people in the
> camp, it was a disaster before fq_codel, where one person saturating either the up or downstream 
> could easily cause 600+ ms of induced latency. fq_codel could keep that to 40-50 ms under load, 
>enough to make it usable for web browsing, at least, and Cake does better.

Thats encouraging! 


>  It’s an interesting question: what can be done as an ISP? Essentially it boils down to the fundamentals 
> of deploying AQM- finding where the queues are forming and placing fq_codel or Cake at the 
> bottleneck links, preferring “hardware” queue management like BQL or in the case of WiFi the ath9k’s 
> driver in LEDE, over soft rate limiting, where possible. When soft rate limiting, the rate limiting strategy 
> and chosen rate are the most CPU intensive and finicky parts of deploying AQM.

What I see as short term posibiliteis for us as ISP's is to push our vendors to include this as a part of the feature set.  We also could do better with the maketing.  Lets steal an idea from the Video area.  HD is often written as 1080P@60.  Why not do the same for internet speed?  60M@80ms.  Where the @80ms would be the larges latency in either direction that queue management would introduce?  (This of cource introduces the risk of artificialy tuning the @xxms to low and ending up with strict policing)


> - I don't understand why ADSL modem vendors don’t just bake BQL-like functionality right into their 
> devices so they can ship AQM without the need for soft rate limiting. AQM is so effective on ADSL's 
> upstream that it seems it would just make a lot of sense. For that matter, why not on the DSLAM as well 
> to shape the customer’s downstream, if that’s also a bottleneck?

I think most ISP's handle shaping on the BRAS level rather than on the DSLAM, as DSLAM's in general have very limited shaping/qos capabilites.

Regarding CPEs, to be fair, up coming devices from Intel (Lantiq) will more or less do away with HW accellerators and do everything in software.  Then the vendors are a lot more free to implement better shapeing strategies.

The trade shows and all sales pitches focuses mostly on next gen stuff.  There are comparatively very little recources dedicated to ADSL, where the best schedulers is most needed. 


-Erik

Re: [Cake] Recomended HW to run cake and fq_codel?

[Pete Heist]

> On May 3, 2017, at 7:59 AM, <erik.taraldsen@telenor.com> <erik.taraldsen@telenor.com> wrote:
> 
>> It’s an interesting question: what can be done as an ISP? Essentially it boils down to the fundamentals 
>> of deploying AQM- finding where the queues are forming and placing fq_codel or Cake at the 
>> bottleneck links, preferring “hardware” queue management like BQL or in the case of WiFi the ath9k’s 
>> driver in LEDE, over soft rate limiting, where possible. When soft rate limiting, the rate limiting strategy 
>> and chosen rate are the most CPU intensive and finicky parts of deploying AQM.
> 
> What I see as short term posibiliteis for us as ISP's is to push our vendors to include this as a part of the feature set.  We also could do better with the maketing.  Lets steal an idea from the Video area.  HD is often written as 1080P@60.  Why not do the same for internet speed?  60M@80ms.  Where the @80ms would be the larges latency in either direction that queue management would introduce?  (This of cource introduces the risk of artificialy tuning the @xxms to low and ending up with strict policing)

True, in the same way max throughputs have been pushed up various ways, I wouldn’t want to see a latency war where ‘pfifo limit 2’ is being deployed, and yet I like the idea of spreading awareness about the importance of latency. When I hear users ask “is it stable?”, I think latency is a big part of what they’re asking about without realizing it. There’s a certain “latency stress" that comes when clicking a link on the web and not getting an immediate response. I wonder if anyone has studied that.

>> - I don't understand why ADSL modem vendors don’t just bake BQL-like functionality right into their 
>> devices so they can ship AQM without the need for soft rate limiting. AQM is so effective on ADSL's 
>> upstream that it seems it would just make a lot of sense. For that matter, why not on the DSLAM as well 
>> to shape the customer’s downstream, if that’s also a bottleneck?
> 
> I think most ISP's handle shaping on the BRAS level rather than on the DSLAM, as DSLAM's in general have very limited shaping/qos capabilities.

That makes sense. I’ve never worked with provider side ADSL equipment so I lumped it all under the term “DSLAM”, not knowing what a BRAS was before. :) 

Another option for ISPs (failing AQM support in the devices, and instead of deploying devices on the customer side), could be to provide each customer a queue that’s tuned to their link rate. There could be an HTB tree with classes for each customer and Cake at the leafs. Knowing each customer’s link rate (assuming it’s not variable) you’d set HTB’s rate to something less than that. There would be work to do as each customer is added and removed, but at least it would be transparent to them. AQM is best done at egress where packets originate, so I’m not sure how well it would work in practice.

What’s usually used for an ADSL provider’s backhaul, fiber?

Re: [Cake] Recomended HW to run cake and fq_codel?

[Andy Furniss]

Pete Heist wrote:

> Another option for ISPs (failing AQM support in the devices, and
> instead of deploying devices on the customer side), could be to
> provide each customer a queue that’s tuned to their link rate. There
> could be an HTB tree with classes for each customer and Cake at the
> leafs. Knowing each customer’s link rate (assuming it’s not
> variable)

FWIW, on the narrow knowing variable rate point, in the UK DSLAMs use (I
think) TR101 to report sync rates to ISPs.
It has always been the case that BRAS is set with this, my ISP does also
use it to do downstream (for me, egress for them) QOS.
Of course they need (expensive?) kit to mark, and DPI/rules to classify
traffic ....
They use Ellacoyas. I don't know if they do the actual shaping or just
the classification and marking.

Re: [Cake] Recomended HW to run cake and fq_codel?

[erik.taraldsen]

> Fra: Pete Heist <peteheist@gmail.com>
>
> Another option for ISPs (failing AQM support in the devices, and instead of deploying devices on the customer 
> side), could be to provide each customer a queue that’s tuned to their link rate. There could be an HTB tree 
> with classes for each customer and Cake at the leafs. Knowing each customer’s link rate (assuming it’s not 
> variable) you’d set HTB’s rate to something less than that. There would be work to do as each customer is 
> added and removed, but at least it would be transparent to them. AQM is best done at egress where packets 
> originate, so I’m not sure how well it would work in practice.

We are limited by what the HW can do here.  The Juniper ERX's we use aggregate multiple 10.000's of users.   So Queue handeling must be supported in silicone.  We do have a queue pr customer, but we cant support HTB.


> What’s usually used for an ADSL provider’s backhaul, fiber?

I'm not sure what our competition use, but we mainly use fiber.  There are some edge cases using alternative backhaul such as radio.  Norway has challenging topology for infrastructure, so some will have to suffer unfortunately. :) 



-Erik

Re: [Cake] Recomended HW to run cake and fq_codel?

[Dave Taht]

>What I see as short term posibiliteis for us as ISP's is to push our vendors to include this as a part of the feature set.  We also could do >better with the maketing.  Lets steal an idea from the Video area.  HD is often written as 1080P@60.  Why not do the same for internet >speed?  60M@80ms.  Where the @80ms would be the larges latency in either direction that queue management would introduce?  (This >of cource introduces the risk of artificialy tuning the @xxms to low and ending up with strict policing)

I like this.

900M @ 1.2ms. Taking the 99th percentile from:

http://www.drhleny.cz/bufferbloat/cake/round0/tor_rrultor_eg_cake_950mbit/rrul_torrent-ping_cdf.svg

Since this is a measure of flow switching time:

950Mbit @ 1.2 "FQ"

better, I think, scaled by a reference to pifo on the same test and
test conditions,
(8ms/1.2ms in this case)

http://www.drhleny.cz/bufferbloat/cake/round0/tor_rrultor_eg_pfifo_950mbit/rrul_torrent-ping_cdf.svg

cake: XMbit @ 6.666 FQ!



On Tue, May 2, 2017 at 10:59 PM,  <erik.taraldsen@telenor.com> wrote:
>> Fra: Pete Heist <peteheist@gmail.com>
>> - As for low bandwidth, in my experience AQM works great on low bandwidth ADSL. A few years ago I
>> used fq_codel at a campground to shape a 0.5 / 5 Mbps ADSL connection. With up to 130 people in the
>> camp, it was a disaster before fq_codel, where one person saturating either the up or downstream
>> could easily cause 600+ ms of induced latency. fq_codel could keep that to 40-50 ms under load,
>>enough to make it usable for web browsing, at least, and Cake does better.
>
> Thats encouraging!
>
>
>>  It’s an interesting question: what can be done as an ISP? Essentially it boils down to the fundamentals
>> of deploying AQM- finding where the queues are forming and placing fq_codel or Cake at the
>> bottleneck links, preferring “hardware” queue management like BQL or in the case of WiFi the ath9k’s
>> driver in LEDE, over soft rate limiting, where possible. When soft rate limiting, the rate limiting strategy
>> and chosen rate are the most CPU intensive and finicky parts of deploying AQM.
>
> What I see as short term posibiliteis for us as ISP's is to push our vendors to include this as a part of the feature set.  We also could do better with the maketing.  Lets steal an idea from the Video area.  HD is often written as 1080P@60.  Why not do the same for internet speed?  60M@80ms.  Where the @80ms would be the larges latency in either direction that queue management would introduce?  (This of cource introduces the risk of artificialy tuning the @xxms to low and ending up with strict policing)
>
>
>> - I don't understand why ADSL modem vendors don’t just bake BQL-like functionality right into their
>> devices so they can ship AQM without the need for soft rate limiting. AQM is so effective on ADSL's
>> upstream that it seems it would just make a lot of sense. For that matter, why not on the DSLAM as well
>> to shape the customer’s downstream, if that’s also a bottleneck?
>
> I think most ISP's handle shaping on the BRAS level rather than on the DSLAM, as DSLAM's in general have very limited shaping/qos capabilites.
>
> Regarding CPEs, to be fair, up coming devices from Intel (Lantiq) will more or less do away with HW accellerators and do everything in software.  Then the vendors are a lot more free to implement better shapeing strategies.
>
> The trade shows and all sales pitches focuses mostly on next gen stuff.  There are comparatively very little recources dedicated to ADSL, where the best schedulers is most needed.
>
>
> -Erik
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] Recomended HW to run cake and fq_codel?

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 370 bytes --]

My pet suggestion here is to represent latency as its inverse,
"responsiveness" with units of Hz.  This has the dual advantages of bigger
numbers being better, and the figures being directly comparable with
framerates.

As you say, the methodology will need to be very carefully specified, so
that we get a meaningful measurement that's hard to game.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 444 bytes --]

Re: [Cake] Recomended HW to run cake and fq_codel?

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 1087 bytes --]


> On Nov 27, 2017, at 1:04 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
> My pet suggestion here is to represent latency as its inverse, "responsiveness" with units of Hz.  This has the dual advantages of bigger numbers being better, and the figures being directly comparable with framerates.
> 
> As you say, the methodology will need to be very carefully specified, so that we get a meaningful measurement that's hard to game.
> 
I like that idea...

Then it’s how to measure it. 1 / latency where latency is what…the maximum value you’ll see considering all traffic as besteffort at a fixed number of concurrent flows? Otherwise it would have do be expressed differently for different traffic classes, which is probably already too complicated for most people.

Food for thought, I know this is the opposite direction, but I’ve always liked in Europe how car “mileage” is expressed as consumption (L/100km) instead of efficiency (miles/gallon). Yes, then a lower number is better, but it’s easier to calculate how much gas you’ll use for a given trip.

[-- Attachment #2: Type: text/html, Size: 1658 bytes --]

Re: [Cake] Recomended HW to run cake and fq_codel?

[Sebastian Moeller]

Well,

how about keeping it simple and just give the latency increment under full (bidirectional) link saturation (I guess a catchy acronym might be found)? Yes this is a number where lower is better, but it also has immediate information (like: "mmmh, at an added 3seconds under load, VoIP might suffer a bit if I start heavy torrenting...").

I am not opposed to the inverse per se and I also like the "bigger is better" property, but mental division is hard and the period seems to be more informative than the frequency. But at this point anything that will get some traction will be a winner...



Best Regards
	Sebastian

> On Nov 27, 2017, at 13:47, Pete Heist <peteheist@gmail.com> wrote:
> 
> 
>> On Nov 27, 2017, at 1:04 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
>> My pet suggestion here is to represent latency as its inverse, "responsiveness" with units of Hz.  This has the dual advantages of bigger numbers being better, and the figures being directly comparable with framerates.
>> 
>> As you say, the methodology will need to be very carefully specified, so that we get a meaningful measurement that's hard to game.
>> 
> I like that idea...
> 
> Then it’s how to measure it. 1 / latency where latency is what…the maximum value you’ll see considering all traffic as besteffort at a fixed number of concurrent flows? Otherwise it would have do be expressed differently for different traffic classes, which is probably already too complicated for most people.
> 
> Food for thought, I know this is the opposite direction, but I’ve always liked in Europe how car “mileage” is expressed as consumption (L/100km) instead of efficiency (miles/gallon). Yes, then a lower number is better, but it’s easier to calculate how much gas you’ll use for a given trip.
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Recomended HW to run cake and fq_codel?

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 1497 bytes --]


> On Nov 27, 2017, at 4:54 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> how about keeping it simple and just give the latency increment under full (bidirectional) link saturation (I guess a catchy acronym might be found)? Yes this is a number where lower is better, but it also has immediate information (like: "mmmh, at an added 3seconds under load, VoIP might suffer a bit if I start heavy torrenting...”).

Couldn’t the number of flows contributing to the saturation affect the results though, so that it would have to be specified?

I think this gets to the crux of the original thinking behind the RRUL specification. The RRUL “Score” section contains a lot of detail for an “optimum result”, and further admissions that it isn’t easy to assess: https://www.bufferbloat.net/projects/bloat/wiki/RRUL_Spec/ <https://www.bufferbloat.net/projects/bloat/wiki/RRUL_Spec/>.

If we could come up with one all-encompassing and reliable metric for measuring the “goodness” of queueing behavior, it would also make testing much easier. I really wish for such a test, and sometimes try to figure out how it would look, but I don’t think it’s an easy problem to solve.

> I am not opposed to the inverse per se and I also like the "bigger is better" property, but mental division is hard and the period seems to be more informative than the frequency. But at this point anything that will get some traction will be a winner...
> 
> Best Regards
> 	Sebastian


[-- Attachment #2: Type: text/html, Size: 2320 bytes --]

Re: [Cake] Recomended HW to run cake and fq_codel?

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 1560 bytes --]

An important factor when designing the test is the difference between
intra-flow and inter-flow induced latencies, as well as the baseline
latency.

In general, AQM by itself controls intra-flow induced latency, while flow
isolation (commonly FQ) controls inter-flow induced latency.  I consider
the latter to be more important to measure.

Baseline latency is a factor of the underlying network topology, and is the
type of latency most often measured.  It should be measured in the no-load
condition, but the choice of remote endpoint is critical.  Large ISPs could
gain an unfair advantage if they can provide a qualifying endpoint within
their network, closer to the last mile links than most realistic Internet
services.  Conversely, ISPs are unlikely to endorse a measurement scheme
which places the endpoints too far away from them.

One reasonable possibility is to use DNS lookups to randomly-selected gTLDs
as the benchmark.  There are gTLD DNS servers well-placed in essentially
all regions of interest, and effective DNS caching is a legitimate means
for an ISP to improve their customers' internet performance.  Random
lookups (especially of domains which are known to not exist) should defeat
the effects of such caching.

Induced latency can then be measured by applying a load and comparing the
new latency measurement to the baseline.  This load can simultaneously be
used to measure available throughput.  The tests on dslreports offer a
decent example of how to do this, but it would be necessary to standardise
the load.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 1682 bytes --]

Re: [Cake] Recomended HW to run cake and fq_codel?

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 2752 bytes --]


> On Nov 27, 2017, at 7:28 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
> An important factor when designing the test is the difference between intra-flow and inter-flow induced latencies, as well as the baseline latency.
> 
> In general, AQM by itself controls intra-flow induced latency, while flow isolation (commonly FQ) controls inter-flow induced latency.  I consider the latter to be more important to measure.
> 
Intra-flow induced latency should also be important for web page load time and websockets, for example. Maybe not as important as inter-flow, because there you’re talking about how voice, videoconferencing and other interactive apps work together with other traffic, which is what people are affected by the most when it doesn’t work.

I don’t think it’s too much to include one public metric for each. People are used to “upload” and “download”, maybe they’d one day get used to “reactivity” and “interactivity”, or some more accessible terms.
> Baseline latency is a factor of the underlying network topology, and is the type of latency most often measured.  It should be measured in the no-load condition, but the choice of remote endpoint is critical.  Large ISPs could gain an unfair advantage if they can provide a qualifying endpoint within their network, closer to the last mile links than most realistic Internet services.  Conversely, ISPs are unlikely to endorse a measurement scheme which places the endpoints too far away from them.
> 
> One reasonable possibility is to use DNS lookups to randomly-selected gTLDs as the benchmark.  There are gTLD DNS servers well-placed in essentially all regions of interest, and effective DNS caching is a legitimate means for an ISP to improve their customers' internet performance.  Random lookups (especially of domains which are known to not exist) should defeat the effects of such caching.
> 
> Induced latency can then be measured by applying a load and comparing the new latency measurement to the baseline.  This load can simultaneously be used to measure available throughput.  The tests on dslreports offer a decent example of how to do this, but it would be necessary to standardise the load.
> 
It would be good to know what an average worst case heavy load is on a typical household Internet connection and standardize on that. Windows updates for example can be pretty bad (many flows).

DNS is an interesting possibility. On the one hand all you get is RTT, but on the other hand your server infrastructure is already available. I use the dslreports speedtest pretty routinely as it’s decent, although results can vary significantly between runs. If they’re using DNS to measure latency, I hadn’t realized it.

[-- Attachment #2: Type: text/html, Size: 3409 bytes --]

Re: [Cake] Recomended HW to run cake and fq_codel?

[Pete Heist]

> Message: 6
> Date: Wed, 3 May 2017 11:14:14 +0000
> From: <erik.taraldsen@telenor.com>
> To: <moeller0@gmx.de>
> Cc: <me@lochnair.net>, <cake@lists.bufferbloat.net>
> Subject: Re: [Cake] Recomended HW to run cake and fq_codel?
> Message-ID: <1493810054966.42941@telenor.com>
> Content-Type: text/plain; charset="iso-8859-1"
> 
>> Fra: Sebastian Moeller <moeller0@gmx.de>
>>       Well, compared to most on this list you have a huge impact on the chipset vendors, so I hope for the best. 
>> Is there anybode besides Broadcom and Intel actually producing VDSL chipsets or is that the set of vendors 
>> that need to be convinced?
> 
> There are other chipset vendors, but they have very little precense in the west.  They typically aim for super low-cost volum in Asia.  Pure bridge units which in extreme cases can't push packets trough at sync speed.  So for all intents and purposes I think the focus for DSL has to be Broadcom and Intel.

If someone takes the time to write a petition, I would sign it as I’m sure would others. Delivered to the right people (the other tricky part), it could mean something. I feel the same way about Cake on the ER-X.