[Bloat] Bloat goes away, but with ~25% speed loss?

[Bloat] Bloat goes away, but with ~25% speed loss?

[Rich Brown]

I'm supporting people re: SQM/fq_codel on some of the boards, and came across a refractory problem, and I'd like to get some advice.

Summary: A person is using OpenWrt 14.07 (same code base as CeroWrt 3.10.50-1) and SQM. Turning on SQM decreases, but doesn't eliminate, bufferbloat. They also lose a significant fraction of their bandwidth (from ~12-13 mbps down to ~9-10mpbs down, with similar decrease on the upload side).

Original report: http://www.techsupportforum.com/forums/f31/bufferbloat-wont-go-away-997842.html 

- I have confirmed that it's set up right and that there doesn't seem to be any other shaping in play. 

- Also, the ISP has confirmed that the tower involved does get overloaded, but I'm not sure how that would affect the SQM rates while leaving unchanged the unshaped measurements...

What other thoughts/advice could I offer? Thanks!

Rich

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 1274 bytes --]

I think he may be seeing a complex interaction of various different queue
components and bottlenecks, which in total manages to confuse TCP
congestion control sufficiently to cause reduced throughput.

I suspect that there is a shaper at the ISP end which limits the bandwidth
available to any single subscriber. This is separate from the queue serving
the tower itself, which is what has been admitted to be periodically
overloaded. However, "overloaded" in network engineer parlance just means a
multi-user link that is saturated. There might well be enough elasticity to
allow one subscriber their full allocation by pushing others out of the
way. In this condition, the tower queue will be full and so will the shaper
queue. I imagine the shaper queue contributes the most to induced latency.

However, this "pushing others out of the way" on a drop-tail queue is done
by allowing the congestion window to grow to large values, facilitated by
the deep ISP shaper queue. This effect is defeated (by design) by running
an ingress AQM shaper, which keeps the ISP shaper queue empty.

A useful exercise might be to log the idle latency over a long period of
time, and correlate it to peak load periods, as A&A do.
http://aa.net.uk/kb-broadband-cqm.html .

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 1443 bytes --]

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[Kevin Darbyshire-Bryant]

[-- Attachment #1.1: Type: text/plain, Size: 819 bytes --]



On 04/06/15 21:01, Jonathan Morton wrote:
>
> A useful exercise might be to log the idle latency over a long period 
> of time, and correlate it to peak load periods, as A&A do. 
> http://aa.net.uk/kb-broadband-cqm.html .
>
Minor claim to 'infamy'.  The line graph shown on that page is my ADSL 
line from 4ish years ago :-)  I was having a hell of a time trying to 
convince BT to solve the packet loss issue caused by a faulty PSU in a 
satellite receiver (one whole street away and affecting a LOT of 
people)  Andrews & Arnold (ISP) were the only people capable of hitting 
BT with a sufficiently large cluebat.
>
> - Jonathan Morton
>
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat


[-- Attachment #1.2: Type: text/html, Size: 1884 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4791 bytes --]

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[Dave Taht]

On Fri, Jun 5, 2015 at 7:33 AM, Kevin Darbyshire-Bryant
<kevin@darbyshire-bryant.me.uk> wrote:
>
>
> On 04/06/15 21:01, Jonathan Morton wrote:
>
> A useful exercise might be to log the idle latency over a long period of
> time, and correlate it to peak load periods, as A&A do.
> http://aa.net.uk/kb-broadband-cqm.html .
>
> Minor claim to 'infamy'.  The line graph shown on that page is my ADSL line
> from 4ish years ago :-)  I was having a hell of a time trying to convince BT
> to solve the packet loss issue caused by a faulty PSU in a satellite
> receiver (one whole street away and affecting a LOT of people)  Andrews &
> Arnold (ISP) were the only people capable of hitting BT with a sufficiently
> large cluebat.

A&A struck me as an extremely clueful ISP (I think they have had ipv6
/48s for forever?)
and it has been my impression that folk like that were using things
like HFSC + SFQ already
in their "rate limiters", and had experimented also with fq_codel by now.

> - Jonathan Morton
>
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>



-- 
Dave Täht
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[Kevin Darbyshire-Bryant]

[-- Attachment #1.1: Type: text/plain, Size: 1368 bytes --]

On 05/06/2015 17:19, Dave Taht wrote:
> On Fri, Jun 5, 2015 at 7:33 AM, Kevin Darbyshire-Bryant
> <kevin@darbyshire-bryant.me.uk> wrote:
>>
>> On 04/06/15 21:01, Jonathan Morton wrote:
>>
>> A useful exercise might be to log the idle latency over a long period of
>> time, and correlate it to peak load periods, as A&A do.
>> http://aa.net.uk/kb-broadband-cqm.html .
>>
>> Minor claim to 'infamy'.  The line graph shown on that page is my ADSL line
>> from 4ish years ago :-)  I was having a hell of a time trying to convince BT
>> to solve the packet loss issue caused by a faulty PSU in a satellite
>> receiver (one whole street away and affecting a LOT of people)  Andrews &
>> Arnold (ISP) were the only people capable of hitting BT with a sufficiently
>> large cluebat.
> A&A struck me as an extremely clueful ISP (I think they have had ipv6
> /48s for forever?)
> and it has been my impression that folk like that were using things
> like HFSC + SFQ already
> in their "rate limiters", and had experimented also with fq_codel by now.

Adrian Kennard the owner writes code for their ISP grade routers etc (Firebricks).  Very clever chap.  And yes very early adopters and promoters of IPv6.  No idea what qdiscs etc they've experimented with.  I'll try to ask.  Would be interesting to get fq_codel or even cake into the ISP side of things!

[-- Attachment #1.2: 0x9DE2334A.asc --]
[-- Type: application/pgp-keys, Size: 3596 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[Dave Taht]

On Fri, Jun 5, 2015 at 10:20 AM, Kevin Darbyshire-Bryant
<kevin@darbyshire-bryant.me.uk> wrote:
> On 05/06/2015 17:19, Dave Taht wrote:
>> On Fri, Jun 5, 2015 at 7:33 AM, Kevin Darbyshire-Bryant
>> <kevin@darbyshire-bryant.me.uk> wrote:
>>>
>>> On 04/06/15 21:01, Jonathan Morton wrote:
>>>
>>> A useful exercise might be to log the idle latency over a long period of
>>> time, and correlate it to peak load periods, as A&A do.
>>> http://aa.net.uk/kb-broadband-cqm.html .
>>>
>>> Minor claim to 'infamy'.  The line graph shown on that page is my ADSL line
>>> from 4ish years ago :-)  I was having a hell of a time trying to convince BT
>>> to solve the packet loss issue caused by a faulty PSU in a satellite
>>> receiver (one whole street away and affecting a LOT of people)  Andrews &
>>> Arnold (ISP) were the only people capable of hitting BT with a sufficiently
>>> large cluebat.
>> A&A struck me as an extremely clueful ISP (I think they have had ipv6
>> /48s for forever?)
>> and it has been my impression that folk like that were using things
>> like HFSC + SFQ already
>> in their "rate limiters", and had experimented also with fq_codel by now.
>
> Adrian Kennard the owner writes code for their ISP grade routers etc (Firebricks).  Very clever chap.  And yes very early adopters and promoters of IPv6.  No idea what qdiscs etc they've experimented with.  I'll try to ask.  Would be interesting to get fq_codel or even cake into the ISP side of things!

Yes, I met him at one of the uknofs. Very clever chap.

So far as I knew the firebrick (which has a GREAT reputation, btw),
was bsd based.

-- 
Dave Täht
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[Adrian Kennard]

On 05/06/2015 18:23, Dave Taht wrote:
>> Adrian Kennard the owner writes code for their ISP grade routers etc (Firebricks).  Very clever chap.  And yes very early adopters and promoters of IPv6.  No idea what qdiscs etc they've experimented with.  I'll try to ask.  Would be interesting to get fq_codel or even cake into the ISP side of things!
> Yes, I met him at one of the uknofs. Very clever chap.
> 
> So far as I knew the firebrick (which has a GREAT reputation, btw),
> was bsd based.

The FireBrick tries to shift packets as they arrive and not run any
queues. It is not generally an endpoint for IP traffic and so buffer
bloat should not be an issue. The shapers all work on predicted queue at
specified speed and do not actually delay traffic (so are policers
rather than shapers) but get used for bonding multiple links.

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[Adrian Kennard]

On 05/06/2015 18:23, Dave Taht wrote:
>>> A&A struck me as an extremely clueful ISP (I think they have had ipv6
>>> /48s for forever?)
>>> and it has been my impression that folk like that were using things
>>> like HFSC + SFQ already
>>> in their "rate limiters", and had experimented also with fq_codel by now.

Oh, I meant to say, the policers used on broadband links have a very
very simple logic of small packets (<1000) are allowed more predicted
lag, and so VoIP "just works" as does DNS, interactive key presses, TCP
ACK packets and all sorts of time critical stuff.

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 552 bytes --]

On 05/06/2015 18:27, Adrian Kennard wrote:
> On 05/06/2015 18:23, Dave Taht wrote:
>>>> in their "rate limiters", and had experimented also with fq_codel by now.
> Oh, I meant to say, the policers used on broadband links have a very
> very simple logic of small packets (<1000) are allowed more predicted
> lag, and so VoIP "just works" as does DNS, interactive key presses, TCP
> ACK packets and all sorts of time critical stuff.

Blimey, the world's a small place, the Internet doubly so!  And I didn't even say 'Shibboleet'! ;-)

Kevin


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4791 bytes --]

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[Dave Taht]

On Fri, Jun 5, 2015 at 10:27 AM, Adrian Kennard <a@k.gg> wrote:
> On 05/06/2015 18:23, Dave Taht wrote:
>>>> A&A struck me as an extremely clueful ISP (I think they have had ipv6
>>>> /48s for forever?)
>>>> and it has been my impression that folk like that were using things
>>>> like HFSC + SFQ already
>>>> in their "rate limiters", and had experimented also with fq_codel by now.
>
> Oh, I meant to say, the policers used on broadband links have a very
> very simple logic of small packets (<1000) are allowed more predicted
> lag, and so VoIP "just works" as does DNS, interactive key presses, TCP
> ACK packets and all sorts of time critical stuff.

Yes, I have thought about many improvements to the linux based policer
with "bobbie", including this one.

Still, prior to running out of cpu with inbound shaping + fq_codel on
the cpe side, we were doing so well that I basically figured that the
same stuff was also very applicable to the isp's side, particularly as
we approached higher rates.

Matt Mathis is always whinging about bad policer implementations
showing up in the google mlabs datasets... certainly linux's is
insanely primitive even compared to the rfcs.

So I am curious as to how well A&A's AS20712 (?) clients are doing on
the new dslreports.com/speedtest, which shows the "bloat" grade and
actual behavior over time?

And I think dslreports is now publishing summary statistics somewhere??

 shibbolet!

-- 
Dave Täht
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[Adrian Kennard]

On 05/06/2015 18:48, Dave Taht wrote:
> So I am curious as to how well A&A's AS20712 (?) clients are doing on
> the new dslreports.com/speedtest, which shows the "bloat" grade and
> actual behavior over time?

As I say, we try not to be a factor in buffering, so would be
interesting to see. I suspect it is the equipment at bottlenecks (DSL
modems) that are the biggest concern apart from endpoints. Our logic
tries to stop any downlink kit being a buffering point by use of a
policer though, and that seems to work.

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 1000 bytes --]

On 05/06/2015 18:51, Adrian Kennard wrote:
> On 05/06/2015 18:48, Dave Taht wrote:
>> So I am curious as to how well A&A's AS20712 (?) clients are doing on
>> the new dslreports.com/speedtest, which shows the "bloat" grade and
>> actual behavior over time?
> As I say, we try not to be a factor in buffering, so would be
> interesting to see. I suspect it is the equipment at bottlenecks (DSL
> modems) that are the biggest concern apart from endpoints. Our logic
> tries to stop any downlink kit being a buffering point by use of a
> policer though, and that seems to work.
>
It was the uplink side and the recent adoption of Zyxel kit which made me wonder out loud to AA-Andrew earlier today regarding A&A bufferbloat experiences/testing on that side of things with the new modems.  You're an ISP that would have some clue in that regard and hence hopefully a bit of clout with the OEM to do things right (see baby jumbo frames support)  It's me being curious again...sorry!

Kevin


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4791 bytes --]

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[Adrian Kennard]

On 05/06/2015 18:57, Kevin Darbyshire-Bryant wrote:
> It was the uplink side and the recent adoption of Zyxel kit which
> made me wonder out loud to AA-Andrew earlier today regarding A&A
> bufferbloat experiences/testing on that side of things with the new
> modems.  You're an ISP that would have some clue in that regard and
> hence hopefully a bit of clout with the OEM to do things right (see
> baby jumbo frames support)  It's me being curious again...sorry!

We can try... Working hard to fix some showstoppers like MTU on bridging
and the like, first.

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[Dave Taht]

On Fri, Jun 5, 2015 at 10:59 AM, Adrian Kennard <a@k.gg> wrote:
> On 05/06/2015 18:57, Kevin Darbyshire-Bryant wrote:
>> It was the uplink side and the recent adoption of Zyxel kit which
>> made me wonder out loud to AA-Andrew earlier today regarding A&A
>> bufferbloat experiences/testing on that side of things with the new
>> modems.  You're an ISP that would have some clue in that regard and
>> hence hopefully a bit of clout with the OEM to do things right (see
>> baby jumbo frames support)  It's me being curious again...sorry!
>
> We can try... Working hard to fix some showstoppers like MTU on bridging
> and the like, first.

I found dslreports.com's summary stats page:
http://www.dslreports.com/speedtest/results/isp/AS20712

not enough samples, but pretty good results. Comparisons were
interesting also. I love a competitive marketplace! (And am admiring
all the tools for continuous link monitoring AA does)

On the downlink, I am relatively uninformed until today. A lot of ISPs
have mentioned HFSC+SFQ without much details.

There are several things, conflated together, that are hurting dsl
performance on the uplink nowadays.

1) A lot of DSL modem firmware used to some form of fq buried deep in
the driver. FT used to mandate SQF, for example.

2) a lot of modems would exert ethernet hardware flow control at very
minimal packet depths. (hardware flow control is so correct for a
cable, fiber, or dsl "modem" - but as manufacturers started embedding
switches into the modem, this feature has been getting lost)

3) connecting routers used to have a default packet depth of 100 (or
less!) and thus responded to flow control more sanely than the current
linux default of 1000. I would be surprised if firebrick had a packet
depth that high.

as for 2 and 3, I know a LOT of people that passionately hold onto
their old dsl modems because they are "better" than anything newer
they've tried. I know one guy that treasures his circa-1998 one...

however, as fq_codel and pie (any latency sensitive aqm) work GREAT
with hardware flow control, this would work better than any fixed
packet limit. A metric ton of people have reported results like ipfire
did in this case:

http://planet.ipfire.org/post/ipfire-2-13-tech-preview-fighting-bufferbloat

(And I keep hoping the DCB people in DCs are taking notice.)

4) In order to get higher reliability (for things like multicast udp
tv), a lot of DSL providers use "interleaving", which incurs quite a
bit of added latency on the link.

5? anyone?

--
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[Dave Taht]

63% F bloat grade for
http://www.dslreports.com/speedtest/results/isp/r3895-Orange%20Broadband

I was disappointed to see the numbers for free, but wish I had insight
into up vs down for their bloat scores.

http://www.dslreports.com/speedtest/results/isp/r2816-Free%20France

but... so wonderful to sit on a vantage point across the world! Way to
go justin!

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[jb]

[-- Attachment #1: Type: text/plain, Size: 779 bytes --]

Dave
Those result pages I pushed out this week, and they are are a work in
progress
I expect to be adding more depth to them, stay tuned.

Unrelated to buffer bloat results, there is a global speed map available
too:
   http://www.dslreports.com/speedtest/results/country

With click features for comparing multiple countries.


On Sat, Jun 6, 2015 at 6:06 AM, Dave Taht <dave.taht@gmail.com> wrote:

> 63% F bloat grade for
> http://www.dslreports.com/speedtest/results/isp/r3895-Orange%20Broadband
>
> I was disappointed to see the numbers for free, but wish I had insight
> into up vs down for their bloat scores.
>
> http://www.dslreports.com/speedtest/results/isp/r2816-Free%20France
>
> but... so wonderful to sit on a vantage point across the world! Way to
> go justin!
>

[-- Attachment #2: Type: text/html, Size: 1465 bytes --]

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[Dave Taht]

Feature requests: (in your copious spare time)

A) be able to break the bloat grade out up and down. as one example
free.fr has limited control on the down (their IPtv implementation
makes it impossible to fix), but total control on the up, and I
figured that they would do better than they did on up.

B) bloat trendline over time...

C) I am loving being able to break out AS numbers and cruise around
the world. fascinating.


On Fri, Jun 5, 2015 at 3:45 PM, jb <justin@dslr.net> wrote:
> Dave
> Those result pages I pushed out this week, and they are are a work in
> progress
> I expect to be adding more depth to them, stay tuned.
>
> Unrelated to buffer bloat results, there is a global speed map available
> too:
>    http://www.dslreports.com/speedtest/results/country
>
> With click features for comparing multiple countries.
>
>
> On Sat, Jun 6, 2015 at 6:06 AM, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> 63% F bloat grade for
>> http://www.dslreports.com/speedtest/results/isp/r3895-Orange%20Broadband
>>
>> I was disappointed to see the numbers for free, but wish I had insight
>> into up vs down for their bloat scores.
>>
>> http://www.dslreports.com/speedtest/results/isp/r2816-Free%20France
>>
>> but... so wonderful to sit on a vantage point across the world! Way to
>> go justin!
>
>



-- 
Dave Täht
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[jb]

[-- Attachment #1: Type: text/plain, Size: 815 bytes --]

It's supposed to be GPRS
but the graph library is not cooperating for a reason that I can't work out
at the moment.

> Interesting stuff.  What is that label to the left of "3G" meant to be?
It seems to show-up in the column charts for all countries.


On Sat, Jun 6, 2015 at 8:49 AM, Rick Jones <rick.jones2@hp.com> wrote:

> On 06/05/2015 03:45 PM, jb wrote:
>
>> Dave
>> Those result pages I pushed out this week, and they are are a work in
>> progress
>> I expect to be adding more depth to them, stay tuned.
>>
>> Unrelated to buffer bloat results, there is a global speed map available
>> too:
>> http://www.dslreports.com/speedtest/results/country
>>
>
> Interesting stuff.  What is that label to the left of "3G" meant to be?
> It seems to show-up in the column charts for all countries.
>
> rick jones
>
>

[-- Attachment #2: Type: text/html, Size: 1575 bytes --]

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[Rick Jones]

On 06/05/2015 05:32 PM, jb wrote:
> It's supposed to be GPRS
> but the graph library is not cooperating for a reason that I can't work
> out at the moment.

Just a guess - doesn't like having something go past the left edge 
perhaps?  You could try swapping positions with G3 and see if that makes 
things happier.

rick


>  > Interesting stuff.  What is that label to the left of "3G" meant to
> be?  It seems to show-up in the column charts for all countries.
>
>
> On Sat, Jun 6, 2015 at 8:49 AM, Rick Jones <rick.jones2@hp.com
> <mailto:rick.jones2@hp.com>> wrote:
>
>     On 06/05/2015 03:45 PM, jb wrote:
>
>         Dave
>         Those result pages I pushed out this week, and they are are a
>         work in
>         progress
>         I expect to be adding more depth to them, stay tuned.
>
>         Unrelated to buffer bloat results, there is a global speed map
>         available
>         too:
>         http://www.dslreports.com/speedtest/results/country
>
>
>     Interesting stuff.  What is that label to the left of "3G" meant to
>     be?  It seems to show-up in the column charts for all countries.
>
>     rick jones
>
>

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 761 bytes --]

Huh, those results are rather different from mine when I had free.fr:

http://burntchrome.blogspot.com/2014/01/bufferbloat-or-lack-thereof-on-freefr.html

-Aaron

On Fri, Jun 5, 2015 at 1:06 PM, Dave Taht <dave.taht@gmail.com> wrote:

> 63% F bloat grade for
> http://www.dslreports.com/speedtest/results/isp/r3895-Orange%20Broadband
>
> I was disappointed to see the numbers for free, but wish I had insight
> into up vs down for their bloat scores.
>
> http://www.dslreports.com/speedtest/results/isp/r2816-Free%20France
>
> but... so wonderful to sit on a vantage point across the world! Way to
> go justin!
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>

[-- Attachment #2: Type: text/html, Size: 1649 bytes --]

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[jb]

[-- Attachment #1: Type: text/plain, Size: 1020 bytes --]

It is a whole lot better than other ISPs.

55% are As and Bs
including Cs (which by the standard of most ISPs, is still decent)
take the total to 90% ..


On Sat, Jun 6, 2015 at 1:54 PM, Aaron Wood <woody77@gmail.com> wrote:

> Huh, those results are rather different from mine when I had free.fr:
>
>
> http://burntchrome.blogspot.com/2014/01/bufferbloat-or-lack-thereof-on-freefr.html
>
> -Aaron
>
> On Fri, Jun 5, 2015 at 1:06 PM, Dave Taht <dave.taht@gmail.com> wrote:
>
>> 63% F bloat grade for
>> http://www.dslreports.com/speedtest/results/isp/r3895-Orange%20Broadband
>>
>> I was disappointed to see the numbers for free, but wish I had insight
>> into up vs down for their bloat scores.
>>
>> http://www.dslreports.com/speedtest/results/isp/r2816-Free%20France
>>
>> but... so wonderful to sit on a vantage point across the world! Way to
>> go justin!
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>>
>
>

[-- Attachment #2: Type: text/html, Size: 2449 bytes --]

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[Kevin Darbyshire-Bryant]

[-- Attachment #1.1: Type: text/plain, Size: 5545 bytes --]

On 05/06/2015 21:06, Dave Taht wrote:
> 63% F bloat grade for
> http://www.dslreports.com/speedtest/results/isp/r3895-Orange%20Broadband
>
> I was disappointed to see the numbers for free, but wish I had insight
> into up vs down for their bloat scores.
>
> http://www.dslreports.com/speedtest/results/isp/r2816-Free%20France
>
> but... so wonderful to sit on a vantage point across the world! Way to
> go justin!
Hi Dave,

I'd like to urge caution about using 'whole ISP' bufferbloat figures.
 In the UK there are quite a few aspects that are quite simply out of an
ISP's control.  If I were an ISP I'd get rather annoyed by the 'ISP
ranked by speed results' sites out there, adding bufferbloat, certainly
without up/down split out is adding insult to injury.  This is going to
be a long post so feel free to skip/delete :-)

In the UK for example the delivery technologies are cable (1 supplier,
Virgin Media), adsl (many suppliers), vdsl (1 supplier, BT)  Virgin have
most control over CPE kit.  ADSL is a free for all.  VDSL2 was down to 2
modems (Huawei & ECI) but this is expanding outward toward free for all
status.

ADSL cpe is mostly 1 box combined modem/router often supplied by ISP. 
VDSL is mostly so far a 2 box solution with 100mbit ethernet twixt
modem/router.  Heading more towards 1 box solution (modem/router/wifi)

In cabled areas Virgin control network.  Each region/cabinet has own
level of contention/congestion.  Where not cabled area they use adsl
solution provided by BT.

ADSL on exchange by exchange basis may have BT only presence or 'a.n.
other suppliers' presence, known as LLU eg Talk Talk, Sky etc.  If
supplier has no local presence then offers service via BT kit.

VDSL at present is BT Openreach only kit in near-to-home cabinets.  BTO
trunk all data back to the exchange and then (I assume) it's split out
across the various ISP backhaul vlans.

Backhaul links between exchanges to ISP POPs may be provisioned by own
supplier (eg Sky, TalkTalk) or BT, all at differing bandwidths etc.

Some ISPs are much better at monitoring lines, latency, usage etc both
CPE and backhaul.  A&A are excellent at doing this, not only do they
have the tools, they actually use them!  They frequently identify
latency increases (and in extremis packet drops) on backhaul links from
their suppliers (mostly BT)

A&A are a good example here: A&A are the ISP but they use BT backhaul
and BT Adsl kit to supply service all over the country.  They monitor
links and provision (ie purchase) bandwidth from BT with the aim of not
being the bottleneck for their customers.  A.N.Other ISP may provide
service in exactly the same way, using the same BT kit but not purchase
sufficient bandwidth.  For the sake of completeness, A&A in 'TalkTalk'
enabled exchanges can also use 'TalkTalk' backhaul

Aside from any ISP backhaul issues or not, 'speed' is fundamentally
limited by 'the last mile' of copper, changing ISP doesn't change that
last mile unless changing technologies (ADSL->Cable->VDSL)  There's a
fundamental lack of understanding in this country as to how 'broadband'
actually works and I get dismayed by the many conversations I hear that
go something like "You should use ISP A, they're great, ISP B are crap. 
Oh but I use ISP B and they're great and ISP A are crap".  This is aside
from 'what do you mean by crap?', slow all the time?, slow when
up/downloading? (ahh, bufferbloat!)  (note to self:  You really should
finish your blog post on this topic Kevin!

BT for all their faults, do run rate limiters on the 'downlink' side so
as to not (hopefully) overfill the pipe from ISP to customer (A&A have a
means whereby they rate limit before BT if desired) so I'd like to think
that 'downlink' bufferbloat should be reasonably controlled in this
country.  Where it all turns to rat shit is on the uplink, many, many
different types of CPE, little under ISP control/specification (3rd
party adsl router/modems available freely in stores)  Add routers
running 3rd party firmware to the mix (OpenWrt, Tomato, Merlin's AsusWrt)

I guess I'm concerned that another means of beating ISPs is being
developed where the signal to noise ratio is actually very low and
sensible interpretation needs to be applied.  If going anywhere near
this I think showing up & down bloat ratings mandatory.

In the interests of full disclosure I'm not actually an A&A customer
(though I was a few years ago)  They're an excellent ISP with superb
customer service and clue.  They are probably more expensive than I
wished to pay for...customer service and clue isn't cheap.  Whilst my
current supplier (Sky...VDSL2 so actually BTO) is good and so far
reasonably reliable I *instantly* lose the will to live and want to
slash my wrists when speaking to their customer service.  I do keep
looking at A&A muttering 'native IPv6, unfiltered, customer service with
clue' though.  Breaking away from the 'triple play' service from Sky
with the associated discounts and benefits is going to be hard.  Clue
isn't cheap.

I've rambled enough.  There's some fun to be had reading Adrian's
battles with his favourite telco, example here:
http://www.revk.uk/2015/02/congestion-case-study.html

-- 
Cheers,

Kevin@Darbyshire-Bryant.me.uk <mailto:Kevin@Darbyshire-Bryant.me.uk>

Theresa May is watching YOU on the internet.  Join ORG
https://www.openrightsgroup.org/blog/2015/this-government-will-put-the-snoopers-charter-and-more-back-on-the-table



[-- Attachment #1.2: Type: text/html, Size: 9420 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4791 bytes --]

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[jb]

[-- Attachment #1: Type: text/plain, Size: 7003 bytes --]

My 2c - I wasn't planning on creating pages listing ISPs in order of
decreasing buffer bloat score.

And for speeds of course in the USA and most markets there are ranges of
products each with their own speed and price attached, so ranking ISPs by
any simple averaging of speeds is pointless as well.

However I think there is value in map-based speed results especially ones
that pin down average speeds and technologies to streets and towns, and if
there is any value at all in grading a single test for bufferbloat (or
latency to major cities, or jitter, or packet loss ..) then there is also
value in combining those statistics.

And even just pure speeds, one can statistically work out products and
create interesting comparisons, both spot, and changes over time. Even if,
at least in the US, there is no way to switch because your local cable
company is your local cable company.

There is also value in showing just how far a few ISPs are ahead of
everyone.

For example, in the USA, any speed ranking would put google fiber far out
in front, and Verizon FIOS far in front for upload speed. Why hide that
information? There may be a few ISPs that really get on top of buffer bloat
as well, and highlighting those, if they exist, makes sense to me. This can
be done without doing a top 100 chart full of nonsense.

On Sat, Jun 6, 2015 at 6:45 PM, Kevin Darbyshire-Bryant <
kevin@darbyshire-bryant.me.uk> wrote:

>  On 05/06/2015 21:06, Dave Taht wrote:
>
> 63% F bloat grade forhttp://www.dslreports.com/speedtest/results/isp/r3895-Orange%20Broadband
>
> I was disappointed to see the numbers for free, but wish I had insight
> into up vs down for their bloat scores.
> http://www.dslreports.com/speedtest/results/isp/r2816-Free%20France
>
> but... so wonderful to sit on a vantage point across the world! Way to
> go justin!
>
>   Hi Dave,
>
>  I'd like to urge caution about using 'whole ISP' bufferbloat figures.
> In the UK there are quite a few aspects that are quite simply out of an
> ISP's control.  If I were an ISP I'd get rather annoyed by the 'ISP ranked
> by speed results' sites out there, adding bufferbloat, certainly without
> up/down split out is adding insult to injury.  This is going to be a long
> post so feel free to skip/delete :-)
>
> In the UK for example the delivery technologies are cable (1 supplier,
> Virgin Media), adsl (many suppliers), vdsl (1 supplier, BT)  Virgin have
> most control over CPE kit.  ADSL is a free for all.  VDSL2 was down to 2
> modems (Huawei & ECI) but this is expanding outward toward free for all
> status.
>
>  ADSL cpe is mostly 1 box combined modem/router often supplied by ISP.
> VDSL is mostly so far a 2 box solution with 100mbit ethernet twixt
> modem/router.  Heading more towards 1 box solution (modem/router/wifi)
>
>  In cabled areas Virgin control network.  Each region/cabinet has own
> level of contention/congestion.  Where not cabled area they use adsl
> solution provided by BT.
>
>  ADSL on exchange by exchange basis may have BT only presence or 'a.n.
> other suppliers' presence, known as LLU eg Talk Talk, Sky etc.  If supplier
> has no local presence then offers service via BT kit.
>
> VDSL at present is BT Openreach only kit in near-to-home cabinets.  BTO
> trunk all data back to the exchange and then (I assume) it's split out
> across the various ISP backhaul vlans.
>
>  Backhaul links between exchanges to ISP POPs may be provisioned by own
> supplier (eg Sky, TalkTalk) or BT, all at differing bandwidths etc.
>
>  Some ISPs are much better at monitoring lines, latency, usage etc both
> CPE and backhaul.  A&A are excellent at doing this, not only do they have
> the tools, they actually use them!  They frequently identify latency
> increases (and in extremis packet drops) on backhaul links from their
> suppliers (mostly BT)
>
> A&A are a good example here: A&A are the ISP but they use BT backhaul and
> BT Adsl kit to supply service all over the country.  They monitor links and
> provision (ie purchase) bandwidth from BT with the aim of not being the
> bottleneck for their customers.  A.N.Other ISP may provide service in
> exactly the same way, using the same BT kit but not purchase sufficient
> bandwidth.  For the sake of completeness, A&A in 'TalkTalk' enabled
> exchanges can also use 'TalkTalk' backhaul
>
> Aside from any ISP backhaul issues or not, 'speed' is fundamentally
> limited by 'the last mile' of copper, changing ISP doesn't change that last
> mile unless changing technologies (ADSL->Cable->VDSL)  There's a
> fundamental lack of understanding in this country as to how 'broadband'
> actually works and I get dismayed by the many conversations I hear that go
> something like "You should use ISP A, they're great, ISP B are crap.  Oh
> but I use ISP B and they're great and ISP A are crap".  This is aside from
> 'what do you mean by crap?', slow all the time?, slow when up/downloading?
> (ahh, bufferbloat!)  (note to self:  You really should finish your blog
> post on this topic Kevin!
>
> BT for all their faults, do run rate limiters on the 'downlink' side so as
> to not (hopefully) overfill the pipe from ISP to customer (A&A have a means
> whereby they rate limit before BT if desired) so I'd like to think that
> 'downlink' bufferbloat should be reasonably controlled in this country.
> Where it all turns to rat shit is on the uplink, many, many different types
> of CPE, little under ISP control/specification (3rd party adsl
> router/modems available freely in stores)  Add routers running 3rd party
> firmware to the mix (OpenWrt, Tomato, Merlin's AsusWrt)
>
> I guess I'm concerned that another means of beating ISPs is being
> developed where the signal to noise ratio is actually very low and sensible
> interpretation needs to be applied.  If going anywhere near this I think
> showing up & down bloat ratings mandatory.
>
> In the interests of full disclosure I'm not actually an A&A customer
> (though I was a few years ago)  They're an excellent ISP with superb
> customer service and clue.  They are probably more expensive than I wished
> to pay for...customer service and clue isn't cheap.  Whilst my current
> supplier (Sky...VDSL2 so actually BTO) is good and so far reasonably
> reliable I *instantly* lose the will to live and want to slash my wrists
> when speaking to their customer service.  I do keep looking at A&A
> muttering 'native IPv6, unfiltered, customer service with clue' though.
> Breaking away from the 'triple play' service from Sky with the associated
> discounts and benefits is going to be hard.  Clue isn't cheap.
>
> I've rambled enough.  There's some fun to be had reading Adrian's battles
> with his favourite telco, example here:
> http://www.revk.uk/2015/02/congestion-case-study.html
>
> --
> Cheers,
>
>  Kevin@Darbyshire-Bryant.me.uk
>
>  Theresa May is watching YOU on the internet.  Join ORG
>
> https://www.openrightsgroup.org/blog/2015/this-government-will-put-the-snoopers-charter-and-more-back-on-the-table
>
>
>

[-- Attachment #2: Type: text/html, Size: 10117 bytes --]

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[Kevin Darbyshire-Bryant]

[-- Attachment #1.1: Type: text/plain, Size: 2494 bytes --]

On 06/06/2015 10:30, jb wrote:
> My 2c - I wasn't planning on creating pages listing ISPs in order of
> decreasing buffer bloat score.
Good :-)
>
> And for speeds of course in the USA and most markets there are ranges
> of products each with their own speed and price attached, so ranking
> ISPs by any simple averaging of speeds is pointless as well.
Absolutely.  I despair in this country because there's a regular 'ISP A
is faster than ISP B' graph/battle...and it really makes no sense.
>
> However I think there is value in map-based speed results especially
> ones that pin down average speeds and technologies to streets and
> towns, and if there is any value at all in grading a single test for
> bufferbloat (or latency to major cities, or jitter, or packet loss ..)
> then there is also value in combining those statistics.
If you're in a geographical/single supplier situation then yes.  In the
UK it simply doesn't work like that, any area, 'any' supplier.
>
> And even just pure speeds, one can statistically work out products and
> create interesting comparisons, both spot, and changes over time. Even
> if, at least in the US, there is no way to switch because your local
> cable company is your local cable company.
Our speeds are fundamentally controlled by how close to the
cabinet/exchange you are, not really ISP controlled.  There are 2
bandings on VDSL though, 80/20, 40/10.  ADSL is a bit more 'best
effort'   Incidentally VDSL is advertised as a 'fibre' service in this
country!  You can get real fibre, but really it's Fibre(rare to the
home), Cable(Virgin Media), VDSL(effectively BT), ADSL(BT&others)
>
> There is also value in showing just how far a few ISPs are ahead of
> everyone. 
>
> For example, in the USA, any speed ranking would put google fiber far
> out in front, and Verizon FIOS far in front for upload speed. Why hide
> that information? There may be a few ISPs that really get on top of
> buffer bloat as well, and highlighting those, if they exist, makes
> sense to me. This can be done without doing a top 100 chart full of
> nonsense.
:-)  No top 100 nonsense yay :-)

I *am* very interested to see up/down bufferbloat split out by
ISP/delivery technology though.  And I am very positive about the
dslreports bloat test, it's really very good and it's great to see
people making the issue of bufferbloat more measurable and more
mainstream.  Apologies if I've sounded less than appreciative.

Kevin

[-- Attachment #1.2: Type: text/html, Size: 4176 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4791 bytes --]

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[jb]

[-- Attachment #1: Type: text/plain, Size: 2948 bytes --]

All input is good. Because the general web using population seems to have
barely enough focus to last the length a tweet compared to even five years
ago.

I do tend to react and type first, then later it does have the desired
impact so sorry if it came over as a bit defensive. I fully agree with your
caution in going for big ranking lists!

On Sat, Jun 6, 2015 at 8:04 PM, Kevin Darbyshire-Bryant <
kevin@darbyshire-bryant.me.uk> wrote:

>  On 06/06/2015 10:30, jb wrote:
>
> My 2c - I wasn't planning on creating pages listing ISPs in order of
> decreasing buffer bloat score.
>
> Good :-)
>
>
>  And for speeds of course in the USA and most markets there are ranges of
> products each with their own speed and price attached, so ranking ISPs by
> any simple averaging of speeds is pointless as well.
>
> Absolutely.  I despair in this country because there's a regular 'ISP A is
> faster than ISP B' graph/battle...and it really makes no sense.
>
>
>  However I think there is value in map-based speed results especially
> ones that pin down average speeds and technologies to streets and towns,
> and if there is any value at all in grading a single test for bufferbloat
> (or latency to major cities, or jitter, or packet loss ..) then there is
> also value in combining those statistics.
>
> If you're in a geographical/single supplier situation then yes.  In the UK
> it simply doesn't work like that, any area, 'any' supplier.
>
>
>  And even just pure speeds, one can statistically work out products and
> create interesting comparisons, both spot, and changes over time. Even if,
> at least in the US, there is no way to switch because your local cable
> company is your local cable company.
>
> Our speeds are fundamentally controlled by how close to the
> cabinet/exchange you are, not really ISP controlled.  There are 2 bandings
> on VDSL though, 80/20, 40/10.  ADSL is a bit more 'best effort'
> Incidentally VDSL is advertised as a 'fibre' service in this country!  You
> can get real fibre, but really it's Fibre(rare to the home), Cable(Virgin
> Media), VDSL(effectively BT), ADSL(BT&others)
>
>
>  There is also value in showing just how far a few ISPs are ahead of
> everyone.
>
>  For example, in the USA, any speed ranking would put google fiber far
> out in front, and Verizon FIOS far in front for upload speed. Why hide that
> information? There may be a few ISPs that really get on top of buffer bloat
> as well, and highlighting those, if they exist, makes sense to me. This can
> be done without doing a top 100 chart full of nonsense.
>
> :-)  No top 100 nonsense yay :-)
>
> I *am* very interested to see up/down bufferbloat split out by
> ISP/delivery technology though.  And I am very positive about the
> dslreports bloat test, it's really very good and it's great to see people
> making the issue of bufferbloat more measurable and more mainstream.
> Apologies if I've sounded less than appreciative.
>
> Kevin
>

[-- Attachment #2: Type: text/html, Size: 4569 bytes --]

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 1555 bytes --]

What about the link type?  If there are extra overheads going on, that's
going to muck with the calculations (possibly adding latency, but shouldn't
be cutting bandwidth), since the throttling calculations will be wrong.
His ISP may be able to help with that.

It would be interesting to see what would happen if he set the bandwidth
limits at half his expected rate, and see if the latency is still there or
not.

-Aaron

On Wed, Jun 3, 2015 at 2:18 PM, Rich Brown <richb.hanover@gmail.com> wrote:

> I'm supporting people re: SQM/fq_codel on some of the boards, and came
> across a refractory problem, and I'd like to get some advice.
>
> Summary: A person is using OpenWrt 14.07 (same code base as CeroWrt
> 3.10.50-1) and SQM. Turning on SQM decreases, but doesn't eliminate,
> bufferbloat. They also lose a significant fraction of their bandwidth (from
> ~12-13 mbps down to ~9-10mpbs down, with similar decrease on the upload
> side).
>
> Original report:
> http://www.techsupportforum.com/forums/f31/bufferbloat-wont-go-away-997842.html
>
> - I have confirmed that it's set up right and that there doesn't seem to
> be any other shaping in play.
>
> - Also, the ISP has confirmed that the tower involved does get overloaded,
> but I'm not sure how that would affect the SQM rates while leaving
> unchanged the unshaped measurements...
>
> What other thoughts/advice could I offer? Thanks!
>
> Rich
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>

[-- Attachment #2: Type: text/html, Size: 2207 bytes --]

Re: [Bloat] Bloat goes away, but with ~25% speed loss?

[Rick Jones]

On 06/03/2015 02:18 PM, Rich Brown wrote:
> What other thoughts/advice could I offer? Thanks!

What sorts of CPU utilizations are being experienced during these tests?

rick jones