[Bloat] an observation from the field

[Bloat] an observation from the field

[Dave Taht]

In looking over the increasingly vast sqm-related deployment, there's
a persistent data point that pops up regarding inbound shaping at high
rates.

We give users a choice - run out of cpu at those rates or do inbound
sqm at a rate their cpu can afford.  A remarkable percentage are
willing to give up tons of bandwidth in order to avoid latency
excursions (oft measured, even in these higher speed 200+Mbit
deployments, in the 100s of ms) -

At least some users want low delay always. It's just the theorists
that want high utilization right at the edge of capacity. Users are
forgiving about running out of cpu - disgruntled, but forgiving.

Certainly I'm back at the point of recommending tbf+fq_codel for
inbound shaping at higher rates - and looking at restoring the high
speed version of cake - and I keep thinking a better policer is
feasible.

-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Bloat] an observation from the field

[Jonathan Foulkes]

Dave, very interesting to hear. In my dataset, I find that non-technical users respond positively to the benefits of low-latency, even if the speedtest metrics show much lower numbers than their plan indicates. Stuff happens quicker, and more consistently,  therefore they are happy.

It’s the semi-techies and hard-core geeks that are a challenge, as they insist on getting the ‘speed’ they pay for, and no amount of explaining satisfies them.

Interestingly, we see some 200+ Mbps lines that show low bloat on the inbound leg with QoS off during tests, but if QoS is left disabled, speed is high, but real-world use suffers and QoS has to be reinstated on the inbound path. Seems the transient bloat on these lines affects usability to the point where users will now accept lower throughput in exchange for goodput.
We see this mainly on Cable systems, not so much on (well deployed) fiber.

I see the challenge as needing to continue to socialize the benefits of low latency vs capacity to the tech crowd. And I still think we need a good end-user accessible test that would prove that point in a way non-techies would get.

Cheers,

Jonathan Foulkes
CEO - Evenroute.com

> On Aug 28, 2018, at 1:07 PM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> In looking over the increasingly vast sqm-related deployment, there's
> a persistent data point that pops up regarding inbound shaping at high
> rates.
> 
> We give users a choice - run out of cpu at those rates or do inbound
> sqm at a rate their cpu can afford.  A remarkable percentage are
> willing to give up tons of bandwidth in order to avoid latency
> excursions (oft measured, even in these higher speed 200+Mbit
> deployments, in the 100s of ms) -
> 
> At least some users want low delay always. It's just the theorists
> that want high utilization right at the edge of capacity. Users are
> forgiving about running out of cpu - disgruntled, but forgiving.
> 
> Certainly I'm back at the point of recommending tbf+fq_codel for
> inbound shaping at higher rates - and looking at restoring the high
> speed version of cake - and I keep thinking a better policer is
> feasible.
> 
> -- 
> 
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] an observation from the field

[David Collier-Brown]

On 2018-08-28 1:07 p.m., Dave Taht wrote:
> In looking over the increasingly vast sqm-related deployment, there's
> a persistent data point that pops up regarding inbound shaping at high
> rates.
>
> We give users a choice - run out of cpu at those rates or do inbound
> sqm at a rate their cpu can afford.  A remarkable percentage are
> willing to give up tons of bandwidth in order to avoid latency
> excursions (oft measured, even in these higher speed 200+Mbit
> deployments, in the 100s of ms) -

Humans experience delays directly, and so perceive systems with high 
latency as "slow". The proverbial "man on the Clapham omnibus" therefor 
responds to high-latency systems with disgust.

A trained scientist, however, runs the risk of choosing something that 
requires complicated measurement schemes, and might well choose to 
optimize for throughput, as that sounds like a desirable measure, one 
matching their intuitions of what "fast" means.

Alas, in this case the scientist's intuition is far poorer than the 
random person's direct experience.

> At least some users want low delay always. It's just the theorists
> that want high utilization right at the edge of capacity. Users are
> forgiving about running out of cpu - disgruntled, but forgiving.
>
> Certainly I'm back at the point of recommending tbf+fq_codel for
> inbound shaping at higher rates - and looking at restoring the high
> speed version of cake - and I keep thinking a better policer is
> feasible.
>
My advice to engineers? First, go for things you can both experience and 
measure, and only then things you have to measure.

--dave

-- 
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
davecb@spamcop.net           |                      -- Mark Twain

Re: [Bloat] an observation from the field

[Jonathan Morton]

> On 29 Aug, 2018, at 2:53 am, David Collier-Brown <davec-b@rogers.com> wrote:
> 
> Humans experience delays directly, and so perceive systems with high latency as "slow". The proverbial "man on the Clapham omnibus" therefor responds to high-latency systems with disgust.
> 
> A trained scientist, however, runs the risk of choosing something that requires complicated measurement schemes, and might well choose to optimize for throughput, as that sounds like a desirable measure, one matching their intuitions of what "fast" means.

The correct approach, for scientists, is to observe that for many applications, response time (a form of latency) is the *only* relevant metric.  In some cases, higher bandwidth correlates with reduced response time, such as for software updates.  In other cases, bandwidth is essentially irrelevant, except as it pertains to serialisation delay of single packets.

Conversely, there are some applications for which sufficient bandwidth is not a matter of response time, but a threshold prerequisite for correct operation.  We can refer to these as isochronous applications, or choose another term if you prefer.  Video streaming is an example of this; given an a-priori chosen video codec setting, if the data it produces cannot be transferred as fast as it is produced, the receiver will not be able to play it back in synchrony.

YouTube can reliably stream Full-HD (1080p60) video down a 10Mbps debloated pipe.  The broadband standard in the US claims that 25Mbps is necessary for this precise application.  Draw your own conclusions.

 - Jonathan Morton

Re: [Bloat] an observation from the field

[Jonas Mårtensson]

[-- Attachment #1: Type: text/plain, Size: 2352 bytes --]

Hi Jonathan,

On Wed, Aug 29, 2018 at 2:16 AM Jonathan Morton <chromatix99@gmail.com>
wrote:

> > On 29 Aug, 2018, at 2:53 am, David Collier-Brown <davec-b@rogers.com>
> wrote:
> >
> > Humans experience delays directly, and so perceive systems with high
> latency as "slow". The proverbial "man on the Clapham omnibus" therefor
> responds to high-latency systems with disgust.
> >
> > A trained scientist, however, runs the risk of choosing something that
> requires complicated measurement schemes, and might well choose to optimize
> for throughput, as that sounds like a desirable measure, one matching their
> intuitions of what "fast" means.
>
> The correct approach, for scientists, is to observe that for many
> applications, response time (a form of latency) is the *only* relevant
> metric.  In some cases, higher bandwidth correlates with reduced response
> time, such as for software updates.  In other cases, bandwidth is
> essentially irrelevant, except as it pertains to serialisation delay of
> single packets.
>

Yes, exactly, thank you for bringing some actual scientific reasoning into
the discussion. It would actually be nice to have a tool for measuring
"response time" for different applications


>
> Conversely, there are some applications for which sufficient bandwidth is
> not a matter of response time, but a threshold prerequisite for correct
> operation.  We can refer to these as isochronous applications, or choose
> another term if you prefer.  Video streaming is an example of this; given
> an a-priori chosen video codec setting, if the data it produces cannot be
> transferred as fast as it is produced, the receiver will not be able to
> play it back in synchrony.
>
> YouTube can reliably stream Full-HD (1080p60) video down a 10Mbps
> debloated pipe.  The broadband standard in the US claims that 25Mbps is
> necessary for this precise application.


No, it doesn't. It claims the opposite, i.e. that 10Mbps is sufficient for
streaming one HD video but with 25Mbps you can stream two HD videos or one
4K video, see Table 1 in the FCC report:

https://docs.fcc.gov/public/attachments/FCC-15-10A1.pdf

/Jonas


> Draw your own conclusions.
>
>  - Jonathan Morton
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>

[-- Attachment #2: Type: text/html, Size: 3417 bytes --]

Re: [Bloat] an observation from the field

[Dave Taht]

On Wed, Aug 29, 2018 at 1:20 AM Jonas Mårtensson
<martensson.jonas@gmail.com> wrote:
>
> Hi Jonathan,
>
> On Wed, Aug 29, 2018 at 2:16 AM Jonathan Morton <chromatix99@gmail.com> wrote:
>>
>> > On 29 Aug, 2018, at 2:53 am, David Collier-Brown <davec-b@rogers.com> wrote:
>> >
>> > Humans experience delays directly, and so perceive systems with high latency as "slow". The proverbial "man on the Clapham omnibus" therefor responds to high-latency systems with disgust.
>> >
>> > A trained scientist, however, runs the risk of choosing something that requires complicated measurement schemes, and might well choose to optimize for throughput, as that sounds like a desirable measure, one matching their intuitions of what "fast" means.
>>
>> The correct approach, for scientists, is to observe that for many applications, response time (a form of latency) is the *only* relevant metric.  In some cases, higher bandwidth correlates with reduced response time, such as for software updates.  In other cases, bandwidth is essentially irrelevant, except as it pertains to serialisation delay of single packets.
>
>
> Yes, exactly, thank you for bringing some actual scientific reasoning into the discussion. It would actually be nice to have a tool for measuring "response time" for different applications

We used to use the chrome web page benchmarker a lot, but it broke. In
flent, we use VOIP MOS scores.
We use the rrul test as a quick diagnostic of a dozen things that can
be wrong on a link. There's a PLT test as well but it takes work to
setup. We're discussing over here:
https://github.com/tohojo/flent/issues/148 ways to improve and revise
our existing tests, if you have any suggestions?

Scientist: This new compression algorithm can fit 10% more angels on
the head of a pin!
Engineer:  Great! Let's go get some angels and a couple pins and try
it out. Does it work on devils, too?

Jim's also always pointed to a lot of human factors research. I'm
always saying that the 20ms interval for voip is massively inferior to
old switched networks and we should at the very least be aiming for
2ms now. 20ms was an engineering compromise based on how much stress
users could handle...

We learned RTT is what dominates page load time above 40mbit several years back.

I'm an itinerant engineer that is really bugged by the lack of
rigorous experimentation and repeatable results that plague "science"
today. I read paper after paper and want some sort of web based red
rubber stamp to mark up all the dubious things I've had to read so
that perhaps others would poke holes in them, that there'd be some
forward and backward trail in time that could sort out the good ideas
from the bad.

I gave a talk once on this:

https://conferences.sigcomm.org/sigcomm/2014/doc/slides/137.pdf

Sigcomm's not invited me back.

Instead of the vigor of public debate, we get researchgate, a "safe
space", for science as usual.
I want my little red rubber stamps.

A scientist worries about what can go wrong every 2^32 times in an
algorithm, and uses saturating math. The engineer looks at the lost 6
ns * 2^32-1 and says "f**k, I can't live with that", and goes to ask
the scientist if the universe will explode if he makes that
optimization.

In starting this thread, I should have perhaps said: Of the .00001% of
humanity aware of bufferbloat, perhaps .000002% are completely willing
to sacrifice bandwidth for latency because they are unwilling or
unable to spend 300 bucks on a router.

A follow up experiment is: does this hold for the rest of humanity? At
what price point or level of inconvenience or other variable tips the
scales for 25% of humanity?

I'd love a psychology experiment:

What happens to people when locked up in a room with a deadline with
lousy internet? Measure stress hormone levels afterwards.

There's lots of anecdotal evidence about good and bad wifi out there...

The other day i sat in a coffeeshop next to a lady that took a
videocall at the table I was at. Seeing her nod, shake her head no or
yes, and emit all sorts of body language that is utterly impossible to
use on a phone call was *really fascinating*... good videoconferencing
totally changes the internet interaction, I gleaned she was working
for facebook...

And I couldn't help myself. I fired up a rrul test in the middle of
her con-call. It disrupted her conversation almost immediately, and to
watch her dismay, disorientation and frustration was painful to watch.
~25 seconds in, I had to abort the test. It took her, oh, 8 seconds to
regain her footing.

I introduced myself to her after the call, told her that her glitch
was probably bufferbloat, but didn't
tell her it was my fault.


>>
>>
>> Conversely, there are some applications for which sufficient bandwidth is not a matter of response time, but a threshold prerequisite for correct operation.  We can refer to these as isochronous applications, or choose another term if you prefer.  Video streaming is an example of this; given an a-priori chosen video codec setting, if the data it produces cannot be transferred as fast as it is produced, the receiver will not be able to play it back in synchrony.
>>
>> YouTube can reliably stream Full-HD (1080p60) video down a 10Mbps debloated pipe.  The broadband standard in the US claims that 25Mbps is necessary for this precise application.
>
>
> No, it doesn't. It claims the opposite, i.e. that 10Mbps is sufficient for streaming one HD video but with 25Mbps you can stream two HD videos or one 4K video, see Table 1 in the FCC report:
>
> https://docs.fcc.gov/public/attachments/FCC-15-10A1.pdf

Good to know.

> /Jonas
>
>>
>> Draw your own conclusions.
>>
>>  - Jonathan Morton
>>
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619