[Bloat] How do we shift the market?

[Bloat] How do we shift the market?

[Jim Gettys]

A) Today, "speed" has been conflated with "bandwidth", where we now know 
that that is naive and hurtful, and that latency under load is key for a 
large variety of applications.

Shifting this discussion from speed == bandwidth in the marketplace to 
something more nuanced and sensible is key; ultimately money talks.


B) Tests demonstrate the problems anyone can run.

There are several tests likely to come on line over the next months to 
help the situation:
	o The FCC SamKnows tests are putting in/have put in a latency under 
load test; I don't know when early results will appear.
	o at sometime later this year, the Ookla folks (e.g. speedtest.net) may 
add a test; they are in the middle of a major platform upgrade though, 
and the dust from that needs to settle before they can put effort there.

I'd really like to get really good easy demonstrations on-line and not 
wait/rely on others here.

Unless people can easily see if they are suffering, it remains a 
hypothetical.  The sooner they can reliably induce suffering on 
themselves, the sooner it becomes concrete.

Also note that both of the commercial tests above don't help network 
operators much (e.g. corporate networks), with diagnosing *where* the 
problems are.  That's why something like pingplotter is sooo important, 
we want heat going toward the right problems, not simply heat entering 
the system without pointing a finger where the problem is located!

C) direct market pressure:

The challenges I see include:
	a) ensuring that various manufacturers understand that their feet are 
going to be held to the fire on this metric so that maybe they start 
putting engineering resources into fixing their product. When they do, 
having working examples (e.g. OpenWRT, and 3g home gateway and others) 
that shows why it can be a competitive differentiator in the short run 
and will become and existential issue in the long run (you won't be able 
to sell bloated hardware any more).
	Certainly we need to hit up both the engineering press, but also 
mainstream business press so that the senior management of those now 
very large companies start to pay attention.
	b) make sure those who do testing and recommendation of products to 
consumers understand and start shining the light of day on the latency 
problems. So there's a lot of work to do to talk to and educate the 
likes of cnet, pcmag, smallnetbuilder, andatech, Tom's hardware etc.

It's not clear to me that this to reach out to the consumer kit 
reviewers should be started until we have at least existence proofs of 
properly working debloated kit in hand.  But as soon as we do, I'd love 
to put a properly working router into the hands of such people, and say: 
run this simple test on both this kit, and the other stuff you 
review.....  Bingo, the case gets made...

D) apropos of other discussions here: A lot of SLA's only talk about 
bandwidth, or packet loss (currently preferably zero with all that 
implies); education toward those who write those agreements that latency 
under load must be a metric in those agreements.

But if we don't shift the market discussion, bufferbloat won't get 
fixed, nor avoided in the future.
		- Jim

Re: [Bloat] How do we shift the market?

[richard]

On Sun, 2011-02-06 at 11:42 -0500, Jim Gettys wrote:
> A) Today, "speed" has been conflated with "bandwidth", where we now know 
> that that is naive and hurtful, and that latency under load is key for a 
> large variety of applications.
> 
> Shifting this discussion from speed == bandwidth in the marketplace to 
> something more nuanced and sensible is key; ultimately money talks.
> 
We should probably come up with a list of key words/phrases that such
tests and comments and complaints and such can be easily categorized
under - terms that can be used in a marketing sense.

Things like "multi-mode stress test" or "bandwidth-latency test"
Or how about a set of classifications of equipment based on what they
can deal with: 1-user throughput, family-capable throughput or???

How about "twitch latency" for the gamer market?

It's hard to talk cohesively about the problem if we don't all use the
same terms with the same implied (and defined) words. Getting at least
some of them nailed down now will make a difference in the long run.

I see wiki.bufferbloat.net has the "It Works" up on it - a page here on
terms would be a good thing.

> 
> B) Tests demonstrate the problems anyone can run.
> 
> There are several tests likely to come on line over the next months to 
> help the situation:
> 	o The FCC SamKnows tests are putting in/have put in a latency under 
> load test; I don't know when early results will appear.
> 	o at sometime later this year, the Ookla folks (e.g. speedtest.net) may 
> add a test; they are in the middle of a major platform upgrade though, 
> and the dust from that needs to settle before they can put effort there.
> 
> I'd really like to get really good easy demonstrations on-line and not 
> wait/rely on others here.
> 
> Unless people can easily see if they are suffering, it remains a 
> hypothetical.  The sooner they can reliably induce suffering on 
> themselves, the sooner it becomes concrete.
> 
> Also note that both of the commercial tests above don't help network 
> operators much (e.g. corporate networks), with diagnosing *where* the 
> problems are.  That's why something like pingplotter is sooo important, 
> we want heat going toward the right problems, not simply heat entering 
> the system without pointing a finger where the problem is located!
> 
> C) direct market pressure:
> 
> The challenges I see include:
> 	a) ensuring that various manufacturers understand that their feet are 
> going to be held to the fire on this metric so that maybe they start 
> putting engineering resources into fixing their product. When they do, 
> having working examples (e.g. OpenWRT, and 3g home gateway and others) 
> that shows why it can be a competitive differentiator in the short run 
> and will become and existential issue in the long run (you won't be able 
> to sell bloated hardware any more).

Getting the paying public to all start talking to the tech support
departments with the same concerns and same key words would help
tremendously.
"when are you going to fix my router's buffer bloat problem - I'm a
multi-use family customer and when my daughter uploads her camera phone
my son screams about the twitch latency?"

> 	Certainly we need to hit up both the engineering press, but also 
> mainstream business press so that the senior management of those now 
> very large companies start to pay attention.

Headline: ISPs demand new software from router manufacturers and
customers threaten class action suits over buffer bloat and poor twitch
latency. :)


> 	b) make sure those who do testing and recommendation of products to 
> consumers understand and start shining the light of day on the latency 
> problems. So there's a lot of work to do to talk to and educate the 
> likes of cnet, pcmag, smallnetbuilder, andatech, Tom's hardware etc.
> 
> It's not clear to me that this to reach out to the consumer kit 
> reviewers should be started until we have at least existence proofs of 
> properly working debloated kit in hand.  But as soon as we do, I'd love 
> to put a properly working router into the hands of such people, and say: 
> run this simple test on both this kit, and the other stuff you 
> review.....  Bingo, the case gets made...
> 
> D) apropos of other discussions here: A lot of SLA's only talk about 
> bandwidth, or packet loss (currently preferably zero with all that 
> implies); education toward those who write those agreements that latency 
> under load must be a metric in those agreements.
> 
> But if we don't shift the market discussion, bufferbloat won't get 
> fixed, nor avoided in the future.
> 		- Jim
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
-- 
Richard C. Pitt                 Pacific Data Capture
rcpitt@pacdat.net               604-644-9265
http://digital-rag.com          www.pacdat.net
PGP Fingerprint: FCEF 167D 151B 64C4 3333  57F0 4F18 AF98 9F59 DD73

Re: [Bloat] How do we shift the market?

[Eric Raymond]

richard <richard@pacdat.net>:
> We should probably come up with a list of key words/phrases that such
> tests and comments and complaints and such can be easily categorized
> under - terms that can be used in a marketing sense.
> 
> Things like "multi-mode stress test" or "bandwidth-latency test"
> Or how about a set of classifications of equipment based on what they
> can deal with: 1-user throughput, family-capable throughput or???
> 
> How about "twitch latency" for the gamer market?
> 
> It's hard to talk cohesively about the problem if we don't all use the
> same terms with the same implied (and defined) words. Getting at least
> some of them nailed down now will make a difference in the long run.
> 
> I see wiki.bufferbloat.net has the "It Works" up on it - a page here on
> terms would be a good thing.

I am *so* there! :-)

I'll start a glossary page.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: [Bloat] How do we shift the market?

[John W. Linville]

On Sun, Feb 06, 2011 at 04:03:17PM -0500, Eric Raymond wrote:
> richard <richard@pacdat.net>:
> > We should probably come up with a list of key words/phrases that such
> > tests and comments and complaints and such can be easily categorized
> > under - terms that can be used in a marketing sense.
> > 
> > Things like "multi-mode stress test" or "bandwidth-latency test"
> > Or how about a set of classifications of equipment based on what they
> > can deal with: 1-user throughput, family-capable throughput or???
> > 
> > How about "twitch latency" for the gamer market?
> > 
> > It's hard to talk cohesively about the problem if we don't all use the
> > same terms with the same implied (and defined) words. Getting at least
> > some of them nailed down now will make a difference in the long run.
> > 
> > I see wiki.bufferbloat.net has the "It Works" up on it - a page here on
> > terms would be a good thing.
> 
> I am *so* there! :-)
> 
> I'll start a glossary page.

(Not strictly direct at Eric...)

Is there any sort of standard metric for "latency under load"?
If not, should we define one?

What would be meaningful?  If you achieve a low latency at some
high percentage of bandwidth usage, does that always imply you can
expect similarly low latencies with lower bandwidth usage?  If not,
how should our "LUL" metric account for such variance?

Sorry if these are dumb questions -- remember, I'm an L2
knuckle-dragger... :-)

John
-- 
John W. Linville		Someday the world will need a hero, and you
linville@tuxdriver.com			might be all we have.  Be ready.

Re: [Bloat] How do we shift the market?

[Jim Gettys]

On 02/08/2011 11:09 AM, John W. Linville wrote:
> On Sun, Feb 06, 2011 at 04:03:17PM -0500, Eric Raymond wrote:
>> richard<richard@pacdat.net>:
>>> We should probably come up with a list of key words/phrases that such
>>> tests and comments and complaints and such can be easily categorized
>>> under - terms that can be used in a marketing sense.
>>>
>>> Things like "multi-mode stress test" or "bandwidth-latency test"
>>> Or how about a set of classifications of equipment based on what they
>>> can deal with: 1-user throughput, family-capable throughput or???
>>>
>>> How about "twitch latency" for the gamer market?
>>>
>>> It's hard to talk cohesively about the problem if we don't all use the
>>> same terms with the same implied (and defined) words. Getting at least
>>> some of them nailed down now will make a difference in the long run.
>>>
>>> I see wiki.bufferbloat.net has the "It Works" up on it - a page here on
>>> terms would be a good thing.
>>
>> I am *so* there! :-)
>>
>> I'll start a glossary page.
>
> (Not strictly direct at Eric...)
>
> Is there any sort of standard metric for "latency under load"?
> If not, should we define one?
>
> What would be meaningful?  If you achieve a low latency at some
> high percentage of bandwidth usage, does that always imply you can
> expect similarly low latencies with lower bandwidth usage?  If not,
> how should our "LUL" metric account for such variance?
>
> Sorry if these are dumb questions -- remember, I'm an L2
> knuckle-dragger... :-)
>

Hi John,

I don't think anyone here has tried to define a formal metric; mine has 
been just measure latency when a link is fully saturated; I've generally 
just used scp to saturate most links, for convenience, though I used 
nttcp when testing 100Mbps switches to ensure I had enough bandwidth.  I 
generally use ICMP ping myself, being of similarly low level frame of 
mind.  Given all the FUD around ICMP ping, however, I worked with 
Folkert VanHeusden to get persistent connections implemented in httping 
so that a TCP based ping would be available to confirm the results (so 
far, it has in my tests).

And indeed, jitter is as important as latency for many applications, as 
latency + some sort of metric for jitter is generally the minimum 
latency you can run most real time applications at.  Jitter is, of 
course, a statistical measurement.

But since the expected outcome for latency is depends (slightly) on the 
number of flows (e.g. how many packets might typically be ahead of you 
competing for a link), a more formal definition is in order.

Let me check around and see if we can get someone from the internet 
measurement community to get involved and give us something well 
understood for formal testing.
			- Jim

Re: [Bloat] How do we shift the market?

[Dave Täht]

Eric Raymond <esr@thyrsus.com> writes:

> richard <richard@pacdat.net>:
>> I see wiki.bufferbloat.net has the "It Works" up on it - a page here on
>> terms would be a good thing.

There *is* no wiki.bufferbloat.net site. 

Successful lookups for that are artifact of bufferbloat.net's current
DNS provider, name.com, which not only never serves up a NXDOMAIN
record, but supplies a *random*, *invalid* ipv6 address for it.

d@cruithne:$ host wiki.bufferbloat.net

  wiki.bufferbloat.net has address 149.20.54.81 

  wiki.bufferbloat.net has IPv6 address 1d00:6574:63:6f6d:6c:2e63:2800:0

d@cruithne:$ host a.random.site.bufferbloat.net

  a.random.site.bufferbloat.net has address 149.20.54.81
  a.random.site.bufferbloat.net has IPv6 address
  7262:6c6f:6174:36e:6574:2036:3034:3830

I first noticed this behavior a few weeks ago. Originally it was worse -
it would wildcard all invalid names to an ad-filled site.

The number of things that this behavior breaks is rather large: email
rbls, people that default to ipv6, the fact that I cannot use short
hostnames, and more than once a typo sent me to the wrong machine
entirely.

I've been meaning to take these issues up with name.com once I can do so
calmly, and rationally, with tact, and diplomacy.

For now, please stick with www.bufferbloat.net as your entry point to
the site!

>
> I am *so* there! :-)
>
> I'll start a glossary page.

There's one started. I don't know if redmine can do a media-wiki like
thing and let us mark articles as belonging to a given catagory? So I
setup the glossary page to just {{include(other pages)}} for now.

I would love to see all the terms we are using tightly defined, with
definitions on the wiki... Y'all 

In particular I would like to extend "Dark Buffers" to include the
concept of retries - or come up with another term for it.

-- 
Dave Taht
http://nex-6.taht.net

Re: [Bloat] How do we shift the market?

[Eric Raymond]

Jim Gettys <jg@freedesktop.org>:
> But if we don't shift the market discussion, bufferbloat won't get
> fixed, nor avoided in the future.

Quite right.  I followed through pretty much the same logic in
propagating open source out of the geek ghetto.  Morte emphasis on the
business and finance press in my case; I don't think that would be
as appropriate here.

We need to continue to think about this, but I believe there isn't any need 
to make any tactical commitments about which media outlets to pursue
until we've got more test tools and demonstrations.

jg, you sound like you've got your hands free doing live demos and
talks.  If you want to hand off any of the writing projects to me,
tell me.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>