Re: [Bloat] fixing bufferbloat in 2017

Re: [Bloat] fixing bufferbloat in 2017

[Rich Brown]

I feel particularly acutely the fact that we don't have a good "simple to deliver" solution today for the curious (not deeply commited) person. I was trying to help a friend with a TP-Link Archer C7, but was stymied because I can't simply install OpenWrt CC because of their "FCC fix".

Besides, OpenWrt CC is now eight months old: although it's stable, it probably doesn't have the latest bufferbloat/make-wifi-fast fixes. The DD development and release process is uncertain. DD-Wrt is of unknown quality (does it include fq_codel? make-wifi-fast? will it ever?), with a really ugly and not terribly-welcoming forum. I have great hopes for LEDE, but it is still sprinting toward its first stable release, so we can't count on it today.

So we are stuck in the short term with a "lack of a product". I have faith that it will happen, but we need to build our plan around its arrival date.

Rich

Re: [Bloat] fixing bufferbloat in 2017

[David Lang]

On Wed, 23 Nov 2016, Rich Brown wrote:

> I feel particularly acutely the fact that we don't have a good "simple to 
> deliver" solution today for the curious (not deeply commited) person. I was 
> trying to help a friend with a TP-Link Archer C7, but was stymied because I 
> can't simply install OpenWrt CC because of their "FCC fix".

The FCC has issued an order that TP-Link open their devices to being flashed by 
third pary firmware. We just need to wait for this to trickle back down to the 
supply chain. It may be that they have a new firmware that you can update to 
that will then let you install OpenWRT

> Besides, OpenWrt CC is now eight months old: although it's stable, it probably 
> doesn't have the latest bufferbloat/make-wifi-fast fixes. The DD development 
> and release process is uncertain. DD-Wrt is of unknown quality (does it 
> include fq_codel? make-wifi-fast? will it ever?), with a really ugly and not 
> terribly-welcoming forum. I have great hopes for LEDE, but it is still 
> sprinting toward its first stable release, so we can't count on it today.

CC doesn't have the latest, but it does have fq_codel, so it's a huge step up 
from the factory firmware.

Most of the OpenWRT developers left to fork the project and now have the LEDE 
project. Most of the development and testing of new stuff is happening there.

Neither project has made a release since the split, but LEDE is working on 
setting up the infrastructure to do a release.

In the meantime, you can install the nightly builds or compile your own.


> So we are stuck in the short term with a "lack of a product". I have faith 
> that it will happen, but we need to build our plan around its arrival date.

Part of the question boils down to what you define as the "product" that you are 
waiting for.

OpenWRT/LEDE are very unlikly to have rapid releases, if they do one a year they 
will be doing much better than they have in the past.

Do you need a device that ships with the fixes in it from the factory?

Do you need a stable release of OpenWRT/LEDE?

Do you need the fixes in the master git repository for OpenWRT/LEDE?

Do you need the latest and greatest fixes?

the last one in incompatible with the earlier ones :-)

David Lang

Re: [Bloat] fixing bufferbloat in 2017

[Jonathan Morton]

> On 23 Nov, 2016, at 20:46, David Lang <david@lang.hm> wrote:
> 
> Do you need a device that ships with the fixes in it from the factory?

I think this is what we should aim for.  That would greatly simplify installation for Joe Average, and it could serve as an anchor point in the wider industry.

Ideally, it would have official mutual support with LEDE or OpenWRT, so that updates to the latter automatically propagate.  Even if it takes a month between writing a patch and being sure it’s reached end-users, that’s still a major improvement over the status quo.

It’s much harder to rely on firmware replacement on existing devices, because the latter can change hardware on the same model number without notice, or simply discontinue manufacture in favour of a new model.  That makes a minefield for networking novices to wade through.  So it’s fine for experimentation among ourselves, and for a certain type of early-adopter, but not for the average gamer.

 - Jonathan Morton

Re: [Bloat] fixing bufferbloat in 2017

[David Lang]

[-- Attachment #1: Type: text/plain, Size: 1212 bytes --]

On Wed, 23 Nov 2016, Jonathan Morton wrote:

>> On 23 Nov, 2016, at 20:46, David Lang <david@lang.hm> wrote:
>>
>> Do you need a device that ships with the fixes in it from the factory?
>
> I think this is what we should aim for.  That would greatly simplify installation for Joe Average, and it could serve as an anchor point in the wider industry.

I don't disagee that we should be aiming for this in the long run. But acting as 
if we have no solution at all until this is achieved is wrong.

David Lang

> Ideally, it would have official mutual support with LEDE or OpenWRT, so that updates to the latter automatically propagate.  Even if it takes a month between writing a patch and being sure it’s reached end-users, that’s still a major improvement over the status quo.
>
> It’s much harder to rely on firmware replacement on existing devices, because the latter can change hardware on the same model number without notice, or simply discontinue manufacture in favour of a new model.  That makes a minefield for networking novices to wade through.  So it’s fine for experimentation among ourselves, and for a certain type of early-adopter, but not for the average gamer.
>
> - Jonathan Morton
>
>

Re: [Bloat] fixing bufferbloat in 2017

[Toke Høiland-Jørgensen]

Jonathan Morton <chromatix99@gmail.com> writes:

>> On 23 Nov, 2016, at 20:46, David Lang <david@lang.hm> wrote:
>> 
>> Do you need a device that ships with the fixes in it from the factory?
>
> I think this is what we should aim for. That would greatly simplify
> installation for Joe Average, and it could serve as an anchor point in
> the wider industry.
>
> Ideally, it would have official mutual support with LEDE or OpenWRT,
> so that updates to the latter automatically propagate. Even if it
> takes a month between writing a patch and being sure it’s reached
> end-users, that’s still a major improvement over the status quo.

Well, this does actually exist now, in the Turris Omnia. The problem is
the price tag, though... Who is going to pay €300 for a router?

-Toke

Re: [Bloat] Fixing bufferbloat in 2017

[Rich Brown]

Dave Täht attempts to refocus the group, and asks: 

> Can I encourage folk to think big and out of the technical box?
> 
> On Tue, Nov 22, 2016 at 7:32 AM, Dave Taht <dave.taht@gmail.com> wrote:
>> What's left to do?
>> 
>> What else can we do?
>> 
>> What should we stop doing?
>> 
>> What can we do better?

Lots of good thoughts on this thread. 

My impression is that we have reached a strong technical point. We have solved some really hard, really significant problems. We are in a position to Declare Victory on a large part of the problem, even though there are loads of details to clean up.

Most of the suggestions in this thread deal with Getting the Word Out. That's good - that's the declaring victory part. The bad news is that this is not our collective skill set. 

Some thoughts about what we *can* do:

1) Toke et al published (are publishing?) a scholarly paper on the make-wifi-fast efforts that "looks like real academic research" (by *actually being* academic research :-) This makes it credible to other academicians, and throws down the gauntlet with a low latency value that others need to improve upon. (No more academic papers that say, "We really worked hard, and got latency down to 100 ms. Aren't you proud of us?")

Are there other papers bottled up inside team members?

2) I wonder if we would gain credibility by updating the bufferbloat web site. I see two things that could be done.
	a) Change the www.bufferbloat.net home page to use a one-page design (see, just as an example, https://bootstrapmade.com/demo/Baker/) with sections that address our primary constituencies: Home users, Gamers, Manufacturers, Software Developers, and Network Researchers. It adds a bit of polish, while keeping our message simple. People can drill down into the (existing) pages for more information. 
	b) We should make a pass through the site, organizing according those constituencies, and removing content that is no longer relevant.
	c) I also grabbed the DNS name "makewififast.com" in case we want to use it.

3) I think it's great to contact reviewers - ArsTechnica and AnandTech were mentioned. (I did reach out to Wirecutter and ask that they incorporate bufferbloat tests in their router recommendations. I was disappointed by the total radio silence.)

4) Do we know people at any of the cell phone companies, or router vendors on whom we could try one last push?

As part of organizing my thoughts for this note, I also collected the following ideas from this thread. I add my $0.02 below.

Rich

1) I don't see that Ookla has much incentive to include bufferbloat measurements in their test, since they private-label it for lots of ISPs who (presumably) wouldn't want their CPE to be proven crappy. ("It is difficult to get a man to understand something, when his salary depends upon his not understanding it!" -Upton Sinclair)

2) The gamer community seems like such a perfect target for these improvements. But I fear that the thought leaders are so wrapped up in the fame generated by their own clever QoS tricks that they can't believe that fq_codel plus the make-wifi-fast fixes could possibly address such a complicated subject. (Upton Sinclair, again.)

3) On the other hand, Comcast (whose DOCSIS modems *might* someday support PIE or other SQM) is in a position to benefit from an increased awareness of the phenomenon, leaving a little ray of hope.

[Note - I wrote 4 & 5 below before I learned of IQrouter... I'm still skeptical of the mainline router vendors adopting this technology anytime soon into their stock firmware.] 

4) I do wish that there were a way to we could stop saying, "Just update your router firmware (trust us...)"  as a solution. It would be so much better to say, "Just buy this low-cost (or medium-cost) router that'll make you supremely happy."

5) But I'm not hopeful that any of the COTS router vendors are going to adopt these techniques, simply because they've been impervious to our earlier entreaties. That doesn't mean we shouldn't try again - it'd be a helluva competitive advantage to incorporate the 25-50 man years of intense software development that has gone into this work.

6) It *is* a good idea to think about attracting the attention of vendors who are hurt by bufferbloat - VoIP, video streaming folks, gaming companies, etc. But it feels like the wrong end of the lever - a gaming company can't fix crappy CPE, and they're stuck saying 

7) Cell phones are another place that obviously would benefit, although, again, it's hard to break through the notion that "It's always been like that..."

What else?

Rich

Re: [Bloat] Fixing bufferbloat in 2017

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 1192 bytes --]

> 5) But I'm not hopeful that any of the COTS router vendors are going to
> adopt these techniques, simply because they've been impervious to our
> earlier entreaties. That doesn't mean we shouldn't try again - it'd be a
> helluva competitive advantage to incorporate the 25-50 man years of intense
> software development that has gone into this work.
>

(painting with a very large brush):

But they aren't in the best position, really, to do this.  Mostly their
ODMs get the BSP Linux from the chipset vendor to work, slap a UI on it,
and call it a day.  And those BSPs are _ancient_.  I wouldn't be surprised
to see 2.6 still coming out on new models, let alone 4.0.

To get the COTS devices to have this, we need to be able to back-port
things far enough back that we can talk QCA, Broadcom, Marvell, RTLink,
etc. to incorporate them into the BSP.  But in the Sinclair vain, they like
their proprietary things (like the hardware offload engines that ignore the
linux stack as much as possible).

So, I think we need to talk to the _right_ people at the chipset vendors to
get traction on this.  Smaller COTS vendors might be swayable (part of
their value over the big companies).

-Aaron

[-- Attachment #2: Type: text/html, Size: 1542 bytes --]

Re: [Bloat] Fixing bufferbloat in 2017

[Mikael Abrahamsson]

On Sat, 26 Nov 2016, Aaron Wood wrote:

> and call it a day.  And those BSPs are _ancient_.  I wouldn't be 
> surprised to see 2.6 still coming out on new models, let alone 4.0.

Most seem to be on 3.2 and 3.4, but I've heard people say Broadcom now has 
BSP for 4.1.

However, since basically all high-speed devices use a hardware packet 
accelerator, even with newer kernels you might not get anti-bufferbloat 
benefit because these packet accelerators have their own buffer handling.

I might be in the position to test one of these broadcom 4.1 based devices 
in the next few months, I'll run some tests and report back if that 
happens.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] Fixing bufferbloat in 2017

[Dave Taht]

There *are* 430+ other minds on this mailing list, and probably a few AIs.

Sometimes I worry that most of our postings go into spamboxes now, or
that we've somehow completely burned people out since our heyday in
2012.

knock, knock - is this mic on?

On Sat, Nov 26, 2016 at 7:33 AM, Rich Brown <richb.hanover@gmail.com> wrote:
> Dave Täht attempts to refocus the group, and asks:
>
>> Can I encourage folk to think big and out of the technical box?
>>
>> On Tue, Nov 22, 2016 at 7:32 AM, Dave Taht <dave.taht@gmail.com> wrote:
>>> What's left to do?
>>>
>>> What else can we do?
>>>
>>> What should we stop doing?
>>>
>>> What can we do better?
>
> Lots of good thoughts on this thread.
>
> My impression is that we have reached a strong technical point. We have solved some really hard, really significant problems. We are in a position to Declare Victory on a large part of the problem, even though there are loads of details to clean up.
>
> Most of the suggestions in this thread deal with Getting the Word Out. That's good - that's the declaring victory part. The bad news is that this is not our collective skill set.
>
> Some thoughts about what we *can* do:
>
> 1) Toke et al published (are publishing?) a scholarly paper on the make-wifi-fast efforts that "looks like real academic research" (by *actually being* academic research :-) This makes it credible to other academicians, and throws down the gauntlet with a low latency value that others need to improve upon. (No more academic papers that say, "We really worked hard, and got latency down to 100 ms. Aren't you proud of us?")
>
> Are there other papers bottled up inside team members?
>
> 2) I wonder if we would gain credibility by updating the bufferbloat web site. I see two things that could be done.
>         a) Change the www.bufferbloat.net home page to use a one-page design (see, just as an example, https://bootstrapmade.com/demo/Baker/) with sections that address our primary constituencies: Home users, Gamers, Manufacturers, Software Developers, and Network Researchers. It adds a bit of polish, while keeping our message simple. People can drill down into the (existing) pages for more information.
>         b) We should make a pass through the site, organizing according those constituencies, and removing content that is no longer relevant.
>         c) I also grabbed the DNS name "makewififast.com" in case we want to use it.
>
> 3) I think it's great to contact reviewers - ArsTechnica and AnandTech were mentioned. (I did reach out to Wirecutter and ask that they incorporate bufferbloat tests in their router recommendations. I was disappointed by the total radio silence.)
>
> 4) Do we know people at any of the cell phone companies, or router vendors on whom we could try one last push?
>
> As part of organizing my thoughts for this note, I also collected the following ideas from this thread. I add my $0.02 below.
>
> Rich
>
> 1) I don't see that Ookla has much incentive to include bufferbloat measurements in their test, since they private-label it for lots of ISPs who (presumably) wouldn't want their CPE to be proven crappy. ("It is difficult to get a man to understand something, when his salary depends upon his not understanding it!" -Upton Sinclair)
>
> 2) The gamer community seems like such a perfect target for these improvements. But I fear that the thought leaders are so wrapped up in the fame generated by their own clever QoS tricks that they can't believe that fq_codel plus the make-wifi-fast fixes could possibly address such a complicated subject. (Upton Sinclair, again.)
>
> 3) On the other hand, Comcast (whose DOCSIS modems *might* someday support PIE or other SQM) is in a position to benefit from an increased awareness of the phenomenon, leaving a little ray of hope.
>
> [Note - I wrote 4 & 5 below before I learned of IQrouter... I'm still skeptical of the mainline router vendors adopting this technology anytime soon into their stock firmware.]
>
> 4) I do wish that there were a way to we could stop saying, "Just update your router firmware (trust us...)"  as a solution. It would be so much better to say, "Just buy this low-cost (or medium-cost) router that'll make you supremely happy."
>
> 5) But I'm not hopeful that any of the COTS router vendors are going to adopt these techniques, simply because they've been impervious to our earlier entreaties. That doesn't mean we shouldn't try again - it'd be a helluva competitive advantage to incorporate the 25-50 man years of intense software development that has gone into this work.
>
> 6) It *is* a good idea to think about attracting the attention of vendors who are hurt by bufferbloat - VoIP, video streaming folks, gaming companies, etc. But it feels like the wrong end of the lever - a gaming company can't fix crappy CPE, and they're stuck saying
>
> 7) Cell phones are another place that obviously would benefit, although, again, it's hard to break through the notion that "It's always been like that..."
>
> What else?
>
> Rich
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Bloat] Fixing bufferbloat in 2017

[Kathleen Nichols]

I never have any problem hearing you, Dave.

Random stuff in-line.

On 11/27/16 1:24 PM, Dave Taht wrote:
> There *are* 430+ other minds on this mailing list, and probably a few
> AIs.
> 
> Sometimes I worry that most of our postings go into spamboxes now,
> or that we've somehow completely burned people out since our heyday
> in 2012.
> 
> knock, knock - is this mic on?
> 
> On Sat, Nov 26, 2016 at 7:33 AM, Rich Brown <richb.hanover@gmail.com>
> wrote:
...
>> 
>> My impression is that we have reached a strong technical point. We
>> have solved some really hard, really significant problems. We are
>> in a position to Declare Victory on a large part of the problem,
>> even though there are loads of details to clean up.

I think this is important. Some really good work has been done by a lot of
people on this list and I have found it interesting, enlightening and
gratifying to
put some small bit of a solution out there and have people grab it,
improve it,
add to it and make it real. So I think people doing that work should pat
themselves on the
back.
>> 
>> Most of the suggestions in this thread deal with Getting the Word
>> Out. That's good - that's the declaring victory part. The bad news
>> is that this is not our collective skill set.

So that's the hard part. Who do you need to Get the Word Out to and what
do you expect them to do? It sounds like there are some edge router
improvements coming. It's possible that some companies are using the
work but they are advertising what it does for the customer not where it
came from or what it is technically. So that might be a victory.
>> 
...
>> 4) Do we know people at any of the cell phone companies, or router
>> vendors on whom we could try one last push?
>> 
>> As part of organizing my thoughts for this note, I also collected
>> the following ideas from this thread. I add my $0.02 below.

Well, getting cellular networks on board would be a coup.
>> 
>> Rich
>> 
>> 1) I don't see that Ookla has much incentive to include bufferbloat
>> measurements in their test, since they private-label it for lots of
>> ISPs who (presumably) wouldn't want their CPE to be proven crappy.
>> ("It is difficult to get a man to understand something, when his
>> salary depends upon his not understanding it!" -Upton Sinclair)

This is, sadly, likely correct.
>> 
>> 2) The gamer community seems like such a perfect target for these
>> improvements. But I fear that the thought leaders are so wrapped up
>> in the fame generated by their own clever QoS tricks that they
>> can't believe that fq_codel plus the make-wifi-fast fixes could
>> possibly address such a complicated subject. (Upton Sinclair,
>> again.)

But where do you find who benefits and who can have an effect? I don't
know anything about these traffic patterns but would be interested in
seeing them if possible.
>> 
>> 3) On the other hand, Comcast (whose DOCSIS modems *might* someday
>> support PIE or other SQM) is in a position to benefit from an
>> increased awareness of the phenomenon, leaving a little ray of
>> hope.

I don't know. I think it has to be a more serious goal at Comcast. The
bufferbloat measurement devices they sent out were electrically
problematic, taking our signal down and reducing bandwidth.  This seems
like one step above skunkworks.
...
>> 6) It *is* a good idea to think about attracting the attention of
>> vendors who are hurt by bufferbloat - VoIP, video streaming folks,
>> gaming companies, etc. But it feels like the wrong end of the lever
>> - a gaming company can't fix crappy CPE, and they're stuck saying

Yes, it's hard for the victims unless there is an alternative or they
wield a large amount of coordinated monetary power. The video streaming
folks, from my measurements, are trashing themselves. Why are they
creating such huge bursts? Why send out bursts that are going to arrive
at the same time? This isn't a bufferbloat problem really, it's a clue
problem.
>> 
>> 7) Cell phones are another place that obviously would benefit,
>> although, again, it's hard to break through the notion that "It's
>> always been like that..."

Yes, but who would benefit? Is it a content company that could put
pressure on some carrier?
>> 
>> What else?

This is good thinking, Rich, but the business side of the current
"ecosystem" seems disincentivized to progress.
>> 
>> Rich
>> 
>> _______________________________________________ Bloat mailing list 
>> Bloat@lists.bufferbloat.net 
>> https://lists.bufferbloat.net/listinfo/bloat
> 
> 
> 

Re: [Bloat] Fixing bufferbloat in 2017

[Jim Gettys]

[-- Attachment #1: Type: text/plain, Size: 5414 bytes --]

This went by in a previous posting I made:

Ookla may have made themselves long term irrelevant by their recent
behavior.  When your customers start funding development of a replacement
(as Comcast has), you know they aren't happy.

So I don't sweat Ookla: helping out the Comcast test effort is probably the
best way to get bufferbloat in front of everyone, and best yet, the code
for the tests is out there.

                                                     - Jim


On Sun, Nov 27, 2016 at 9:11 PM, Kathleen Nichols <nichols@pollere.com>
wrote:

>
> I never have any problem hearing you, Dave.
>
> Random stuff in-line.
>
> On 11/27/16 1:24 PM, Dave Taht wrote:
> > There *are* 430+ other minds on this mailing list, and probably a few
> > AIs.
> >
> > Sometimes I worry that most of our postings go into spamboxes now,
> > or that we've somehow completely burned people out since our heyday
> > in 2012.
> >
> > knock, knock - is this mic on?
> >
> > On Sat, Nov 26, 2016 at 7:33 AM, Rich Brown <richb.hanover@gmail.com>
> > wrote:
> ...
> >>
> >> My impression is that we have reached a strong technical point. We
> >> have solved some really hard, really significant problems. We are
> >> in a position to Declare Victory on a large part of the problem,
> >> even though there are loads of details to clean up.
>
> I think this is important. Some really good work has been done by a lot of
> people on this list and I have found it interesting, enlightening and
> gratifying to
> put some small bit of a solution out there and have people grab it,
> improve it,
> add to it and make it real. So I think people doing that work should pat
> themselves on the
> back.
> >>
> >> Most of the suggestions in this thread deal with Getting the Word
> >> Out. That's good - that's the declaring victory part. The bad news
> >> is that this is not our collective skill set.
>
> So that's the hard part. Who do you need to Get the Word Out to and what
> do you expect them to do? It sounds like there are some edge router
> improvements coming. It's possible that some companies are using the
> work but they are advertising what it does for the customer not where it
> came from or what it is technically. So that might be a victory.
> >>
> ...
> >> 4) Do we know people at any of the cell phone companies, or router
> >> vendors on whom we could try one last push?
> >>
> >> As part of organizing my thoughts for this note, I also collected
> >> the following ideas from this thread. I add my $0.02 below.
>
> Well, getting cellular networks on board would be a coup.
> >>
> >> Rich
> >>
> >> 1) I don't see that Ookla has much incentive to include bufferbloat
> >> measurements in their test, since they private-label it for lots of
> >> ISPs who (presumably) wouldn't want their CPE to be proven crappy.
> >> ("It is difficult to get a man to understand something, when his
> >> salary depends upon his not understanding it!" -Upton Sinclair)
>
> This is, sadly, likely correct.
> >>
> >> 2) The gamer community seems like such a perfect target for these
> >> improvements. But I fear that the thought leaders are so wrapped up
> >> in the fame generated by their own clever QoS tricks that they
> >> can't believe that fq_codel plus the make-wifi-fast fixes could
> >> possibly address such a complicated subject. (Upton Sinclair,
> >> again.)
>
> But where do you find who benefits and who can have an effect? I don't
> know anything about these traffic patterns but would be interested in
> seeing them if possible.
> >>
> >> 3) On the other hand, Comcast (whose DOCSIS modems *might* someday
> >> support PIE or other SQM) is in a position to benefit from an
> >> increased awareness of the phenomenon, leaving a little ray of
> >> hope.
>
> I don't know. I think it has to be a more serious goal at Comcast. The
> bufferbloat measurement devices they sent out were electrically
> problematic, taking our signal down and reducing bandwidth.  This seems
> like one step above skunkworks.
> ...
> >> 6) It *is* a good idea to think about attracting the attention of
> >> vendors who are hurt by bufferbloat - VoIP, video streaming folks,
> >> gaming companies, etc. But it feels like the wrong end of the lever
> >> - a gaming company can't fix crappy CPE, and they're stuck saying
>
> Yes, it's hard for the victims unless there is an alternative or they
> wield a large amount of coordinated monetary power. The video streaming
> folks, from my measurements, are trashing themselves. Why are they
> creating such huge bursts? Why send out bursts that are going to arrive
> at the same time? This isn't a bufferbloat problem really, it's a clue
> problem.
> >>
> >> 7) Cell phones are another place that obviously would benefit,
> >> although, again, it's hard to break through the notion that "It's
> >> always been like that..."
>
> Yes, but who would benefit? Is it a content company that could put
> pressure on some carrier?
> >>
> >> What else?
>
> This is good thinking, Rich, but the business side of the current
> "ecosystem" seems disincentivized to progress.
> >>
> >> Rich
> >>
> >> _______________________________________________ Bloat mailing list
> >> Bloat@lists.bufferbloat.net
> >> https://lists.bufferbloat.net/listinfo/bloat
> >
> >
> >
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>

[-- Attachment #2: Type: text/html, Size: 7626 bytes --]

Re: [Bloat] Fixing bufferbloat in 2017

[Jan Ceuleers]

On 28/11/16 03:16, Jim Gettys wrote:
> Ookla may have made themselves long term irrelevant by their recent
> behavior.  When your customers start funding development of a
> replacement (as Comcast has), you know they aren't happy.
> 
> So I don't sweat Ookla: helping out the Comcast test effort is probably
> the best way to get bufferbloat in front of everyone, and best yet, the
> code for the tests is out there.

I do hope you're right Jim, but I still worry that Ookla is heavily
entrenched in carriers' test labs. This position has, I believe, come
about not because of Ookla's expertise in network testing but rather
because of market pull (i.e. speedtest.net's huge popularity with
end-users).

As long as both of these positions remain (i.e. Ookla's mind share of
end-users and their resulting market share in the labs of large
purchasers of CPE) their lack of interest in bufferbloat is going to
keep this topic off the agenda in a large part of the industry.

Unless Ookla can be coerced somehow.

I have previously suggested standardising network throughput testing
methods and "grading" criteria. If there's an RFC on this subject
carriers are going to be interested in conformance to it and will
pressure their suppliers (of network testing gear, of CPE etc).

Re: [Bloat] Fixing bufferbloat in 2017

[David Collier-Brown]

A short RFC with a clear summary would change the ground on which we stand.
Include me in if you're planning one.

--dave

On 28/11/16 01:00 AM, Jan Ceuleers wrote:
> On 28/11/16 03:16, Jim Gettys wrote:
>> Ookla may have made themselves long term irrelevant by their recent
>> behavior.  When your customers start funding development of a
>> replacement (as Comcast has), you know they aren't happy.
>>
>> So I don't sweat Ookla: helping out the Comcast test effort is probably
>> the best way to get bufferbloat in front of everyone, and best yet, the
>> code for the tests is out there.
> I do hope you're right Jim, but I still worry that Ookla is heavily
> entrenched in carriers' test labs. This position has, I believe, come
> about not because of Ookla's expertise in network testing but rather
> because of market pull (i.e. speedtest.net's huge popularity with
> end-users).
>
> As long as both of these positions remain (i.e. Ookla's mind share of
> end-users and their resulting market share in the labs of large
> purchasers of CPE) their lack of interest in bufferbloat is going to
> keep this topic off the agenda in a large part of the industry.
>
> Unless Ookla can be coerced somehow.
>
> I have previously suggested standardising network throughput testing
> methods and "grading" criteria. If there's an RFC on this subject
> carriers are going to be interested in conformance to it and will
> pressure their suppliers (of network testing gear, of CPE etc).
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat


-- 
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
davecb@spamcop.net           |                      -- Mark Twain

Re: [Bloat] Fixing bufferbloat in 2017

[Dave Taht]

On Mon, Nov 28, 2016 at 4:48 AM, David Collier-Brown <davec-b@rogers.com> wrote:
> A short RFC with a clear summary would change the ground on which we stand.
> Include me in if you're planning one.

Call me grumpy. Call me disaffected. But it's been 4 years into the
IETF RFC process with codel and fq_codel with still no end in sight.

I presently have no hope that a "short RFC" can enter and exit the
IETF in a reasonable amount of time, without being watered down into
unreadability.


>
> --dave
>
>
> On 28/11/16 01:00 AM, Jan Ceuleers wrote:
>>
>> On 28/11/16 03:16, Jim Gettys wrote:
>>>
>>> Ookla may have made themselves long term irrelevant by their recent
>>> behavior.  When your customers start funding development of a
>>> replacement (as Comcast has), you know they aren't happy.
>>>
>>> So I don't sweat Ookla: helping out the Comcast test effort is probably
>>> the best way to get bufferbloat in front of everyone, and best yet, the
>>> code for the tests is out there.
>>
>> I do hope you're right Jim, but I still worry that Ookla is heavily
>> entrenched in carriers' test labs. This position has, I believe, come
>> about not because of Ookla's expertise in network testing but rather
>> because of market pull (i.e. speedtest.net's huge popularity with
>> end-users).
>>
>> As long as both of these positions remain (i.e. Ookla's mind share of
>> end-users and their resulting market share in the labs of large
>> purchasers of CPE) their lack of interest in bufferbloat is going to
>> keep this topic off the agenda in a large part of the industry.
>>
>> Unless Ookla can be coerced somehow.
>>
>> I have previously suggested standardising network throughput testing
>> methods and "grading" criteria. If there's an RFC on this subject
>> carriers are going to be interested in conformance to it and will
>> pressure their suppliers (of network testing gear, of CPE etc).
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>
>
>
> --
> David Collier-Brown,         | Always do right. This will gratify
> System Programmer and Author | some people and astonish the rest
> davecb@spamcop.net           |                      -- Mark Twain
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Bloat] Fixing bufferbloat in 2017

[Wesley Eddy]

On 11/28/2016 10:12 AM, Dave Taht wrote:
> On Mon, Nov 28, 2016 at 4:48 AM, David Collier-Brown <davec-b@rogers.com> wrote:
>> A short RFC with a clear summary would change the ground on which we stand.
>> Include me in if you're planning one.
> Call me grumpy. Call me disaffected. But it's been 4 years into the
> IETF RFC process with codel and fq_codel with still no end in sight.
>


Hi Dave, while it's been undeniably slow, "no end in sight" is not 
really accurate.  Here is the correct status, since I am document 
shepherd for both:

- The fq-codel draft is completely and totally done in terms of IETF 
process, and has been in the RFC Editor's queue simply awaiting the 
codel draft to arrive.  This is what the "MISSREF" state indicates in 
that IETF datatracker tool.  It completed the IETF last call and IESG 
balloting in March/April earlier this year.

- The codel document completed AQM working group last call, and I 
believe is awaiting the authors to make changes requested by the Area 
Director in order to go for IETF Last Call and IESG balloting.

The end is most definitely within sight!

Re: [Bloat] Fixing bufferbloat in 2017

[Dave Taht]

On Mon, Nov 28, 2016 at 7:23 AM, Wesley Eddy <wes@mti-systems.com> wrote:
> On 11/28/2016 10:12 AM, Dave Taht wrote:
>>
>> On Mon, Nov 28, 2016 at 4:48 AM, David Collier-Brown <davec-b@rogers.com>
>> wrote:
>>>
>>> A short RFC with a clear summary would change the ground on which we
>>> stand.
>>> Include me in if you're planning one.
>>
>> Call me grumpy. Call me disaffected. But it's been 4 years into the
>> IETF RFC process with codel and fq_codel with still no end in sight.
>>
>
>
> Hi Dave, while it's been undeniably slow, "no end in sight" is not really
> accurate.  Here is the correct status, since I am document shepherd for
> both:
>
> - The fq-codel draft is completely and totally done in terms of IETF
> process, and has been in the RFC Editor's queue simply awaiting the codel
> draft to arrive.  This is what the "MISSREF" state indicates in that IETF
> datatracker tool.  It completed the IETF last call and IESG balloting in
> March/April earlier this year.
>
> - The codel document completed AQM working group last call, and I believe is
> awaiting the authors to make changes requested by the Area Director in order
> to go for IETF Last Call and IESG balloting.
>
> The end is most definitely within sight!

Thank you for the update! I will, however, believe it when I see it
(and heave a great sigh of relief).

I see from looking over this preliminary draft,

http://snapon.lab.bufferbloat.net/~d/draft-taht-home-gateway-best-practices-00.html

that since I wrote it, we have made a serious dent in dealing with nat
and in per host fq with the cake project, as well as dealing well with
rate shaping (and diffserv classification to the best of our
understandings)

current man page for cake: http://static.lochnair.net/bufferbloat/tc-cake.8.html
Some tech detail (does not include the new de-natting code or per host
fq (triple-isolate)):
https://www.bufferbloat.net/projects/codel/wiki/CakeTechnical/

There is *no way* I want to submit cake to the RFC process (the code
is dual licensed), but an updated form of this attempt at a  best
practices document might be worthwhile, if not as a wg submission,
then as an individual submission.


>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Bloat] Fixing bufferbloat in 2017

[Stephen Hemminger]

My experience has been that the media and developer attention span is short
lived, and maybe that is part of the problem. Gaming is a niche market, and therefore is easily ignored; plus the classic gaming market is dying and I am not sure anyone is really investing in it. 

The current hot topic use case seems to be machine learning and voice interaction. I wonder if we could build a use case something related to that? "Ok Google, my house is on fire!!" -- "Sorry can not call fire department, network is congested".

Re: [Bloat] Fixing bufferbloat in 2017

[Dave Taht]

On Mon, Nov 28, 2016 at 8:58 AM, Stephen Hemminger
<stephen@networkplumber.org> wrote:
> My experience has been that the media and developer attention span is short
> lived, and maybe that is part of the problem.

Well, in portions of the market, the only way to get attention is to
buy it, with press releases. I certainly plan a press release for
whenever the wifi airtime fairness paper is published, and perhaps
doing one sooner than that is warranted. I hope that wifi devices now
being sold from some of the newer players all incorporate it and ship
better products in 2017.

>Gaming is a niche market, and therefore is easily ignored; plus the classic gaming market is dying and I am not sure anyone is really investing in it.

But a big one, and it doesn't matter if you use a dedicated device or
not to game with. I think doing a "fixing your lag" talk at an E3 and
trying to get more cloudy gaming companies (as well as VR ones) behind
fixing the Internet would be a good thing to try.

> The current hot topic use case seems to be machine learning and voice interaction. I wonder if we could build a use case something related to that? "Ok Google, my house is on fire!!" -- "Sorry can not call fire department, network is congested".

Well, I still like videoconferencing as a key thing we're enabling.
There's been much progress there - I just learned that jitsy now has
the "google congestion control" algorithm in it. The discussion was
here:

https://www.vuc.me/

And in other news, on needing regular, reliable software updates -
this just in. 41 million routers have this port exposed.

https://isc.sans.edu/forums/diary/Port+7547+SOAP+Remote+Code+Execution+Attack+Against+DSL+Modems/21759

thought this fall's DDOSes were bad?


>
>



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Bloat] Fixing bufferbloat in 2017

[Jonathan Morton]

> On 28 Nov, 2016, at 19:07, Dave Taht <dave.taht@gmail.com> wrote:
> 
> (as well as VR ones)

VR gaming *is* pretty hot right now.  Even the consoles are trying to get in on it, though I’m skeptical if even the just-upgraded consoles have the horsepower to really keep up.

The big thing about VR is that latency and framerate matter about 100x more than they do with a normal, 2D-projected game, because the realism factor is stepped up by using “natural” movements to control viewpoint and aimpoint instead of interpolating a mouse and keyboard.  Most of the (reputable) VR headset guys recommend a solid 90fps and absolutely minimal input-to-display lag to avoid motion sickness.

To put that into perspective, most monitors on peoples’ desks right now top out at  60Hz, and the equivalent responsiveness of a “fairly good” Internet connection is 20-30Hz.  Fortunately, the latter mostly impacts the movements of other objects in the game world, not the player himself - but there are counterexamples, especially in competitive multiplayer games.

I’m still keen on representing network latency as its reciprocal, “responsiveness”, which carries the same unit as framerate and is thereby made intuitively intelligible to gaming enthusiasts.  Admittedly most multiplayer gamers are by now familiar with “ping” in milliseconds, but they probably haven’t done the arithmetic to relate it to framerate and thus understand its relative importance.

 - Jonathan Morton

Re: [Bloat] Fixing bufferbloat in 2017

[Kathleen Nichols]

Well, it would be good to know where the congestion is coming from, i.e.
saying that "the network is congested" doesn't say which network. Since
our downlink got upgraded, there is rarely an issue there but from time
to time the comcast network just "goes down" in that it seems that
nothing gets out (the Nextdoor list then pops up all these postings "is
anyone else having internet problems?") but this is infrequent. We've
been able to find all sorts of interesting things in the home network
including some substandard cabling (which has been replaced) and various
wifi oddities (which would be easier to characterize if I could get
better data on the clients, particularly wrt to Google wifi but I have
an idea on this....)

Guess if your house is on fire, you stop watching the cat videos, right?

	Kathie

On 11/28/16 8:58 AM, Stephen Hemminger wrote:
> My experience has been that the media and developer attention span is
> short lived, and maybe that is part of the problem. Gaming is a niche
> market, and therefore is easily ignored; plus the classic gaming
> market is dying and I am not sure anyone is really investing in it.
> 
> The current hot topic use case seems to be machine learning and voice
> interaction. I wonder if we could build a use case something related
> to that? "Ok Google, my house is on fire!!" -- "Sorry can not call
> fire department, network is congested".
> 
> 
> _______________________________________________ Bloat mailing list 
> Bloat@lists.bufferbloat.net 
> https://lists.bufferbloat.net/listinfo/bloat
> 

Re: [Bloat] Fixing bufferbloat in 2017

[Pedro Tumusok]

[-- Attachment #1: Type: text/plain, Size: 2343 bytes --]

On Mon, Nov 28, 2016 at 1:48 PM, David Collier-Brown <davec-b@rogers.com>
wrote:

> A short RFC with a clear summary would change the ground on which we stand.
> Include me in if you're planning one.
>
> --dave
>
>
There are some RFCs that vendors uses for throughput testing, RFC2544 I
have seen on a few Firewall vendors datasheets atleast, not seen any
reference RFC3511.

Pedro



>
> On 28/11/16 01:00 AM, Jan Ceuleers wrote:
>
>> On 28/11/16 03:16, Jim Gettys wrote:
>>
>>> Ookla may have made themselves long term irrelevant by their recent
>>> behavior.  When your customers start funding development of a
>>> replacement (as Comcast has), you know they aren't happy.
>>>
>>> So I don't sweat Ookla: helping out the Comcast test effort is probably
>>> the best way to get bufferbloat in front of everyone, and best yet, the
>>> code for the tests is out there.
>>>
>> I do hope you're right Jim, but I still worry that Ookla is heavily
>> entrenched in carriers' test labs. This position has, I believe, come
>> about not because of Ookla's expertise in network testing but rather
>> because of market pull (i.e. speedtest.net's huge popularity with
>> end-users).
>>
>> As long as both of these positions remain (i.e. Ookla's mind share of
>> end-users and their resulting market share in the labs of large
>> purchasers of CPE) their lack of interest in bufferbloat is going to
>> keep this topic off the agenda in a large part of the industry.
>>
>> Unless Ookla can be coerced somehow.
>>
>> I have previously suggested standardising network throughput testing
>> methods and "grading" criteria. If there's an RFC on this subject
>> carriers are going to be interested in conformance to it and will
>> pressure their suppliers (of network testing gear, of CPE etc).
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>>
>
>
> --
> David Collier-Brown,         | Always do right. This will gratify
> System Programmer and Author | some people and astonish the rest
> davecb@spamcop.net           |                      -- Mark Twain
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>



-- 
Best regards / Mvh
Jan Pedro Tumusok

[-- Attachment #2: Type: text/html, Size: 4000 bytes --]

Re: [Bloat] Fixing bufferbloat in 2017

[Dave Taht]

On Mon, Nov 28, 2016 at 7:14 AM, Pedro Tumusok <pedro.tumusok@gmail.com> wrote:
>
>
> On Mon, Nov 28, 2016 at 1:48 PM, David Collier-Brown <davec-b@rogers.com>
> wrote:
>>
>> A short RFC with a clear summary would change the ground on which we
>> stand.
>> Include me in if you're planning one.

For the record, the rfc output of the aqm working group is here:

https://tools.ietf.org/wg/aqm/

And I'd made an attempt at developing a short RFC

http://snapon.lab.bufferbloat.net/~d/draft-taht-home-gateway-best-practices-00.html

but encountered so much flack from the aqm-ers on the list, that I
gave up, and decided to concentrate on getting the code out there on
multiple platforms, to finding the problems in the algorithms in the
real world, and creating de-facto standards.


>>
>> --dave
>>
>
> There are some RFCs that vendors uses for throughput testing, RFC2544 I have
> seen on a few Firewall vendors datasheets atleast, not seen any reference
> RFC3511.
>
> Pedro
>
>
>>
>>
>> On 28/11/16 01:00 AM, Jan Ceuleers wrote:
>>>
>>> On 28/11/16 03:16, Jim Gettys wrote:
>>>>
>>>> Ookla may have made themselves long term irrelevant by their recent
>>>> behavior.  When your customers start funding development of a
>>>> replacement (as Comcast has), you know they aren't happy.
>>>>
>>>> So I don't sweat Ookla: helping out the Comcast test effort is probably
>>>> the best way to get bufferbloat in front of everyone, and best yet, the
>>>> code for the tests is out there.
>>>
>>> I do hope you're right Jim, but I still worry that Ookla is heavily
>>> entrenched in carriers' test labs. This position has, I believe, come
>>> about not because of Ookla's expertise in network testing but rather
>>> because of market pull (i.e. speedtest.net's huge popularity with
>>> end-users).
>>>
>>> As long as both of these positions remain (i.e. Ookla's mind share of
>>> end-users and their resulting market share in the labs of large
>>> purchasers of CPE) their lack of interest in bufferbloat is going to
>>> keep this topic off the agenda in a large part of the industry.
>>>
>>> Unless Ookla can be coerced somehow.
>>>
>>> I have previously suggested standardising network throughput testing
>>> methods and "grading" criteria. If there's an RFC on this subject
>>> carriers are going to be interested in conformance to it and will
>>> pressure their suppliers (of network testing gear, of CPE etc).
>>> _______________________________________________
>>> Bloat mailing list
>>> Bloat@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/bloat
>>
>>
>>
>> --
>> David Collier-Brown,         | Always do right. This will gratify
>> System Programmer and Author | some people and astonish the rest
>> davecb@spamcop.net           |                      -- Mark Twain
>>
>>
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>
>
>
>
> --
> Best regards / Mvh
> Jan Pedro Tumusok
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Bloat] Fixing bufferbloat in 2017

[Dave Taht]

The biggest problem I see with speedtest-like network testing... is
the tests don't last long enough.

I've begun making that joke at every presentation - pointing at the
bloat spike at T+18 or T+22 seconds -
how many of you only use your networks for 30 seconds a day?

Re: [Bloat] Fixing bufferbloat in 2017

[David Lang]

On Sun, 27 Nov 2016, Kathleen Nichols wrote:

>>> Most of the suggestions in this thread deal with Getting the Word
>>> Out. That's good - that's the declaring victory part. The bad news
>>> is that this is not our collective skill set.
>
> So that's the hard part. Who do you need to Get the Word Out to and what
> do you expect them to do? It sounds like there are some edge router
> improvements coming. It's possible that some companies are using the
> work but they are advertising what it does for the customer not where it
> came from or what it is technically. So that might be a victory.

I think the biggest thing that's needed in terms of getting the word out is to 
have something that is recognized as being orders of magnatude better than 
what's out there, even if not perfect, and agree not to talk it down because you 
are working on something that may be better.

If there's ever a case of 'perfect being the enemy of good enough', it's the 
development here.

David Lang

[Bloat] fixing bufferbloat in 2017

[Dave Taht]

What's left to do?

What else can we do?

What should we stop doing?

What can we do better?

-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Bloat] fixing bufferbloat in 2017

[Jan Ceuleers]

On 22/11/16 16:32, Dave Taht wrote:
> What's left to do?

Furthering adoption of the code that contains the bloat-related
improvements.

In my view, the single biggest potential contributor towards driving
such adoption would be for Ookla to start measuring and reporting
bufferbloat, thereby putting it on the agenda of both end users and
carriers (who use Ookla's servers to qualify equipment in their labs).
That would in turn put it on the agenda of equipment manufacturers.

Re: [Bloat] fixing bufferbloat in 2017

[Dave Taht]

On Tue, Nov 22, 2016 at 8:19 AM, Jan Ceuleers <jan.ceuleers@gmail.com> wrote:
> On 22/11/16 16:32, Dave Taht wrote:
>> What's left to do?
>
> Furthering adoption of the code that contains the bloat-related
> improvements.
>
> In my view, the single biggest potential contributor towards driving
> such adoption would be for Ookla to start measuring and reporting
> bufferbloat, thereby putting it on the agenda of both end users and
> carriers (who use Ookla's servers to qualify equipment in their labs).
> That would in turn put it on the agenda of equipment manufacturers.

We've been after ookla for years. Samknows, also. The FCC, too. But
how to raise awareness further? An open letter? A petition to the
white house? A press release?

Until then, there's always dslreports and a few other sites.

I would like to see the industries most affected by bufferbloat -
voip/videoconferencing/gaming,web gain a good recognition of the
problem, how to fix it, and who to talk to about it (router makers and
ISPs)



> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Bloat] fixing bufferbloat in 2017

[Jim Gettys]

[-- Attachment #1: Type: text/plain, Size: 2229 bytes --]

On Tue, Nov 22, 2016 at 11:25 AM, Dave Taht <dave.taht@gmail.com> wrote:

> On Tue, Nov 22, 2016 at 8:19 AM, Jan Ceuleers <jan.ceuleers@gmail.com>
> wrote:
> > On 22/11/16 16:32, Dave Taht wrote:
> >> What's left to do?
> >
> > Furthering adoption of the code that contains the bloat-related
> > improvements.
> >
> > In my view, the single biggest potential contributor towards driving
> > such adoption would be for Ookla to start measuring and reporting
> > bufferbloat, thereby putting it on the agenda of both end users and
> > carriers (who use Ookla's servers to qualify equipment in their labs).
> > That would in turn put it on the agenda of equipment manufacturers.
>

​May not matter too much longer...​


>
> We've been after ookla for years. Samknows, also. The FCC, too. But
> how to raise awareness further? An open letter? A petition to the
> white house? A press release?
>
> Until then, there's always dslreports and a few other sites.
>

​Comcast has/is funding development of an open source speed test, and the
intent is to
have a bufferbloat test in it.

Several ISP's are on board with it in addition to Comcast and are helping
too.

You can improve the code...  I can dig up the link if someone is hot to
trot.
Right now, we're discussing how to license the code, however.  I don't know
what the final
outcome of that will be.
​


>
> I would like to see the industries most affected by bufferbloat -
> voip/videoconferencing/gaming,web gain a good recognition of the
> problem, how to fix it, and who to talk to about it (router makers and
> ISPs)
>

​Getting cellular manufacturers to understand would also be a good thing,
IMHO.
                                               - Jim
​


>
>
>
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
>
>
>
> --
> Dave Täht
> Let's go make home routers and wifi faster! With better software!
> http://blog.cerowrt.org
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>

[-- Attachment #2: Type: text/html, Size: 4784 bytes --]

Re: [Bloat] fixing bufferbloat in 2017

[Mikael Abrahamsson]

On Tue, 22 Nov 2016, Dave Taht wrote:

> I would like to see the industries most affected by bufferbloat - 
> voip/videoconferencing/gaming,web gain a good recognition of the 
> problem, how to fix it, and who to talk to about it (router makers and 
> ISPs)

It would be great if the realtime communications people (gaming, video, 
audio etc) had some kind of help page where people could be pointed to 
understand the problem.

I saw a Youtube video btw, where they had problems with gaming because 
"I'm uploading a youtube video at the same time as I am gaming, stupid 
me". People don't even realise this is not the way it has to be.

My take on this is that the problem is fairly well understood in "our" 
circles, but the wider audience still doesn't know, and even if they know, 
there is nowhere to go to fix it.

If we can find a product that solves the gaming community problem (they're 
one of the people who have "ping" in their applications and who 
immediately notices when it's bad), we could perhaps approach someone 
prominent in that gaming community and making a video on how to solve the 
problem.

"Look here, I did <X> and now I can game and upload a youtube video at the 
same time without problems!!!!1111oneoneone"

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] fixing bufferbloat in 2017

[Jonathan Morton]

> On 23 Nov, 2016, at 10:28, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> 
> If we can find a product that solves the gaming community problem (they're one of the people who have "ping" in their applications and who immediately notices when it's bad), we could perhaps approach someone prominent in that gaming community and making a video on how to solve the problem.

I can immediately think of one or two people who might be receptive to such a thing, if we can put together something approachable and practical.

The real problem is that currently our instructions start with “reflash your router firmware”, not with “buy this inexpensive router that Just Works".

 - Jonathan Morton

Re: [Bloat] fixing bufferbloat in 2017

[Kelvin Edmison]

[-- Attachment #1: Type: text/plain, Size: 2415 bytes --]




Sent from my iPhone
> On Nov 23, 2016, at 3:28 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> 
>> On Tue, 22 Nov 2016, Dave Taht wrote:
>> 
>> I would like to see the industries most affected by bufferbloat - voip/videoconferencing/gaming,web gain a good recognition of the problem, how to fix it, and who to talk to about it (router makers and ISPs)
> 
> It would be great if the realtime communications people (gaming, video, audio etc) had some kind of help page where people could be pointed to understand the problem.
> 
> I saw a Youtube video btw, where they had problems with gaming because "I'm uploading a youtube video at the same time as I am gaming, stupid me". People don't even realise this is not the way it has to be.
> 
> My take on this is that the problem is fairly well understood in "our" circles, but the wider audience still doesn't know, and even if they know, there is nowhere to go to fix it.
> 
> If we can find a product that solves the gaming community problem (they're one of the people who have "ping" in their applications and who immediately notices when it's bad), we could perhaps approach someone prominent in that gaming community and making a video on how to solve the problem.
> 
> "Look here, I did <X> and now I can game and upload a youtube video at the same time without problems!!!!1111oneoneone"
> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

The gaming/youtube video is an excellent use case on whim to build. Simple, compelling, and can easily demonstrate the bufferbloat problem. 

One suggestion would be to talk to the people/web sites that publish reviews of router and gaming performance. There should be only one or two people there to convince regarding bufferbloat. If those people can be convinced and write about it then the gaming community at large will learn about it. With any luck, these reviews will also generate talk inside the router vendors as to how they perform and what they have to do to compete.

AnandTech or ArsTechnica might be a good place to start. They have pretty literate staff, and AnandTech even had access to an Ixia wifi tester at one point. http://www.anandtech.com/show/10081/wifi-testing-with-ixia-wavedevice

Regards,
  Kelvin

[-- Attachment #2: Type: text/html, Size: 3309 bytes --]

Re: [Bloat] fixing bufferbloat in 2017

[Pedro Tumusok]

[-- Attachment #1: Type: text/plain, Size: 3141 bytes --]

On Wed, Nov 23, 2016 at 12:59 PM, Kelvin Edmison <kelvin@edmison.net> wrote:

>
>
>
> Sent from my iPhone
> On Nov 23, 2016, at 3:28 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>
> On Tue, 22 Nov 2016, Dave Taht wrote:
>
> I would like to see the industries most affected by bufferbloat -
> voip/videoconferencing/gaming,web gain a good recognition of the problem,
> how to fix it, and who to talk to about it (router makers and ISPs)
>
>
> It would be great if the realtime communications people (gaming, video,
> audio etc) had some kind of help page where people could be pointed to
> understand the problem.
>
> I saw a Youtube video btw, where they had problems with gaming because
> "I'm uploading a youtube video at the same time as I am gaming, stupid me".
> People don't even realise this is not the way it has to be.
>
> My take on this is that the problem is fairly well understood in "our"
> circles, but the wider audience still doesn't know, and even if they know,
> there is nowhere to go to fix it.
>
> If we can find a product that solves the gaming community problem (they're
> one of the people who have "ping" in their applications and who immediately
> notices when it's bad), we could perhaps approach someone prominent in that
> gaming community and making a video on how to solve the problem.
>
> "Look here, I did <X> and now I can game and upload a youtube video at the
> same time without problems!!!!1111oneoneone"
>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
>
> The gaming/youtube video is an excellent use case on whim to build.
> Simple, compelling, and can easily demonstrate the bufferbloat problem.
>
> One suggestion would be to talk to the people/web sites that publish
> reviews of router and gaming performance.
>

Who actually cares about or read router reviews? I am guessing a subset of
the people that care about lag in FPS games.

How about approaching the people that make the post popular games that are
affected.

Battlefield series is from Dice and they have a main office in Stockholm,
Sweden. Given them a demo of a bloat free game and a bloated game and
hopefully get them to go to the players through their channels on how to
get a better game experience would most likely give a better result. The
game developers have their own conferences where they share, so if somebody
in a studio ended up doing a presentation at these ones, it would also get
the word out and about.

There are other games, but I am not a big FPS gamer, so I googled and this
came up

http://www.gamersdecide.com/pc-game-news/15-most-played-fps-games-2016-pc

But we would need a couple of things

1. Establishing contact with the different game development studios
2. Somebody to go and do a demo, shake hands and talk with them. Video, I
assume would be less effective.

If this something we should try, I can help out with the first point, but
the second one probably needs local bufferbloat evangelists.

-- 
Best regards / Mvh
Jan Pedro Tumusok

[-- Attachment #2: Type: text/html, Size: 4719 bytes --]

Re: [Bloat] fixing bufferbloat in 2017

[Mário Sérgio Fujikawa Ferreira]

[-- Attachment #1: Type: text/plain, Size: 4358 bytes --]



On 23/11/2016 11:31, Pedro Tumusok wrote:
>
>
> On Wed, Nov 23, 2016 at 12:59 PM, Kelvin Edmison <kelvin@edmison.net 
> <mailto:kelvin@edmison.net>> wrote:
>
>
>
>
>     Sent from my iPhone
>     On Nov 23, 2016, at 3:28 AM, Mikael Abrahamsson <swmike@swm.pp.se
>     <mailto:swmike@swm.pp.se>> wrote:
>
>>     On Tue, 22 Nov 2016, Dave Taht wrote:
>>
>>>     I would like to see the industries most affected by bufferbloat
>>>     - voip/videoconferencing/gaming,web gain a good recognition of
>>>     the problem, how to fix it, and who to talk to about it (router
>>>     makers and ISPs)
>>
>>     It would be great if the realtime communications people (gaming,
>>     video, audio etc) had some kind of help page where people could
>>     be pointed to understand the problem.
>>
>>     I saw a Youtube video btw, where they had problems with gaming
>>     because "I'm uploading a youtube video at the same time as I am
>>     gaming, stupid me". People don't even realise this is not the way
>>     it has to be.
>>
>>     My take on this is that the problem is fairly well understood in
>>     "our" circles, but the wider audience still doesn't know, and
>>     even if they know, there is nowhere to go to fix it.
>>
>>     If we can find a product that solves the gaming community problem
>>     (they're one of the people who have "ping" in their applications
>>     and who immediately notices when it's bad), we could perhaps
>>     approach someone prominent in that gaming community and making a
>>     video on how to solve the problem.
>>
>>     "Look here, I did <X> and now I can game and upload a youtube
>>     video at the same time without problems!!!!1111oneoneone"
>>
>>     -- 
>>     Mikael Abrahamsson    email: swmike@swm.pp.se
>>     <mailto:swmike@swm.pp.se>
>>     _______________________________________________
>>     Bloat mailing list
>>     Bloat@lists.bufferbloat.net <mailto:Bloat@lists.bufferbloat.net>
>>     https://lists.bufferbloat.net/listinfo/bloat
>>     <https://lists.bufferbloat.net/listinfo/bloat>
>
>     The gaming/youtube video is an excellent use case on whim to
>     build. Simple, compelling, and can easily demonstrate the
>     bufferbloat problem.
>
>     One suggestion would be to talk to the people/web sites that
>     publish reviews of router and gaming performance.
>
>
> Who actually cares about or read router reviews? I am guessing a 
> subset of the people that care about lag in FPS games.
>
> How about approaching the people that make the post popular games that 
> are affected.
>
> Battlefield series is from Dice and they have a main office in 
> Stockholm, Sweden. Given them a demo of a bloat free game and a 
> bloated game and hopefully get them to go to the players through their 
> channels on how to get a better game experience would most likely give 
> a better result. The game developers have their own conferences where 
> they share, so if somebody in a studio ended up doing a presentation 
> at these ones, it would also get the word out and about.
>
> There are other games, but I am not a big FPS gamer, so I googled and 
> this came up
>
> http://www.gamersdecide.com/pc-game-news/15-most-played-fps-games-2016-pc
>
> But we would need a couple of things
>
> 1. Establishing contact with the different game development studios
> 2. Somebody to go and do a demo, shake hands and talk with them. 
> Video, I assume would be less effective.
>
> If this something we should try, I can help out with the first point, 
> but the second one probably needs local bufferbloat evangelists.

   IF we're heading on to gaming, perhaps attacking a huge vendor such 
as Valve's Steam. I have no idea how approachable they're of course.

   As for FPS gaming, I can tell you that the most prolific gamers will 
go to great lenghts to optimize their experience. They are far easier to 
convince to flash a router, modify windows drivers parameters, change 
broadband providers then the bigger audiance. As soon as others notice 
this "unfair" advantage they begin petitioning the game developers so 
that the "advantage" comes baked in rather than be a special recipe.

>
> -- 
> Best regards / Mvh
> Jan Pedro Tumusok
>
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat


[-- Attachment #2: Type: text/html, Size: 9200 bytes --]

Re: [Bloat] fixing bufferbloat in 2017

[Mikael Abrahamsson]

On Wed, 23 Nov 2016, Pedro Tumusok wrote:

> If this something we should try, I can help out with the first point, 
> but the second one probably needs local bufferbloat evangelists.

I am not worried about getting these people on board to show a solution.

I'm worried that we do not have a solution that is easily deployable for 
"normal" people. If someone has X/Y megabit/s Comcast Internet connection, 
what solution do we have to offer them? I can't think of one that actually 
solves the problem for real.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] fixing bufferbloat in 2017

[Benjamin Cronce]

[-- Attachment #1: Type: text/plain, Size: 1534 bytes --]

On Wed, Nov 23, 2016 at 11:20 AM, Mikael Abrahamsson <swmike@swm.pp.se>
wrote:

> On Wed, 23 Nov 2016, Pedro Tumusok wrote:
>
> If this something we should try, I can help out with the first point, but
>> the second one probably needs local bufferbloat evangelists.
>>
>
> I am not worried about getting these people on board to show a solution.
>
> I'm worried that we do not have a solution that is easily deployable for
> "normal" people. If someone has X/Y megabit/s Comcast Internet connection,
> what solution do we have to offer them? I can't think of one that actually
> solves the problem for real.


If there is a simple affordable solution, say Open/DD-WRT distro based
bridge that all you do is configure your up/down bandwidth and it applies
Codel/fq-Codel/Cake, then all you need to do is drive up awareness. A good
channel for awareness would be getting in contact with popular Twitch or
YouTube gaming streamers. But I wouldn't put much effort into driving up
awareness until there is a device that people can easily acquire, use, and
afford. At first I was thinking of telling people to use *-WRT supporting
routers, but changing the firmware on your router requires too much
research, and many people care about bleeding edge features. You need
something that works in tangent with whatever they are using.


>
>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>

[-- Attachment #2: Type: text/html, Size: 2445 bytes --]

Re: [Bloat] fixing bufferbloat in 2017

[Mikael Abrahamsson]

On Wed, 23 Nov 2016, Benjamin Cronce wrote:

> If there is a simple affordable solution, say Open/DD-WRT distro based 
> bridge that all you do is configure your up/down bandwidth and it 
> applies Codel/fq-Codel/Cake, then all you need to do is drive up 
> awareness. A good channel for awareness would be getting in contact with 
> popular Twitch or YouTube gaming streamers. But I wouldn't put much 
> effort into driving up awareness until there is a device that people can 
> easily acquire, use, and afford. At first I was thinking of telling 
> people to use *-WRT supporting routers, but changing the firmware on 
> your router requires too much research, and many people care about 
> bleeding edge features. You need something that works in tangent with 
> whatever they are using.

If Comcast sells you 100/20 (I have no idea if this is a thing), you set 
your upstream on this box to 18 meg fq_codel, and then Comcast 
oversubscribes you so you only get 15 meg up part of the time, then you're 
still bloated by the modem. This is not a solution.

I don't think "buy $thing, install *WRT on it, configure it like this" is 
above most gamers, but I'm afraid we don't even have a working solution 
for someone with that kind of skillset.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] fixing bufferbloat in 2017

[Sebastian Moeller]

Well,

> On Nov 23, 2016, at 18:31, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> 
> On Wed, 23 Nov 2016, Benjamin Cronce wrote:
> 
>> If there is a simple affordable solution, say Open/DD-WRT distro based bridge that all you do is configure your up/down bandwidth and it applies Codel/fq-Codel/Cake, then all you need to do is drive up awareness. A good channel for awareness would be getting in contact with popular Twitch or YouTube gaming streamers. But I wouldn't put much effort into driving up awareness until there is a device that people can easily acquire, use, and afford. At first I was thinking of telling people to use *-WRT supporting routers, but changing the firmware on your router requires too much research, and many people care about bleeding edge features. You need something that works in tangent with whatever they are using.
> 
> If Comcast sells you 100/20 (I have no idea if this is a thing), you set your upstream on this box to 18 meg fq_codel, and then Comcast oversubscribes you so you only get 15 meg up part of the time, then you're still bloated by the modem. This is not a solution.

since Comcast will roll-out DOCSIS3.1 modems any time now, the inbuilt PIE AQM will solve the variable upstream problem relatively well, but that still does not fix the downstream. Also oversubscription/congestion is not unique on DOCSIS systems, most ISPs have somewhere some bandwidth link that is oversubscribed (e.g. the uplink connection of aDSLAM) and there the problem is the same as on cable, namely we can only control the local bottleneck. 


> 
> I don't think "buy $thing, install *WRT on it, configure it like this" is above most gamers, but I'm afraid we don't even have a working solution for someone with that kind of skillset.

	Well we have something that can solve a big part of the problem “self-induuced” congestion on the last mile, while not a solution to all latency increase under load certainly a worthwhile improvement. Or put differently this will not remedy bandwidth competition with other subscribers, but it can solve the bandwidth competition with other local users (behind the same CPE).
Maybe we need to document better the scope of what we can and can not “deliver”?

Best Regards
	Sebastian

> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] fixing bufferbloat in 2017

[Noah Causin]

There is a company called Netduma which sells a product called the 
Netduma R1 Router.  It's main feature is reducing lag.  It does this 
through QOS and GEO-IP Filtering.  (Limiting available servers to your 
local region = reduced RTT)

It seems relatively popular in the gaming world, especially console.

It is based on OpenWRT Chaos Calmer: https://netduma.com/opensource/

It has an advanced QOS system that already uses FQ_Codel.

Here are the hardware specs:

https://netduma.com/features/hardware/

I assume it has an ath9k.

Maybe they could implement the ath9k fq_codel and airtime patches.

The user base that buys this product seems like they would be more 
familiar with setting up routers than the average person.

On 11/23/2016 12:31 PM, Mikael Abrahamsson wrote:
> On Wed, 23 Nov 2016, Benjamin Cronce wrote:
>
>> If there is a simple affordable solution, say Open/DD-WRT distro 
>> based bridge that all you do is configure your up/down bandwidth and 
>> it applies Codel/fq-Codel/Cake, then all you need to do is drive up 
>> awareness. A good channel for awareness would be getting in contact 
>> with popular Twitch or YouTube gaming streamers. But I wouldn't put 
>> much effort into driving up awareness until there is a device that 
>> people can easily acquire, use, and afford. At first I was thinking 
>> of telling people to use *-WRT supporting routers, but changing the 
>> firmware on your router requires too much research, and many people 
>> care about bleeding edge features. You need something that works in 
>> tangent with whatever they are using.
>
> If Comcast sells you 100/20 (I have no idea if this is a thing), you 
> set your upstream on this box to 18 meg fq_codel, and then Comcast 
> oversubscribes you so you only get 15 meg up part of the time, then 
> you're still bloated by the modem. This is not a solution.
>
> I don't think "buy $thing, install *WRT on it, configure it like this" 
> is above most gamers, but I'm afraid we don't even have a working 
> solution for someone with that kind of skillset.
>

Re: [Bloat] fixing bufferbloat in 2017

[David Lang]

that doesn't even do 5GHz, so your wifi performance will be cripped by 
interference and the lack of available bandwidth.

On Wed, 23 Nov 2016, Noah Causin wrote:

> There is a company called Netduma which sells a product called the 
> Netduma R1 Router.  It's main feature is reducing lag.  It does this 
> through QOS and GEO-IP Filtering.  (Limiting available servers to your 
> local region = reduced RTT)
>
> It seems relatively popular in the gaming world, especially console.
>
> It is based on OpenWRT Chaos Calmer: https://netduma.com/opensource/
>
> It has an advanced QOS system that already uses FQ_Codel.
>
> Here are the hardware specs:
>
> https://netduma.com/features/hardware/
>
> I assume it has an ath9k.
>
> Maybe they could implement the ath9k fq_codel and airtime patches.
>
> The user base that buys this product seems like they would be more 
> familiar with setting up routers than the average person.
>
> On 11/23/2016 12:31 PM, Mikael Abrahamsson wrote:
>> On Wed, 23 Nov 2016, Benjamin Cronce wrote:
>>
>>> If there is a simple affordable solution, say Open/DD-WRT distro 
>>> based bridge that all you do is configure your up/down bandwidth and 
>>> it applies Codel/fq-Codel/Cake, then all you need to do is drive up 
>>> awareness. A good channel for awareness would be getting in contact 
>>> with popular Twitch or YouTube gaming streamers. But I wouldn't put 
>>> much effort into driving up awareness until there is a device that 
>>> people can easily acquire, use, and afford. At first I was thinking 
>>> of telling people to use *-WRT supporting routers, but changing the 
>>> firmware on your router requires too much research, and many people 
>>> care about bleeding edge features. You need something that works in 
>>> tangent with whatever they are using.
>>
>> If Comcast sells you 100/20 (I have no idea if this is a thing), you 
>> set your upstream on this box to 18 meg fq_codel, and then Comcast 
>> oversubscribes you so you only get 15 meg up part of the time, then 
>> you're still bloated by the modem. This is not a solution.
>>
>> I don't think "buy $thing, install *WRT on it, configure it like this" 
>> is above most gamers, but I'm afraid we don't even have a working 
>> solution for someone with that kind of skillset.
>>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>

Re: [Bloat] fixing bufferbloat in 2017

[Benjamin Cronce]

[-- Attachment #1: Type: text/plain, Size: 4063 bytes --]

On Wed, Nov 23, 2016 at 11:56 AM, David Lang <david@lang.hm> wrote:

> that doesn't even do 5GHz, so your wifi performance will be cripped by
> interference and the lack of available bandwidth.
>
>
> On Wed, 23 Nov 2016, Noah Causin wrote:
>
> There is a company called Netduma which sells a product called the Netduma
>> R1 Router.  It's main feature is reducing lag.  It does this through QOS
>> and GEO-IP Filtering.  (Limiting available servers to your local region =
>> reduced RTT)
>>
>> It seems relatively popular in the gaming world, especially console.
>>
>> It is based on OpenWRT Chaos Calmer: https://netduma.com/opensource/
>>
>> It has an advanced QOS system that already uses FQ_Codel.
>>
>> Here are the hardware specs:
>>
>> https://netduma.com/features/hardware/
>>
>> I assume it has an ath9k.
>>
>> Maybe they could implement the ath9k fq_codel and airtime patches.
>>
>> The user base that buys this product seems like they would be more
>> familiar with setting up routers than the average person.
>>
>> On 11/23/2016 12:31 PM, Mikael Abrahamsson wrote:
>>
>>> On Wed, 23 Nov 2016, Benjamin Cronce wrote:
>>>
>>> If there is a simple affordable solution, say Open/DD-WRT distro based
>>>> bridge that all you do is configure your up/down bandwidth and it applies
>>>> Codel/fq-Codel/Cake, then all you need to do is drive up awareness. A good
>>>> channel for awareness would be getting in contact with popular Twitch or
>>>> YouTube gaming streamers. But I wouldn't put much effort into driving up
>>>> awareness until there is a device that people can easily acquire, use, and
>>>> afford. At first I was thinking of telling people to use *-WRT supporting
>>>> routers, but changing the firmware on your router requires too much
>>>> research, and many people care about bleeding edge features. You need
>>>> something that works in tangent with whatever they are using.
>>>>
>>>
>>> If Comcast sells you 100/20 (I have no idea if this is a thing), you set
>>> your upstream on this box to 18 meg fq_codel, and then Comcast
>>> oversubscribes you so you only get 15 meg up part of the time, then you're
>>> still bloated by the modem. This is not a solution.
>>>
>>> I don't think "buy $thing, install *WRT on it, configure it like this"
>>> is above most gamers, but I'm afraid we don't even have a working solution
>>> for someone with that kind of skillset.
>>>
>>
I would be curious to know what the 80/20 rule is. Can we reach it with
what I described? The other way to handle the situation you mentioned is to
tell the end users they can trade more bandwidth for a less likely chance
of having high latency, depending on the stability of their ISP.

There is also the strange issue of crazy high bursts from video streaming
services. I know Netflix is working on the packet-pacing problem with
FreeBSD, but I've done packet-dumps from several streaming providers and
the issue seems to be with TCP with transient activity and data transfers
that with in the TCP transmit window. A 5Mb/s average really turns into a
40Gb/s burst of 256KiB of data 3 times a second. Since the buffers are
large, they don't drop anything. The bigger issue is the end-user sees an
"average" ping that is low, but they get constant transient oddities while
gaming and can't figure out why someone streaming 5Mb/s is hosing their
100Mb connection.

Most people only have a 1Gb network link, so a 40Gb burst won't get through
anyway, but they will see a 1Gb burst dragged out 40x longer, giving a
bridging device time to drop a packet or two and signal TCP to back-off.
Looking at my WAN port, I actually see back-to-back packets at 1Gb
line-rate from Netflix, Hulu, Youtube, and Twitch for long lived
connections that have periodic activity.

>
>>>
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>>
>> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>

[-- Attachment #2: Type: text/html, Size: 5962 bytes --]

Re: [Bloat] fixing bufferbloat in 2017

[David Lang]

On Wed, 23 Nov 2016, Benjamin Cronce wrote:

> Most people only have a 1Gb network link,

umm, no, most people have FAR slower links, by an order or two of magnatude.

David Lang

Re: [Bloat] fixing bufferbloat in 2017

[Benjamin Cronce]

[-- Attachment #1: Type: text/plain, Size: 1210 bytes --]

I meant the actual physical link, not the provisioned rate. Most last mile
tech encapsulates Ethernet frames into larger super-frames. Decapsulating
the Ethernet frames is pretty much at 1Gb line rate. On my GPON link, with
shaping in PFSense turned off, I regularly see 4,000+ 1500byte datagrams at
full 1Gb link rate hitting my WAN before my ISP starts shaping to my
150Mb/s average. When I had a cable modem, I would see similar, except
instead of a constant 1Gb/s, I would see bursts of multiple packets, then a
small pause, but eventually the packets started to get more and more spaced
as the provisioned average started to kick in. I assumed this was because
my DOCSIS3 8x bonded link could only max out around 300Mb/s, even though my
provisioned rate was only 30Mb/s.

It really depended on the quantum of the shaping algorithm, calculated
average window size, and how elastic or strict it was.

Ahh, packet packing, what a fun problem.

On Wed, Nov 23, 2016 at 12:38 PM, David Lang <david@lang.hm> wrote:

> On Wed, 23 Nov 2016, Benjamin Cronce wrote:
>
> Most people only have a 1Gb network link,
>>
>
> umm, no, most people have FAR slower links, by an order or two of
> magnatude.
>
> David Lang
>

[-- Attachment #2: Type: text/html, Size: 1783 bytes --]

Re: [Bloat] fixing bufferbloat in 2017

[David Lang]

Most people not only aren't operating at 1Gb/sec, they can't buy a 1Gb/sec line 
at any cost.

100Mb is getting more common, but the majority of people cannot buy a line this 
fast for any amount of money.

10-30 Mb is probably the range that "most people" have, with a large number stll 
having <10Mb available, no matter what they are willing to pay

David Lang

On Wed, 23 Nov 2016, Benjamin Cronce wrote:

> I meant the actual physical link, not the provisioned rate. Most last mile
> tech encapsulates Ethernet frames into larger super-frames. Decapsulating
> the Ethernet frames is pretty much at 1Gb line rate. On my GPON link, with
> shaping in PFSense turned off, I regularly see 4,000+ 1500byte datagrams at
> full 1Gb link rate hitting my WAN before my ISP starts shaping to my
> 150Mb/s average. When I had a cable modem, I would see similar, except
> instead of a constant 1Gb/s, I would see bursts of multiple packets, then a
> small pause, but eventually the packets started to get more and more spaced
> as the provisioned average started to kick in. I assumed this was because
> my DOCSIS3 8x bonded link could only max out around 300Mb/s, even though my
> provisioned rate was only 30Mb/s.
>
> It really depended on the quantum of the shaping algorithm, calculated
> average window size, and how elastic or strict it was.
>
> Ahh, packet packing, what a fun problem.
>
> On Wed, Nov 23, 2016 at 12:38 PM, David Lang <david@lang.hm> wrote:
>
>> On Wed, 23 Nov 2016, Benjamin Cronce wrote:
>>
>> Most people only have a 1Gb network link,
>>>
>>
>> umm, no, most people have FAR slower links, by an order or two of
>> magnatude.
>>
>> David Lang
>>
>

Re: [Bloat] fixing bufferbloat in 2017

[Dave Taht]

On Wed, Nov 23, 2016 at 11:42 AM, David Lang <david@lang.hm> wrote:
> Most people not only aren't operating at 1Gb/sec, they can't buy a 1Gb/sec
> line at any cost.

I too have seen the bursty behavior that OP is describing. It's one
reason why cake's estimator can't get a good result on cable.

>
> 100Mb is getting more common, but the majority of people cannot buy a line
> this fast for any amount of money.
>
> 10-30 Mb is probably the range that "most people" have, with a large number
> stll having <10Mb available, no matter what they are willing to pay
>
> David Lang
>
> On Wed, 23 Nov 2016, Benjamin Cronce wrote:
>
>> I meant the actual physical link, not the provisioned rate. Most last mile
>> tech encapsulates Ethernet frames into larger super-frames. Decapsulating
>> the Ethernet frames is pretty much at 1Gb line rate. On my GPON link, with
>> shaping in PFSense turned off, I regularly see 4,000+ 1500byte datagrams
>> at
>> full 1Gb link rate hitting my WAN before my ISP starts shaping to my
>> 150Mb/s average. When I had a cable modem, I would see similar, except
>> instead of a constant 1Gb/s, I would see bursts of multiple packets, then
>> a
>> small pause, but eventually the packets started to get more and more
>> spaced
>> as the provisioned average started to kick in. I assumed this was because
>> my DOCSIS3 8x bonded link could only max out around 300Mb/s, even though
>> my
>> provisioned rate was only 30Mb/s.
>>
>> It really depended on the quantum of the shaping algorithm, calculated
>> average window size, and how elastic or strict it was.
>>
>> Ahh, packet packing, what a fun problem.
>>
>> On Wed, Nov 23, 2016 at 12:38 PM, David Lang <david@lang.hm> wrote:
>>
>>> On Wed, 23 Nov 2016, Benjamin Cronce wrote:
>>>
>>> Most people only have a 1Gb network link,
>>>>
>>>>
>>>
>>> umm, no, most people have FAR slower links, by an order or two of
>>> magnatude.
>>>
>>> David Lang
>>>
>>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Bloat] fixing bufferbloat in 2017

[David Lang]

On Wed, 23 Nov 2016, Mikael Abrahamsson wrote:

> If Comcast sells you 100/20 (I have no idea if this is a thing), you set 
> your upstream on this box to 18 meg fq_codel, and then Comcast 
> oversubscribes you so you only get 15 meg up part of the time, then you're 
> still bloated by the modem. This is not a solution.
>
> I don't think "buy $thing, install *WRT on it, configure it like this" is 
> above most gamers, but I'm afraid we don't even have a working solution 
> for someone with that kind of skillset.

perfect is the enemy of good enough

stop trying for perfection, you will never be able to address all problems.

But we already have several options that would address the very real problems 
people are having if they could just be deployed.

you don't even need to use cake and worry about what your uplink bandwidth is if 
you can just use fq_codel on the real edge device. The fact that it is almost 
impossible to buy a router that has a DSL interface on it, so you have to try to 
artificially move the bottleneck is a problem.

Deploy what we already know to work on the real edge devices and things get 
vastly simpler.

David Lang

Re: [Bloat] fixing bufferbloat in 2017

[Mikael Abrahamsson]

On Wed, 23 Nov 2016, David Lang wrote:

> Deploy what we already know to work on the real edge devices and things 
> get vastly simpler.

Sure! Sounds Great. How?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] fixing bufferbloat in 2017

[Sebastian Moeller]

> On Nov 23, 2016, at 19:09, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> 
> On Wed, 23 Nov 2016, David Lang wrote:
> 
>> Deploy what we already know to work on the real edge devices and things get vastly simpler.

	One problem will be that the actual edge devices are often ISP supplied and hence extremely cost sensitive; combined with increasing bandwidth in many ISP offerings, having  CPEs that can perform ingress shaping at “modern” rates looks challenging if the same CPEs also need to be very cheap. Egress shaping is a different kettle of fish though and for most asymmetric plans the CPE either should have enough punch or might be amendable to a BQL-like solution that could e actually relatively computationally cheap. But for that to happen we would need to convince ISPs and/or CPE chipset manufacturer (or better those engineers that create the drivers for the SDKs).

> 
> Sure! Sounds Great. How?

	Getting a big ISP on board would already be a great start, if combined with a tiny bit of PR…?

Best Regards
	Sebastian


> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] fixing bufferbloat in 2017

[Dave Taht]

On Wed, Nov 23, 2016 at 10:18 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
>
>> On Nov 23, 2016, at 19:09, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>>
>> On Wed, 23 Nov 2016, David Lang wrote:
>>
>>> Deploy what we already know to work on the real edge devices and things get vastly simpler.
>
>         One problem will be that the actual edge devices are often ISP supplied and hence extremely cost sensitive; combined with increasing bandwidth in many ISP offerings, having  CPEs that can perform ingress shaping at “modern” rates looks challenging if the same CPEs also need to be very cheap. Egress shaping is a different kettle of fish though and for most asymmetric plans the CPE either should have enough punch or might be amendable to a BQL-like solution that could e actually relatively computationally cheap. But for that to happen we would need to convince ISPs and/or CPE chipset manufacturer (or better those engineers that create the drivers for the SDKs).

One benefit of the fq_codel on wifi work is that for homes that are
primarily wifi, you no longer need inbound rate shaping to work, you
can do it on the wifi naturally.

A long term plan might be to try to develop code that could be used at
the ISP, a transparent bridge that would take over customer rate
shaping and queue management. Hardware "good enough" to do this,
ranging from high end xeons to the next generation NXP products, to
mellonox's bluefield thing, is arriving, and I anticipate being able
to effectively fq and shape 1000s of customers with gear that costs
less than 10k - maybe not with linux, but with vpp and/or dpdk.
>>
>> Sure! Sounds Great. How?
>
>         Getting a big ISP on board would already be a great start, if combined with a tiny bit of PR…?
>
> Best Regards
>         Sebastian
>
>
>>
>> --
>> Mikael Abrahamsson    email: swmike@swm.pp.se
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Bloat] fixing bufferbloat in 2017

[Toke Høiland-Jørgensen]

Dave Taht <dave.taht@gmail.com> writes:

> On Wed, Nov 23, 2016 at 10:18 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
>>
>>> On Nov 23, 2016, at 19:09, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>>>
>>> On Wed, 23 Nov 2016, David Lang wrote:
>>>
>>>> Deploy what we already know to work on the real edge devices and things get vastly simpler.
>>
>>         One problem will be that the actual edge devices are often ISP
>>         supplied and hence extremely cost sensitive; combined with increasing
>>         bandwidth in many ISP offerings, having CPEs that can perform ingress
>>         shaping at “modern” rates looks challenging if the same CPEs also need
>>         to be very cheap. Egress shaping is a different kettle of fish though
>>         and for most asymmetric plans the CPE either should have enough punch
>>         or might be amendable to a BQL-like solution that could e actually
>>         relatively computationally cheap. But for that to happen we would need
>>         to convince ISPs and/or CPE chipset manufacturer (or better those
>>         engineers that create the drivers for the SDKs).
>
> One benefit of the fq_codel on wifi work is that for homes that are
> primarily wifi, you no longer need inbound rate shaping to work, you
> can do it on the wifi naturally.
>
> A long term plan might be to try to develop code that could be used at
> the ISP, a transparent bridge that would take over customer rate
> shaping and queue management. Hardware "good enough" to do this,
> ranging from high end xeons to the next generation NXP products, to
> mellonox's bluefield thing, is arriving, and I anticipate being able
> to effectively fq and shape 1000s of customers with gear that costs
> less than 10k - maybe not with linux, but with vpp and/or dpdk.

I happen to know of at least one company that offers this kind of middle
box and also ships CoDel and BLUE (I think it was). Those are DPI boxes,
though (*shudder*) -- and I think the price tag is quite a bit more than
$10k... :/

-Toke

Re: [Bloat] fixing bufferbloat in 2017

[David Lang]

[-- Attachment #1: Type: text/plain, Size: 1553 bytes --]

On Wed, 23 Nov 2016, Sebastian Moeller wrote:

>> On Nov 23, 2016, at 19:09, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>>
>> On Wed, 23 Nov 2016, David Lang wrote:
>>
>>> Deploy what we already know to work on the real edge devices and things get vastly simpler.
>
> 	One problem will be that the actual edge devices are often ISP supplied 
> and hence extremely cost sensitive; combined with increasing bandwidth in many 
> ISP offerings, having CPEs that can perform ingress shaping at “modern” rates 
> looks challenging if the same CPEs also need to be very cheap. Egress shaping 
> is a different kettle of fish though and for most asymmetric plans the CPE 
> either should have enough punch or might be amendable to a BQL-like solution 
> that could e actually relatively computationally cheap. But for that to happen 
> we would need to convince ISPs and/or CPE chipset manufacturer (or better 
> those engineers that create the drivers for the SDKs).

if the ISPs are on board, you don't need to do ingres shaping, the ISP does that 
on their end.

but even if you don't do any ingres shaping, just handling egress will make a 
huge difference.

David Lang

>>
>> Sure! Sounds Great. How?
>
> 	Getting a big ISP on board would already be a great start, if combined with a tiny bit of PR…?
>
> Best Regards
> 	Sebastian
>
>
>>
>> --
>> Mikael Abrahamsson    email: swmike@swm.pp.se
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>
>

Re: [Bloat] fixing bufferbloat in 2017

[Dave Taht]

On Wed, Nov 23, 2016 at 9:54 AM, David Lang <david@lang.hm> wrote:
> On Wed, 23 Nov 2016, Mikael Abrahamsson wrote:
>
>> If Comcast sells you 100/20 (I have no idea if this is a thing), you set
>> your upstream on this box to 18 meg fq_codel, and then Comcast
>> oversubscribes you so you only get 15 meg up part of the time, then you're
>> still bloated by the modem. This is not a solution.
>>
>> I don't think "buy $thing, install *WRT on it, configure it like this" is
>> above most gamers, but I'm afraid we don't even have a working solution for
>> someone with that kind of skillset.
>
>
> perfect is the enemy of good enough

I agree that we are getting close to the point where we can recommend
several off the shelf products to home and small business users. In
addition to the netduma, there's the edgerouters, and the asus-merlin
on arm, and turris omnia. There is also ipfire, and pfsense is coming
along as well.

There are multiple other commercial products whose methods could be
benchmarked and evaluated, notably meraki, google wifi, eero, plume,
and portal. The latter two i believe are openwrt based.  eero is
debian. google wifi is it's own thing, not fully buildroot, debian,
chromeos, or brillo. There are more than a few others - linksys itself
is closely tracking the work nowadays - and in most cases folk are
using automated tests to configure things. I'm pretty sure eero has
got the qos work working along the edge, and the google folk are well
aware of the work and I hope pushing towards deployment on the gfiber
and onhub products (the "google wifi" stuff ships dec 7th, don't know
a lot about it). Meraki has a fq_codel like solution and airtime
fairness being rolled out in beta mode now, I'm told.

There's also the longstanding qca streamboost stuff.

I have an issue with these latter products in that most are claiming
the benefit comes from their cloud based analytics software, rather
than better algorithms, and I'm allergic to snoopy stuff on my core
network equipment sharing anything about my network activity with a
"private" cloud.  This is one of the few places where my politics has
got in the way of doing better science. Google has been good with
their privacy policy and has learned much from insights into home
networks, as no doubt eero has, but I've never been in a position
where ethically I could deal with this in our projects (nor, if I did,
could I cope with the potential liability). I have been grateful for
what bits of data google was willing to publish.

I would love to see a barracuda ship something for small to medium
business, or something of that caliber from a cisco

> stop trying for perfection, you will never be able to address all problems.
>
> But we already have several options that would address the very real
> problems people are having if they could just be deployed.
>
> you don't even need to use cake and worry about what your uplink bandwidth
> is if you can just use fq_codel on the real edge device. The fact that it is
> almost impossible to buy a router that has a DSL interface on it, so you
> have to try to artificially move the bottleneck is a problem.

I do regard the total lack of one bql-enabled dsl device firmware as a
real market disappointment.

Similarly, a lack of transparency into whatever is shipping in
multiple big ISP's gear - notably the FIOS folk are very good about
the GPL side of their stuff, but it's never been clear that they've
been paying attention - at least one FIOS device I tested had sfq on
outbound by default.

I have spent a bit of time on an upcoming cablemodem product that I
can't talk about but I am presently very frustrated with how it abuses
linux's network stack.
>
> Deploy what we already know to work on the real edge devices and things get
> vastly simpler.
>
> David Lang
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Bloat] fixing bufferbloat in 2017

[Pedretti Fabio]

The bufferbloat site has lot of informations, but many are outdated
(and also not very well organized/formatted). I'd suggest removing (or
clearly tagging as such) all obsolete infos (I would also consider
obsolete everything that was only needed before current Debian
stable).

I'd also like to have updated how to, for a linux router (on a generic
distro like Debian) and (if still needed) clients.

BQL for vmxnet3 (if possible). Virtual router are becoming common.

Thanks.

2016-11-22 16:32 GMT+01:00 Dave Taht <dave.taht@gmail.com>:
> What's left to do?
>
> What else can we do?
>
> What should we stop doing?
>
> What can we do better?
>
> --
> Dave Täht
> Let's go make home routers and wifi faster! With better software!
> http://blog.cerowrt.org
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] fixing bufferbloat in 2017

[Stephen Hemminger]

> BQL for vmxnet3 (if possible). Virtual router are becoming common.

BQL could be implemented in vmxnet3, but probably not in virtio.
virtio defers freeing packets to try and have better cache behavior

Re: [Bloat] fixing bufferbloat in 2017

[Dave Taht]

Can I encourage folk to think big and out of the technical box?

On Tue, Nov 22, 2016 at 7:32 AM, Dave Taht <dave.taht@gmail.com> wrote:
> What's left to do?
>
> What else can we do?
>
> What should we stop doing?
>
> What can we do better?
>
> --
> Dave Täht
> Let's go make home routers and wifi faster! With better software!
> http://blog.cerowrt.org



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org