[Cerowrt-devel] Perfection vs. Good Enough

[Cerowrt-devel] Perfection vs. Good Enough

[Rich Brown]

Folks,

I am so pleased with the state of CeroWrt. The software has improved enormously, to the point that we all get really good performance from our routers at home. If you want a real eyeful of the progress we’ve made, check list at the bottom of the Release Notes: http://www.bufferbloat.net/projects/cerowrt/wiki/CeroWrt_310_Release_Notes

CeroWrt is working great. We have two great testimonials for how it has improved network performance (from Fred Stratton and David Personnette, see  https://lists.bufferbloat.net/pipermail/cerowrt-devel/2014-January/001961.html and https://lists.bufferbloat.net/pipermail/cerowrt-devel/2014-January/001970.html)

I have been using 3.10.24-8 at home without hiccups (after I turned on SQM :-) since it was shipped. We’ve got a really great program.

But - I’m afraid we’re letting perfection be the enemy of the good. Here are a couple indications:

- The rest of the world doesn’t know about this good work. If you look at the front page of the site, we’re recommending CeroWrt 3.7.5-2 from last February. It has Codel, but not much more. Our understanding of the world has expanded by an order of magnitude, but we’re not making it available to anyone.

- The entire discussion of link layers has held us back. That’s why I proposed to cut back the choices to ATM and None, and let people figure out the details if they want to/have time to optimize.

- We have tons of updated modules (dnsmasq, IPv6, quagga, mosh) which we should get out to the world. 

- The entire product is much tighter, works better, and we can be proud of it. As Dave Täht pointed out in a recent note:

> Compared to the orders of magnitude we already get from fq codel, the sum benefit 
> of these [Link Layer Adaptation] fixes is in the very small percentage points.

This is true of the entire CeroWrt build.

Proposal:

We should “finish up the last bits” to make 3.10.24-8 (or a close derivative) be a stable release. It has been working fine AFAIK for lots and lots of us. It certainly has been as well tested as other branches. I see the following:

- Look through the release notes (very bottom of the page at the URL above) and review the items that Dave was worried about for the 3.10.24-8 release

- Make a decision on Link Layer Adaptation choices, and implement it.

- What else? 

Best,

Rich

Re: [Cerowrt-devel] Perfection vs. Good Enough

[Theodore Ts'o]

On Sat, Jan 11, 2014 at 11:31:59AM -0500, Rich Brown wrote:
> Folks,
> 
> I am so pleased with the state of CeroWrt. The software has improved
> enormously, to the point that we all get really good performance
> from our routers at home. If you want a real eyeful of the progress
> we’ve made, check list at the bottom of the Release Notes:
> http://www.bufferbloat.net/projects/cerowrt/wiki/CeroWrt_310_Release_Notes

I hate to do an AOL, but I was thinking exactly the same thing.  In
fact, just this morning I was wondering whether it would make sense to
start a stablization / maintenance branch if there was more
development work that people wanted to do.

					- Ted

Re: [Cerowrt-devel] Perfection vs. Good Enough

[Sebastian Moeller]

Hi Rich,

On Jan 11, 2014, at 17:31 , Rich Brown <richb.hanover@gmail.com> wrote:

> Folks,
> 
> I am so pleased with the state of CeroWrt. The software has improved enormously, to the point that we all get really good performance from our routers at home. If you want a real eyeful of the progress we’ve made, check list at the bottom of the Release Notes: http://www.bufferbloat.net/projects/cerowrt/wiki/CeroWrt_310_Release_Notes
> 
> CeroWrt is working great. We have two great testimonials for how it has improved network performance (from Fred Stratton and David Personnette, see  https://lists.bufferbloat.net/pipermail/cerowrt-devel/2014-January/001961.html and https://lists.bufferbloat.net/pipermail/cerowrt-devel/2014-January/001970.html)
> 
> I have been using 3.10.24-8 at home without hiccups (after I turned on SQM :-) since it was shipped. We’ve got a really great program.
> 
> But - I’m afraid we’re letting perfection be the enemy of the good. Here are a couple indications:
> 
> - The rest of the world doesn’t know about this good work. If you look at the front page of the site, we’re recommending CeroWrt 3.7.5-2 from last February. It has Codel, but not much more. Our understanding of the world has expanded by an order of magnitude, but we’re not making it available to anyone.
> 
> - The entire discussion of link layers has held us back. That’s why I proposed to cut back the choices to ATM and None, and let people figure out the details if they want to/have time to optimize.
> 
> - We have tons of updated modules (dnsmasq, IPv6, quagga, mosh) which we should get out to the world. 
> 
> - The entire product is much tighter, works better, and we can be proud of it. As Dave Täht pointed out in a recent note:
> 
>> Compared to the orders of magnitude we already get from fq codel, the sum benefit 
>> of these [Link Layer Adaptation] fixes is in the very small percentage points.

	I do not agree with this sentiment, as I understood Dave was talking about different modifications to fq_codel (nfq_codel and efq_codel), this was not about the link layer; for an ATM link if you get the link layer wrong the shaper does at best work stochastically; and if the shaper does not work well we are back at square one: badly managed buffers out of our control filling up causing delays worth seconds. So unless you shape down to ~50% of link rate, you will get at least temporary buffer bloat on an ATM link, unless you take all the ATM peculiarities into account (basically what link layer ATM is doing).

> 
> This is true of the entire CeroWrt build.
> 
> Proposal:
> 
> We should “finish up the last bits” to make 3.10.24-8 (or a close derivative) be a stable release. It has been working fine AFAIK for lots and lots of us. It certainly has been as well tested as other branches. I see the following:
> 
> - Look through the release notes (very bottom of the page at the URL above) and review the items that Dave was worried about for the 3.10.24-8 release
> 
> - Make a decision on Link Layer Adaptation choices, and implement it.

	It is quite clear to me, that I failed to explain the matters surrounding ATM links properly. But if I can not explain this to a small group of technical experts there is no chance for me to explain this to lay persons. I will try my best to contribute to the "more than you ever wanted to know about link layer adaptation" page.

Best Regards
	Sebastian

> 
> - What else? 
> 
> Best,
> 
> Rich
> 
> 
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] Perfection vs. Good Enough

[Rich Brown]

Hi Sebastian,

> 	It is quite clear to me, that I failed to explain the matters surrounding ATM links properly. But if I can not explain this to a small group of technical experts there is no chance for me to explain this to lay persons. I will try my best to contribute to the "more than you ever wanted to know about link layer adaptation" page.

Oh dear, please don’t think that. 

You really have done a great job describing the problems of ATM links. I have snagged all the relevant points from the Link Layer discussion, and plan to include them in the “more than you want to know…” page. Your points A-G describing link framing in https://lists.bufferbloat.net/pipermail/cerowrt-devel/2014-January/001963.html will be a major part of the discussion.

But the fact that we can see many ways to improve the software shouldn’t stop us from celebrating our immense success to date. One way to do this would be by shipping a stable version to the world to try. And to do that, we need to provide good enough instructions that’ll work for new people, and all the details for those who want to dig further.

On the other hand, I won’t say No to any help you want to provide. :-) I know you’re busy, so I’m happy to take a shot at it later this week. Thanks for all you’ve contributed to the conversation.

Rich

Re: [Cerowrt-devel] Perfection vs. Good Enough

[Sebastian Moeller]

Hi Rich,


On Jan 11, 2014, at 19:47 , Rich Brown <richb.hanover@gmail.com> wrote:

> Hi Sebastian,
> 
>> 	It is quite clear to me, that I failed to explain the matters surrounding ATM links properly. But if I can not explain this to a small group of technical experts there is no chance for me to explain this to lay persons. I will try my best to contribute to the "more than you ever wanted to know about link layer adaptation" page.
> 
> Oh dear, please don’t think that. 

	Oh, I am sure there are people with more insight, who could explain the complications away, just I am not one of those :) I think you strike a good balance in the description of being informative and improving the typical user's experience with frightening people away with too much detail. I am happy to help with the scary part on the details page...

> 
> You really have done a great job describing the problems of ATM links. I have snagged all the relevant points from the Link Layer discussion, and plan to include them in the “more than you want to know…” page. Your points A-G describing link framing in https://lists.bufferbloat.net/pipermail/cerowrt-devel/2014-January/001963.html will be a major part of the discussion.

	Note, I do not do networking for a living, so there will be inaccuracies in there (I tried my best though)

> 
> But the fact that we can see many ways to improve the software shouldn’t stop us from celebrating our immense success to date. One way to do this would be by shipping a stable version to the world to try. And to do that, we need to provide good enough instructions that’ll work for new people, and all the details for those who want to dig further.

	Oh, yes, and I think your wiki page gets this balance right.

> 
> On the other hand, I won’t say No to any help you want to provide. :-) I know you’re busy, so I’m happy to take a shot at it later this week. Thanks for all you’ve contributed to the conversation.
> 

Best
	Sebastian

> Rich

Re: [Cerowrt-devel] Perfection vs. Good Enough

[David Lang]

On Sat, 11 Jan 2014, Sebastian Moeller wrote:

>>> Compared to the orders of magnitude we already get from fq codel, the sum benefit
>>> of these [Link Layer Adaptation] fixes is in the very small percentage points.
>
> 	I do not agree with this sentiment, as I understood Dave was talking 
> about different modifications to fq_codel (nfq_codel and efq_codel), this was 
> not about the link layer; for an ATM link if you get the link layer wrong the 
> shaper does at best work stochastically; and if the shaper does not work well 
> we are back at square one: badly managed buffers out of our control filling up 
> causing delays worth seconds. So unless you shape down to ~50% of link rate, 
> you will get at least temporary buffer bloat on an ATM link, unless you take 
> all the ATM peculiarities into account (basically what link layer ATM is 
> doing).

the question boils down to

compared to stock firmware,

how much of a beneifit is openwrt trunk (and how risky)
how much of a benefit is cerowrt without the link-level tuning

how much additional benefit do we expect to get from the added work.

What we have now is a huge step forward compared to stock firmware, a 
significant portion of this benefit has been pushed upstream to openwrt.

but right now there is a lot of work to make things perfect, and as a result, 
people are stuck using stock firmware or openwrt releases (unless they compile 
their own image from trunk), and so they are missing everything that's been 
done.

There is a lot of value in getting another release out that brings all the gains 
that we have now to everyone. Then development can continue trying to optimize 
things more.

David Lang

Re: [Cerowrt-devel] Perfection vs. Good Enough

[Theodore Ts'o]

On Sun, Jan 12, 2014 at 04:10:17PM -0800, David Lang wrote:
> 
> the question boils down to
> 
> compared to stock firmware,
> 
> how much of a beneifit is openwrt trunk (and how risky)

"how risky" is I think the biggest question.  You can always make
things better.  But it seems pretty clear the current version is
better than the last stable version, in terms of benefits, right?

So the question is, are people confident that it is more reliable, and
handles various corner cases that users might have in their homes,
better than the last stable version?  If it is, then we should do a
stable release.

Something that might be worth considering is something akin to Debian
"testing".  If a development release has gone for more than a week
without having any release critical bugs filed against it, promote it
to testing.  There will be people like me who aren't willing to run
the development branch on their home router, especially if so often it
ends up taking down their home infrastructure completely.

As a result, I'm currently stuck on a stable release which is almost a
year old at this point.  I'm not willing to take a development
release, because of the potential of instabilities, but I *am* willing
to try out a testing release --- especially if I'm given reassurances
that if I take a backup snapshot of my config files, I will be able to
roll back to the last stable release, or a previous testing release,
and basic stuff such as firewall rules and DHCP MAC address to IP
address static assignments will be preserved.

As long as people understand what risks and shortcomings are with
respect to rolling forward from a previous stable or testing release,
to a newer testing release, and what the shortcomings are with rolling
back to an earlier release while preserving their configuration
information, I suspect you'll find a much larger pool of people who
are willing to test.

Or if I know that I'm going to have to type in all of my configuration
in from scratch, that's might also be fine --- then I'll know to wait
until have an hour or two, just in case I have to keypunch in all of
the Cerowrt config by hand.  (I know about the backup function; I just
don't know how reliable it is across different versions.)  Right now,
one of the reasons why I havne't gone to the development release is
because I don't know what the worst case situation will be if
everything goes south and my home environment goes down for the count.

Regards,

						- Ted

Re: [Cerowrt-devel] Perfection vs. Good Enough

[Michael Richardson]

Theodore Ts'o <tytso@mit.edu> wrote:
    > I take a backup snapshot of my config files, I will be able to roll
    > back to the last stable release, or a previous testing release, and
    > basic stuff such as firewall rules and DHCP MAC address to IP address
    > static assignments will be preserved.

I was thinking as I read this, that perhaps I need to write a patch to
include dynamic DHCP assignments in the config backup as well.  I don't think
they survived.

I like to upgrade by swapping hardware, but I noticed some systems got a
different address afterwards... it's okay, all non-techies use names...

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        | network architect  [
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [

Re: [Cerowrt-devel] Perfection vs. Good Enough

[David Lang]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2643 bytes --]

On Sat, 11 Jan 2014, Rich Brown wrote:

> Folks,
>
> I am so pleased with the state of CeroWrt. The software has improved enormously, to the point that we all get really good performance from our routers at home. If you want a real eyeful of the progress we’ve made, check list at the bottom of the Release Notes: http://www.bufferbloat.net/projects/cerowrt/wiki/CeroWrt_310_Release_Notes
>
> CeroWrt is working great. We have two great testimonials for how it has improved network performance (from Fred Stratton and David Personnette, see  https://lists.bufferbloat.net/pipermail/cerowrt-devel/2014-January/001961.html and https://lists.bufferbloat.net/pipermail/cerowrt-devel/2014-January/001970.html)
>
> I have been using 3.10.24-8 at home without hiccups (after I turned on SQM :-) since it was shipped. We’ve got a really great program.
>
> But - I’m afraid we’re letting perfection be the enemy of the good. Here are a couple indications:

Strong agreement from me as well.

David Lang


> - The rest of the world doesn’t know about this good work. If you look at the front page of the site, we’re recommending CeroWrt 3.7.5-2 from last February. It has Codel, but not much more. Our understanding of the world has expanded by an order of magnitude, but we’re not making it available to anyone.
>
> - The entire discussion of link layers has held us back. That’s why I proposed to cut back the choices to ATM and None, and let people figure out the details if they want to/have time to optimize.
>
> - We have tons of updated modules (dnsmasq, IPv6, quagga, mosh) which we should get out to the world.
>
> - The entire product is much tighter, works better, and we can be proud of it. As Dave Täht pointed out in a recent note:
>
>> Compared to the orders of magnitude we already get from fq codel, the sum benefit
>> of these [Link Layer Adaptation] fixes is in the very small percentage points.
>
> This is true of the entire CeroWrt build.
>
> Proposal:
>
> We should “finish up the last bits” to make 3.10.24-8 (or a close derivative) be a stable release. It has been working fine AFAIK for lots and lots of us. It certainly has been as well tested as other branches. I see the following:
>
> - Look through the release notes (very bottom of the page at the URL above) and review the items that Dave was worried about for the 3.10.24-8 release
>
> - Make a decision on Link Layer Adaptation choices, and implement it.
>
> - What else?
>
> Best,
>
> Rich
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>