[Cerowrt-devel] Will there ever be a stable version of cerowrt?

[Cerowrt-devel] Will there ever be a stable version of cerowrt?

[Stephen Hemminger]

Any progress on getting a stable version? Recent versions seem to be targeted
at IPv6 and other things which do not impact me for home use. What does matter
is fixing the wireless hanging issue on 2.4Ghz.

Re: [Cerowrt-devel] Will there ever be a stable version of cerowrt?

[Jim Gettys]

[-- Attachment #1: Type: text/plain, Size: 2629 bytes --]

On Fri, Feb 7, 2014 at 12:01 PM, Stephen Hemminger <
stephen@networkplumber.org> wrote:

> Any progress on getting a stable version? Recent versions seem to be
> targeted
> at IPv6 and other things which do not impact me for home use. What does
> matter
> is fixing the wireless hanging issue on 2.4Ghz.
>

Dave's (and I) are pedalling as hard and fast as he can at  the moment.
 And Dave has other responsibilitys that consume time.

The IPv6 problem consumed a lot of time, and you do care.

Comcast has lit up a big chunk of their network with Native IPv6 and
tickled a latent disastrous bug (now fixed upstream in OpenWrt). That we
simultaneously had a disk failure on our machine that does much of
OpenWrt's and CeroWrt's builds is more than unfortunate and so we end up
spending  our time on trying to get the build situation fixed once and for
all (by getting machines and people to set up and manage the build system
going forward at a scale previously unimaginable to OpenWrt/CeroWrt).

And yes, we're making progress on all fronts.  2 juicy servers are teed up
awaiting a single signature (thanks Mikael!) and I have a strong lead on
both large amounts of additional cloud cycles and volunteers to help admin
all this so it doesn't happen again and the build system is as fast as it
should be (continuous across all supported architectures)

It would be very help if this mailing list could at least nominate bugs
that they think are showstoppers before declaring it "soup"....

But please don't jiggle the chef (Dave's) arm expecting immediate involved
response in that discussion while sorting through it; it will reduce the
time to see what if anything remains.

We keep finding more and more people running CeroWrt (not subscribed to
this list necessarily) we had no clue were doing so.

I know I have one test I have to run on the first build he can give me (one
Mike Frankston ran into) that might be in that category if I can easily
reproduce it, as I think I ran into it myself recently.  It would cause a
support nightmare as it's a "new to CeroWrt" thing triggered from an
obvious thing you want and often need to configure.

But we gotta be able to build it, close out those show stoppers and test
the fixes, and get that build installed on enough boxes in different
environments for Dave send out an announcement that people should try an
upgrade.

We know it has to be ASAP.

                                                - Jim


> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>

[-- Attachment #2: Type: text/html, Size: 4716 bytes --]

Re: [Cerowrt-devel] Will there ever be a stable version of cerowrt?

[Dave Taht]

On Fri, Feb 7, 2014 at 12:01 PM, Stephen Hemminger
<stephen@networkplumber.org> wrote:
> Any progress on getting a stable version? Recent versions seem to be targeted
> at IPv6 and other things which do not impact me for home use. What does matter
> is fixing the wireless hanging issue on 2.4Ghz.

Do I detect a plainative note? I share it...

I'm not aware of any problems on 2.4ghz in the last specific release I
did for comcast. It's been in day-to-day use here since I put it out.

http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/comcast/3.10.28-4/

Feel free to give it a shot. The stuff that made it comcast specific
(co-existence with  HE tunnels) is solved, so the next gen will be
generic.

I AM painfully aware that ht40+ at 5ghz appears to have broken on
several channels (Seems to work on 36 and fail on 44) There was a
fresh merge from wireless-testing recently, that I hope fixes it.

I DO plan a stable release this month if possible, and at the very
least a code freeze fairly soon.

in cerowrt head are fixes for ipv6 tunnel co-existence, mostly
everything now using procd, fixed upnp support (think that landed in
the release above), cleanups of the SQM code for eventual submittal to
mainline landed also.

*Please note* that a source of hangs in the last several cero releases
was an oopsy in turning off logging for ahcp and babeld, which filled
up available memory rapidly. EVERYBODY: Do turn send those logs to
/dev/null if you haven't already in the relevant /etc/config/ files
and look for other sources of memory growth from logs.

the remaining problems are:

got competing work requirements terribly pressing, final pie release
needs to be added, (kind of wanted sch_fq and the other new qdisc from
3.13, too) a bunch of ietf drafts are in the nascent stage and stuck
there, a build server crashed and a whole string of openwrt builds
failed to build, still need /etc/init.d/babeld support for source
specific routing and babel to run out of procd, package signing is
broken, and worst of the cerowrt build broke last week due to some
changes in the openwrt build process I still don't understand.

There is still one unaligned_instruction trap left to beat that
appears to be triggered by odhcpd.

root@comcast-gw:~# cat /sys/kernel/debug/mips/unaligned_instructions
18000

One of the things that scares me is that a bunch of openwrt targets
are migrating to 3.13 and if I miss a window  the very stable 3.10.28
release is going to get replaced upstream with 3.13 and resultant
headaches. (and benefits)

As my own deadlines slip everywhere, I made a big push earlier this
week to appeal to many folk to get dnsmasq + dnssec more widely tested
on other platforms in the hope that whenever I got unburied it would
be stable enough to toss into cero.  (that's going along swimmingly on
x86 and arm so far)

I hope to replace the failed server this tonight and burn sunday on
trying to resolve the problems above. The currently borked srcs are on
github.


-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] Will there ever be a stable version of cerowrt?

[Sebastian Moeller]

Hi Dave,


On Feb 7, 2014, at 18:57 , Dave Taht <dave.taht@gmail.com> wrote:

> On Fri, Feb 7, 2014 at 12:01 PM, Stephen Hemminger
> <stephen@networkplumber.org> wrote:
>> Any progress on getting a stable version? Recent versions seem to be targeted
>> at IPv6 and other things which do not impact me for home use. What does matter
>> is fixing the wireless hanging issue on 2.4Ghz.
> 
> Do I detect a plainative note? I share it...
> 
> I'm not aware of any problems on 2.4ghz in the last specific release I
> did for comcast. It's been in day-to-day use here since I put it out.
> 
> http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/comcast/3.10.28-4/
> 
> Feel free to give it a shot. The stuff that made it comcast specific
> (co-existence with  HE tunnels) is solved, so the next gen will be
> generic.
> 
> I AM painfully aware that ht40+ at 5ghz appears to have broken on
> several channels (Seems to work on 36 and fail on 44) There was a
> fresh merge from wireless-testing recently, that I hope fixes it.

	With ht40+ on 44 I get MCS index 7 with transmigrate 150 reported on my macbook, that according to the table in http://en.wikipedia.org/wiki/IEEE_802.11n-2009 means 40MHz channel, with netperf-wrapper I get an accumulate of ~100Mbits/s (which actually sounds too good to be true I would have expected 75 tops, but with the macbook hogging all tops, there seems to be less wasted airtime for direction changes). So at least with country code GE channel 44 is capable of operating in ht40+ mode.

> 
> I DO plan a stable release this month if possible, and at the very
> least a code freeze fairly soon.
> 
> in cerowrt head are fixes for ipv6 tunnel co-existence, mostly
> everything now using procd, fixed upnp support (think that landed in
> the release above), cleanups of the SQM code for eventual submittal to
> mainline landed also.
> 
> *Please note* that a source of hangs in the last several cero releases
> was an oopsy in turning off logging for ahcp and babeld, which filled
> up available memory rapidly. EVERYBODY: Do turn send those logs to
> /dev/null if you haven't already in the relevant /etc/config/ files
> and look for other sources of memory growth from logs.
> 
> the remaining problems are:
> 
> got competing work requirements terribly pressing, final pie release
> needs to be added, (kind of wanted sch_fq and the other new qdisc from
> 3.13, too) a bunch of ietf drafts are in the nascent stage and stuck
> there, a build server crashed and a whole string of openwrt builds
> failed to build, still need /etc/init.d/babeld support for source
> specific routing and babel to run out of procd, package signing is
> broken, and worst of the cerowrt build broke last week due to some
> changes in the openwrt build process I still don't understand.
> 
> There is still one unaligned_instruction trap left to beat that
> appears to be triggered by odhcpd.
> 
> root@comcast-gw:~# cat /sys/kernel/debug/mips/unaligned_instructions
> 18000

	So with my non-working IP6 on 3.10.28-1, I get:
root@nacktmulle:~# cat /sys/kernel/debug/mips/unaligned_instructions
0
root@nacktmulle:~# uptime
 19:35:13 up 11 days, 22:46,  load average: 0.09, 0.43, 1.06

so most of the traps seem fixed!


> 
> One of the things that scares me is that a bunch of openwrt targets
> are migrating to 3.13 and if I miss a window  the very stable 3.10.28
> release is going to get replaced upstream with 3.13 and resultant
> headaches. (and benefits)

	Do yu know what their plan is? Put out a stable version an maintain that or switch to a rolling release cycle?


Best Regards & many Thanks for doing all this
	Sebastian

> 
> As my own deadlines slip everywhere, I made a big push earlier this
> week to appeal to many folk to get dnsmasq + dnssec more widely tested
> on other platforms in the hope that whenever I got unburied it would
> be stable enough to toss into cero.  (that's going along swimmingly on
> x86 and arm so far)
> 
> I hope to replace the failed server this tonight and burn sunday on
> trying to resolve the problems above. The currently borked srcs are on
> github.
> 
> 
> -- 
> Dave Täht
> 
> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] Will there ever be a stable version of cerowrt?

[Dave Taht]

On Fri, Feb 7, 2014 at 1:37 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
> Hi Dave,
>
>
> On Feb 7, 2014, at 18:57 , Dave Taht <dave.taht@gmail.com> wrote:
>
>> On Fri, Feb 7, 2014 at 12:01 PM, Stephen Hemminger
>> <stephen@networkplumber.org> wrote:
>>> Any progress on getting a stable version? Recent versions seem to be targeted
>>> at IPv6 and other things which do not impact me for home use. What does matter
>>> is fixing the wireless hanging issue on 2.4Ghz.
>>
>> Do I detect a plainative note? I share it...
>>
>> I'm not aware of any problems on 2.4ghz in the last specific release I
>> did for comcast. It's been in day-to-day use here since I put it out.
>>
>> http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/comcast/3.10.28-4/
>>
>> Feel free to give it a shot. The stuff that made it comcast specific
>> (co-existence with  HE tunnels) is solved, so the next gen will be
>> generic.
>>
>> I AM painfully aware that ht40+ at 5ghz appears to have broken on
>> several channels (Seems to work on 36 and fail on 44) There was a
>> fresh merge from wireless-testing recently, that I hope fixes it.
>
>         With ht40+ on 44 I get MCS index 7 with transmigrate 150 reported on my macbook, that according to the table in http://en.wikipedia.org/wiki/IEEE_802.11n-2009 means 40MHz channel, with netperf-wrapper I get an accumulate of ~100Mbits/s (which actually sounds too good to be true I would have expected 75 tops, but with the macbook hogging all tops, there seems to be less wasted airtime for direction changes). So at least with country code GE channel 44 is capable of operating in ht40+ mode.

What release exactly? I confess to have merely seen the bug on a release
or two back.

You should get MCS15, btw.

>> There is still one unaligned_instruction trap left to beat that
>> appears to be triggered by odhcpd.
>>
>> root@comcast-gw:~# cat /sys/kernel/debug/mips/unaligned_instructions
>> 18000
>
>         So with my non-working IP6 on 3.10.28-1, I get:
> root@nacktmulle:~# cat /sys/kernel/debug/mips/unaligned_instructions
> 0
> root@nacktmulle:~# uptime
>  19:35:13 up 11 days, 22:46,  load average: 0.09, 0.43, 1.06

No, not fixed, it's triggered by ipv6 traffic.

>
> so most of the traps seem fixed!
>
>
>>
>> One of the things that scares me is that a bunch of openwrt targets
>> are migrating to 3.13 and if I miss a window  the very stable 3.10.28
>> release is going to get replaced upstream with 3.13 and resultant
>> headaches. (and benefits)
>
>         Do yu know what their plan is? Put out a stable version an maintain that or switch to a rolling release cycle?

I have had few detailed conversations with the #openwrt-devel folk of
late. I have been ignoring irc in favor of doing work.

There is a ton of good stuff landing in openwrt, let me forward
something in a second...

>
>
> Best Regards & many Thanks for doing all this
>         Sebastian
>
>>
>> As my own deadlines slip everywhere, I made a big push earlier this
>> week to appeal to many folk to get dnsmasq + dnssec more widely tested
>> on other platforms in the hope that whenever I got unburied it would
>> be stable enough to toss into cero.  (that's going along swimmingly on
>> x86 and arm so far)
>>
>> I hope to replace the failed server this tonight and burn sunday on
>> trying to resolve the problems above. The currently borked srcs are on
>> github.
>>
>>
>> --
>> Dave Täht
>>
>> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] Will there ever be a stable version of cerowrt?

[Sebastian Moeller]

Hi Dave,

On Feb 7, 2014, at 19:43 , Dave Taht <dave.taht@gmail.com> wrote:

> On Fri, Feb 7, 2014 at 1:37 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
>> Hi Dave,
>> 
>> 
>> On Feb 7, 2014, at 18:57 , Dave Taht <dave.taht@gmail.com> wrote:
>> 
>>> On Fri, Feb 7, 2014 at 12:01 PM, Stephen Hemminger
>>> <stephen@networkplumber.org> wrote:
>>>> Any progress on getting a stable version? Recent versions seem to be targeted
>>>> at IPv6 and other things which do not impact me for home use. What does matter
>>>> is fixing the wireless hanging issue on 2.4Ghz.
>>> 
>>> Do I detect a plainative note? I share it...
>>> 
>>> I'm not aware of any problems on 2.4ghz in the last specific release I
>>> did for comcast. It's been in day-to-day use here since I put it out.
>>> 
>>> http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/comcast/3.10.28-4/
>>> 
>>> Feel free to give it a shot. The stuff that made it comcast specific
>>> (co-existence with  HE tunnels) is solved, so the next gen will be
>>> generic.
>>> 
>>> I AM painfully aware that ht40+ at 5ghz appears to have broken on
>>> several channels (Seems to work on 36 and fail on 44) There was a
>>> fresh merge from wireless-testing recently, that I hope fixes it.
>> 
>>        With ht40+ on 44 I get MCS index 7 with transmigrate 150 reported on my macbook, that according to the table in http://en.wikipedia.org/wiki/IEEE_802.11n-2009 means 40MHz channel, with netperf-wrapper I get an accumulate of ~100Mbits/s (which actually sounds too good to be true I would have expected 75 tops, but with the macbook hogging all tops, there seems to be less wasted airtime for direction changes). So at least with country code GE channel 44 is capable of operating in ht40+ mode.
> 
> What release exactly? I confess to have merely seen the bug on a release
> or two back.

	Oh 3.10.28-1, my working place and the router position are badly matched, so I only get 80 to 150 mbits/s there, but it is nice to cover the full flat with one router, so I will have to live with that. Question to all, I have set up my WNDR3700 on the vertical stand on a shelf above my head, is that the ideal position or has anyone other recommendations how to rotate the router? (The overhead position is a given, since I want this to be safely out of reach of my two toddlers).

> 
> You should get MCS15, btw.

	On the occasional day I get a bit more, also dependent on how the other people in the apartment are located ;)


> 
>>> There is still one unaligned_instruction trap left to beat that
>>> appears to be triggered by odhcpd.
>>> 
>>> root@comcast-gw:~# cat /sys/kernel/debug/mips/unaligned_instructions
>>> 18000
>> 
>>        So with my non-working IP6 on 3.10.28-1, I get:
>> root@nacktmulle:~# cat /sys/kernel/debug/mips/unaligned_instructions
>> 0
>> root@nacktmulle:~# uptime
>> 19:35:13 up 11 days, 22:46,  load average: 0.09, 0.43, 1.06
> 
> No, not fixed, it's triggered by ipv6 traffic.

	Ah, okay I will then dart looking at those again once I can exercise odhcp with ip6 traffic...

> 
>> 
>> so most of the traps seem fixed!
>> 
>> 
>>> 
>>> One of the things that scares me is that a bunch of openwrt targets
>>> are migrating to 3.13 and if I miss a window  the very stable 3.10.28
>>> release is going to get replaced upstream with 3.13 and resultant
>>> headaches. (and benefits)
>> 
>>        Do yu know what their plan is? Put out a stable version an maintain that or switch to a rolling release cycle?
> 
> I have had few detailed conversations with the #openwrt-devel folk of
> late. I have been ignoring irc in favor of doing work.
> 
> There is a ton of good stuff landing in openwrt, let me forward
> something in a second…

	Much appreciated.

Best Regards
	Sebastian

> 
>> 
>> 
>> Best Regards & many Thanks for doing all this
>>        Sebastian
>> 
>>> 
>>> As my own deadlines slip everywhere, I made a big push earlier this
>>> week to appeal to many folk to get dnsmasq + dnssec more widely tested
>>> on other platforms in the hope that whenever I got unburied it would
>>> be stable enough to toss into cero.  (that's going along swimmingly on
>>> x86 and arm so far)
>>> 
>>> I hope to replace the failed server this tonight and burn sunday on
>>> trying to resolve the problems above. The currently borked srcs are on
>>> github.
>>> 
>>> 
>>> --
>>> Dave Täht
>>> 
>>> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html
>>> _______________________________________________
>>> Cerowrt-devel mailing list
>>> Cerowrt-devel@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> 
> 
> 
> 
> -- 
> Dave Täht
> 
> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html