[Cerowrt-devel] notes on going for a stable release

[Dave Taht]

I am in strong agreement that cerowrt is close to being ready for a
stable release.

However there are problems...

* Gating factors
** Sync with opewrt's release schedule

I have not been tracking what openwrt's plan is for releasing a stable
version of "Barrier Breaker". Attitude Adjustment (AA) got stalled on
finding a stable maintainer, and took a really long time to become
stable after that.

** Native IPv6 and dhcpv6-pd support
I am very happy with comcast's huge rollout of ipv6 across their
network. (over 25% of their base now) http://www.comcast6.net .

Hearing that cero isn't working on comcast anymore bugs me. I don't
seem to have ipv6 on my directly controlled comcast nodes. Yet.

Setting up a dhcpv6-pd server and some testing is required to make
sure it isn't cerowrt that's busted. I'd like to not go another year
without ipv6.

** Instruction traps

The instruction trap problem has resurfaced on boot. It is unknown
what triggers it. It doesn't happen very much after boot in my limited
testing.

The last time it bit me was on doing tests on a busy ipv6-enabled
network where it thoroughly blew up the tests. (even when not doing
ipv6 itself) It also made cerowrt unreliable.

oot at davedesk:~# cd /sys/kernel/debug/mips/
root at davedesk:/sys/kernel/debug/mips# cat unaligned_instructions
7884

What values do you see, both on boot and after some uptime?

For more details on how to actually fix the bug:
http://www.bufferbloat.net/issues/419
** IPv6 vs THC

Go blow up ipv6 some more, please, and look at instruction traps...

and run stuff from this

https://www.thc.org/download.php?t=r&f=thc-ipv6-2.5.tar.gz

** src/dst routing via babels

   In the last (3.10.24 dev release I switched to babels from quagga.
Either nobody but me uses babel (?), or it "just worked".  That said,
the whole point of doing that was to be able to test multiple exit
nodes with tcp and mptcp (ipv4 and ipv6) and packet encapsulations
(6rd, native, 6in4)... and I haven't got round2it.

** Random number testing

I forget when I slammed in ralf's improved mips random number stuff,
but it's in cero, and should be verified if it is working properly
before being pushed to openwrt or shipped. Ted's other random stuff
was already picked up by openwrt.

** Refresh of some test packages

  shaperprobe and uftp4 need an update in particular. I havent been
tracking what else needs an update.

* DEFER to next release cycle thoughts
** IWL crash

The iwl (http://www.iwl.com) test suite caused a sporadic reboot of
cerowrt back in november. as I haven't been able to reproduce it, I
can let this pass.  (I did witness the semi-repeatable crash with my
own two eyes)

** make-wifi-fast
   There is a huge amount of work that can now be done to improve wifi
   performance. This will be the year of make-wifi-fast. I would very much
   like to declare a 3.10.X cerowrt stable and go off and work in x86
and arm land for
   a while to hack at 802.11ac on the ath10k and 802.11n ath9k.
Perhaps "make-wifi-fast" would be something that sells to funders
better than "fixing bufferbloat" or "fixing security problems".

** dnsmasq + dnssec support
   dnsmasq 2.69test3 has sortof working dnssec support, and I'd like
   to start testing that soon.

* Hot topics

** What is a stable release?

To me a "stable release" is something that has been extensively
tested, benchmarked, and will have a series of updates and security
fixes for 1-2 years. It has a maintainer, a bug database, a means for
dealing with major security issues, and so on.

** What is CeroWrt?

Originally intended to prove out a bunch of AQM and scheduling ideas,
it's done that. We proved dnssec was feasible, and simon kelly is
doing that. ISC and openwrt got signed updates working recently, the
only major update-in-the-field problem for openwrt is on updating
kernels.

CeroWrt is ALSO useful for day-to-day use, presently.

** CeroWrt could use a non-profit foundation
   although we have achieved stable hosting with ISC for the next
year, we've been unable to find a permanent "home" for the project, or
funding for it or bufferbloat.net.

** CeroWrt needs a new maintainer
   After 2 years of doing this my interest in solving "cross
compilation problem of the week", has declined considerably. I burn
out frequently, and only recover when I'm driven by the contributions
of you, the cerowrt-devel folk. I was delighted by the increase in
interest and in the new stuff that arrived from everybody on the sqm
front over the last couple months - but my own motivation was in
seeing sqm-scripts and the gui pushed up to openwrt, not so much on
making Cero usable by your mom.

Right now I try to dedicate only my sundays and early monday mornings
(many openwrt contributions land on sundays in the wee hours on German
time) to it now. In the early days it was 4-5 days a week, in addition
to running bufferbloat.net. Obviously progress has slowed as I have
slowed. It would get worse if I also had to maintain a stable release.

I strongly believe in the CI cycle that cerowrt does, at least once a
week, integrating changes from upstream and at the very least, compile
testing. This nips small problems in the bud before they become big
ones.

My interest in maintaining a "stable" release as well as continuing
development is slim.
I would like my sundays back. I would like to be able to work on 5
rfcs, some new code replacing htb, better analysis and qa tools, and a
couple papers... and also my day job is very different from
maintaining cero, and that is what is putting food on the table now.

Without stable funding, a non-burnt-out maintainer, and a non-profit
setup to manage the org, I don't know what the future could hold for
cero as an independent OS. Certainly I feel the pains of you, bruce,
the ISPs and vendors for the costs of continual maintenance and
security vigilance:

https://plus.google.com/u/0/+JimGettys/posts/SprUcpmDa1W

But who should pay to keep the internet edge working right? I don't
think it can be done by volunteers. What would happen if the NYT
covered cero and there were 10,000 new users all at once?

at the current ~$140/mo in donations cerowrt and bufferbloat.net are
not viable entities. I've been working with a new nonprofit that
daydreams of using kickstarter campaigns to get stuff done, but I
think the appeal of running a kickstarter to pay for a maintainer for
a year is limited.

Debian had a LOT more users than cero ever will before it got to where
it had a group np running it.

By all means, we should do a stable release soon, but what happens after?

** The wndr3800 is obsolete and fixing the next generation soon would be good

the chipset it is based on keeps going strong, and so far we've been
able to find versions of it on amazon pretty regularly.

But this past christmas everything was 802.11ac, running on arm. The
ath10k is out and getting some love, too.

** Finishing the new SQM stuff
*** emulate bfifo DSLAM, CMTS
I have code for this just never have pushed it out
*** emulate busted wondershaper
I was annoyed enough at wondershaper to write a newer version of it
*** Push to openwrt of SQM and other patches

in order to replace the aqm-scripts it need to do packet classification better.

* Fit and finish issues that would be good to have done before a stable release

** BCP38 compliance
Cerowrt does not currently stop unknown rfc1918 addresses from going out ge00.
** Squash incoming diffserv bits
many providers pee on the diffserv bits. It would be good to detect it
and reset to BE incoming packets. (note: IPv6 is far less peed on.).
There was a nice idea discussed last year on using conntrack to match
incoming with outgoing diffserv bits.

** SSL support for the configuration interfaces
All the plumbing exists for this in cero, it just has to be made to
work. the key generation routine needs to be fixed in uci-defaults and
lighttpd config updated. It's embarrassing to not have SSL running.

** On board documentation updates
still has a lot of obsolete information on it.
I just updated the credits file.

* Bufferbloat.net problems
the bufferbloat.net servers are undermaintained and obsolete. I long
ago swapped out my sysadmin and ruby skills for other things.

** huchra replacement (one disk currently crashed, the other going)
In addition to running this mailing list this used to be 1/5th of the
openwrt build cluster.

lists needs to move to a virtual server ASAP.

openwrt could really use a good build cluster. been running most of
theirs now for a couple years, out of machines pulled from the junk
bin.

** Web Site updates
   the redmine implementation on bufferbloat.net has been overrrun by
spam and I stopped
   accepting new contributors that didn't contact me also via email
   long ago.

   given how hard it would be to update the present website, perhaps
moving to cerowrt.org
   on a virtual server will be simpler.


-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html