[Cerowrt-devel] CeroWrt Releases from 3.7.5-2 to current

Richard E. Brown richb.hanover at gmail.com
Fri Nov 1 19:13:58 EDT 2013 

In preparation for a strong beta release (I can feel one brewing), I decided to go through the various "3.x.x released” notes on the cerowrt-devel list and collate the info. I plan to use this to create a new Release Notes page for 3.10 that follows the model of http://www.bufferbloat.net/projects/cerowrt/wiki/CeroWrt_37_Release_Notes

I’m not sure that I’ve captured all the goodness that was accomplished, so I’d love to have more eyeballs on (and admiration of) the work we’ve done over the last 9 months. See the list below - it’s pretty impressive. 

Please let me know about anything else that’s significant, and of course, correct assertions that simply aren’t true… :-) I’ll collate the first round of responses via responses to the list, and post the resulting document to the CeroWrt wiki for further refinement. 

Best,

Rich

======= Draft of Intro stuff for new Release Notes for 3.10.x ========

Features:

The CeroWrt 3.10 series of builds include the following features and capabilities:

* High performance routing in an inexpensive “home” router. (What stats can we brag about?)
* A major improvement to the problem of bufferbloat. VoIP, Skype, gaming, and other latency-sensitive applications continue to work well even during heavy up/download.
* IPv6 support. Another major goal of CeroWrt is to make IPv6 networking in the home as simple as IPv4.
* Linux 3.10.x kernel. Many of the fixes for bufferbloat have been implemented in mainline Linux. This means that bufferbloat is improving for the rest of the world. http://kernel.org
* The "CoDel":http://www.bufferbloat.net/projects/codel/wiki algorithm from Kathie Nichols and Van Jacobson along with Eric Dumazet's adaptation of Fair Queueing (fq_codel) on top. These in turn rely on the Byte Queue Limits that were implemented in the Linux 3.3 kernel. These techniques replace earlier Active Queue Management fixes for bufferbloat including: Stochastic Fair Queueing-Random Early Drop (SFQRED), but for comparison previous technologies such as SFQ and RED continue to be included. WHAT SHOULD BE SAID HERE?
* Test releases of Cisco’s PIE active queue management code. http://tools.ietf.org/html/draft-pan-aqm-pie-00 and ftp://ftpeng.cisco.com/pie/documents/pie_hpsr2013_final.pdf WHAT’S TRUE HERE?
* Babel mesh routing protocol (Quagga-babeld). Other protocols such as ra, ospf, and bgp are also available. WHAT’S TRUE HERE?
* DNS, DNSSEC  - WHAT’S TRUE HERE?
* More Entropy! (Is this true?)
* OpenWrt features with the attractive LuCI web GUI for configuration. We track the OpenWrt development code base (“Barrier Breaker”) and incorporate the capabilities of that distribution. We actively push our changes/enhancements back toward the OpenWrt trunk. http://openwrt.org and http://wiki.openwrt.org/doc/howto/luci.essentials
* CeroWrt has the broad set of useful packages built-in or optionally loaded. See the list of Major Packages below.
* WHAT ELSE?

CeroWrt remains a vehicle for research around many aspects of networking, both in SOHO and high-performance settings. But if you just want to use it (and we think you should), here's a _link to simple installation and configuration instructions_

What has Changed since 3.7.5-2:

* Cerowrt defaults to fq_codel/sfq_codel/something else? on all interfaces/some interfaces/?
* Linux 3.10 kernel which has incorporated many fixes to bufferbloat, as well as finding many long-standing errors in the TCP/IP stack.
* A GUI for setting Active Queue Management (AQM) parameters for slower links
* Updates for:
       - babel/mesh networking;
       - IPv6 (native, 6in4, 6to4, etc);
       - DNS & DNSSEC;
       - mDNS;
       - nftables;
       - ipv6 nat :-(
* Incorporates mosh ssh replacement
* Much work to support the current dnsmasq for both DNS naming as well as IPv4/IPv6 address assignment
* Deep scrutiny of the entire linux networking stack has identified a number of errors which are fixed in CeroWrt and also pushed back into the Linux kernel, including TSO handling; improvements of RTT computations; fixed many unaligned access traps in the IPv6 code;
* Incorporates work to improve the entropy for /dev/random and get_cycles()
* Firewall improvements; block external access to SNMP (port 161) by default; uses pattern matching syntax to simply/decrease number of filter rules.
* Includes recent Cisco PIE queue discipline for comparison with fq_codel
* Signed packages
* WHAT ELSE?

Major Packages distributed with CeroWrt:

[What packages should be mentioned?]

======= END OF Draft of Intro stuff of new Release Notes for 3.10.x ========


========= The following appeared in “3.x.x … released” messages on the cerowrt-devel list =======

---------------------------------------
3.7.5-2 - 3 Feb 2013

Previous stable "Modena" release

---------------------------------------
3.8.6-2 - 7 Apr 2013

Up to Openwrt head
** DONE update to dnsmasq 2.66rc4
** DONE update iptables
 But is there npt66 support?
** DONE fix igmp patch
** DONE update quagga, netperf,
** TODO babel refresh
** DONE Change name to berlin
** DONE Fix kernel config for additional TCPs

---------------------------------------
3.8.6-3 - 10 Apr 2013

This has a merge from openwrt from over the weekend (fixes to qos-scripts, some ipv6 gui support, I forget what else)

also the requested mtr package is built and available via opkg.
the openvpn gui didn't build.

---------------------------------------
3.8.8-4 - 24 Apr 2013

+ Refresh to openwrt barrier breaker head

 this now contains nearly all the patches formerly separately in cerowrt!

 ++ fq_codel is on by default on ALL interfaces with default quantum of 300
      (yes, openwrt has obsoleted pfifo_fast!)
 ++ unaligned access patches, etc, etc
 + dhcp-pd SERVER support
the usual multitude of other openwrt fixes... all tested extensively
at the battlemesh conference.

+ Update to dnsmasq 2.67test2

Toke got really busy in building his own version of cero and adding

+ AQM scripts and gui
+ tahoe-lafs added (untested)
+ uftp4 updated

- no upnp/ssdp fix because I'm clueless

---------------------------------------
3.8.13-3 - 18 May 2013

Very much a development release - I want to clearly note that I can crash the router over wifi using the rrul test easily. I can (furthermore) crash the x86 linux-3.9.2 iwl driver on my laptop even more easier than I can crash the router. The combination of the two problems are making debugging impossible.

So... pretty please... with sugar on top... don't install this on your default gw?

If on the other hand, you have a jtag debugger handy, and don't have a iwl card on your laptop, and can look into the wifi issues, please do so... (all you have to do is bump up /etc/xinetd.d/netserver to 16 and run the netperf-wrapper against it for a few minutes)

There are otherwise a huge number of interesting things that have accumulated for this release cycle.

I was very happy that most of what was in Modena has landed in openwrt and the mainline linux kernels last month. Relieved, actually. I felt that I could take a break... even thought I could quit... spent a few days on a beach in Morocco and got bored to death... so....

The BIG new thing in this release is a version CISCO's PIE AQM algorithm, which after nearly a year of development and analysis was released as open source last week. The version of pie I just put in cero has not been fully verified to be correct, but has the additional features of ECN and TSQ support over the original. I hope to bake this a lot more over the coming week. (the wifi issue is annoying but secondary at the moment to finally! finally! fiddling with PIE)

There was the usual huge resync with openwrt. dslite landed recently in particular, but there have just been a huge number of updates across the board that I've lost track of. FW3 for example, is a fast, in-c replacement for the old firewall scripts, and openwrt is now using multi-table support in preparation for handling src/dst routing better.

Toke contributed tahoe-lafs and suggested trying out the tinc vpn system, so those are available as an optional package. tinc is kind of neat. a meshy vpn system. Never heard of it before now.

Toke also has been a great help elsewhere, notably in getting a gui and scripts going for the backend AQM system, working on a new build script to make it easier for others to build cero, and lots, lots more.
Rich Brown & Toke updated the onboard documentation significantly
Electra convinced me to make batman-adv available (but not enabled) by default
Babeld 1.4 has a new convergence smoothing algorithm (but quagga-babeld is still the default)
OpenWrt's QOS web page and backend scripts have been replaced by the new AQM page
The AQM scripts are now correct for EF and ECN.
fq_codel is now the default on everything with a quantum of 300

---------------------------------------
3.8.13-7 - 12 June 2013

I've had it up and running a few days on a couple routers,

and yes, I'm still trying to take some time off but:

+ can't crash it over wifi anymore
+ AQM + gui is coming along, am looking at gargoyle's methods a bit now...

- Known bug: 6in4 does not work via the gui or openwrt config file - this bug has existed for about a month now
and I haven't looked into it. I did look into fixing fq_codel performance under 6in4, and that patch is in here,
so after a bit more testing I'll try to get that upstream...

- the results I get from 802.11e are even more dismal than usual when the VI and VO queues are in full use.
+ For purely best effort wifi traffic, things look pretty good.

I am seriously considering disabling 802.11e negotiation in the next release.

I did prove 6in4 is working with the std-from-hurricane-electric script, so it's a bug in netifd, cero's config, or elsewhere at the openwrt level...

modprobe ipv6
ip tunnel add he-ipv6 mode sit remote $the_he_tunnel  local $my_local_ip ttl 255 tos inherit

# Note that I don't know if openwrt turns on tos inherit or not, btw, need to look into it. It's potentially useful

ip link set he-ipv6 up
ip addr add $mylink/64 dev he-ipv6
ip route add ::/0 dev he-ipv6
ip -f inet6 addr

---------------------------------------
??? - Mid June 2013

- Work on htb queuing (Only affected ATM?) - lots of problems, helped straighten out in CeroWrt and also other distros/kernel?

- Tweak for Windows file sharing (see Robert Bradley, 21 Jun 2013)

- Toke's note re: CeroWrt build script - 30 Jun 2013

---------------------------------------
3.10.10-1 - 9 Sep 2013

+ readlink fix (hopefully fixes sysupgrade)
+ usual merge with openwrt head (tons of ath9k changes)
+ dnsmasq 2.67test10
+ ipv6subtrees back in
+ the final htb atm patches
+ eliminated maxpacket check in codel

- did not fold in edumazet's new fq code
- 100% totally untested. May a braver soul than I give it a shot. I won't be near a cero box til thursday, otherwise.

http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.10-1/

-I'm not sure if I got the "last" of the aqm gui patches in there or not...

...

Anyway... I had hopes to get a stable release out in august. I AM very happy about the major stuff that got fixed, instead... but...

Since we didn't... I now have a ton of other matters piled up. Not least of which is a pending trip to england and the eu.

So for the next month I don't see how I'm going to be able to put more than a day a week into cerowrt. Tops. So I have tagged up this "release" and pushed all the baked portions of the sources to github. I'm still a little dubious of the ipv6 subtrees bit....


---------------------------------------
3.10.13-2 - 1 Oct 2013

+ Proved it is possible to build an OS release on a "Narrowboat"
-  but not test one without hacking at the 12v power supply off the solar panel
+ merge with openwrt head
+ dnsmasq 2.67test17
+ ipv6subtrees now part of 3.10.12
+ htb adsl fixes also
+ Simon kelly is starting to finalize dnsmasq 2.67 now that summer is over

- still no fix for the sysupgrade bug
- Most of the get_cycles() and /dev/random keruffle has settled down
but I did not fold the latest patchset for that into this. The
discussion on PRNGs was very illuminating and worth reading.There were
multiple threads on this topic on lkml, this is one:

https://lkml.org/lkml/2013/9/10/188


- I'd meant to push out some fixes to codel to the kernel mainline, didn't.
- PIE was submitted to the kernel mainline a few days ago but was
kicked back, also that version as submitted is pretty different from
what is in cero
- Been trying to find a sane answer for dns-sd support and haven't found one.

I will be returning to the US a bit early (tomorrow) and hope to gain
a week to solidify cero some more towards getting towards an honest
beta. But: If you are happy with previous dev builds I don't think
there is reason to use this one.

---------------------------------------
3.10.15-5 - 14Oct2013

totally untested. I will be back in front of a router in the yurtlab
monday morning  PDT.

+ resync with openwrt
- revert back to dnsmasq 2.66 (openwrt head)

Judging from the conversation it sounds like the dnsmasq bug may well
not be the latest dnsmasq at all! but a modern openwrt not interacting
with the multiple devices correctly. So I've reverted dnsmasq to
openwrt head to test that assumption...

... in the morning. Unless someone beats me to it.


--------
3.10.17-1 - 20 Oct 2013

+ sync with openwrt
+ dnsmasq 2.67rc4
+ get_cycles() and /dev/random fixes
+ mild firewall changes
+ actually sort of tested
-  sysupgrade still busted
- didn't package the jitter rng

The simple expedient of putting a script in /etc/rc.local to restart
pimd, minissdpd, and dnsmasq 60 seconds after boot appears to get us a
working dhcp/dns on the wifi interfaces once again.

dnsmasq wasn't busted, it was how it interfaces to netifd. the march
down to something deployable resumes with rc4.

This is the first test that I know of, of some of the RNG fixes
upstream, notably the mips code does the right thing with a highly
optimized "get_cycles()".

There are two changes to the firewall code

1) There has been a long-standing error in not blocking port 161
(snmp) from the outside world. It is now blocked by default.

Although I am not aware of any exploits of this (besides the
information leakage) I would recommend blocking this port by default
on your existing builds, also, or disabling the snmp daemon entirely
if you do not use it.

2) Usage of the "pattern matching syntax" on various firewall rules.

Instead of 3 rules for se00,sw00,sw10, and 4 for gw00,gw10,gw01,gw11
there are now 1 rule for s+ and one rule for gw+

This does not show up in the web interface correctly. I'd also like to
get to a more efficient rule set for the blocked ports, perhaps with
ipset...

...

It's sort of my hope that with these fixes that the march towards a
stable release can resume, and we get some fresh shiny new bugs out of
this.

Upcoming next are a revised version of pie, more random number fixes,
and I forget what else.

--------------------------------------
3.10.17-2 - 20 Oct 2013

- lighttpd didn't work

---------------------------------------
3.10.17-3 - 21 Oct 2013


+ this fixes the lighttppd bug noted in -2.
+ has support for signed packages
+ better random support
+ tested long enough to check for the -2 regression
+ Added (slow implementation of) port-mirroring http://code.google.com/p/port-mirroring/

- doesn't do https yet
- still abuses rc.local for starting up late daemons

---------------------------------------
3.10.17-5 - 30 Oct 2013

3.10.17-5 has the "final" version of cisco's pie, the "final" version
of dnsmasq 2.67, and imho was finally feature complete.

regrettably it still has the sysupgrade bug and a bug was found in
dnsmasq that has not been fully addressed yet, and I haven't had the
chance to evaluate the differences between this version of pie and the
last.

It seems wise to stick with 3.10.17-3 for now unless you specifically
want to play with pie.

===============================================================================================





-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

More information about the Cerowrt-devel mailing list