[Cerowrt-devel] Is there a particular reason cerowrt isn't using UBIFS?

[Cerowrt-devel] Is there a particular reason cerowrt isn't using UBIFS?

[Theodore Ts'o]

Potentially stupid question.  I was taking a look at 

	http://wiki.openwrt.org/doc/techref/filesystems

and there was discussion there about how using raw squashfs doesn't
deal with wear leveling and bad flash blocks, and that openwrt is now
using ubifs for all targets with raw NAND flash --- and my
understanding is that the WNDR 3800 uses raw NAND flash.  Is there any
particular reason why Cerowrt isn't using ubifs, or squashfs over ubi,
other than purely historical and/or this wasn't the research focus of Cerowrt?

Thanks,

					- Ted

Re: [Cerowrt-devel] Is there a particular reason cerowrt isn't using UBIFS?

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 1288 bytes --]

I tried ubifs in the early days. It doesn't squeeze stuff down even as good
as jffs2, so the load of cerowrt exceeded 15mbyte. It does look to be an
ever more reasonable answer once you have flash sizes greater than 128mbyte.

I see a lot of ext4 on much larger flashes....

A thing that irks me in the age of 4G flash becoming fairly common is the
general lack of compression aside from an option to btrfs. Debian barely
fits into 2 gb.
On Aug 30, 2014 2:35 PM, "Theodore Ts'o" <tytso@mit.edu> wrote:

> Potentially stupid question.  I was taking a look at
>
>         http://wiki.openwrt.org/doc/techref/filesystems
>
> and there was discussion there about how using raw squashfs doesn't
> deal with wear leveling and bad flash blocks, and that openwrt is now
> using ubifs for all targets with raw NAND flash --- and my
> understanding is that the WNDR 3800 uses raw NAND flash.  Is there any
> particular reason why Cerowrt isn't using ubifs, or squashfs over ubi,
> other than purely historical and/or this wasn't the research focus of
> Cerowrt?
>
> Thanks,
>
>                                         - Ted
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>

[-- Attachment #2: Type: text/html, Size: 1883 bytes --]

Re: [Cerowrt-devel] Is there a particular reason cerowrt isn't using UBIFS?

[Theodore Ts'o]

On Sat, Aug 30, 2014 at 02:57:05PM -0700, Dave Taht wrote:
> I tried ubifs in the early days. It doesn't squeeze stuff down even as good
> as jffs2, so the load of cerowrt exceeded 15mbyte. It does look to be an
> ever more reasonable answer once you have flash sizes greater than 128mbyte.

Another possible solution might be squashfs on top of ubi (instead of
mtdblock).  Ubi will provide the wear levelling and bad block
remapping, which mtdblock doesn't do.

> A thing that irks me in the age of 4G flash becoming fairly common is the
> general lack of compression aside from an option to btrfs. Debian barely
> fits into 2 gb

It depends on what you have installed, of course.  I have a debian
test image which gets used for ext4 testing which is 189 megabytes
uncompressed, and 57 megabytes using qcow2 compression (it gets run
using qemu/kvm)[1].  It's a basic debootstrap image plus a handful of
packages[2] plus xfstests (which is 22 megabytes uncompressed).

[1] ftp://ftp.kernel.org/pub/linux/kernel/people/tytso/kvm-xfstests/
[2] https://git.kernel.org/cgit/fs/ext2/xfstests-bld.git/tree/kvm-xfstests/test-appliance/packages

This is still much larger than 7 megabytes in the Cerowrt's root
image, granted, but it is possible to make a relatively svelte
debian-based image.


I've considered implementing MacOS X style compression (immutable
files, compression which happens in userspace, with decompression in
the kernel.)  The main reason why I haven't is that for most use
cases, space hasn't really been that much of an issue, or most of the
files are already compressed (i.e., Java or Dalvik classpath files
which are already zip compressed).  It wouldn't be _that_ hard to do,
but it's just not that high up on most people's priority lists.

    	      	       	       	  - Ted

Re: [Cerowrt-devel] Is there a particular reason cerowrt isn't using UBIFS?

[David Lang]

On Sat, 30 Aug 2014, Theodore Ts'o wrote:

>> A thing that irks me in the age of 4G flash becoming fairly common is the
>> general lack of compression aside from an option to btrfs. Debian barely
>> fits into 2 gb
>
> It depends on what you have installed, of course.  I have a debian
> test image which gets used for ext4 testing which is 189 megabytes
> uncompressed, and 57 megabytes using qcow2 compression (it gets run
> using qemu/kvm)[1].  It's a basic debootstrap image plus a handful of
> packages[2] plus xfstests (which is 22 megabytes uncompressed).
>
> [1] ftp://ftp.kernel.org/pub/linux/kernel/people/tytso/kvm-xfstests/
> [2] https://git.kernel.org/cgit/fs/ext2/xfstests-bld.git/tree/kvm-xfstests/test-appliance/packages
>
> This is still much larger than 7 megabytes in the Cerowrt's root
> image, granted, but it is possible to make a relatively svelte
> debian-based image.
>
>
> I've considered implementing MacOS X style compression (immutable
> files, compression which happens in userspace, with decompression in
> the kernel.)  The main reason why I haven't is that for most use
> cases, space hasn't really been that much of an issue, or most of the
> files are already compressed (i.e., Java or Dalvik classpath files
> which are already zip compressed).  It wouldn't be _that_ hard to do,
> but it's just not that high up on most people's priority lists.

One other place this sort of thing is likely to be useful is for Raspberry Pi 
and other small (embedded by some defintions) systems that use SD cards for 
their OS system. The I/O to the storage is so slow that the saved I/O time is 
likely to more than cover the cost of the decompression.

Raspberry Pi systems have had to move to 4G cards as their base because it's 
just not possible to have the standard install do more than boot on a 2G card.

David Lang

Re: [Cerowrt-devel] Is there a particular reason cerowrt isn't using UBIFS?

[Theodore Ts'o]

On Sun, Aug 31, 2014 at 12:05:52AM -0700, David Lang wrote:
> 
> One other place this sort of thing is likely to be useful is for Raspberry
> Pi and other small (embedded by some defintions) systems that use SD cards
> for their OS system. The I/O to the storage is so slow that the saved I/O
> time is likely to more than cover the cost of the decompression.

Yeah, that's the main argument I've heard for wanting to do
decompression; it's to speed up I/O when using HDD's and cheap flash
that has a minimal number of flash channels.

> Raspberry Pi systems have had to move to 4G cards as their base because it's
> just not possible to have the standard install do more than boot on a 2G
> card.

2G SD cards --  $42.95 for a 10-pack
4G SD cards --  $53.95 for a 10-pack

I have a design for adding compression, but except as a hobby effort,
I've had a lot of trouble finding a company who would be willing to
support an engineer to actually do the effort.  :-( 

Unless the company is making a truly vast number of devices, and are
very sensitive to the BOM cost, it might not make sense from an
engineering / business plan point of view.

So it will probably be something I do "for fun", when I can find the
time.... and there are a lot of other projects where companies are
willing to sponsor engineers to add new features, such as encryption,
reflink support, data checksums, etc., where if I can help them land
those features, it's unfortunately going to be higher priority than my
personally working on compression support for ext4.

But if someone is interested in working on it, they should talk to me;
I'd be happy to work with someone interested in working on the
project.

							- Ted

Re: [Cerowrt-devel] Is there a particular reason cerowrt isn't using UBIFS?

[Dave Taht]

Ted:

Certainly making the big cpe makers and ISPs better able to meet their
market requirements in the open source age is high on my list of
things to fix.

In the bad old days, a cpe maker would put out a big RFP with their
requirements, some big ODM or ODMs would bid on it, and after 5 years
the next product would come out, 2+ years late, there would be
acrimony and finger pointing on all sides, and the product would not
be as good as it could be.

In the present day, it really pays for the cpe maker (or ISP) to "feed
the penguins" - to identify key technologies *and their primary
authors* during their RFP process, and to try to put a floor under
them directly, to get the code more "out there" in an 18 month
timescale... and thus thoroughly tested before it has to be integrated
and then burned into flash by the ODM.

I think this approach chops of *years* of development time off of
shipping new stuff into embedded systems, and results in a much higher
quality product overall as the new features have the elegance, and
robustness that only an author with time, taste, and resources, can
produce.


On Sun, Aug 31, 2014 at 9:45 AM, Theodore Ts'o <tytso@mit.edu> wrote:
> On Sun, Aug 31, 2014 at 12:05:52AM -0700, David Lang wrote:
>>
>> One other place this sort of thing is likely to be useful is for Raspberry
>> Pi and other small (embedded by some defintions) systems that use SD cards
>> for their OS system. The I/O to the storage is so slow that the saved I/O
>> time is likely to more than cover the cost of the decompression.
>
> Yeah, that's the main argument I've heard for wanting to do
> decompression; it's to speed up I/O when using HDD's and cheap flash
> that has a minimal number of flash channels.

Faster boot times are good.

I also tend to think "using less flash" out of your total might extend the
service life. I do not have a good awareness of what filesystem(s)
people are planning to ship in the next generation of embedded systems,
certainly I see a lot of ext4, I've seen some try btrfs, and the old standbys
of squashfs and jffs2 are still good choices when you are below 32MB flaash.

Better tailoring a fs (as ubi did) for better speed, higher
reliability and easy updates in the field strikes me as a very good
idea.

>
>> Raspberry Pi systems have had to move to 4G cards as their base because it's
>> just not possible to have the standard install do more than boot on a 2G
>> card.
>
> 2G SD cards --  $42.95 for a 10-pack
> 4G SD cards --  $53.95 for a 10-pack

It really is astounding isn't it!

>
> I have a design for adding compression, but except as a hobby effort,
> I've had a lot of trouble finding a company who would be willing to
> support an engineer to actually do the effort.  :-(
>
> Unless the company is making a truly vast number of devices, and are
> very sensitive to the BOM cost, it might not make sense from an
> engineering / business plan point of view.

It's a small matter of finding a company or companies with sufficient
vision, moving
potentially millions of units, that want to save 20 cents each, realizing that
they can't go it alone.

I am loving the hipnet, homenet and homewrt's efforts to further their goals,
which are much wider than "saving money" - but in making new technologies
more available on cpe.

http://www.cablelabs.com/the-future-of-home-networking-putting-the-hip-in-hipnet/

http://www.homewrt.org/doku.php

These groups have realized that for certain key new features, once identified,
they can't get them from the ODM, but have to go direct to the authors of the
technologies in our new age modern open source world in order to get them
done enough - and "out there" enough and tested enough - for the ODM
to integrate
and ship, and put floors under various things needed.

(I was delighted comcast stepped up to get dnssec into dnsmasq,
in particular. So many people were whining about lack of dnssec adoption...

and there has been much work on routing protocols as an outgrowth of
all that, also... and so much behind the scenes integration and automation
work in that quest for extra nines of reliability. )


>
> So it will probably be something I do "for fun", when I can find the
> time.... and there are a lot of other projects where companies are
> willing to sponsor engineers to add new features, such as encryption,
> reflink support, data checksums, etc., where if I can help them land
> those features, it's unfortunately going to be higher priority than my
> personally working on compression support for ext4.
>
> But if someone is interested in working on it, they should talk to me;
> I'd be happy to work with someone interested in working on the
> project.

I will put possible work on ext4 compression and on ubifs on the
possible roadmap for the next phase of the cerowrt effort.

>                                                         - Ted



-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article

Re: [Cerowrt-devel] Is there a particular reason cerowrt isn't using UBIFS?

[David Lang]

On Sun, 31 Aug 2014, Theodore Ts'o wrote:

> On Sun, Aug 31, 2014 at 12:05:52AM -0700, David Lang wrote:
>>
>> One other place this sort of thing is likely to be useful is for Raspberry
>> Pi and other small (embedded by some defintions) systems that use SD cards
>> for their OS system. The I/O to the storage is so slow that the saved I/O
>> time is likely to more than cover the cost of the decompression.
>
> Yeah, that's the main argument I've heard for wanting to do
> decompression; it's to speed up I/O when using HDD's and cheap flash
> that has a minimal number of flash channels.

As a datapoint in this discussin, take a look at the openwrt thread
[OpenWrt-Devel] Toolchain issue: Significant decrease in performance of binaries 
produced by Barrier Breaker relative to Attitude Adjustment

Apparently with BB they changed the gcc options on mips to produce 16 bit code 
because it results in smaller binaries, but this cripples performance in some 
cases

It doesn't answer your funding issue, but it shows another case of people being 
very sensitive to storage size.

David Lang

>> Raspberry Pi systems have had to move to 4G cards as their base because it's
>> just not possible to have the standard install do more than boot on a 2G
>> card.
>
> 2G SD cards --  $42.95 for a 10-pack
> 4G SD cards --  $53.95 for a 10-pack
>
> I have a design for adding compression, but except as a hobby effort,
> I've had a lot of trouble finding a company who would be willing to
> support an engineer to actually do the effort.  :-(
>
> Unless the company is making a truly vast number of devices, and are
> very sensitive to the BOM cost, it might not make sense from an
> engineering / business plan point of view.
>
> So it will probably be something I do "for fun", when I can find the
> time.... and there are a lot of other projects where companies are
> willing to sponsor engineers to add new features, such as encryption,
> reflink support, data checksums, etc., where if I can help them land
> those features, it's unfortunately going to be higher priority than my
> personally working on compression support for ext4.
>
> But if someone is interested in working on it, they should talk to me;
> I'd be happy to work with someone interested in working on the
> project.
>
> 							- Ted
>

Re: [Cerowrt-devel] Is there a particular reason cerowrt isn't using UBIFS?

[Theodore Ts'o]

On Sun, Aug 31, 2014 at 12:37:07PM -0700, David Lang wrote:
> As a datapoint in this discussin, take a look at the openwrt thread
> [OpenWrt-Devel] Toolchain issue: Significant decrease in performance of
> binaries produced by Barrier Breaker relative to Attitude Adjustment
> 
> Apparently with BB they changed the gcc options on mips to produce 16 bit
> code because it results in smaller binaries, but this cripples performance
> in some cases
> 
> It doesn't answer your funding issue, but it shows another case of people
> being very sensitive to storage size.

Right, but *wrt is a bit of a special case, since what we're trying to
do here is shoehorn a different software load unto pre-existing
hardware where someone else has already cost-optimized the BOM so that
it only has, say, 8MB of flash instead of 16MB of flash.

But if company is designing a new product from scratch, the delta to
the BOM cost of using 16MB of flash versus 8MB of flash (unless the
company is making gazillions of units) is probably not going to
justify sponsoring an engineer to do open source work on a new file
system feature.

So I'm sure there are cases where people are sensitive to storage size
--- including for example, handset companies who want to be able to
install a newer version of Android on an older handset which doesn't
have enough flash for Kit Kat, or some such....  (and I hear the
peanut gallery burst into laughter at this point --- what?  handset
vendors caring about software updates?  They *want* people to buy a
new phone since that's the only way they get money out of the deal.
And if they can use the excuse of, "so sorry, the handset doesn't have
enough flash storage space to upgrade to Kit Kat", so much the
better. :-)

:-/

						- Ted

Re: [Cerowrt-devel] Is there a particular reason cerowrt isn't using UBIFS?

[Dave Taht]

On Sun, Aug 31, 2014 at 12:05 AM, David Lang <david@lang.hm> wrote:
> On Sat, 30 Aug 2014, Theodore Ts'o wrote:
>
>>> A thing that irks me in the age of 4G flash becoming fairly common is the
>>> general lack of compression aside from an option to btrfs. Debian barely
>>> fits into 2 gb
>>
>>
>> It depends on what you have installed, of course.  I have a debian
>> test image which gets used for ext4 testing which is 189 megabytes
>> uncompressed, and 57 megabytes using qcow2 compression (it gets run
>> using qemu/kvm)[1].  It's a basic debootstrap image plus a handful of
>> packages[2] plus xfstests (which is 22 megabytes uncompressed).
>>
>> [1] ftp://ftp.kernel.org/pub/linux/kernel/people/tytso/kvm-xfstests/
>> [2]
>> https://git.kernel.org/cgit/fs/ext2/xfstests-bld.git/tree/kvm-xfstests/test-appliance/packages
>>
>> This is still much larger than 7 megabytes in the Cerowrt's root
>> image, granted, but it is possible to make a relatively svelte
>> debian-based image.
>>
>>
>> I've considered implementing MacOS X style compression (immutable
>> files, compression which happens in userspace, with decompression in
>> the kernel.)  The main reason why I haven't is that for most use
>> cases, space hasn't really been that much of an issue, or most of the
>> files are already compressed (i.e., Java or Dalvik classpath files
>> which are already zip compressed).  It wouldn't be _that_ hard to do,
>> but it's just not that high up on most people's priority lists.
>
>
> One other place this sort of thing is likely to be useful is for Raspberry
> Pi and other small (embedded by some defintions) systems that use SD cards
> for their OS system. The I/O to the storage is so slow that the saved I/O
> time is likely to more than cover the cost of the decompression.
>
> Raspberry Pi systems have had to move to 4G cards as their base because it's
> just not possible to have the standard install do more than boot on a 2G
> card.

Beaglebone blacks updated to 4G also due to the same issue in building a useful
debian system in < 2G.

... and they increased the price 10 bucks.

The CPE market is highly price sensitive (and generally has preferred flash
soldiered to the board vs something removable)

Other things that bug me (on the x86 front), is that the onboard flash
there is usually
just large enough for the BIOS. I had kind of figured that flash sizes
there would have
increased to 8 or more MB in the general case by now, which would make
it more possible
to boot openwrt off of available on board flash (along with openbios, if needed)

Also the new systemd init system alone is bigger than available flash
on most router
platforms shipped last year; same goes for android.

(which is why openwrt went their own way with the json based procd
 system:

http://wiki.openwrt.org/doc/techref/procd

)

Now, I don't know how much time we have left before there are no cost
or power reduction benefits from absurdly small flash sizes  (my gut
feel is we passed those a while ago), but I do feel there are
significant benefits in an age of small caches and
weak processors, in keeping the footprint of the codebase small, and
the centralized uci based configuration system (which works well with
guis) in openwrt is unmatched by anything else - as is (now, in
barrier breaker), the ipv6, and de-bufferbloating support, and other
tight integrations with various cpe concepts.

> David Lang



-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article

Re: [Cerowrt-devel] Is there a particular reason cerowrt isn't using UBIFS?

[David Lang]

On Sat, 30 Aug 2014, Theodore Ts'o wrote:

> Potentially stupid question.  I was taking a look at
>
> 	http://wiki.openwrt.org/doc/techref/filesystems
>
> and there was discussion there about how using raw squashfs doesn't
> deal with wear leveling and bad flash blocks, and that openwrt is now
> using ubifs for all targets with raw NAND flash --- and my
> understanding is that the WNDR 3800 uses raw NAND flash.

The 3800 is the last generation before NAND flash started shipping, the 3700v4, 
4300 etc have NAND flash, and that is why support for them lagged for a long 
time (just hitting this year, even though the routers have been out for a couple 
of years)

David Lang

>  Is there any
> particular reason why Cerowrt isn't using ubifs, or squashfs over ubi,
> other than purely historical and/or this wasn't the research focus of Cerowrt?
>
> Thanks,
>
> 					- Ted
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>