[Cerowrt-devel] wireguard almost takes a bullet

[Theodore Ts'o]

On Tue, Mar 30, 2021 at 09:23:50PM -0400, David P. Reed wrote:
> 
> On the other hand, they are pretty damn high salaries for a
> non-profit. Are they appropriate? Depends. There are no stockholders
> and no profits, just a pretty substantial net worth.

For better or for worse, senior software engineers who work for Big
Tech will be making similar amounts of money.  Without being
inappropriately specific, my total compensation isn't that different
from Linus's, once you take things like equity compensation into
account.  This was true both in my current employer, as well as my
previous employer (IBM).

Is that right or wrong?  Unfortunately, it is what it is.  Part of it
is that it's very much a supply and demand question.  I know, because
I've tried to find talented file system kernel developers for my
team.... and it's hard to find them.

I know that in many ways, it's hugely unfair.  When I was at IBM, I
was the high powered senior developer who could meet with the senior
technical leaders at a major US defense contractor.  I was the one
with ths TS/SCI security clearance, and yes, I was the senior
architect who got an IBM award for my team's work on real-time Linux
capable of running real-time Java for us in the DDG(1000) Zumwalt
class destroyer.  And yet, the vast majority of the work was done by
much more junior engineers, and they didn't get any of the major
awards, and their salary was much less.  It was one of the best teams
I had the pleasure of working with, and I'm glad to see that they are
now working in much more senior roles at other companies.

So while it's easy to criticize the Linux Foundation, it's by no means
unique to it, and to the extent that it's part of a larger tech
ecosystem that pays engineers in a very disproportionate way, it has
to pay its people commensurate with what they could get if they had
decided to go work for a company like Facebook or Amazon.

> Regarding the organizaton of "Linux, Inc." as a hierachical control
> structure - I'll just point out that hierarchical control of the
> development of Linux suggests that it is not at all a "community
> project" (if it ever was). It's a product development organization
> with multiple levels of management.

So I'd argue that *any* successful, very large open source project
needs to have multiple levels of management.  It's *technical*
managers, and I would point out that it's really more servant
leadership more than anything else.  I may be the ext4 maintainer, but
that means that in order to make ext4 successful, I end up doing the
work that no one else finds "fun" to work for, or that companies
aren't willing to pay engineers to do as part of their day job.  So
for example, the test framework[1] for ext4 is something I had to create
and maintain, because no one else would do it.  And code review is not
necessary *fun*, but someone has to do it.  Much of this work actually
happens late at night or on weekend, on my own time, because I care
enough about it that it's something I *choose* to do it.

[1] https://thunk.org/gce-xfstests 

So if your definition of a "community project" is one which has a
non-scalable governance structure, I'm going to have to disagree.  In
the early 90's, when I first getting started with Linux, there were
attempts from senior leaders at NetBSD and GNU HURD who tried to woo
me to do work for their kernel instead.  Let's just say that even
then, after seeing the toxicity/drama of those governance structures,
(and it didn't help that living in Cambridge, I had the ability to
meet and break bread with some of these senior people face-to-face),
one of the primary *reasons* why I declined to work on *BSD and HURD
was due to the leaders of those projects that I would have had to work
with.  This despite the fact that my first OS/systems programming
experience, courtesy of MIT Project Athena, was on BSD 4.3.

> Yet the developers are employees of a small number of major
> corporations. In this sense, it is like a "joint venture" among
> those companies.
>  
> To the extent that those companies gain (partial) control of the
> Linux kernel, as appears to be the case, I think Linux misrepresents
> itself as a "community project", and in particular, the actual users
> of the software may have little say in the direction development
> takes going forwards.

There are certainly still developers in Linux that are hobbyists, and
not everyone works for Big Tech.  In fact, Jason worked at a startup
that was certainly not what I would call an example of Big Tech.
Sure, his startup let him spend a significant amount of his time
working on getting WireGuard upstream, but WireGuard was very much
accepted on the basis of the merits of his work.  It was not because
someone paid $$$ to the Linux Foundation, or anything crazy like that.

I also started out as a hobbyist.  For a long time, being tech lead
for Kerberos at MIT, and doing IETF standards work (e.g., I was ipsec
working group chair) was my day job, and Linux as my hobby.  Sure, I
was the first North American Linux kernel developer, and that got me
invited to a bunch of conferences who were willing to pay my travel
expenses (since I was a starving academic), but I was paid a very
small salary compared to industry.  (We were wondering why MIT kept on
losing people to industry, so my department brought in a salary
consultant who determined that MIT was paying its people at the tenth
percentile of industry at that time.)  I doubled my salary when I went
to work for a startup, and given that VA Linux Systems imploded before
I was able to sell most of my stock, that figure was *before* any
equity compensation.

Some of the people who were smarter than me, at least in terms of
deciding to go out into industry much sooner, and who were able to
benefit from Red Hat's IPO, have multiple expensive houses and can
travel between them as the ski seasons open up.  And I also know
people working in Indiana contributing to Linux who make a tiny
fraction of what one can make in Big Tech.  I try not to get envious
over those who have done financially much better than I, and I also
try not to think that I'm inherently superior just because I've been
incredibly blessed and lucky and have a very privileged existence.
Life is unfair, and all you can do is to try to your best to make the
world a better place than when you entered it.

> There's little safeguard, for example, against "senior management"
> biases in support of certain vendors, if other vendors are excluded
> from effective participation by one of many techniques. In other
> words, there's no way it can be a level playing field for
> innovation.

The safeguard is in the maintainers' hands.  Remember, we "own" our
subsystems and to the extent that we are passionate to let it be
successful, we'll take the help from whereever we can get it.  I might
be at Company A one year, and Company B another, and if I take crappy
code from one Company, I'll end up owning that crap and I'll
ultimately need to fix it later, perhaps when I'm at another company.

It is true that as a someone who manages volunteers (regardless of
whether such volunteers are hobbyists or people who are doing the work
paid by a certain company), we can't force someone to implement a
particular feature or fix a certain bug.  As I learned from my service
on the IETF, the only power of such leaders is to say "No".  But if we
stop a good feature from getting in, that ultimately is going to be to
the detriment of our subsystem.

And if that does happen for some reason, one of the roles that Linus
plays is as an authority that someone can appeal to.  I've never seen
a support for some CPU architecture get denied just because it might
threaten existing "Big Companys", for example.  I'm sure the ARM SOC's
weren't happy to see RISC-V support land in the kernel.  But if there
was an attempt to keep RISC-V out of the kernel, that's a case where
Linus would intervene, since ultimately it's *his* choice to accept a
new subsystem and a new maintainer.

> (one that is not transparent at all about functioning as such -
> preferring to give the impression that the kernel is developed by
> part-time voluntary "contributions").

Actually, the Linux Foundation has been quite transparent about this;
every few years, it relases a "Who Writes Linux Report".  Anyone who
had such an impression hasn't been paying attention:

https://www.linuxfoundation.org/wp-content/uploads/linux-kernel-report-2017.pdf
https://www.linuxfoundation.org/wp-content/uploads/2020_kernel_history_report_082720.pdf

>From these reports, you'll see that in 2017 we had 8.2% of the
contributions coming from people weren't getting financial
contributions (with another 4.1% where the source of financial support
couldn't be determined).  This was down from the 2013 report, where
14.6% of the contributions came from hobbyists.

In the 2020 report, "None" was 11.95%, with the next highest
contributor being Intel at 10%, Red Hat at 8.9%, "Unknown" at 4%, and
IBM at 3.8%.  (Google was much farther down the list, at 2.8%).  Not
to put too fine a point on it, "people who receive no financial
contributions" are #1 on the "Top 20 committers list".

> The contrast with other open source communities is quite sharp
> now. There is little eleemosynary intent that can be detected any
> more. I think that is too bad, but things change.

If you look at the members of the Git, Perl and Python communitiesn, I
believe you'll find that most of the major contributors do work for
companies that pay for at least part of their open source
contributions.  Given that most people do enjoy having food with their
meals, if a OSS project is successful, this really shouldn't be a
surprise.

It is true that there is a huge long tail of OSS projects which have
not been successful, and which only exist as abandonware on
SourceForge or GitHub.  (Or in the case of OpenOffice, as part of the
Apache Consortium :-P)  But just as the vast majorities of startups end
up emploding with less than 1% becoming the IPO unicorns, I'm not so
sure it's anything more than sour graps for people to claim that the
startups which made it big were never "real startups" to begin with,
and that the story of startups is all a Big Lie.

Cheers,

						- Ted