From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 682753B29E; Wed, 31 Mar 2021 12:08:06 -0400 (EDT) Received: from cwcc.thunk.org (pool-72-74-133-215.bstnma.fios.verizon.net [72.74.133.215]) (authenticated bits=0) (User authenticated as tytso@ATHENA.MIT.EDU) by outgoing.mit.edu (8.14.7/8.12.4) with ESMTP id 12VG82e6008005 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 31 Mar 2021 12:08:03 -0400 Received: by cwcc.thunk.org (Postfix, from userid 15806) id 735D215C39CF; Wed, 31 Mar 2021 12:08:02 -0400 (EDT) Date: Wed, 31 Mar 2021 12:08:02 -0400 From: "Theodore Ts'o" To: "David P. Reed" Cc: Dave Taht , Cake List , Make-Wifi-fast , "Jason A. Donenfeld" , cerowrt-devel , bloat Message-ID: References: <1617049691.187521510@apps.rackspace.com> <1617153830.6256867@apps.rackspace.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1617153830.6256867@apps.rackspace.com> X-Mailman-Approved-At: Wed, 31 Mar 2021 12:44:53 -0400 Subject: Re: [Bloat] [Cerowrt-devel] wireguard almost takes a bullet X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Mar 2021 16:08:06 -0000 On Tue, Mar 30, 2021 at 09:23:50PM -0400, David P. Reed wrote: > > On the other hand, they are pretty damn high salaries for a > non-profit. Are they appropriate? Depends. There are no stockholders > and no profits, just a pretty substantial net worth. For better or for worse, senior software engineers who work for Big Tech will be making similar amounts of money. Without being inappropriately specific, my total compensation isn't that different from Linus's, once you take things like equity compensation into account. This was true both in my current employer, as well as my previous employer (IBM). Is that right or wrong? Unfortunately, it is what it is. Part of it is that it's very much a supply and demand question. I know, because I've tried to find talented file system kernel developers for my team.... and it's hard to find them. I know that in many ways, it's hugely unfair. When I was at IBM, I was the high powered senior developer who could meet with the senior technical leaders at a major US defense contractor. I was the one with ths TS/SCI security clearance, and yes, I was the senior architect who got an IBM award for my team's work on real-time Linux capable of running real-time Java for us in the DDG(1000) Zumwalt class destroyer. And yet, the vast majority of the work was done by much more junior engineers, and they didn't get any of the major awards, and their salary was much less. It was one of the best teams I had the pleasure of working with, and I'm glad to see that they are now working in much more senior roles at other companies. So while it's easy to criticize the Linux Foundation, it's by no means unique to it, and to the extent that it's part of a larger tech ecosystem that pays engineers in a very disproportionate way, it has to pay its people commensurate with what they could get if they had decided to go work for a company like Facebook or Amazon. > Regarding the organizaton of "Linux, Inc." as a hierachical control > structure - I'll just point out that hierarchical control of the > development of Linux suggests that it is not at all a "community > project" (if it ever was). It's a product development organization > with multiple levels of management. So I'd argue that *any* successful, very large open source project needs to have multiple levels of management. It's *technical* managers, and I would point out that it's really more servant leadership more than anything else. I may be the ext4 maintainer, but that means that in order to make ext4 successful, I end up doing the work that no one else finds "fun" to work for, or that companies aren't willing to pay engineers to do as part of their day job. So for example, the test framework[1] for ext4 is something I had to create and maintain, because no one else would do it. And code review is not necessary *fun*, but someone has to do it. Much of this work actually happens late at night or on weekend, on my own time, because I care enough about it that it's something I *choose* to do it. [1] https://thunk.org/gce-xfstests So if your definition of a "community project" is one which has a non-scalable governance structure, I'm going to have to disagree. In the early 90's, when I first getting started with Linux, there were attempts from senior leaders at NetBSD and GNU HURD who tried to woo me to do work for their kernel instead. Let's just say that even then, after seeing the toxicity/drama of those governance structures, (and it didn't help that living in Cambridge, I had the ability to meet and break bread with some of these senior people face-to-face), one of the primary *reasons* why I declined to work on *BSD and HURD was due to the leaders of those projects that I would have had to work with. This despite the fact that my first OS/systems programming experience, courtesy of MIT Project Athena, was on BSD 4.3. > Yet the developers are employees of a small number of major > corporations. In this sense, it is like a "joint venture" among > those companies. > > To the extent that those companies gain (partial) control of the > Linux kernel, as appears to be the case, I think Linux misrepresents > itself as a "community project", and in particular, the actual users > of the software may have little say in the direction development > takes going forwards. There are certainly still developers in Linux that are hobbyists, and not everyone works for Big Tech. In fact, Jason worked at a startup that was certainly not what I would call an example of Big Tech. Sure, his startup let him spend a significant amount of his time working on getting WireGuard upstream, but WireGuard was very much accepted on the basis of the merits of his work. It was not because someone paid $$$ to the Linux Foundation, or anything crazy like that. I also started out as a hobbyist. For a long time, being tech lead for Kerberos at MIT, and doing IETF standards work (e.g., I was ipsec working group chair) was my day job, and Linux as my hobby. Sure, I was the first North American Linux kernel developer, and that got me invited to a bunch of conferences who were willing to pay my travel expenses (since I was a starving academic), but I was paid a very small salary compared to industry. (We were wondering why MIT kept on losing people to industry, so my department brought in a salary consultant who determined that MIT was paying its people at the tenth percentile of industry at that time.) I doubled my salary when I went to work for a startup, and given that VA Linux Systems imploded before I was able to sell most of my stock, that figure was *before* any equity compensation. Some of the people who were smarter than me, at least in terms of deciding to go out into industry much sooner, and who were able to benefit from Red Hat's IPO, have multiple expensive houses and can travel between them as the ski seasons open up. And I also know people working in Indiana contributing to Linux who make a tiny fraction of what one can make in Big Tech. I try not to get envious over those who have done financially much better than I, and I also try not to think that I'm inherently superior just because I've been incredibly blessed and lucky and have a very privileged existence. Life is unfair, and all you can do is to try to your best to make the world a better place than when you entered it. > There's little safeguard, for example, against "senior management" > biases in support of certain vendors, if other vendors are excluded > from effective participation by one of many techniques. In other > words, there's no way it can be a level playing field for > innovation. The safeguard is in the maintainers' hands. Remember, we "own" our subsystems and to the extent that we are passionate to let it be successful, we'll take the help from whereever we can get it. I might be at Company A one year, and Company B another, and if I take crappy code from one Company, I'll end up owning that crap and I'll ultimately need to fix it later, perhaps when I'm at another company. It is true that as a someone who manages volunteers (regardless of whether such volunteers are hobbyists or people who are doing the work paid by a certain company), we can't force someone to implement a particular feature or fix a certain bug. As I learned from my service on the IETF, the only power of such leaders is to say "No". But if we stop a good feature from getting in, that ultimately is going to be to the detriment of our subsystem. And if that does happen for some reason, one of the roles that Linus plays is as an authority that someone can appeal to. I've never seen a support for some CPU architecture get denied just because it might threaten existing "Big Companys", for example. I'm sure the ARM SOC's weren't happy to see RISC-V support land in the kernel. But if there was an attempt to keep RISC-V out of the kernel, that's a case where Linus would intervene, since ultimately it's *his* choice to accept a new subsystem and a new maintainer. > (one that is not transparent at all about functioning as such - > preferring to give the impression that the kernel is developed by > part-time voluntary "contributions"). Actually, the Linux Foundation has been quite transparent about this; every few years, it relases a "Who Writes Linux Report". Anyone who had such an impression hasn't been paying attention: https://www.linuxfoundation.org/wp-content/uploads/linux-kernel-report-2017.pdf https://www.linuxfoundation.org/wp-content/uploads/2020_kernel_history_report_082720.pdf >From these reports, you'll see that in 2017 we had 8.2% of the contributions coming from people weren't getting financial contributions (with another 4.1% where the source of financial support couldn't be determined). This was down from the 2013 report, where 14.6% of the contributions came from hobbyists. In the 2020 report, "None" was 11.95%, with the next highest contributor being Intel at 10%, Red Hat at 8.9%, "Unknown" at 4%, and IBM at 3.8%. (Google was much farther down the list, at 2.8%). Not to put too fine a point on it, "people who receive no financial contributions" are #1 on the "Top 20 committers list". > The contrast with other open source communities is quite sharp > now. There is little eleemosynary intent that can be detected any > more. I think that is too bad, but things change. If you look at the members of the Git, Perl and Python communitiesn, I believe you'll find that most of the major contributors do work for companies that pay for at least part of their open source contributions. Given that most people do enjoy having food with their meals, if a OSS project is successful, this really shouldn't be a surprise. It is true that there is a huge long tail of OSS projects which have not been successful, and which only exist as abandonware on SourceForge or GitHub. (Or in the case of OpenOffice, as part of the Apache Consortium :-P) But just as the vast majorities of startups end up emploding with less than 1% becoming the IPO unicorns, I'm not so sure it's anything more than sour graps for people to claim that the startups which made it big were never "real startups" to begin with, and that the story of startups is all a Big Lie. Cheers, - Ted