[Cerowrt-devel] wireguard almost takes a bullet

[Cerowrt-devel] wireguard almost takes a bullet

[Dave Taht]

I am sad about the state of freebsd today, and of companies
contracting outside the authors of the code to get crappy things
committed without review and testing.

https://lwn.net/Articles/850757/

(long rant of mine in the comments).

My hat is off to jason for sinking a frantic week into vastly
improving that wireguard implementation, and I hope he and his team
gets caught up on sleep now.

-- 
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729

Re: [Cerowrt-devel] wireguard almost takes a bullet

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 4541 bytes --]


Dave -
 
I've spent a fair amount of time orbiting the FreeBSD community over the past few years. It's not as sad as you might think.
However, the networking portion of FreeBSD community is quite differently organized than it is in Linux.
 
What tends to shape Linux and FreeBSD, etc. are the money sources that flow into the communities. Of course Linux is quite independently wealthy now. The senior executives of the Linux Foundation are paid nearly a million dollars a year, each.  Which just indicates that major corporations are seriously interested in controlling the evolution of Linux (not the Gnu part, the part that has Linus Torvalds at its center).
 
FreeBSD, in contrast, is a loose alliance of what you might call "embedded hardware vendors" like NetApp as an example. They value an open, portable, efficient operating environment, but not for servers, laptops or smartphones.
 
They overlap at the intersection of network routing and storage platforms, where Linux doesn't seem to fit well, except in the case of "home routers".
 
At least that's my view. The major controllers of architectural elements are not terribly interested in FreeBSD's positive qualities. FreeBSD is not very visible at Intel and ARM at all, interms of their product planning. IBM has no "Power" FreeBSD.
 
Take for example, bufferbloat as an issue that routing and switching hardware ought to address. This is a serious weakness in the FreeBSD community (where it should matter!) There's not been much demand by the major corporate spenders on FreeBSD in fixing bufferbloat. But then again, there's not been much visibility regarding bufferbloat in the IETF, either. I'm not sure Torvalds has ever even heard of it (and I suspect he would try to argue it isn't a problem at all, given his tendency to not think clearly about systems scale issues, so what's caused Linux to even bother is the fringes in OpenWRT land and mesh networking land, plus Jim Gettys).
 
Anyway, FreeBSD and FreeRTOS and a few other very strong but small communities have solutions that are far better for their actual needs than the behemoth mess that Linux has become. And for those communities, they work very well. They are disentangled from Gnu, which is both a good and a bad thing depending on your perspective.
 
I just spent 9 months trying to get a very tiny fix to the Linux kernel into the mainline kernel. I actually gave up, because it seemed utterly pointless, even though it was clearly a design error that I was fixing, and I was trying to meet all the constraints on patches. No one was fighting me, no one said it was wrong. I found the problem in a personal research project where it was a blocking bug, so I had to maintain it as an add-on private patch (and I still do) that I needed to verify every release of the Linux kernel. Why is this? Well, it shows how Linux excludes ideas by the very bureaucracy of its management structure. (and I'd suggest that the mess that "init" has turned into in the OS, which the kernel actually requires in order to be useful, called "systemd", is an example of how not to modularize a portable OS kernel).
 
So FreeBSD, compared to Linux, in some ways, is far more pleasant to deal with. The community doesn't have rude and clueless and entitled members like Torvalds and Alan Cox have been. It isn't being driven by a consortium of F100 companies in a near-cartel.
 
So there are pluses and minuses. I suspect this is why many, many Linux developers actually use macOS as their personal computer for development. A paradox, given that macOS is completely proprietary.
 
 
 
On Sunday, March 28, 2021 11:56am, "Dave Taht" <dave.taht@gmail.com> said:



> I am sad about the state of freebsd today, and of companies
> contracting outside the authors of the code to get crappy things
> committed without review and testing.
> 
> https://lwn.net/Articles/850757/
> 
> (long rant of mine in the comments).
> 
> My hat is off to jason for sinking a frantic week into vastly
> improving that wireguard implementation, and I hope he and his team
> gets caught up on sleep now.
> 
> --
> "For a successful technology, reality must take precedence over public
> relations, for Mother Nature cannot be fooled" - Richard Feynman
> 
> dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 

[-- Attachment #2: Type: text/html, Size: 7499 bytes --]

Re: [Cerowrt-devel] wireguard almost takes a bullet

[Dave Taht]

I've had several good conversations with linus.He cares that his
ath10k works better now.

As for the rest, well, I've always felt more outreach would help. In
particular, getting something bql-like into freebsd would be a start.
a cake port, better.

Re: [Cerowrt-devel] wireguard almost takes a bullet

[Theodore Ts'o]

On Mon, Mar 29, 2021 at 04:28:11PM -0400, David P. Reed wrote:
> 
>  
> What tends to shape Linux and FreeBSD, etc. are the money sources
> that flow into the communities. Of course Linux is quite
> independently wealthy now. The senior executives of the Linux
> Foundation are paid nearly a million dollars a year, each.  Which
> just indicates that major corporations are seriously interested in
> controlling the evolution of Linux (not the Gnu part, the part that
> has Linus Torvalds at its center).

First of all, I don't believe your salary numbers are correct.

https://nonprofitlight.com/ca/san-francisco/linux-foundation

Secondly, the "senior executives" of the Linux Foundation don't have
any control over "the evolution of Linux".  The exception to that are
the "Fellows" (e.g., Linus Torvalds, Greg K-H, etc.) and I can assure
you that they don't take orders from Jim Zemlin, the executive
director, or any one else at the Linux Foundation.

The senior developers of Linux do tend to work for the big
corporations, but culturally, we do try to keep our "corporate hats"
and our "community" hats quite separate, and identify when we our
company hats on.  Many senior developers have transitioned between
multiple companies, and over time, it's been understood that their
primarily allegiance is to Linux, and not to the company.  In fact,
the primary job of maintainers is to say "no" to companies when they
try to push crap code into the kernel.  And that's because it's the
maintainer's responsibility to clean up the mess if they say yes to
code that's Just Not Ready, since they have a long-term responsbility
towards their subsystem, unlike engineers or contractors that only
have a short-term goal to get the code upstream.

This is where having a hierarchial ownership model IMHO works better
than a "core team" model where there can be a diffusion of
responsibility, where anyone with a commit bit can commit anywhere in
the OS.  In contrast, David Miller "owns" the networking area, and so
someone who might be, say, the ext4 or xfs maintainer does not have
the right (read: Linus will reject a pull request from me if I try to
change code in the networking stack with out DaveM's signoff) to
change code outside of their subsystem.

So you're right that Linus probably doesn't know or care about
bufferbloat.  He's delegated pretty much all networking issues to
David Miller as the networking czar, and within networking, David
Miller has his submaintainers with different specialities.  This does
get complicated when there are changes which cross subsystems.  For
example, before Wireguard could land in the kernel, there were changes
needed in both the crypto and networking layers, and Jason had to
negotiate with multiple senior developers in those subsystems, and the
code was subject to quite a lot of review before it could land.  (It
took months, and we didn't try to rush things before a major
release....)

> I just spent 9 months trying to get a very tiny fix to the Linux
> kernel into the mainline kernel. I actually gave up, because it
> seemed utterly pointless, even though it was clearly a design error
> that I was fixing, and I was trying to meet all the constraints on
> patches. No one was fighting me, no one said it was wrong.

It sounds like the real problem is no one was paying attention to you.
There is a *huge* number of changes going into the Linux kernel, and
so the the challenge is getting review bandwidth by the relevant
maintainers.  Blindly posting to the linux-kernel mailing list will
generally not get you very far.

The Linux development process is not really optimized for "drive by
patching".  Knowng where (and to whom) a patch needs to be reviewed is
not necessarily easy for a novice, and while there are tools such as
./scripts/get_maintainer.pl that try to make it a bit easier, I can
see how someone who Just Wants To Get A Single Patch accepted, can see
it as "bureaucracy".

Cheers,

					- Ted

Re: [Cerowrt-devel] wireguard almost takes a bullet

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 7328 bytes --]


Theodore -
 
I appreciate you showing the LF executive salary numbers are not quite as high as I noted. My numbers may have been inflated, but I've definitely seen a $900,000 package for at least one executive reported in the press (an executive who was transferred in from a F100 company which is close to the LF).
 
On the other hand, they are pretty damn high salaries for a non-profit. Are they appropriate? Depends. There are no stockholders and no profits, just a pretty substantial net worth.
 
Regarding the organizaton of "Linux, Inc." as  a hierachical control structure - I'll just point out that hierarchical control of the development of Linux suggests that it is not at all a "community project" (if it ever was). It's a product development organization with multiple levels of management.
 
Yet the developers are employees of a small number of major corporations. In this sense, it is like a "joint venture" among those companies.
 
To the extent that those companies gain (partial) control of the Linux kernel, as appears to be the case, I think Linux misrepresents itself as a "community project", and in particular, the actual users of the software may have little say in the direction development takes going forwards.
 
There's little safeguard, for example, against "senior management" biases in support of certain vendors, if other vendors are excluded from effective participation by one of many techniques. In other words, there's no way it can be a level playing field for innovation.
 
In that sense, the Linux kernel community has reached a point very much like Microsoft Windows development reached in 1990 or so. I note that date because at that point, Microsoft was challenged with a variety of anti-trust actions based on the fact that it used its Windows monopoly status to put competitors in the application space, and competitors producing innovative operating systems out of business (GO Computer Corporation being one example of many).
 
This troubles me. It may not trouble the developers who are in the Linux community and paid by the cartel of companies that control its direction.
 
I have no complaint about the technical competence of individual developers - the quality is pretty high, at least as good as those who worked on Windows and macOS. But it's becoming clear that their is a narrowing of control of an OS that has a lot of influence in a few hands. That those few hands don't work for one company doesn't eliminate its tendency to become a cartel. (one that is not transparent at all about functioning as such - preferring to give the impression that the kernel is developed by part-time voluntary "contributions").
 
The contrast with other open source communities is quite sharp now. There is little eleemosynary intent that can be detected any more. I think that is too bad, but things change.
 
This is just the personal opinion of someone who has been developing systems for 50+ years now. I'm kind of disappointed, but my opinion does not really matter much.
 
David
 
 
 
 
On Monday, March 29, 2021 9:52pm, "Theodore Ts'o" <tytso@mit.edu> said:



> On Mon, Mar 29, 2021 at 04:28:11PM -0400, David P. Reed wrote:
> >
> >
> > What tends to shape Linux and FreeBSD, etc. are the money sources
> > that flow into the communities. Of course Linux is quite
> > independently wealthy now. The senior executives of the Linux
> > Foundation are paid nearly a million dollars a year, each. Which
> > just indicates that major corporations are seriously interested in
> > controlling the evolution of Linux (not the Gnu part, the part that
> > has Linus Torvalds at its center).
> 
> First of all, I don't believe your salary numbers are correct.
> 
> https://nonprofitlight.com/ca/san-francisco/linux-foundation
> 
> Secondly, the "senior executives" of the Linux Foundation don't have
> any control over "the evolution of Linux". The exception to that are
> the "Fellows" (e.g., Linus Torvalds, Greg K-H, etc.) and I can assure
> you that they don't take orders from Jim Zemlin, the executive
> director, or any one else at the Linux Foundation.
> 
> The senior developers of Linux do tend to work for the big
> corporations, but culturally, we do try to keep our "corporate hats"
> and our "community" hats quite separate, and identify when we our
> company hats on. Many senior developers have transitioned between
> multiple companies, and over time, it's been understood that their
> primarily allegiance is to Linux, and not to the company. In fact,
> the primary job of maintainers is to say "no" to companies when they
> try to push crap code into the kernel. And that's because it's the
> maintainer's responsibility to clean up the mess if they say yes to
> code that's Just Not Ready, since they have a long-term responsbility
> towards their subsystem, unlike engineers or contractors that only
> have a short-term goal to get the code upstream.
> 
> This is where having a hierarchial ownership model IMHO works better
> than a "core team" model where there can be a diffusion of
> responsibility, where anyone with a commit bit can commit anywhere in
> the OS. In contrast, David Miller "owns" the networking area, and so
> someone who might be, say, the ext4 or xfs maintainer does not have
> the right (read: Linus will reject a pull request from me if I try to
> change code in the networking stack with out DaveM's signoff) to
> change code outside of their subsystem.
> 
> So you're right that Linus probably doesn't know or care about
> bufferbloat. He's delegated pretty much all networking issues to
> David Miller as the networking czar, and within networking, David
> Miller has his submaintainers with different specialities. This does
> get complicated when there are changes which cross subsystems. For
> example, before Wireguard could land in the kernel, there were changes
> needed in both the crypto and networking layers, and Jason had to
> negotiate with multiple senior developers in those subsystems, and the
> code was subject to quite a lot of review before it could land. (It
> took months, and we didn't try to rush things before a major
> release....)
> 
> > I just spent 9 months trying to get a very tiny fix to the Linux
> > kernel into the mainline kernel. I actually gave up, because it
> > seemed utterly pointless, even though it was clearly a design error
> > that I was fixing, and I was trying to meet all the constraints on
> > patches. No one was fighting me, no one said it was wrong.
> 
> It sounds like the real problem is no one was paying attention to you.
> There is a *huge* number of changes going into the Linux kernel, and
> so the the challenge is getting review bandwidth by the relevant
> maintainers. Blindly posting to the linux-kernel mailing list will
> generally not get you very far.
> 
> The Linux development process is not really optimized for "drive by
> patching". Knowng where (and to whom) a patch needs to be reviewed is
> not necessarily easy for a novice, and while there are tools such as
> ./scripts/get_maintainer.pl that try to make it a bit easier, I can
> see how someone who Just Wants To Get A Single Patch accepted, can see
> it as "bureaucracy".
> 
> Cheers,
> 
> - Ted
> 

[-- Attachment #2: Type: text/html, Size: 11126 bytes --]

Re: [Cerowrt-devel] [Make-wifi-fast] wireguard almost takes a bullet

[David Lang]

[-- Attachment #1: Type: text/plain, Size: 8519 bytes --]

the 'control' that the various companies gain over parts of the kernel is less a 
matter of the company having control and more a matter of them hiring/sponsoring 
a developer who has the control. If the person leaves that company for another 
one, any control moves with that developer.

and while most of the developers do work for a reltively small group of 
companies, the list of developers does shift over time nd people can 'break in' 
by submitting patches.

I'm not thrilled by the Linux Foundation, it was created to be a way to pay 
Linus without him working for a specific company (avoiding even the appearance 
of bias) but it's morphed to present at least the appearance of special access.

David Lang

On Tue, 30 Mar 2021, David P. Reed wrote:

> Date: Tue, 30 Mar 2021 21:23:50 -0400 (EDT)
> From: David P. Reed <dpreed@deepplum.com>
> To: Theodore Ts'o <tytso@mit.edu>
> Cc: Make-Wifi-fast <make-wifi-fast@lists.bufferbloat.net>,
>     Cake List <cake@lists.bufferbloat.net>,
>     cerowrt-devel <cerowrt-devel@lists.bufferbloat.net>,
>     bloat <bloat@lists.bufferbloat.net>
> Subject: Re: [Make-wifi-fast] [Cerowrt-devel] wireguard almost takes a bullet
> 
>
> Theodore -
> 
> I appreciate you showing the LF executive salary numbers are not quite as high as I noted. My numbers may have been inflated, but I've definitely seen a $900,000 package for at least one executive reported in the press (an executive who was transferred in from a F100 company which is close to the LF).
> 
> On the other hand, they are pretty damn high salaries for a non-profit. Are they appropriate? Depends. There are no stockholders and no profits, just a pretty substantial net worth.
> 
> Regarding the organizaton of "Linux, Inc." as  a hierachical control structure - I'll just point out that hierarchical control of the development of Linux suggests that it is not at all a "community project" (if it ever was). It's a product development organization with multiple levels of management.
> 
> Yet the developers are employees of a small number of major corporations. In this sense, it is like a "joint venture" among those companies.
> 
> To the extent that those companies gain (partial) control of the Linux kernel, as appears to be the case, I think Linux misrepresents itself as a "community project", and in particular, the actual users of the software may have little say in the direction development takes going forwards.
> 
> There's little safeguard, for example, against "senior management" biases in support of certain vendors, if other vendors are excluded from effective participation by one of many techniques. In other words, there's no way it can be a level playing field for innovation.
> 
> In that sense, the Linux kernel community has reached a point very much like Microsoft Windows development reached in 1990 or so. I note that date because at that point, Microsoft was challenged with a variety of anti-trust actions based on the fact that it used its Windows monopoly status to put competitors in the application space, and competitors producing innovative operating systems out of business (GO Computer Corporation being one example of many).
> 
> This troubles me. It may not trouble the developers who are in the Linux community and paid by the cartel of companies that control its direction.
> 
> I have no complaint about the technical competence of individual developers - the quality is pretty high, at least as good as those who worked on Windows and macOS. But it's becoming clear that their is a narrowing of control of an OS that has a lot of influence in a few hands. That those few hands don't work for one company doesn't eliminate its tendency to become a cartel. (one that is not transparent at all about functioning as such - preferring to give the impression that the kernel is developed by part-time voluntary "contributions").
> 
> The contrast with other open source communities is quite sharp now. There is little eleemosynary intent that can be detected any more. I think that is too bad, but things change.
> 
> This is just the personal opinion of someone who has been developing systems for 50+ years now. I'm kind of disappointed, but my opinion does not really matter much.
> 
> David
> 
> 
> 
> 
> On Monday, March 29, 2021 9:52pm, "Theodore Ts'o" <tytso@mit.edu> said:
>
>
>
>> On Mon, Mar 29, 2021 at 04:28:11PM -0400, David P. Reed wrote:
>> >
>> >
>> > What tends to shape Linux and FreeBSD, etc. are the money sources
>> > that flow into the communities. Of course Linux is quite
>> > independently wealthy now. The senior executives of the Linux
>> > Foundation are paid nearly a million dollars a year, each. Which
>> > just indicates that major corporations are seriously interested in
>> > controlling the evolution of Linux (not the Gnu part, the part that
>> > has Linus Torvalds at its center).
>> 
>> First of all, I don't believe your salary numbers are correct.
>> 
>> https://nonprofitlight.com/ca/san-francisco/linux-foundation
>> 
>> Secondly, the "senior executives" of the Linux Foundation don't have
>> any control over "the evolution of Linux". The exception to that are
>> the "Fellows" (e.g., Linus Torvalds, Greg K-H, etc.) and I can assure
>> you that they don't take orders from Jim Zemlin, the executive
>> director, or any one else at the Linux Foundation.
>> 
>> The senior developers of Linux do tend to work for the big
>> corporations, but culturally, we do try to keep our "corporate hats"
>> and our "community" hats quite separate, and identify when we our
>> company hats on. Many senior developers have transitioned between
>> multiple companies, and over time, it's been understood that their
>> primarily allegiance is to Linux, and not to the company. In fact,
>> the primary job of maintainers is to say "no" to companies when they
>> try to push crap code into the kernel. And that's because it's the
>> maintainer's responsibility to clean up the mess if they say yes to
>> code that's Just Not Ready, since they have a long-term responsbility
>> towards their subsystem, unlike engineers or contractors that only
>> have a short-term goal to get the code upstream.
>> 
>> This is where having a hierarchial ownership model IMHO works better
>> than a "core team" model where there can be a diffusion of
>> responsibility, where anyone with a commit bit can commit anywhere in
>> the OS. In contrast, David Miller "owns" the networking area, and so
>> someone who might be, say, the ext4 or xfs maintainer does not have
>> the right (read: Linus will reject a pull request from me if I try to
>> change code in the networking stack with out DaveM's signoff) to
>> change code outside of their subsystem.
>> 
>> So you're right that Linus probably doesn't know or care about
>> bufferbloat. He's delegated pretty much all networking issues to
>> David Miller as the networking czar, and within networking, David
>> Miller has his submaintainers with different specialities. This does
>> get complicated when there are changes which cross subsystems. For
>> example, before Wireguard could land in the kernel, there were changes
>> needed in both the crypto and networking layers, and Jason had to
>> negotiate with multiple senior developers in those subsystems, and the
>> code was subject to quite a lot of review before it could land. (It
>> took months, and we didn't try to rush things before a major
>> release....)
>> 
>> > I just spent 9 months trying to get a very tiny fix to the Linux
>> > kernel into the mainline kernel. I actually gave up, because it
>> > seemed utterly pointless, even though it was clearly a design error
>> > that I was fixing, and I was trying to meet all the constraints on
>> > patches. No one was fighting me, no one said it was wrong.
>> 
>> It sounds like the real problem is no one was paying attention to you.
>> There is a *huge* number of changes going into the Linux kernel, and
>> so the the challenge is getting review bandwidth by the relevant
>> maintainers. Blindly posting to the linux-kernel mailing list will
>> generally not get you very far.
>> 
>> The Linux development process is not really optimized for "drive by
>> patching". Knowng where (and to whom) a patch needs to be reviewed is
>> not necessarily easy for a novice, and while there are tools such as
>> ./scripts/get_maintainer.pl that try to make it a bit easier, I can
>> see how someone who Just Wants To Get A Single Patch accepted, can see
>> it as "bureaucracy".
>> 
>> Cheers,
>> 
>> - Ted
>>

[-- Attachment #2: Type: text/plain, Size: 166 bytes --]

_______________________________________________
Make-wifi-fast mailing list
Make-wifi-fast@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/make-wifi-fast

Re: [Cerowrt-devel] wireguard almost takes a bullet

[Dave Taht]

David, I'm sure if you would post your patch for review on
cerowrt-devel someone here would review and help sponsor it to
wherever it belongs in the kernel. One thing I really liked about
what we did with cerowrt and cake is to help a set of new developers
grow to where they could grow in skill and power and influence into
the mainline kernel itself. Linux needs to keep doing
that as we grow older and crunchier.

Aside: While I'm no longer heavily involved in the ipv4 extensions
project it has been kind of fun to delve through the history of ipv4
along with the patchset and fix 30+ yr old bugs(
https://lwn.net/Articles/849970/ )

This is an awfully wide distribution list, but hey, I enjoy water
cooler convos as much as
anybody. Ted, it's nice to see you here.

A bit more below.


On Tue, Mar 30, 2021 at 6:23 PM David P. Reed <dpreed@deepplum.com> wrote:
>
> Theodore -
>
>
>
> I appreciate you showing the LF executive salary numbers are not quite as high as I noted. My numbers may have been inflated, but I've definitely seen a $900,000 package for at least one executive reported in the press (an executive who was transferred in from a F100 company which is close to the LF).

I have generally hoped that LF or some other org of that caliber would
step up to
help clean up the mess that is wifi, the first or second most common
wireless tech
the world uses. Can't do it at any scale without major support and I've almost
given up trying.

Periodically I run for LF's TAB in the hope that embedded linux in general would
gain more visibility in there.

>
>
> On the other hand, they are pretty damn high salaries for a non-profit. Are they appropriate? Depends. There are no stockholders and no profits, just a pretty substantial net worth.
>
>
>
> Regarding the organizaton of "Linux, Inc." as  a hierachical control structure - I'll just point out that hierarchical control of the development of Linux suggests that it is not at all a "community project" (if it ever was). It's a product development organization with multiple levels of management.
>

I remember ICANN and its promise of a democratic internet.

>
>
> Yet the developers are employees of a small number of major corporations. In this sense, it is like a "joint venture" among those companies.
>
>
>
> To the extent that those companies gain (partial) control of the Linux kernel, as appears to be the case, I think Linux misrepresents itself as a "community project", and in particular, the actual users of the software may have little say in the direction development takes going forwards.

Linux seems mostly driven by the data center nowadays.

I would of course like to see more linux use in laptops and the
android messes managed.

As one example of the embedded ecosystem having gone south,
all the security cameras with no linux sources available phoning home to china.

>
>
> There's little safeguard, for example, against "senior management" biases in support of certain vendors, if other vendors are excluded from effective participation by one of many techniques. In other words, there's no way it can be a level playing field for innovation.

I still think that good code makes it into linux better than any other
process we have had. Elsewhere, well, I despair.

>
>
> In that sense, the Linux kernel community has reached a point very much like Microsoft Windows development reached in 1990 or so. I note that date because at that point, Microsoft was challenged with a variety of anti-trust actions based on the fact that it used its Windows monopoly status to put competitors in the application space, and competitors producing innovative operating systems out of business (GO Computer Corporation being one example of many).

I distrust a monoculture in general, and I do wish that we had
processors with better
levels of privilege per our attempts in the 90s to make viable
microkernels. I keep
hoping a grumpy billionaire tired of spectre might find development of
mill computings
architecture, because it's hard to trust any modern computing
environment today, and
certainly not the network.

Recently my new laptop asked if I wanted to install siri, while I was
10 miles out at
sea, and off the internet entirely.

>
>
>
> This troubles me. It may not trouble the developers who are in the Linux community and paid by the cartel of companies that control its direction.
>
>
>
> I have no complaint about the technical competence of individual developers - the quality is pretty high, at least as good as those who worked on Windows and macOS. But it's becoming clear that their is a narrowing of control of an OS that has a lot of influence in a few hands. That those few hands don't work for one company doesn't eliminate its tendency to become a cartel. (one that is not transparent at all about functioning as such - preferring to give the impression that the kernel is developed by part-time voluntary "contributions").
>

It's hard to see much further into the future. I do hope that more
computing ends up on our
own machines and not the cloud. It's why I got so excited about
galene.org - I reallly hated this
past year that sending a videoconference from upstairs to downstairs
had to make a R/T to the cloud. Similarly, dictation, should handled
locally.

this the kind of aiding the disabled thing that I'd like an
institution to be funding and I don't think an LF is going to help
here, even as much as the latter might be good for linux. I'm
frustrated,
being partially blind, how hard it is to use voice prompts and speech
to text in linux,
but I'll be damned if I'll give my early rants to siri. espeak for
emacs is the best I had
and I haven't had it work right in years.


>
>
> The contrast with other open source communities is quite sharp now. There is little eleemosynary intent that can be detected any more. I think that is too bad, but things change.


thx for the new word.

>
>
>
> This is just the personal opinion of someone who has been developing systems for 50+ years now. I'm kind of disappointed, but my opinion does not really matter much.

It helps to tilt at a windmill once in a while. :)

>
>
>
> David
>
>
>
>
>
>
>
>
>
> On Monday, March 29, 2021 9:52pm, "Theodore Ts'o" <tytso@mit.edu> said:
>
> > On Mon, Mar 29, 2021 at 04:28:11PM -0400, David P. Reed wrote:
> > >
> > >
> > > What tends to shape Linux and FreeBSD, etc. are the money sources
> > > that flow into the communities. Of course Linux is quite
> > > independently wealthy now. The senior executives of the Linux
> > > Foundation are paid nearly a million dollars a year, each. Which
> > > just indicates that major corporations are seriously interested in
> > > controlling the evolution of Linux (not the Gnu part, the part that
> > > has Linus Torvalds at its center).
> >
> > First of all, I don't believe your salary numbers are correct.
> >
> > https://nonprofitlight.com/ca/san-francisco/linux-foundation
> >
> > Secondly, the "senior executives" of the Linux Foundation don't have
> > any control over "the evolution of Linux". The exception to that are
> > the "Fellows" (e.g., Linus Torvalds, Greg K-H, etc.) and I can assure
> > you that they don't take orders from Jim Zemlin, the executive
> > director, or any one else at the Linux Foundation.
> >
> > The senior developers of Linux do tend to work for the big
> > corporations, but culturally, we do try to keep our "corporate hats"
> > and our "community" hats quite separate, and identify when we our
> > company hats on. Many senior developers have transitioned between
> > multiple companies, and over time, it's been understood that their
> > primarily allegiance is to Linux, and not to the company. In fact,
> > the primary job of maintainers is to say "no" to companies when they
> > try to push crap code into the kernel. And that's because it's the
> > maintainer's responsibility to clean up the mess if they say yes to
> > code that's Just Not Ready, since they have a long-term responsbility
> > towards their subsystem, unlike engineers or contractors that only
> > have a short-term goal to get the code upstream.
> >
> > This is where having a hierarchial ownership model IMHO works better
> > than a "core team" model where there can be a diffusion of
> > responsibility, where anyone with a commit bit can commit anywhere in
> > the OS. In contrast, David Miller "owns" the networking area, and so
> > someone who might be, say, the ext4 or xfs maintainer does not have
> > the right (read: Linus will reject a pull request from me if I try to
> > change code in the networking stack with out DaveM's signoff) to
> > change code outside of their subsystem.
> >
> > So you're right that Linus probably doesn't know or care about
> > bufferbloat. He's delegated pretty much all networking issues to
> > David Miller as the networking czar, and within networking, David
> > Miller has his submaintainers with different specialities. This does
> > get complicated when there are changes which cross subsystems. For
> > example, before Wireguard could land in the kernel, there were changes
> > needed in both the crypto and networking layers, and Jason had to
> > negotiate with multiple senior developers in those subsystems, and the
> > code was subject to quite a lot of review before it could land. (It
> > took months, and we didn't try to rush things before a major
> > release....)
> >
> > > I just spent 9 months trying to get a very tiny fix to the Linux
> > > kernel into the mainline kernel. I actually gave up, because it
> > > seemed utterly pointless, even though it was clearly a design error
> > > that I was fixing, and I was trying to meet all the constraints on
> > > patches. No one was fighting me, no one said it was wrong.
> >
> > It sounds like the real problem is no one was paying attention to you.
> > There is a *huge* number of changes going into the Linux kernel, and
> > so the the challenge is getting review bandwidth by the relevant
> > maintainers. Blindly posting to the linux-kernel mailing list will
> > generally not get you very far.
> >
> > The Linux development process is not really optimized for "drive by
> > patching". Knowng where (and to whom) a patch needs to be reviewed is
> > not necessarily easy for a novice, and while there are tools such as
> > ./scripts/get_maintainer.pl that try to make it a bit easier, I can
> > see how someone who Just Wants To Get A Single Patch accepted, can see
> > it as "bureaucracy".
> >
> > Cheers,
> >
> > - Ted
> >



--
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729

Re: [Cerowrt-devel] wireguard almost takes a bullet

[Theodore Ts'o]

On Tue, Mar 30, 2021 at 09:23:50PM -0400, David P. Reed wrote:
> 
> On the other hand, they are pretty damn high salaries for a
> non-profit. Are they appropriate? Depends. There are no stockholders
> and no profits, just a pretty substantial net worth.

For better or for worse, senior software engineers who work for Big
Tech will be making similar amounts of money.  Without being
inappropriately specific, my total compensation isn't that different
from Linus's, once you take things like equity compensation into
account.  This was true both in my current employer, as well as my
previous employer (IBM).

Is that right or wrong?  Unfortunately, it is what it is.  Part of it
is that it's very much a supply and demand question.  I know, because
I've tried to find talented file system kernel developers for my
team.... and it's hard to find them.

I know that in many ways, it's hugely unfair.  When I was at IBM, I
was the high powered senior developer who could meet with the senior
technical leaders at a major US defense contractor.  I was the one
with ths TS/SCI security clearance, and yes, I was the senior
architect who got an IBM award for my team's work on real-time Linux
capable of running real-time Java for us in the DDG(1000) Zumwalt
class destroyer.  And yet, the vast majority of the work was done by
much more junior engineers, and they didn't get any of the major
awards, and their salary was much less.  It was one of the best teams
I had the pleasure of working with, and I'm glad to see that they are
now working in much more senior roles at other companies.

So while it's easy to criticize the Linux Foundation, it's by no means
unique to it, and to the extent that it's part of a larger tech
ecosystem that pays engineers in a very disproportionate way, it has
to pay its people commensurate with what they could get if they had
decided to go work for a company like Facebook or Amazon.

> Regarding the organizaton of "Linux, Inc." as a hierachical control
> structure - I'll just point out that hierarchical control of the
> development of Linux suggests that it is not at all a "community
> project" (if it ever was). It's a product development organization
> with multiple levels of management.

So I'd argue that *any* successful, very large open source project
needs to have multiple levels of management.  It's *technical*
managers, and I would point out that it's really more servant
leadership more than anything else.  I may be the ext4 maintainer, but
that means that in order to make ext4 successful, I end up doing the
work that no one else finds "fun" to work for, or that companies
aren't willing to pay engineers to do as part of their day job.  So
for example, the test framework[1] for ext4 is something I had to create
and maintain, because no one else would do it.  And code review is not
necessary *fun*, but someone has to do it.  Much of this work actually
happens late at night or on weekend, on my own time, because I care
enough about it that it's something I *choose* to do it.

[1] https://thunk.org/gce-xfstests 

So if your definition of a "community project" is one which has a
non-scalable governance structure, I'm going to have to disagree.  In
the early 90's, when I first getting started with Linux, there were
attempts from senior leaders at NetBSD and GNU HURD who tried to woo
me to do work for their kernel instead.  Let's just say that even
then, after seeing the toxicity/drama of those governance structures,
(and it didn't help that living in Cambridge, I had the ability to
meet and break bread with some of these senior people face-to-face),
one of the primary *reasons* why I declined to work on *BSD and HURD
was due to the leaders of those projects that I would have had to work
with.  This despite the fact that my first OS/systems programming
experience, courtesy of MIT Project Athena, was on BSD 4.3.

> Yet the developers are employees of a small number of major
> corporations. In this sense, it is like a "joint venture" among
> those companies.
>  
> To the extent that those companies gain (partial) control of the
> Linux kernel, as appears to be the case, I think Linux misrepresents
> itself as a "community project", and in particular, the actual users
> of the software may have little say in the direction development
> takes going forwards.

There are certainly still developers in Linux that are hobbyists, and
not everyone works for Big Tech.  In fact, Jason worked at a startup
that was certainly not what I would call an example of Big Tech.
Sure, his startup let him spend a significant amount of his time
working on getting WireGuard upstream, but WireGuard was very much
accepted on the basis of the merits of his work.  It was not because
someone paid $$$ to the Linux Foundation, or anything crazy like that.

I also started out as a hobbyist.  For a long time, being tech lead
for Kerberos at MIT, and doing IETF standards work (e.g., I was ipsec
working group chair) was my day job, and Linux as my hobby.  Sure, I
was the first North American Linux kernel developer, and that got me
invited to a bunch of conferences who were willing to pay my travel
expenses (since I was a starving academic), but I was paid a very
small salary compared to industry.  (We were wondering why MIT kept on
losing people to industry, so my department brought in a salary
consultant who determined that MIT was paying its people at the tenth
percentile of industry at that time.)  I doubled my salary when I went
to work for a startup, and given that VA Linux Systems imploded before
I was able to sell most of my stock, that figure was *before* any
equity compensation.

Some of the people who were smarter than me, at least in terms of
deciding to go out into industry much sooner, and who were able to
benefit from Red Hat's IPO, have multiple expensive houses and can
travel between them as the ski seasons open up.  And I also know
people working in Indiana contributing to Linux who make a tiny
fraction of what one can make in Big Tech.  I try not to get envious
over those who have done financially much better than I, and I also
try not to think that I'm inherently superior just because I've been
incredibly blessed and lucky and have a very privileged existence.
Life is unfair, and all you can do is to try to your best to make the
world a better place than when you entered it.

> There's little safeguard, for example, against "senior management"
> biases in support of certain vendors, if other vendors are excluded
> from effective participation by one of many techniques. In other
> words, there's no way it can be a level playing field for
> innovation.

The safeguard is in the maintainers' hands.  Remember, we "own" our
subsystems and to the extent that we are passionate to let it be
successful, we'll take the help from whereever we can get it.  I might
be at Company A one year, and Company B another, and if I take crappy
code from one Company, I'll end up owning that crap and I'll
ultimately need to fix it later, perhaps when I'm at another company.

It is true that as a someone who manages volunteers (regardless of
whether such volunteers are hobbyists or people who are doing the work
paid by a certain company), we can't force someone to implement a
particular feature or fix a certain bug.  As I learned from my service
on the IETF, the only power of such leaders is to say "No".  But if we
stop a good feature from getting in, that ultimately is going to be to
the detriment of our subsystem.

And if that does happen for some reason, one of the roles that Linus
plays is as an authority that someone can appeal to.  I've never seen
a support for some CPU architecture get denied just because it might
threaten existing "Big Companys", for example.  I'm sure the ARM SOC's
weren't happy to see RISC-V support land in the kernel.  But if there
was an attempt to keep RISC-V out of the kernel, that's a case where
Linus would intervene, since ultimately it's *his* choice to accept a
new subsystem and a new maintainer.

> (one that is not transparent at all about functioning as such -
> preferring to give the impression that the kernel is developed by
> part-time voluntary "contributions").

Actually, the Linux Foundation has been quite transparent about this;
every few years, it relases a "Who Writes Linux Report".  Anyone who
had such an impression hasn't been paying attention:

https://www.linuxfoundation.org/wp-content/uploads/linux-kernel-report-2017.pdf
https://www.linuxfoundation.org/wp-content/uploads/2020_kernel_history_report_082720.pdf

From these reports, you'll see that in 2017 we had 8.2% of the
contributions coming from people weren't getting financial
contributions (with another 4.1% where the source of financial support
couldn't be determined).  This was down from the 2013 report, where
14.6% of the contributions came from hobbyists.

In the 2020 report, "None" was 11.95%, with the next highest
contributor being Intel at 10%, Red Hat at 8.9%, "Unknown" at 4%, and
IBM at 3.8%.  (Google was much farther down the list, at 2.8%).  Not
to put too fine a point on it, "people who receive no financial
contributions" are #1 on the "Top 20 committers list".

> The contrast with other open source communities is quite sharp
> now. There is little eleemosynary intent that can be detected any
> more. I think that is too bad, but things change.

If you look at the members of the Git, Perl and Python communitiesn, I
believe you'll find that most of the major contributors do work for
companies that pay for at least part of their open source
contributions.  Given that most people do enjoy having food with their
meals, if a OSS project is successful, this really shouldn't be a
surprise.

It is true that there is a huge long tail of OSS projects which have
not been successful, and which only exist as abandonware on
SourceForge or GitHub.  (Or in the case of OpenOffice, as part of the
Apache Consortium :-P)  But just as the vast majorities of startups end
up emploding with less than 1% becoming the IPO unicorns, I'm not so
sure it's anything more than sour graps for people to claim that the
startups which made it big were never "real startups" to begin with,
and that the story of startups is all a Big Lie.

Cheers,

						- Ted

Re: [Cerowrt-devel] [Bloat] wireguard almost takes a bullet

[Jonathan Corbet]

"David P. Reed" <dpreed@deepplum.com> writes:

> Regarding the organizaton of "Linux, Inc." as a hierachical control
> structure - I'll just point out that hierarchical control of the
> development of Linux suggests that it is not at all a "community
> project" (if it ever was). It's a product development organization
> with multiple levels of management.
>  
> Yet the developers are employees of a small number of major
> corporations. In this sense, it is like a "joint venture" among those
> companies.

...where "a small number" == 225 for the 5.11 development cycle; the
biggest of those contributed just under 10% of the patches.

	https://lwn.net/Articles/845831/

It seems rather less concentrated than many projects out there, and it
has become less so over time.

> To the extent that those companies gain (partial) control of the Linux
> kernel, as appears to be the case, I think Linux misrepresents itself
> as a "community project", and in particular, the actual users of the
> software may have little say in the direction development takes going
> forwards.
>  
> There's little safeguard, for example, against "senior management"
> biases in support of certain vendors, if other vendors are excluded
> from effective participation by one of many techniques. In other
> words, there's no way it can be a level playing field for innovation.

I would be curious to hear whether you think there is evidence of
vendors being excluded?  No doubt something has happened somewhere, but
I am unaware if widespread use of "one of many techniques" and would
certainly appreciate being enlightened.

The biggest impediment to innovation in Linux, I think, is the massive
installed user base and the need to never break anything for anybody —
along with the increase in complexity overall.  Just look at what it has
taken (and is still taking) to get us past the year-2038 problem
relative to how some other systems have been able to just deal with it,
for example.

WireGuard, which started this discussion, is a good example to look at
it.  This was definitely an innovative development, brought in by a
developer previously unknown to the community and, as far as I know, not
in the thrall of any of the corporations contributing to the Linux
kernel.  Jason had to jump through all sorts of hoops to get it in, but
that wasn't the result of anybody trying to block it - we wanted it!
But it did need to fit into what we had already, and that took some
work. 

Thanks,

jon

Re: [Cerowrt-devel] wireguard almost takes a bullet

[Stephen Hemminger]

> contributions" are #1 on the "Top 20 committers list".
> 
> > The contrast with other open source communities is quite sharp
> > now. There is little eleemosynary intent that can be detected any
> > more. I think that is too bad, but things change.  
> 
> If you look at the members of the Git, Perl and Python communitiesn, I
> believe you'll find that most of the major contributors do work for
> companies that pay for at least part of their open source
> contributions.  Given that most people do enjoy having food with their
> meals, if a OSS project is successful, this really shouldn't be a
> surprise.
> 
> It is true that there is a huge long tail of OSS projects which have
> not been successful, and which only exist as abandonware on
> SourceForge or GitHub.  (Or in the case of OpenOffice, as part of the
> Apache Consortium :-P)  But just as the vast majorities of startups end
> up emploding with less than 1% becoming the IPO unicorns, I'm not so
> sure it's anything more than sour graps for people to claim that the
> startups which made it big were never "real startups" to begin with,
> and that the story of startups is all a Big Lie.

I get the impression that there are some folks that have a deep seated
bias on how they believe the Linux kernel works and therefore are unable
to change their point of view even when presented with data.

Kind of reminds me of the people I run into the FOSS community
who still believe Microsoft is forever going to be evil.
Scott Hassellman calls those people "You killed my pappy"

Re: [Cerowrt-devel] wireguard almost takes a bullet

[Theodore Ts'o]

On Tue, Mar 30, 2021 at 09:15:04PM -0700, Dave Taht wrote:
> David, I'm sure if you would post your patch for review on
> cerowrt-devel someone here would review and help sponsor it to
> wherever it belongs in the kernel. One thing I really liked about
> what we did with cerowrt and cake is to help a set of new developers
> grow to where they could grow in skill and power and influence into
> the mainline kernel itself. Linux needs to keep doing
> that as we grow older and crunchier.

I'll make the same offer.  I don't specialize in the networking
subsystem, but I'm happy to mentor new kernel contributors and to help
make sure that patches can be directed to right place so they can
receive the attention they deserve.  (This is not a guarantee that the
patches will go in, of course.)

Cheers,

					- Ted

> This is an awfully wide distribution list, but hey, I enjoy water
> cooler convos as much as
> anybody. Ted, it's nice to see you here.

Dave, if you think we've gone too far afield, just say the word.  One
of tehr easons why I stay on some of these lists is because they
aren't terribly high traffic, and I don't want to helping to drive
people away because the signal to nosie ratio on a particular list
gets pushed too low...