[NNagain] The rise and fall of the 90's telecom bubble

Dave Taht dave.taht at gmail.com
Mon Nov 13 06:28:32 EST 2023


On Sun, Nov 12, 2023 at 3:23 PM Nathan Simington via Nnagain
<nnagain at lists.bufferbloat.net> wrote:
>
> Hi Dave,
>
> Re not seeing productivity gains, I'm very interested in seeing telecom start to translate more into productivity, including nuts-and-bolts manufacturing productivity.

This is one of those vast and deep subjects very hard to cover even in
long form emails! I go back to some of the earlier developments in
factory floor automation like the Boeing's "HIP" protocol, and I have
only current familiarity with smart AG than shop automation. Hopefully
we can find more people to comment on the state of the art there. Karl
(a member here) has had a grand time red-teaming  network impairments
into multiple robotics experiments. While his videos of robots failing
are hilarious, the underlying causes ill understood... and in some
cases, fatal, in a real deployment.

> (From a 30-year backward perspective, you could reasonably argue IMHO that much of the ideology that "deindustrialization is good" was generated to justify software companies' valuation multipliers.

Yes.

The short term margins in just software were appealing. But once you
lose control of the underlying platforms your software needs to run
on, things get dicy fast.

How to regain a long term perspective? I have many bugaboos here,
things like shifting academia away from publishing papers to running
code, or repurposing the patent system... but not today.

>The Chinese don't seem to agree that deindustrialization is good or that it's a bad idea to hold production assets on-balance sheet.

Agree.

>Meanwhile, it's been historically much easier to make money in flaky, consumer-grade software than in reliable, infrastructure-grade smart manufacturing/logistics --

I need to unpack three things from the above: "Cheap, consumer-grade
software", for starters.

I sometimes reflect on how well the nation's septic systems work after
100s of years of development, and yet how few understand why an "air
return" is needed until they finally encounter the side effects of a
clogged one.

This week, taking place, is the premier "Linux Plumbers" conference
at 100 S 12th St, Richmond, VA 23219. People can also attend remotely.
see https://lpc.events/ for more details.

It would be good for less technical folk to at least do the hallway
track. They might find technologists worth hiring.

...

Building reliable *infrastructure* software is a different beast than
"consumer", and what it takes, in terms of design, development,
continuous integration, fuzz testing, deployment, and maintenance of
hundreds of millions of lines of code in order to evolve this
fundamental OS component of an "Advanced telecom infrastructure"  is
ill understood ... because we do our job too well most of the time.

Speaking there this week is one of my favorite computer scientists,
Paul McKenney, who after inventing "stochastic fair queuing" (SFQ) in
the 90s, in his "RCU" work with tandem, sequent, ibm, and now meta
since, went on to working on solving classes of paralellism's "million
year problems" - bugs that only happen once in a million years for
multi-core architectures, or every few minutes on some machine out of
a billion, somewhere.

If you know anyone in DC tired of reading or writing law, I recommend
a journey through Paul's lighter book "Is parallel programming hard?"
https://arxiv.org/abs/1701.00854 - particularly the chapter on mere
parallel counting problems.

Parallel software designed for distributed, loosely coupled systems
like the internet is a whole additional shelf of books.

There is another class of realtime software that controls our cars,
milling machines and essential infrastructure.

I am not sure if I understand what you mean by "cheap, consumer
software"? I start by breaking things down into "can this software
accidently kill people, or not?". Others might break it down more into
"will this software improve productivity, or if used, what are the
potential financial damages of a security breach?".

Finally moving to your logistics point... well, I miss older tools for
planning ahead potentially years such as gaant charts.

> even though it's clear that the latter is the real prize, just as steel mills were a bigger prize for the 18th-c UK than faster post-horses or cheaper India ink would have been.)

I boggled at the outsourcing trends that culminated in the Boeing 787
mess. But more on that and QA techniques like TQM, or six sigma, or
skunkworks, later.

I admire the kind of vertical integration Apple and Spacex now do
instead, and the kind of fail-fast testing anyone can now do at
internet scale.

>
> I think a lazy, vague equation between "good 5G" and "Chinese-style smart manufacturing" has had a lot of policy salience in the last 5 years. Would love to spend some time thinking together about what a smart manufacturing system would look like in terms of connectivity, latency, compute availability, anything that occurs to you. I know a guy who does devops for factories, and he has amazing stories -- might be good to make that connection as well.

I would hope there exists a forum, mailing list, or organisation that
focuses more sharply on these issues.

>Also, I think you had a catch-up with Adam from my team about wireless ISPs/improved routers -- hope that went well!

I always enjoy puncturing holes in the legal "network neutral"
conceptions of common carriage with entertaining analogies about how
the ubiquitous tcp slow start (web pages, file transfers) and
voip/gaming/videoconferencing algorithms used to be mutually
incompatible... as I did in four minutes with them:

https://www.youtube.com/watch?v=TWViGcBlnm0&t=16m20s

and I hope pass along.

IF we could somehow get the intuition across to millions more - or a
sufficient number of those in power - that the internet's behavior is
governed by:

" we constantly probe for more bandwidth by throwing ever more packets
until we lose one, slow down, retransmit, gradually increase the speed
until we get another drop, slow down, retransmit and then... imagine
trillions of applications like these governed by these two simple
rules, transmitting data over billions of tubes interconnected by
often overlarge funnels, and designed by madmen..."

We could make enormous progress along the edge in a matter of months,
rather than decades.

But: in an informal survey of 30+ "regular" folk over the past few
weeks, not a single one could describe what "a packet" was, which is
the fundamental underlying "atom" out of which the internet is
constructed. I did not know this before! It never occurred to me
before now that how data got from one place to another was considered
to be akin to magic.

It's a fine mess we're in, ollie.
> Take care,
> Nathan
>
> On Sun, Nov 12, 2023 at 10:49 AM Dave Taht via Nnagain <nnagain at lists.bufferbloat.net> wrote:
>>
>> Aside from me pinning the start of the bubble closer to 1992 when
>> commercial activity was allowed, and M&A for ISPs at insane valuations
>> per subscriber by 1995 (I had co-founded an ISP in 93, but try as I
>> might I cannot remember if it peaked at 50 or 60x1 by 1996 (?) and
>> crashed by 97 (?)), this was a whacking good read, seems accurate, and
>> moves to comparing it across that to the present day AI bubble.
>>
>> https://www.fabricatedknowledge.com/p/lessons-from-history-the-rise-and
>>
>> In the end we sold (my ISP, founded 93) icanect for 3 cents on the
>> dollar in 99, and I lost my shirt (not for the first time) on it, only
>> to move into embedded Linux (Montavista) after the enormous pop
>> redhat's IPO had had in 99. The company I was part of slightly prior
>> (Mediaplex) went public December 12, 1999 and cracked 100/share, only
>> to crash by march, 2000 to half the IPO price (around $7 as I recall),
>> wiping out everyone that had not vested yet. I lost my shirt again on
>> that and Montavista too and decided I would avoid VCs henceforth.
>>
>> I am always interested in anecdotal reports of personal events in this
>> increasingly murky past, and in trying to fact check the above link.
>>
>> So much fiber got laid by 2000 that it is often claimed that it was at
>> least a decade before it was used up, (the article says only 2.7% was
>> in use by 2002) and I have always wondered how much dark, broken,
>> inaccessible fiber remains that nobody knows where it even is anymore
>> due to many lost databases. I hear horror stories...
>>
>> The article also focuses solely on the us sector, and I am wondering
>> what it looked like worldwide.
>>
>> I believed in the 90s we were seeing major productivity gains. The
>> present expansion of the internet in my mind should not be much
>> associated with "productivity gains", as, imho, reducing the general
>> population to two thumbs and a 4 inch screen strikes me as an enormous
>> step backwards.
>>
>> (I have a bad habit of cross posting my mails to where older denizens
>> of the internet reside, sorry! If you end up posting to one of my
>> lists I will add a sender allows filter for you)
>> --
>> :( My old R&D campus is up for sale: https://tinyurl.com/yurtlab
>> Dave Täht CSO, LibreQos
>> _______________________________________________
>> Nnagain mailing list
>> Nnagain at lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/nnagain
>
>
>
> --
> Nathan Simington
> cell: 305-793-6899
> _______________________________________________
> Nnagain mailing list
> Nnagain at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/nnagain



--
:( My old R&D campus is up for sale: https://tinyurl.com/yurtlab
Dave Täht CSO, LibreQos


More information about the Nnagain mailing list