The Markeley Group in Boston charges $2500 to $5000 per month to lease a dark fiber strand into their colo. The labor to install is on the wanna be isp. One has to provide the colo switches and pay Colo fees, peering fees and transit fees. The only long lived parts are the fiber, patch panels and building which are not sufficient by themselves to operate a modern ISP. Switch tech is upgrading on silicon mfg cycles. Similar with optics though the MTBF is less than silicon, hence pluggables. And that's just interconnections and ignores compute.

https://www.markleygroup.com/

Then there are OSP pedestals and actives, optical amps e.g. via doped fiber,  too. Operating an optical OSP requires a skilled labor force.

I don't see a muni funding model that works. Maybe in 20 or 30 years but not today. I wish it were different. I think it is nice as an ideal but the economics don't seem to work. If they did, we would see the munis everywhere already. It's been proposed for over two decades now and nothing substantial has been built.

Bob
On Oct 9, 2023, at 11:13 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
Hi Bob.

On 10 October 2023 02:13:18 CEST, Robert McMahon <rjmcmahon@rjmcmahon.com> wrote:
Hi Sebastian,

The NRE per chip starts at $100M. It's multiplr semiconductors that now define a networks and data centers capabilitied. A small municipal overbuilder is not a market maker.

[SM] Sure, a small outfit is essentially forced to use off the shelf components, though they might innovate a bit on the software side, like libreqos. But for one of the biggest current challenges, getting fiber out to all residences/businesses is that really an issue?



So yes, an overbuilder that can't fund ASIC NRE needs to be intimately aware of both market dynamics and the state of engineering, of today, tomorrow and the next 20-30 years as that's typically the life of the municipal bonds.

[SM] In a dark fiber model, this will not matter too much, no? For a lighted fiber approach 20-30 years require 2-4 technology generations which seems hard to predict... then again the biggest cost is likely the fiber access network, the active tech might by financable out of the cash flow?



Investors aren't govt. bond holders and investor owned companies can take more risk. If low latency offerings don't increase ARPU, the investors lose. If it works, they win. Big difference.

[SM] Building that fiber plant seems like a pretty save bet to me, allowing for longterm financing. Interestigly over here some insurrances got into the FTTH build-out game, obviously considering it a viable long term investment, though population density is higher here than in the US likely affecting cost and amortisation periods...


⁣Bob

On Oct 9, 2023, 12:40 AM, at 12:40 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
Hi Bob,

On Oct 8, 2023, at 22:44, rjmcmahon <rjmcmahon@rjmcmahon.com> wrote:

Yeah, I get it. I think we're just too early for a structural
separation model in comm infrastructure.

[SM] I see one reason why we should not wait, and that is the
future-proofness of the eventually reached FTTH-deployment...



I think when we get to mix & match DSP/optics and point to point
fiber in the OSPs, as done in data centers, it may change. But today
it's PON at best which implies a communal decision process vs
individual one.

[SM] There are IMHO two components to the AON (Point to
MultiPoint/PtMP)/PON (Point to Point/PtP) debate:
a) PONs are cheaper to operate, as they require less power (on the ISP
side) and space (depending on where the passive splitters are located).
b) Structural PONs with splitters out in the field (to realize space
saving in the CO) are less flexible, one can always operate a PtMP
plant as PON, but converting a structural PON to AON likely requires
putting new fibers into the ground.

The first part is something I am not too concerned about, the coming
FTTH access network is going to operate for decades, so I could not
care less about what active technology is going to be used in the next
decade (I assume that ISPs tryto keep the same tech operational for ~ a
decade, but for PON that might be too pessimistic), the second part is
different though... micro-economics favor PtMP with splitters in the
field (lower up front cost* AND less potential for regulatory
intervention**) while the macro-economic perspective makes PtP more
attractive (offering more flexibility over the expected life time of
multiple decades).



*) One big item, the cost for actually deploying the fiber to is not
all that sensitive, if you put fiber into the ground the traditional
way, typically the cost of the earth works dominates over e.g. the cost
of the individual fibers (not that this would stop bean-counter types
to still minimize the number of fiber cables...).

**) With PtP the potential exists that a regulator (likely not the FCC)
could force an ISP to offer dark-fibers to end customers at wholesale
prices, with PtMP the ISP having build out likely will stay in control
of the active tech in each segment (might be forced to offer bitstream
access***) so potential competitors will not be able to offer
better/faster technology on the shared fiber. That is some of the PONs
are backward compatible and in theory on the same PON tree one ISP
might be operating GPON while another ISP might theoreticallu offer
XGS-PON on the same segment, but I think this is a rather theoretical
construct unlikely to happen quantitatively...

***) Not only control of the tech, but offering bitstream access likely
means a larger wholesale price as well.


Communal actions, as seen in both LUS and Glasgow, can take decades
and once done, are slow to change.

[SM] LUS already offer symmetric 10G links... they do not seem to be
lagging behind, the main criticism seems to be that they are somewhat
more expensive than the big ISPs, which is not all that surprising
given that they will not be able to leverage scale effects all that
much simply by being small... Also a small ISP likely can not afford a
price war with a much larger company (that can afford to serve below
cost in areas it competes with smaller ISPs in an attempt to drive
those smaller ones out of the market, after which prices likely
increase again).


The decision process time vs tech timelines exacerbate this. Somebody
has to predict the future - great for investors & speculators, not so
for regulators looking backwards.

[SM] I am not convinced that investors/speculators actually do a much
better job predicting the future, just look at how much VC is wadted on
hare-brained schemes like NFTs/crypto currencies and the like?

Also, engineering & market cadence matching is critical and neither
LUS nor Glasgow solved that.

[SM] But do they need to solve that? Would it not be enough to simply
keep offering something that in their service area is considered good
enough by the customers? Whether they do or do not, I can not tell.


Regards
Sebastian



Bob
Hi Bob,
On Oct 8, 2023, at 21:27, rjmcmahon <rjmcmahon@rjmcmahon.com>
wrote:
Hi Sebastian,
Here's a good link on Glasgow, KY likely the first U.S. muni
network started around 1994. It looks like a one and done type
investment. Their offering was competitive for maybe a decade and now
seems to have fallen behind for the last few decades.
https://www.glasgowepb.com/internet-packages/
https://communitynets.org/content/birth-community-broadband-video
[SM] Looks like they are using DOCSIS and are just about to go
fiber;
not totally unexpected, it takes awhile to amortize the cost of say
a
CMTS to go DOCSIS and only after that period you make some profit,
so
many ISPs will be tempted to operate the active gear a bit longer
longer after break even, as with new active gear revenue will likely
not generate surplus. The challenge is to decide when to upgrade...
My preferred model however is not necessarily having a communal ISP
that sells internet access services (I am not against that), but
have
a communal built-out of the access network and centralize the lines
(preferably fiber) in a few large enough local IXs, so internet
access
providers only need to bring their head-ends and upstream links to
those locations to be able to offer services. In the beginning it
makes probably sense to also offer some sort of GPON/XGSPON bit
stream
access to reduce the up-front cost for ISPs that expect to serve
only
a small portion of customers in such an IX, but that is pure
speculation.... The real idea is to keep those things that will
result
in a natural monopoly to form in communal hands (that already manage
other such monopoly infrastructure anyway) and then try to use the
fact that there is no local 800lb Gorilla ISP owning most lines to
try
to create a larger pool of competing ISPs to light up the fiber
infrastructure... That is I am fine with a market solution, if we
can
assure the market to be big enough to actually deliver on its
promises.
LUS is similar if this article is to be believed.
https://thecurrentla.com/2023/column-lus-fiber-has-lost-its-edge/
[SM] The article notices that comparing things is hard... as the
offers differ considerably from what alternate ISPs offer (e.g. LUS
offers symmetric capacity for down- and upload) and the number
compared seems to be the advertised price, which IIRC in the US is
considerably smaller than what one happens to actually pay month per
month due to additional fees and stuff... (in Germany prices for
end-customers typically are "all inclusive prices", the amount of
VAT/tax is shown singled out in the receipts, but the number we
operate on is typically the final price, but then we have almost no
local taxes that could apply).
The LUS NN site says there is no congestion on their fiber (GPON)
so they don't need AQM or other congestion mgmt mechanisms which I find
suspect. https://www.lusfiber.com/net-neutrality
[SM] Actually intriguing, would I live in their area I would try
them
out, then I could report on the details here :)
Browsing their documentation I am not a big fan of their volume
limits
though, I consider these to be absurd measures of control....(absurd
in that they are too loosely coupled with the relevant measure for
the
actual cost).
This may demonstrate that technology & new requirements are moving
too quickly for municipal approaches.
[SM] That might well be true. I have no insight any more on how
this
affects commercial ISPs in the US either (I only tried two anyway
sonic and charter)
Bob
Hi Sebastian,
The U.S. of late isn't very good with regulatory that motivates
investment into essential comm infrastructure. It seems to go the
other way, regulatory triggers under investment, a tragedy of the
commons.
The RBOCs eventually did overbuild. They used wireless and went to
contract carriage, and special access rate regulation has been
removed. The cable cos did HFC and have always been contract
carriage.
And they are upgrading today.
The tech companies providing content & services are doing fine too
and
have enough power to work things out with the ISPs directly.
The undeserved areas do need support. The BEAD monies may help. I
think these areas shouldn't be relegated to DSL.
Bob
On Oct 8, 2023, at 2:38 AM, Sebastian Moeller <moeller0@gmx.de>
wrote:
Hi Bob,
On 8 October 2023 00:13:07 CEST, rjmcmahon via Nnagain
<nnagain@lists.bufferbloat.net> wrote:
Everybody abandoned my local loop. Twisted pair from multiple
decades ago into antiquated, windowless COs with punch blocks,
with no space nor latency advantage for colocated content &
compute, seems to have killed it off.
[SM] Indeed, throughput for DSL is inversely proportional to loop
length, so providing 'acceptable' capacity requires sufficiently
short wire runs from DSLAM to CPE, and that in turn means moving
DSLAMs closer to the end users... which in a densely populated
area
works well, but in a less densely populated area becomes costly
fast. And doing so will only make sense if you get enough
customers
on such an 'outdoor DSLAM' so might work for the first to built
out,
but becomes prohibitively unattractive for other ISP later.
However
terminating the loops in the field clears up lots of spaces in
the
COs... not that anybody over here moved much compute into
these...
(there exist too many COs to make that an attractive proposition
in
spite of all the hype about moving compute to the edge). As is a
few
well connected data centers for compute seem to work well
enough...
I suspect in some towns one can buy out the local loop copper
with
just a promise of maintenance.
[SM] A clear sign of regulatory failure to me, maintenance of the
copper plant inherited from Bell should never have been left to
the
ISPs to decide about...
The whole CLEC open the loop to competitive access seems to have
failed per costs, antiquated technology, limited colocation, an
outdated waveguide (otherwise things like CDDI would have won
over
Cat 5), and market reasons. The early ISPs didn't collocate, they
bought T1s and E1s and connected the TDM to statistical
multiplexingThe FCC has stated that “rigging or slanting the news is a
most heinous act against the public interest.”
- no major investment there either.
The RBOCs, SBC (now AT&T) & and VZ went to contract carriage and
wireless largely because of the burdens of title II per
regulators
not being able to create an investment into the OSPs. The 2000
blow up was kinda real.
[SM] Again, I see no fault in title 2 here, but in letting ISPs
of
the hook on maintaining their copper plant or replace it with
FTTH...
She starts out by complaining about trying to place her WiFi in
the right place. That's like trying to share a flashlight. She
has
access to the FCC technology group full of capable engineers.
They
should have told her to install some structured wire, place more
APs, set the carrier and turn down the power.
[SM] I rather read this more as an attempt to built a report with
the audience over a shared experience and less as a problem
report
;)
My wife works in the garden now using the garden AP SSID with no
issues. My daughter got her own carrier too per here Dad
dedicating
a front end module for her distance learning needs. I think her
story to justify title II regulation is a bit made up.
[SM] Hmm, while covid19 lockdown wasn't the strongest example, I
agree, I see no good argument for keeping essential
infrastructure
like internet access in private hands without appropriate
oversight.
Especially given the numbers for braodband choice for customers,
clearly the market is not going to solve the issues at hand.
Also, communications have been essential back before the rural
free delivery of mail in 1896. Nothing new here other than
hyperbole to justify a 5 member commission acting as the single
federal regulator over 140M households and 33M businesses,
almost
none of which have any idea about the complexities of the
internet.
[SM] But the access network is quite different than the
internet's
core, so not being experts on the core seems acceptable, no? And
even 5 members is clearly superior to no oversight at all?
I'm not buying it and don't want to hand the keys to the FCC who
couldn't protect journalism nor privacy. Maybe start there,
looking
at what they didn't do versus blaming contract carriage for a
distraction?
[SM] I can speak to the FCC as regulatory agency, but over here
IMHO
the national regulatory agency does a decent job arbitrating
between
the interests of both sides.

https://about.usps.com/who/profile/history/rural-free-delivery.htm#:~:text=On%20October%201%2C%201896%2C%20rural,were%20operating%20in%2029%20states.
Bob
My understanding, though I am not 100% certain, is that the baby
bells
lobbied to have the CLEC equal access provisions revoked/gutted.
Before this, the telephone companies were required to provide
access
to the "last mile" of the copper lines and the switches at
wholesale
costs. Once the equal access provisions were removed, the
telephone
companies started charging the small phone and DSL providers
close
to
the retail price for access. The CLEC DSL providers could not
stay
in
business when they charged a customer $35 / month for Internet
service
while the telephone company charged the DSL ISP $35 / month for
access.
---- On Sat, 07 Oct 2023 17:22:10 -0400 Dave Taht via Nnagain
wrote
---
I have a lot to unpack from this:
https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf
the first two on my mind from 2005 are: "FCC adopted its first
open
internet policy" and "Competitiveness" As best as I recall, (and
please correct me), this led essentially to the departure of all
the
3rd party DSL providers from the field. I had found something
referencing this interpretation that I cannot find right now, but
I
do
clearly remember all the DSL services you could buy from in the
early
00s, and how few you can buy from now. Obviously there are many
other
possible root causes.
DSL continued to get better and evolve, but it definately suffers
from
many reports of degraded copper quality, but does an estimate
exist
for how much working DSL is left?
Q0) How much DSL is in the EU?
Q1) How much DSL is left in the USA?
Q2) What form is it? (VDSL, etc?)
Did competition in DSL vanish because of or not of an FCC related
order?
--
Oct 30:

https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos
-------------------------
Nnagain mailing list
Nnagain@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/nnagain
-------------------------
Nnagain mailing list
Nnagain@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/nnagain
-------------------------
Nnagain mailing list
Nnagain@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/nnagain


Nnagain mailing list
Nnagain@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/nnagain