[LibreQoS] [Starlink] GPON vs Active Fiber Ethernet

Fri Mar 17 16:36:39 EDT 2023

Hi Dave,

> On Mar 17, 2023, at 18:05, Dave Taht via Starlink <starlink at lists.bufferbloat.net> wrote:
> 
> On Fri, Mar 17, 2023 at 9:54 AM David Fernández via Starlink
> <starlink at lists.bufferbloat.net> wrote:
>> 
>> Hi Dave,
>> 
>> Telefonica achieved considerable gains in efficiency dismantling the
>> copper network (85% gains in energy efficiency).
>> 
>> Read here: https://www.xataka.com/empresas-y-economia/paso-cobre-adsl-a-fibra-ha-sido-beneficioso-para-todos-especialmente-para-telefonica
> 
> I am not prepared to make a coherent argument (although I enjoy them),
> but I look at the serialization delay of gpon of 250us and other

	GPON has no strict 250µs serialization delay, it uses time slots of 250µs... for its accounting... that is IMHO the timescale for the grant-request cycle and considerably below what DOCSIS gets away with today.

> potential multiplexing problems,

	For downstream the OLT simply encrypts with the target ONT's key and "broadcasts it to all ONTs that are listening on the PON-tree. For upstream we have a rather typical request grant scheme like in DOCSIS where the OLT assigns transmit slots to the ONT's so these do not step on each other (the ONTs as far as I can tell can not see/hear each other's transmissions so the OLT needs to arbitrate.)

> and go, "yuck". It really, really,
> really is the RTT that dominates the performance of all our networking
> technologies, and no matter how much bandwidth you have, nearly every
> per-sub tech I know of has a floor above 250us today. Multiple hops of
> mmwave are particularly bad here.

	I would assume that the actual delay from sending a request to sending the first bit is probably more than one cycle, so the access delay is likely >> 250µs, compared to other delays this still seems to be quite fine to me... (DSL operate with a 4 KHz "clock" as well, but without request-grant since the last mile is not directly shared*)

*) Vectoring bring in a certain fate sharing, but still does not bring request grant delays.

> Modern active fiber ethernet is 10000x quicker (nanoseconds!), very
> standardized, increasingly low power (with a ton of bang for the
> buck/gbit), with gear available everywhere that "just works",
> leveraging the past decade of rapid development in the data center,
> with USED 1Gbit SFPs going into trashbins everywhere.
> 
> ... while gpon is, if anything, more proprietary than cable, the gear,
> more expensive, and did I mention the latency?
> 
> /me dons flame retardant suit

	No need, I am not a PON fan, but given its cost benefits (power and central office space, a single OLT port can terminate anywhere from 64 (GPON) to 128 (XGSPON) ONTs**, and finally regulatory control***) that is what we are going to get, and honestly it could be worse.

	I am not sure GPON is more expensive than cable stuff, it seems less proprietary than DOCSIS (which is CableLabs or nothing) and is described in ITU standards. I think there is an open source OLT software stack (voltha?) so this looks not that bad compared to what I know about DOCSIS... and latency wise it smokes DOCSIS... heck look at low-latency docsis instead of tackling the elephant in the room the slow request-grant cycling they essentially added L4S... (they also propose a slight speed-up of the grant mechanism IIRC, but nothing close to 250µs epochs).

Regards
	Sebastian

**) The big incumbent over here mosty aims for <= 32 users on an OLT port...
***) An ISP operating PONs can no really be forced to rent out dark fibers to competitors, with a PON realistic worst case is having to offer some sort of bitstream access where the original ISP is still tightly in control. As enduser I do not think this to be a good outcome, yet it seems to factor in technology decisions ISPs make. E.g. deutsche glasfaser started with AON but mostly switched to PON for new build outs.

P.S.: One thing I always thought great about any kind of AON is that direct traffic to my neighbors would never have to leave the local switch, but that is not how ISPs over here operate, all links are terminated somewhere semi-centrally (incumbent ~900 places, my ISP ~9 places) and any traffic will only cross over at that central site anyway.

> 
>> 
>> Regards,
>> 
>> David
>> 
>>> Date: Fri, 17 Mar 2023 09:38:03 -0700
>>> From: Dave Taht <dave.taht at gmail.com>
>>> To: Mike Puchol <mike at starlink.sx>
>>> Cc: Dave Taht via Starlink <starlink at lists.bufferbloat.net>, Rpm
>>>      <rpm at lists.bufferbloat.net>,  libreqos
>>>      <libreqos at lists.bufferbloat.net>, bloat <bloat at lists.bufferbloat.net>
>>> Subject: Re: [Starlink] [Rpm]  On FiWi
>>> Message-ID:
>>>      <CAA93jw6EWH19Jo-pUJMX7QCi=GfHD8iWTDKCcRY=0tEOe4Vt1Q at mail.gmail.com>
>>> Content-Type: text/plain; charset="utf-8"
>>> 
>>> This is a pretty neat box:
>>> 
>>> https://mikrotik.com/product/netpower_lite_7r
>>> 
>>> What are the compelling arguments for fiber vs copper, again?
>>> 
>>> 
>>> On Tue, Mar 14, 2023 at 4:10 AM Mike Puchol via Rpm <
>>> rpm at lists.bufferbloat.net> wrote:
>>> 
>>>> Hi Bob,
>>>> 
>>>> You hit on a set of very valid points, which I'll complement with my views
>>>> on where the industry (the bit of it that affects WISPs) is heading, and
>>>> what I saw at the MWC in Barcelona. Love the FiWi term :-)
>>>> 
>>>> I have seen the vendors that supply WISPs, such as Ubiquiti, Cambium, and
>>>> Mimosa, but also newer entrants such as Tarana, increase the performance
>>>> and on-paper specs of their equipment. My examples below are centered on
>>>> the African market, if you operate in Europe or the US, where you can
>>>> charge customers a higher install fee, or even charge them a break-up fee
>>>> if they don't return equipment, the economics work.
>>>> 
>>>> Where currently a ~$500 sector radio could serve ~60 endpoints, at a cost
>>>> of ~$50 per endpoint (I use this term in place of ODU/CPE, the antenna
>>>> that
>>>> you mount on the roof), and supply ~2.5 Mbps CIR per endpoint, the
>>>> evolution is now a ~$2,000+ sector radio, a $200 endpoint, capability for
>>>> ~150 endpoints per sector, and ~25 Mbps CIR per endpoint.
>>>> 
>>>> If every customer a WISP installs represents, say, $100 CAPEX at install
>>>> time ($50 for the antenna + cabling, router, etc), and you charge a $30
>>>> install fee, you have $70 to recover, and you recover from the monthly
>>>> contribution the customer makes. If the contribution after OPEX is, say,
>>>> $10, it takes you 7 months to recover the full install cost. Not bad,
>>>> doable even in low-income markets.
>>>> 
>>>> Fast-forward to the next-generation version. Now, the CAPEX at install is
>>>> $250, you need to recover $220, and it will take you 22 months, which is
>>>> above the usual 18 months that investors look for.
>>>> 
>>>> The focus, thereby, has to be the lever that has the largest effect on the
>>>> unit economics - which is the per-customer cost. I have drawn what my
>>>> ideal
>>>> FiWi network would look like:
>>>> 
>>>> 
>>>> 
>>>> Taking you through this - we start with a 1-port, low-cost EPON OLT (or
>>>> you could go for 2, 4, 8 ports as you add capacity). This OLT has capacity
>>>> for 64 ONUs on its single port. Instead of connecting the typical fiber
>>>> infrastructure with kilometers of cables which break, require maintenance,
>>>> etc. we insert an EPON to Ethernet converter (I added "magic" because
>>>> these
>>>> don't exist AFAIK).
>>>> 
>>>> This converter allows us to connect our $2k sector radio, and serve the
>>>> $200 endpoints (ODUs) over wireless point-to-multipoint up to 10km away.
>>>> Each ODU then has a reverse converter, which gives us EPON again.
>>>> 
>>>> Once we are back on EPON, we can insert splitters, for example,
>>>> pre-connectorized outdoor 1:16 boxes. Every customer install now involves
>>>> a
>>>> 100 meter roll of pre-connectorized 2-core drop cable, and a $20 EPON ONU.
>>>> 
>>>> Using this deployment method, we could connect up to 16 customers to a
>>>> single $200 endpoint, so the enpoint CAPEX per customer is now $12.5. Add
>>>> the ONU, cable, etc. and we have a per-install CAPEX of $82.5 (assuming
>>>> the
>>>> same $50 of extras we had before), and an even shorter break-even. In
>>>> addition, as the endpoints support higher capacity, we can provision at
>>>> least the same, if not more, capacity per customer.
>>>> 
>>>> Other advantages: the $200 ODU is no longer customer equipment and CAPEX,
>>>> but network equipment, and as such, can operate under a longer break-even
>>>> timeline, and be financed by infrastructure PE funds, for example. As a
>>>> result, churn has a much lower financial impact on the operator.
>>>> 
>>>> The main reason why this wouldn't work today is that EPON, as we know, is
>>>> synchronous, and requires the OLT to orchestrate the amount of time each
>>>> ONU can transmit, and when. Having wireless hops and media conversions
>>>> will
>>>> introduce latencies which can break down the communications (e.g. one ONU
>>>> may transmit, get delayed on the radio link, and end up overlapping
>>>> another
>>>> ONU that transmitted on the next slot). Thus, either the "magic" box needs
>>>> to account for this, or an new hybrid EPON-wireless protocol developed.
>>>> 
>>>> My main point here: the industry is moving away from the unconnected. All
>>>> the claims I heard and saw at MWC about "connecting the unconnected" had
>>>> zero resonance with the financial drivers that the unconnected really
>>>> operate under, on top of IT literacy, digital skills, devices, power...
>>>> 
>>>> Best,
>>>> 
>>>> Mike
>>>> On Mar 14, 2023 at 05:27 +0100, rjmcmahon via Starlink <
>>>> starlink at lists.bufferbloat.net>, wrote:
>>>> 
>>>> To change the topic - curious to thoughts on FiWi.
>>>> 
>>>> Imagine a world with no copper cable called FiWi (Fiber,VCSEL/CMOS
>>>> Radios, Antennas) and which is point to point inside a building
>>>> connected to virtualized APs fiber hops away. Each remote radio head
>>>> (RRH) would consume 5W or less and only when active. No need for things
>>>> like zigbee, or meshes, or threads as each radio has a fiber connection
>>>> via Corning's actifi or equivalent. Eliminate the AP/Client power
>>>> imbalance. Plastics also can house smoke or other sensors.
>>>> 
>>>> Some reminders from Paul Baran in 1994 (and from David Reed)
>>>> 
>>>> o) Shorter range rf transceivers connected to fiber could produce a
>>>> significant improvement - - tremendous improvement, really.
>>>> o) a mixture of terrestrial links plus shorter range radio links has the
>>>> effect of increasing by orders and orders of magnitude the amount of
>>>> frequency spectrum that can be made available.
>>>> o) By authorizing high power to support a few users to reach slightly
>>>> longer distances we deprive ourselves of the opportunity to serve the
>>>> many.
>>>> o) Communications systems can be built with 10dB ratio
>>>> o) Digital transmission when properly done allows a small signal to
>>>> noise ratio to be used successfully to retrieve an error free signal.
>>>> o) And, never forget, any transmission capacity not used is wasted
>>>> forever, like water over the dam. Not using such techniques represent
>>>> lost opportunity.
>>>> 
>>>> And on waveguides:
>>>> 
>>>> o) "Fiber transmission loss is ~0.5dB/km for single mode fiber,
>>>> independent of modulation"
>>>> o) “Copper cables and PCB traces are very frequency dependent. At
>>>> 100Gb/s, the loss is in dB/inch."
>>>> o) "Free space: the power density of the radio waves decreases with the
>>>> square of distance from the transmitting antenna due to spreading of the
>>>> electromagnetic energy in space according to the inverse square law"
>>>> 
>>>> The sunk costs & long-lived parts of FiWi are the fiber and the CPE
>>>> plastics & antennas, as CMOS radios+ & fiber/laser, e.g. VCSEL could be
>>>> pluggable, allowing for field upgrades. Just like swapping out SFP in a
>>>> data center.
>>>> 
>>>> This approach basically drives out WiFi latency by eliminating shared
>>>> queues and increases capacity by orders of magnitude by leveraging 10dB
>>>> in the spatial dimension, all of which is achieved by a physical design.
>>>> Just place enough RRHs as needed (similar to a pop up sprinkler in an
>>>> irrigation system.)
>>>> 
>>>> Start and build this for an MDU and the value of the building improves.
>>>> Sadly, there seems no way to capture that value other than over long
>>>> term use. It doesn't matter whether the leader of the HOA tries to
>>>> capture the value or if a last mile provider tries. The value remains
>>>> sunk or hidden with nothing on the asset side of the balance sheet.
>>>> We've got a CAPEX spend that has to be made up via "OPEX returns" over
>>>> years.
>>>> 
>>>> But the asset is there.
>>>> 
>>>> How do we do this?
>>>> 
>>>> Bob
>>>> _______________________________________________
>>>> Starlink mailing list
>>>> Starlink at lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/starlink
>>>> 
>>>> _______________________________________________
>>>> Rpm mailing list
>>>> Rpm at lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/rpm
>>>> 
>>> 
>>> 
>>> --
>>> Come Heckle Mar 6-9 at: https://www.understandinglatency.com
>>> <https://www.understandinglatency.com/Dave>/
>>> Dave Täht CEO, TekLibre, LLC
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>> URL:
>>> <https://lists.bufferbloat.net/pipermail/starlink/attachments/20230317/c9c62be3/attachment.html>
>>> -------------- next part --------------
>>> A non-text attachment was scrubbed...
>>> Name: Hybrid EPON-Wireless network.png
>>> Type: image/png
>>> Size: 149871 bytes
>>> Desc: not available
>>> URL:
>>> <https://lists.bufferbloat.net/pipermail/starlink/attachments/20230317/c9c62be3/attachment.png>
>>> 
>>> ------------------------------
>>> 
>>> Subject: Digest Footer
>>> 
>>> _______________________________________________
>>> Starlink mailing list
>>> Starlink at lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/starlink
>>> 
>>> 
>>> ------------------------------
>>> 
>>> End of Starlink Digest, Vol 24, Issue 39
>>> ****************************************
>>> 
>> _______________________________________________
>> Starlink mailing list
>> Starlink at lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/starlink
> 
> 
> 
> -- 
> Come Heckle Mar 6-9 at: https://www.understandinglatency.com/
> Dave Täht CEO, TekLibre, LLC
> _______________________________________________
> Starlink mailing list
> Starlink at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink