[NNagain] The Whys of the Wichita IXP Project

[Dave Taht]

On Wed, Feb 21, 2024 at 8:02 PM Brent Legg via Nnagain
<nnagain at lists.bufferbloat.net> wrote:
>
> First, let me offer a public THANK YOU to Dave Taht for reaching out to us about the specifics of our Wichita IXP project, and for inviting me to join this group.  It’s been disheartening to see folks talk about us & the project on public forums like LinkedIn without first engaging us in conversation to learn the specifics of what we’re actually doing.  I’d like to think that those who have been disparaging have only done so because they don’t understand what we’re trying to achieve.

I was wildly enthusiastic to see what you were proposing appear in the
press. It was a breath of potentially fresh air in an otherwise
depressing post RDOF, post BEAD environment where it seemed like the
only metrics were speedtests and passings.

I try very hard to get people of wildly disparate backgrounds to
converse, and escape the bubbles they are in. I have tried to gather
together on this old-fashioned email *discussion* list both
technologists and policy-makers to clear the air in ways that cannot
be encapsulated in 240 characters. These two groups (a lot of old
internet experts here) have not been communicating very well of late,
ironically, over the best communication medium ever invented.

It is sad that email lists have so been in decline the past 20+ years,
overwhelmed by marketing and spam, as an email address is the only
universal identifier we have for so many other transactions. The
advantage of a discussion list, over all the faddy technologies, are:
you retain a copy of what you said, everyone else does also, and the
internet at least used to make it searchable into the far future.
Remembering that I had a dispute or discussion with @randomperson and
finding them again via the technology-of-the-day (g+ anyone?, slack?
disquis? hackernews?) is really hard otherwise, and I do hope that
email makes a comeback.

But someones need to start maintaining it better.

>
>
> To begin, I think there is confusion in the terminology being used.  When we say “IXP,” we mean the facility (building, venue) where interconnection & peering occurs.  The “IX” is the ethernet switch in the building.  When someone says an IXP can be built for $8k, that’s apples-to-oranges with what we’re doing.  Yes, a switch can be procured for $8k.  But where does it go?  What if there is no safe, secure, neutral place for it to go?  Then such a place must be built.  That’s what we’re building in Wichita.

To not annoy us old farts, clarifying that you mean a carrier neutral
facility or datacenter with an IXP would go a long way. :)

Too many in the past built gold-plated IXPs, ending up with an
appalling cost model that attracts nobody. This total plan,
at this cost, is a *very good* one, and my hope would be, commoditized
and widely replicated to even more than the 120 locations you project
- but my hope is that the IXP component will mirror the successful IXP
models already existing in the USA.

The costs of interconnecting networks have fallen dramatically, and
can fall further.

>
>
> Saying an IXP can be built for $8k is enormously confusing to many policymakers who do not understand the issue or how interconnection & peering actually work, yet have enormous power to set policy and spend money that will affect the future of the Internet for generations.

Operational expense needs to be discussed. The underlying technologies
used to "make it happen", need to be selected. It is amazing what a
modern cheap 100GB 32 port switch can do. IPv6 is mandatory nowadays
while still finding a way to carry what little remains of IPv4 space
efficiently is needed. It would help if there was a local mirror of
one or more of the root DNS servers. Some really tough design choices
regarding what forms of active ethernet fiber vs a vs gpon need to be
made. And so on. Who makes those decisions?

>
>
>
> We began this whole initiative by asking a series of questions to help us arrive at our model for IXP (building) proliferation.  I’ll use Wichita as the context for these questions, but these could just as easily apply to any other similar city that is home to a large public research university:

Thank you for sharing this last criterion. I had done a similar (much
briefer) study targetting latency and resilience primarily, and what
it would cost to do more "rural IXPs" - call them RXPs - every 50
miles or so - on the cheap as an outgrowth of BEAD. But that would be
a subject for another thread.

But I did not limit it to "research" universities, but to areas that
had universities. Certainly there is high demand for sexy AI-related
things, but the nuts and bolts of how to design and build networks, is
lacking.

I regard network design and operations to be a branch of civil
engineering nowadays, and most operations people are quite leery of
letting grad students loose with operational networks. I would love to
see more universities actually teaching the skills to be a decent
sysadmin (or SRE), because basic knowledge of packets, routing, tcp,
bgp, resiliency, and so on is in the decline. Being a BOFH requires
far more skills than a electrician and is actually comparable in
skills and stress to being a doctor. (SREs get paid pretty well, but
most fall into the profession rather than being directly trained on
it)

Instead, I have been coping (as part of bead), at 6 week educational
programs intended to train people how to splice fiber.

So I would broaden your targets to places that also intend to teach
people how to design and maintain civil infrastructure, and plan ahead
for disaster recovery. This includes connecting up governments and
emergency services. Reusing old postal buildings is an option, as are
other lower grades of schools.

I would love to see curricula for the next generation of BOFHs that
included formerly basic things like how to decode a packet capture and
teachings from TCP/ip volume 3, illustrated, and everything
in-between.

Obligatory xkcd: https://xkcd.com/705/

>
>
>
> Should Wichita, with a regional metro population of 600k+, be literally dependent, from an interconnection standpoint, on Kansas City and Denver forever?  No.
> Okay, then what type of facility does Wichita need?  Ideally, something that can meet current needs and scale to meet future needs.
> What are the attributes of such a facility?
>
> Does it need to be carrier-neutral?  Yes.
> Does it need to be secure?  Yes.
> Does it need to provide a level-playing field for networks of all types?  Yes.
> Does it need to be able to convey rights to, and protect the rights of, its tenants?  Yes.
> Does it need to be a facility that networks can rely on to remain “up” in the wake of adverse events?  Yes.
>
> Resilient from power outages?  Yes.
> Resilient from cooling equipment failures? Yes.
> Resistant to wind damage?  Yes.
> Resistant to vandalism or ballistics damage?  Yes.
>
> Does it need to be financially sustainable?  Yes.

So that is the good question. How do you do opex?

> Is “best effort” good enough?  No.

Redundancy helps.

> Then does it need to be professionally managed?  Yes.

Where will they come from? What software do they have to manage the
facility? Who writes the software?

>
> Is there an existing facility in Wichita that can meet those needs?  No.

In general I use latency as a proxy for where interconnects should go.
Historically this has been about 500 miles. I thought it was
interesting to explore what (as part of Biden´s ev charger program)
what it would take to have an old fashioned IXP ever 50 miles. Turns
out that is pretty close 8k in gear + a lot of fiber.

> So one must be built?  Yes.
> Where should it be built?  Where a concentration of eyeball traffic already exists that can grow a peering ecosystem faster than it might otherwise, and that is also proximate to existing fiber plant, and where diverse manholes can be placed on the edge of public right-of-way.
>
>
>
> In the case of Wichita, that’s at Wichita State University.

Do they teach how to run a network?

>
>
>
> Creating a secure, neutral, resilient interconnection facility with proper cooling, power systems, lockable cabinet space, diverse manholes and POE isn’t cheap.  The whole project is actually more than the $5M grant we received.  We’re putting in over $800k in cash, plus additional in-kind match.
>
>
>
> We’ve done the data analyses necessary to determine which communities need such facilities, and that’s how we came up with our list of 125 target communities.  Most of them are home to public research universities, but have no IXP or IX.  Not all of those communities are equal in terms of priority, but all of them have a need, and we’re actively seeking pathways to scale that preserve our core principles and avoid the need for grants.  But that’s a big challenge.
>
>
>
> I really appreciate the opportunity to provide clarity on the project and I’m happy to answer your questions.  Surely we agree on much more than we disagree.
>
>
>
> --Brent Legg, Connected Nation
>
> _______________________________________________
> Nnagain mailing list
> Nnagain at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/nnagain



-- 
40 years of net history, a couple songs:
https://www.youtube.com/watch?v=D9RGX6QFm5E
Dave Täht CSO, LibreQos