[LibreQoS] transit and peering costs projections

[LibreQoS] transit and peering costs projections

[Dave Taht]

This set of trendlines was very interesting. Unfortunately the data
stops in 2015. Does anyone have more recent data?

https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php

I believe a gbit circuit that an ISP can resell still runs at about
$900 - $1.4k (?) in the usa? How about elsewhere?

...

I am under the impression that many IXPs remain very successful,
states without them suffer, and I also find the concept of doing micro
IXPs at the city level, appealing, and now achievable with cheap gear.
Finer grained cross connects between telco and ISP and IXP would lower
latencies across town quite hugely...

PS I hear ARIN is planning on dropping the price for, and bundling 3
BGP AS numbers at a time, as of the end of this year, also.



-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: [LibreQoS] [NNagain] transit and peering costs projections

[Dave Cohen]

I’m a couple years removed from dealing with this on the provider side but the focus has shifted rapidly to adding core capacity and large capacity ports to the extent that smaller capacity ports like 1 Gbps aren’t going to see much more price compression. Cost per bit will come down at higher tiers but there simply isn’t enough focus at lower levels at the hardware providers to afford carriers more price compression at 1 Gbps, even 10 Gbps. I would expect further price compression in access costs but not really in transit costs below 10 Gbps. 

In general I agree that IXs continue to proliferate relative to quantity, throughput and geographic reach, almost to the degree that mainland Europe has been covered for years. In my home market of Atlanta, I’m aware of at least four IXs that have been established here or entered the market in the last three years - there were only two major ones prior to that. This is a net positive for a wide variety of reasons but I don’t think it’s created much of an impact in terms of pulling down transit prices. There are a few reasons for this, but primarily because that growth hasn’t really displaced transit demand (at least in my view) and has really been more about a relatively stable set of IX participants creating more resiliency and driving other performance improvements in that leg of the peering ecosystem. 

Dave Cohen
craetdave@gmail.com

> On Oct 14, 2023, at 7:02 PM, Dave Taht via Nnagain <nnagain@lists.bufferbloat.net> wrote:
> 
> This set of trendlines was very interesting. Unfortunately the data
> stops in 2015. Does anyone have more recent data?
> 
> https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php
> 
> I believe a gbit circuit that an ISP can resell still runs at about
> $900 - $1.4k (?) in the usa? How about elsewhere?
> 
> ...
> 
> I am under the impression that many IXPs remain very successful,
> states without them suffer, and I also find the concept of doing micro
> IXPs at the city level, appealing, and now achievable with cheap gear.
> Finer grained cross connects between telco and ISP and IXP would lower
> latencies across town quite hugely...
> 
> PS I hear ARIN is planning on dropping the price for, and bundling 3
> BGP AS numbers at a time, as of the end of this year, also.
> 
> 
> 
> -- 
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos
> _______________________________________________
> Nnagain mailing list
> Nnagain@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/nnagain

Re: [LibreQoS] transit and peering costs projections

[Tim Burke]

I would say that a 1Gbit IP transit in a carrier neutral DC can be had for a good bit less than $900 on the wholesale market.

Sadly, IXP’s are seemingly turning into a pay to play game, with rates almost costing as much as transit in many cases after you factor in loop costs. 

For example, in the Houston market (one of the largest and fastest growing regions in the US!), we do not have a major IX, so to get up to Dallas it’s several thousand for a 100g wave, plus several thousand for a 100g port on one of those major IXes. Or, a better option, we can get a 100g flat internet transit for just a little bit more. 

Fortunately, for us as an eyeball network, there are a good number of major content networks that are allowing for private peering in markets like Houston for just the cost of a cross connect and a QSFP if you’re in the right DC, with Google and some others being the outliers.

So for now, we'll keep paying for transit to get to the others (since it’s about as much as transporting IXP from Dallas), and hoping someone at Google finally sees Houston as more than a third rate city hanging off of Dallas. Or… someone finally brings a worthwhile IX to Houston that gets us more than peering to Kansas City. Yeah, I think the former is more likely. 😊

See y’all in San Diego this week,
Tim

On Oct 14, 2023, at 18:04, Dave Taht <dave.taht@gmail.com> wrote:
> 
> This set of trendlines was very interesting. Unfortunately the data
> stops in 2015. Does anyone have more recent data?
> 
> https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php
> 
> I believe a gbit circuit that an ISP can resell still runs at about
> $900 - $1.4k (?) in the usa? How about elsewhere?
> 
> ...
> 
> I am under the impression that many IXPs remain very successful,
> states without them suffer, and I also find the concept of doing micro
> IXPs at the city level, appealing, and now achievable with cheap gear.
> Finer grained cross connects between telco and ISP and IXP would lower
> latencies across town quite hugely...
> 
> PS I hear ARIN is planning on dropping the price for, and bundling 3
> BGP AS numbers at a time, as of the end of this year, also.
> 
> 
> 
> --
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos

Re: [LibreQoS] transit and peering costs projections

[Ryan Hamel]

[-- Attachment #1: Type: text/plain, Size: 3998 bytes --]

Why not place the routers in Dallas, aggregate the transit, IXP, and PNI's there, and backhaul it over redundant dark fiber with DWDM waves or 400G OpenZR?

Ryan

________________________________
From: NANOG <nanog-bounces+ryan=rkhtech.org@nanog.org> on behalf of Tim Burke <tim@mid.net>
Sent: Saturday, October 14, 2023 8:45 PM
To: Dave Taht <dave.taht@gmail.com>
Cc: Network Neutrality is back! Let´s make the technical aspects heard this time! <nnagain@lists.bufferbloat.net>; libreqos <libreqos@lists.bufferbloat.net>; NANOG <nanog@nanog.org>
Subject: Re: transit and peering costs projections

Caution: This is an external email and may be malicious. Please take care when clicking links or opening attachments.


I would say that a 1Gbit IP transit in a carrier neutral DC can be had for a good bit less than $900 on the wholesale market.

Sadly, IXP’s are seemingly turning into a pay to play game, with rates almost costing as much as transit in many cases after you factor in loop costs.

For example, in the Houston market (one of the largest and fastest growing regions in the US!), we do not have a major IX, so to get up to Dallas it’s several thousand for a 100g wave, plus several thousand for a 100g port on one of those major IXes. Or, a better option, we can get a 100g flat internet transit for just a little bit more.

Fortunately, for us as an eyeball network, there are a good number of major content networks that are allowing for private peering in markets like Houston for just the cost of a cross connect and a QSFP if you’re in the right DC, with Google and some others being the outliers.

So for now, we'll keep paying for transit to get to the others (since it’s about as much as transporting IXP from Dallas), and hoping someone at Google finally sees Houston as more than a third rate city hanging off of Dallas. Or… someone finally brings a worthwhile IX to Houston that gets us more than peering to Kansas City. Yeah, I think the former is more likely. 😊

See y’all in San Diego this week,
Tim

On Oct 14, 2023, at 18:04, Dave Taht <dave.taht@gmail.com> wrote:
>
> This set of trendlines was very interesting. Unfortunately the data
> stops in 2015. Does anyone have more recent data?
>
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdrpeering.net%2Fwhite-papers%2FInternet-Transit-Pricing-Historical-And-Projected.php&data=05%7C01%7Cryan%40rkhtech.org%7Cc8ebae9f0ecd4b368dcb08dbcd319880%7C81c24bb4f9ec4739ba4d25c42594d996%7C0%7C0%7C638329385118876648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nQeWrGi%2BblMmtiG9u7SdF3JOi1h9Fni7xXo%2FusZRopA%3D&reserved=0<https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php>
>
> I believe a gbit circuit that an ISP can resell still runs at about
> $900 - $1.4k (?) in the usa? How about elsewhere?
>
> ...
>
> I am under the impression that many IXPs remain very successful,
> states without them suffer, and I also find the concept of doing micro
> IXPs at the city level, appealing, and now achievable with cheap gear.
> Finer grained cross connects between telco and ISP and IXP would lower
> latencies across town quite hugely...
>
> PS I hear ARIN is planning on dropping the price for, and bundling 3
> BGP AS numbers at a time, as of the end of this year, also.
>
>
>
> --
> Oct 30: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnetdevconf.info%2F0x17%2Fnews%2Fthe-maestro-and-the-music-bof.html&data=05%7C01%7Cryan%40rkhtech.org%7Cc8ebae9f0ecd4b368dcb08dbcd319880%7C81c24bb4f9ec4739ba4d25c42594d996%7C0%7C0%7C638329385118876648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ROLgtoeiBgfAG40UZqS8Zd8vMK%2B0HQB7RV%2FhQRvIcFM%3D&reserved=0<https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html>
> Dave Täht CSO, LibreQos

[-- Attachment #2: Type: text/html, Size: 5578 bytes --]

Re: [LibreQoS] transit and peering costs projections

[Tim Burke]

[-- Attachment #1: Type: text/plain, Size: 4522 bytes --]

It’s better for customer experience to keep it local instead of adding 200 miles to the route. All of the competition hauls all of their traffic up to Dallas, so we easily have a nice 8-10ms latency advantage by keeping transit and peering as close to the customer as possible.

Plus, you can’t forget to mention another ~$10k MRC per pair in DF costs to get up to Dallas, not including colo, that we can spend on more transit or better gear!

On Oct 14, 2023, at 23:03, Ryan Hamel <ryan@rkhtech.org> wrote:


Why not place the routers in Dallas, aggregate the transit, IXP, and PNI's there, and backhaul it over redundant dark fiber with DWDM waves or 400G OpenZR?

Ryan

________________________________
From: NANOG <nanog-bounces+ryan=rkhtech.org@nanog.org> on behalf of Tim Burke <tim@mid.net>
Sent: Saturday, October 14, 2023 8:45 PM
To: Dave Taht <dave.taht@gmail.com>
Cc: Network Neutrality is back! Let´s make the technical aspects heard this time! <nnagain@lists.bufferbloat.net>; libreqos <libreqos@lists.bufferbloat.net>; NANOG <nanog@nanog.org>
Subject: Re: transit and peering costs projections

Caution: This is an external email and may be malicious. Please take care when clicking links or opening attachments.


I would say that a 1Gbit IP transit in a carrier neutral DC can be had for a good bit less than $900 on the wholesale market.

Sadly, IXP’s are seemingly turning into a pay to play game, with rates almost costing as much as transit in many cases after you factor in loop costs.

For example, in the Houston market (one of the largest and fastest growing regions in the US!), we do not have a major IX, so to get up to Dallas it’s several thousand for a 100g wave, plus several thousand for a 100g port on one of those major IXes. Or, a better option, we can get a 100g flat internet transit for just a little bit more.

Fortunately, for us as an eyeball network, there are a good number of major content networks that are allowing for private peering in markets like Houston for just the cost of a cross connect and a QSFP if you’re in the right DC, with Google and some others being the outliers.

So for now, we'll keep paying for transit to get to the others (since it’s about as much as transporting IXP from Dallas), and hoping someone at Google finally sees Houston as more than a third rate city hanging off of Dallas. Or… someone finally brings a worthwhile IX to Houston that gets us more than peering to Kansas City. Yeah, I think the former is more likely. 😊

See y’all in San Diego this week,
Tim

On Oct 14, 2023, at 18:04, Dave Taht <dave.taht@gmail.com> wrote:
>
> This set of trendlines was very interesting. Unfortunately the data
> stops in 2015. Does anyone have more recent data?
>
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdrpeering.net%2Fwhite-papers%2FInternet-Transit-Pricing-Historical-And-Projected.php&data=05%7C01%7Cryan%40rkhtech.org%7Cc8ebae9f0ecd4b368dcb08dbcd319880%7C81c24bb4f9ec4739ba4d25c42594d996%7C0%7C0%7C638329385118876648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nQeWrGi%2BblMmtiG9u7SdF3JOi1h9Fni7xXo%2FusZRopA%3D&reserved=0<https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php>
>
> I believe a gbit circuit that an ISP can resell still runs at about
> $900 - $1.4k (?) in the usa? How about elsewhere?
>
> ...
>
> I am under the impression that many IXPs remain very successful,
> states without them suffer, and I also find the concept of doing micro
> IXPs at the city level, appealing, and now achievable with cheap gear.
> Finer grained cross connects between telco and ISP and IXP would lower
> latencies across town quite hugely...
>
> PS I hear ARIN is planning on dropping the price for, and bundling 3
> BGP AS numbers at a time, as of the end of this year, also.
>
>
>
> --
> Oct 30: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnetdevconf.info%2F0x17%2Fnews%2Fthe-maestro-and-the-music-bof.html&data=05%7C01%7Cryan%40rkhtech.org%7Cc8ebae9f0ecd4b368dcb08dbcd319880%7C81c24bb4f9ec4739ba4d25c42594d996%7C0%7C0%7C638329385118876648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ROLgtoeiBgfAG40UZqS8Zd8vMK%2B0HQB7RV%2FhQRvIcFM%3D&reserved=0<https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html>
> Dave Täht CSO, LibreQos

[-- Attachment #2: Type: text/html, Size: 6292 bytes --]

Re: [LibreQoS] transit and peering costs projections

[Dave Taht]

On Sat, Oct 14, 2023 at 9:12 PM Tim Burke <tim@mid.net> wrote:
>
> It’s better for customer experience to keep it local instead of adding 200 miles to the route. All of the competition hauls all of their traffic up to Dallas, so we easily have a nice 8-10ms latency advantage by keeping transit and peering as close to the customer as possible.
>
> Plus, you can’t forget to mention another ~$10k MRC per pair in DF costs to get up to Dallas, not including colo, that we can spend on more transit or better gear!

Texas's BEAD funding and broadband offices are looking for proposals
and seem to have dollars to spend. I have spent much of the past few
years attempting to convince these entities that what was often more
needed was better, more local IXPs. Have you reached out to them?


> On Oct 14, 2023, at 23:03, Ryan Hamel <ryan@rkhtech.org> wrote:
>
> 
> Why not place the routers in Dallas, aggregate the transit, IXP, and PNI's there, and backhaul it over redundant dark fiber with DWDM waves or 400G OpenZR?
>
> Ryan
>
> ________________________________
> From: NANOG <nanog-bounces+ryan=rkhtech.org@nanog.org> on behalf of Tim Burke <tim@mid.net>
> Sent: Saturday, October 14, 2023 8:45 PM
> To: Dave Taht <dave.taht@gmail.com>
> Cc: Network Neutrality is back! Let´s make the technical aspects heard this time! <nnagain@lists.bufferbloat.net>; libreqos <libreqos@lists.bufferbloat.net>; NANOG <nanog@nanog.org>
> Subject: Re: transit and peering costs projections
>
> Caution: This is an external email and may be malicious. Please take care when clicking links or opening attachments.
>
>
> I would say that a 1Gbit IP transit in a carrier neutral DC can be had for a good bit less than $900 on the wholesale market.
>
> Sadly, IXP’s are seemingly turning into a pay to play game, with rates almost costing as much as transit in many cases after you factor in loop costs.
>
> For example, in the Houston market (one of the largest and fastest growing regions in the US!), we do not have a major IX, so to get up to Dallas it’s several thousand for a 100g wave, plus several thousand for a 100g port on one of those major IXes. Or, a better option, we can get a 100g flat internet transit for just a little bit more.
>
> Fortunately, for us as an eyeball network, there are a good number of major content networks that are allowing for private peering in markets like Houston for just the cost of a cross connect and a QSFP if you’re in the right DC, with Google and some others being the outliers.
>
> So for now, we'll keep paying for transit to get to the others (since it’s about as much as transporting IXP from Dallas), and hoping someone at Google finally sees Houston as more than a third rate city hanging off of Dallas. Or… someone finally brings a worthwhile IX to Houston that gets us more than peering to Kansas City. Yeah, I think the former is more likely. 😊
>
> See y’all in San Diego this week,
> Tim
>
> On Oct 14, 2023, at 18:04, Dave Taht <dave.taht@gmail.com> wrote:
> >
> > This set of trendlines was very interesting. Unfortunately the data
> > stops in 2015. Does anyone have more recent data?
> >
> > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdrpeering.net%2Fwhite-papers%2FInternet-Transit-Pricing-Historical-And-Projected.php&data=05%7C01%7Cryan%40rkhtech.org%7Cc8ebae9f0ecd4b368dcb08dbcd319880%7C81c24bb4f9ec4739ba4d25c42594d996%7C0%7C0%7C638329385118876648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nQeWrGi%2BblMmtiG9u7SdF3JOi1h9Fni7xXo%2FusZRopA%3D&reserved=0
> >
> > I believe a gbit circuit that an ISP can resell still runs at about
> > $900 - $1.4k (?) in the usa? How about elsewhere?
> >
> > ...
> >
> > I am under the impression that many IXPs remain very successful,
> > states without them suffer, and I also find the concept of doing micro
> > IXPs at the city level, appealing, and now achievable with cheap gear.
> > Finer grained cross connects between telco and ISP and IXP would lower
> > latencies across town quite hugely...
> >
> > PS I hear ARIN is planning on dropping the price for, and bundling 3
> > BGP AS numbers at a time, as of the end of this year, also.
> >
> >
> >
> > --
> > Oct 30: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnetdevconf.info%2F0x17%2Fnews%2Fthe-maestro-and-the-music-bof.html&data=05%7C01%7Cryan%40rkhtech.org%7Cc8ebae9f0ecd4b368dcb08dbcd319880%7C81c24bb4f9ec4739ba4d25c42594d996%7C0%7C0%7C638329385118876648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ROLgtoeiBgfAG40UZqS8Zd8vMK%2B0HQB7RV%2FhQRvIcFM%3D&reserved=0
> > Dave Täht CSO, LibreQos



-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: [LibreQoS] transit and peering costs projections

[dan]

[-- Attachment #1: Type: text/plain, Size: 5900 bytes --]

the 900-1400 for 1G is right at what I'm seeing in the rockies region.
price scales decently at 10G.   transit costs just as much as DIA or more
because of port costs on each side.

Also, with zayo and lumen, traversing their MPLS networks a few hundred
miles probably costs you 10-15ms latency.  That's what I'm seeing.  So
unless you're doing a wave or something that isn't getting battered in
their neglected networks centralizing could be worse overall.



On Sat, Oct 14, 2023 at 10:19 PM Dave Taht via LibreQoS <
libreqos@lists.bufferbloat.net> wrote:

> On Sat, Oct 14, 2023 at 9:12 PM Tim Burke <tim@mid.net> wrote:
> >
> > It’s better for customer experience to keep it local instead of adding
> 200 miles to the route. All of the competition hauls all of their traffic
> up to Dallas, so we easily have a nice 8-10ms latency advantage by keeping
> transit and peering as close to the customer as possible.
> >
> > Plus, you can’t forget to mention another ~$10k MRC per pair in DF costs
> to get up to Dallas, not including colo, that we can spend on more transit
> or better gear!
>
> Texas's BEAD funding and broadband offices are looking for proposals
> and seem to have dollars to spend. I have spent much of the past few
> years attempting to convince these entities that what was often more
> needed was better, more local IXPs. Have you reached out to them?
>
>
> > On Oct 14, 2023, at 23:03, Ryan Hamel <ryan@rkhtech.org> wrote:
> >
> > 
> > Why not place the routers in Dallas, aggregate the transit, IXP, and
> PNI's there, and backhaul it over redundant dark fiber with DWDM waves or
> 400G OpenZR?
> >
> > Ryan
> >
> > ________________________________
> > From: NANOG <nanog-bounces+ryan=rkhtech.org@nanog.org> on behalf of Tim
> Burke <tim@mid.net>
> > Sent: Saturday, October 14, 2023 8:45 PM
> > To: Dave Taht <dave.taht@gmail.com>
> > Cc: Network Neutrality is back! Let´s make the technical aspects heard
> this time! <nnagain@lists.bufferbloat.net>; libreqos <
> libreqos@lists.bufferbloat.net>; NANOG <nanog@nanog.org>
> > Subject: Re: transit and peering costs projections
> >
> > Caution: This is an external email and may be malicious. Please take
> care when clicking links or opening attachments.
> >
> >
> > I would say that a 1Gbit IP transit in a carrier neutral DC can be had
> for a good bit less than $900 on the wholesale market.
> >
> > Sadly, IXP’s are seemingly turning into a pay to play game, with rates
> almost costing as much as transit in many cases after you factor in loop
> costs.
> >
> > For example, in the Houston market (one of the largest and fastest
> growing regions in the US!), we do not have a major IX, so to get up to
> Dallas it’s several thousand for a 100g wave, plus several thousand for a
> 100g port on one of those major IXes. Or, a better option, we can get a
> 100g flat internet transit for just a little bit more.
> >
> > Fortunately, for us as an eyeball network, there are a good number of
> major content networks that are allowing for private peering in markets
> like Houston for just the cost of a cross connect and a QSFP if you’re in
> the right DC, with Google and some others being the outliers.
> >
> > So for now, we'll keep paying for transit to get to the others (since
> it’s about as much as transporting IXP from Dallas), and hoping someone at
> Google finally sees Houston as more than a third rate city hanging off of
> Dallas. Or… someone finally brings a worthwhile IX to Houston that gets us
> more than peering to Kansas City. Yeah, I think the former is more likely.
> 😊
> >
> > See y’all in San Diego this week,
> > Tim
> >
> > On Oct 14, 2023, at 18:04, Dave Taht <dave.taht@gmail.com> wrote:
> > >
> > > This set of trendlines was very interesting. Unfortunately the data
> > > stops in 2015. Does anyone have more recent data?
> > >
> > >
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdrpeering.net%2Fwhite-papers%2FInternet-Transit-Pricing-Historical-And-Projected.php&data=05%7C01%7Cryan%40rkhtech.org%7Cc8ebae9f0ecd4b368dcb08dbcd319880%7C81c24bb4f9ec4739ba4d25c42594d996%7C0%7C0%7C638329385118876648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nQeWrGi%2BblMmtiG9u7SdF3JOi1h9Fni7xXo%2FusZRopA%3D&reserved=0
> > >
> > > I believe a gbit circuit that an ISP can resell still runs at about
> > > $900 - $1.4k (?) in the usa? How about elsewhere?
> > >
> > > ...
> > >
> > > I am under the impression that many IXPs remain very successful,
> > > states without them suffer, and I also find the concept of doing micro
> > > IXPs at the city level, appealing, and now achievable with cheap gear.
> > > Finer grained cross connects between telco and ISP and IXP would lower
> > > latencies across town quite hugely...
> > >
> > > PS I hear ARIN is planning on dropping the price for, and bundling 3
> > > BGP AS numbers at a time, as of the end of this year, also.
> > >
> > >
> > >
> > > --
> > > Oct 30:
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnetdevconf.info%2F0x17%2Fnews%2Fthe-maestro-and-the-music-bof.html&data=05%7C01%7Cryan%40rkhtech.org%7Cc8ebae9f0ecd4b368dcb08dbcd319880%7C81c24bb4f9ec4739ba4d25c42594d996%7C0%7C0%7C638329385118876648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ROLgtoeiBgfAG40UZqS8Zd8vMK%2B0HQB7RV%2FhQRvIcFM%3D&reserved=0
> > > Dave Täht CSO, LibreQos
>
>
>
> --
> Oct 30:
> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos
> _______________________________________________
> LibreQoS mailing list
> LibreQoS@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/libreqos
>

[-- Attachment #2: Type: text/html, Size: 8444 bytes --]

Re: [LibreQoS] transit and peering costs projections

[Bill Woodcock]

[-- Attachment #1: Type: text/plain, Size: 4839 bytes --]

Exactly.  Speed x distance = cost.  This is _exactly_ why IXPs get set up.  To avoid backhauling bandwidth from Dallas, or wherever.  Loss, latency, out-of-order delivery, and jitter.  All lower when you source your bandwidth closer.

                                -Bill



> On Oct 15, 2023, at 06:12, Tim Burke <tim@mid.net> wrote:
> 
> It’s better for customer experience to keep it local instead of adding 200 miles to the route. All of the competition hauls all of their traffic up to Dallas, so we easily have a nice 8-10ms latency advantage by keeping transit and peering as close to the customer as possible.
> 
> Plus, you can’t forget to mention another ~$10k MRC per pair in DF costs to get up to Dallas, not including colo, that we can spend on more transit or better gear!
> 
>> On Oct 14, 2023, at 23:03, Ryan Hamel <ryan@rkhtech.org> wrote:
>> 
>>  Why not place the routers in Dallas, aggregate the transit, IXP, and PNI's there, and backhaul it over redundant dark fiber with DWDM waves or 400G OpenZR?
>> 
>> Ryan
>> 
>> From: NANOG <nanog-bounces+ryan=rkhtech.org@nanog.org> on behalf of Tim Burke <tim@mid.net>
>> Sent: Saturday, October 14, 2023 8:45 PM
>> To: Dave Taht <dave.taht@gmail.com>
>> Cc: Network Neutrality is back! Let´s make the technical aspects heard this time! <nnagain@lists.bufferbloat.net>; libreqos <libreqos@lists.bufferbloat.net>; NANOG <nanog@nanog.org>
>> Subject: Re: transit and peering costs projections   Caution: This is an external email and may be malicious. Please take care when clicking links or opening attachments.
>> 
>> 
>> I would say that a 1Gbit IP transit in a carrier neutral DC can be had for a good bit less than $900 on the wholesale market.
>> 
>> Sadly, IXP’s are seemingly turning into a pay to play game, with rates almost costing as much as transit in many cases after you factor in loop costs.
>> 
>> For example, in the Houston market (one of the largest and fastest growing regions in the US!), we do not have a major IX, so to get up to Dallas it’s several thousand for a 100g wave, plus several thousand for a 100g port on one of those major IXes. Or, a better option, we can get a 100g flat internet transit for just a little bit more.
>> 
>> Fortunately, for us as an eyeball network, there are a good number of major content networks that are allowing for private peering in markets like Houston for just the cost of a cross connect and a QSFP if you’re in the right DC, with Google and some others being the outliers.
>> 
>> So for now, we'll keep paying for transit to get to the others (since it’s about as much as transporting IXP from Dallas), and hoping someone at Google finally sees Houston as more than a third rate city hanging off of Dallas. Or… someone finally brings a worthwhile IX to Houston that gets us more than peering to Kansas City. Yeah, I think the former is more likely. 😊
>> 
>> See y’all in San Diego this week,
>> Tim
>> 
>> On Oct 14, 2023, at 18:04, Dave Taht <dave.taht@gmail.com> wrote:
>> >
>> > This set of trendlines was very interesting. Unfortunately the data
>> > stops in 2015. Does anyone have more recent data?
>> >
>> > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdrpeering.net%2Fwhite-papers%2FInternet-Transit-Pricing-Historical-And-Projected.php&data=05%7C01%7Cryan%40rkhtech.org%7Cc8ebae9f0ecd4b368dcb08dbcd319880%7C81c24bb4f9ec4739ba4d25c42594d996%7C0%7C0%7C638329385118876648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nQeWrGi%2BblMmtiG9u7SdF3JOi1h9Fni7xXo%2FusZRopA%3D&reserved=0
>> >
>> > I believe a gbit circuit that an ISP can resell still runs at about
>> > $900 - $1.4k (?) in the usa? How about elsewhere?
>> >
>> > ...
>> >
>> > I am under the impression that many IXPs remain very successful,
>> > states without them suffer, and I also find the concept of doing micro
>> > IXPs at the city level, appealing, and now achievable with cheap gear.
>> > Finer grained cross connects between telco and ISP and IXP would lower
>> > latencies across town quite hugely...
>> >
>> > PS I hear ARIN is planning on dropping the price for, and bundling 3
>> > BGP AS numbers at a time, as of the end of this year, also.
>> >
>> >
>> >
>> > --
>> > Oct 30: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnetdevconf.info%2F0x17%2Fnews%2Fthe-maestro-and-the-music-bof.html&data=05%7C01%7Cryan%40rkhtech.org%7Cc8ebae9f0ecd4b368dcb08dbcd319880%7C81c24bb4f9ec4739ba4d25c42594d996%7C0%7C0%7C638329385118876648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ROLgtoeiBgfAG40UZqS8Zd8vMK%2B0HQB7RV%2FhQRvIcFM%3D&reserved=0
>> > Dave Täht CSO, LibreQos


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [LibreQoS] transit and peering costs projections

[Mike Hammett]

[-- Attachment #1: Type: text/plain, Size: 3287 bytes --]

Houston is tricky as due to it's geographic scope, it's quite expensive to build an IX that goes into enough facilities to achieve meaningful scale. CDN 1 is in facility A. CDN 2 in facility B. CDN 3 is in facility C. When I last looked, it was about 80 driving miles to have a dark fiber ring that encompassed all of the facilities one would need to be in. 




----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

----- Original Message -----

From: "Tim Burke" <tim@mid.net> 
To: "Dave Taht" <dave.taht@gmail.com> 
Cc: "Network Neutrality is back! Let´s make the technical aspects heard this time!" <nnagain@lists.bufferbloat.net>, "libreqos" <libreqos@lists.bufferbloat.net>, "NANOG" <nanog@nanog.org> 
Sent: Saturday, October 14, 2023 10:45:47 PM 
Subject: Re: transit and peering costs projections 

I would say that a 1Gbit IP transit in a carrier neutral DC can be had for a good bit less than $900 on the wholesale market. 

Sadly, IXP’s are seemingly turning into a pay to play game, with rates almost costing as much as transit in many cases after you factor in loop costs. 

For example, in the Houston market (one of the largest and fastest growing regions in the US!), we do not have a major IX, so to get up to Dallas it’s several thousand for a 100g wave, plus several thousand for a 100g port on one of those major IXes. Or, a better option, we can get a 100g flat internet transit for just a little bit more. 

Fortunately, for us as an eyeball network, there are a good number of major content networks that are allowing for private peering in markets like Houston for just the cost of a cross connect and a QSFP if you’re in the right DC, with Google and some others being the outliers. 

So for now, we'll keep paying for transit to get to the others (since it’s about as much as transporting IXP from Dallas), and hoping someone at Google finally sees Houston as more than a third rate city hanging off of Dallas. Or… someone finally brings a worthwhile IX to Houston that gets us more than peering to Kansas City. Yeah, I think the former is more likely. 😊 

See y’all in San Diego this week, 
Tim 

On Oct 14, 2023, at 18:04, Dave Taht <dave.taht@gmail.com> wrote: 
> 
> This set of trendlines was very interesting. Unfortunately the data 
> stops in 2015. Does anyone have more recent data? 
> 
> https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php 
> 
> I believe a gbit circuit that an ISP can resell still runs at about 
> $900 - $1.4k (?) in the usa? How about elsewhere? 
> 
> ... 
> 
> I am under the impression that many IXPs remain very successful, 
> states without them suffer, and I also find the concept of doing micro 
> IXPs at the city level, appealing, and now achievable with cheap gear. 
> Finer grained cross connects between telco and ISP and IXP would lower 
> latencies across town quite hugely... 
> 
> PS I hear ARIN is planning on dropping the price for, and bundling 3 
> BGP AS numbers at a time, as of the end of this year, also. 
> 
> 
> 
> -- 
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html 
> Dave Täht CSO, LibreQos 


[-- Attachment #2: Type: text/html, Size: 3875 bytes --]

Re: [LibreQoS] transit and peering costs projections

[Tim Burke]

[-- Attachment #1: Type: text/plain, Size: 3710 bytes --]

I’ve found that most of the CDNs that matter are in one facility in Houston, the Databank West (formerly Cyrus One) campus. We are about to light up a POP there so we’ll at least be able to get PNIs to them. There is even an IX in the facility, but it’s relatively small (likely because the operator wants near-transit pricing to get on it) so we’ll just PNI what we can for now.

On Oct 15, 2023, at 08:50, Mike Hammett <nanog@ics-il.net> wrote:


Houston is tricky as due to it's geographic scope, it's quite expensive to build an IX that goes into enough facilities to achieve meaningful scale. CDN 1 is in facility A. CDN 2 in facility B. CDN 3 is in facility C. When I last looked, it was about 80 driving miles to have a dark fiber ring that encompassed all of the facilities one would need to be in.



-----
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

________________________________
From: "Tim Burke" <tim@mid.net>
To: "Dave Taht" <dave.taht@gmail.com>
Cc: "Network Neutrality is back! Let´s make the technical aspects heard this time!" <nnagain@lists.bufferbloat.net>, "libreqos" <libreqos@lists.bufferbloat.net>, "NANOG" <nanog@nanog.org>
Sent: Saturday, October 14, 2023 10:45:47 PM
Subject: Re: transit and peering costs projections

I would say that a 1Gbit IP transit in a carrier neutral DC can be had for a good bit less than $900 on the wholesale market.

Sadly, IXP’s are seemingly turning into a pay to play game, with rates almost costing as much as transit in many cases after you factor in loop costs.

For example, in the Houston market (one of the largest and fastest growing regions in the US!), we do not have a major IX, so to get up to Dallas it’s several thousand for a 100g wave, plus several thousand for a 100g port on one of those major IXes. Or, a better option, we can get a 100g flat internet transit for just a little bit more.

Fortunately, for us as an eyeball network, there are a good number of major content networks that are allowing for private peering in markets like Houston for just the cost of a cross connect and a QSFP if you’re in the right DC, with Google and some others being the outliers.

So for now, we'll keep paying for transit to get to the others (since it’s about as much as transporting IXP from Dallas), and hoping someone at Google finally sees Houston as more than a third rate city hanging off of Dallas. Or… someone finally brings a worthwhile IX to Houston that gets us more than peering to Kansas City. Yeah, I think the former is more likely. 😊

See y’all in San Diego this week,
Tim

On Oct 14, 2023, at 18:04, Dave Taht <dave.taht@gmail.com> wrote:
>
> This set of trendlines was very interesting. Unfortunately the data
> stops in 2015. Does anyone have more recent data?
>
> https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php
>
> I believe a gbit circuit that an ISP can resell still runs at about
> $900 - $1.4k (?) in the usa? How about elsewhere?
>
> ...
>
> I am under the impression that many IXPs remain very successful,
> states without them suffer, and I also find the concept of doing micro
> IXPs at the city level, appealing, and now achievable with cheap gear.
> Finer grained cross connects between telco and ISP and IXP would lower
> latencies across town quite hugely...
>
> PS I hear ARIN is planning on dropping the price for, and bundling 3
> BGP AS numbers at a time, as of the end of this year, also.
>
>
>
> --
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos


[-- Attachment #2: Type: text/html, Size: 4725 bytes --]

Re: [LibreQoS] transit and peering costs projections

[dan]

[-- Attachment #1: Type: text/plain, Size: 6311 bytes --]

Just want to rewind back to the IX map above.  The problem is that it's
really misleading.  Dive down on a number of those (a big number) and they
are registered as an IX but they have few tier1 providers in them.  The
closest one to me is essentially just fed by Zayo.   Not much of an IX when
there's only one path out on one provider...

If big providers, or at least multiple providers linking to other IX'
aren't participating, then the purpose isn't met.  Zayo isn't offering IX
rates in these for lack of competition so the incentive to build out from
there is very low, ie I can get Zayo in a road-side hut for basically the
same price and not have to share access.  I also realize that getting 2-4
providers into a shack in the middle of no-where doesn't make sense either,
but population dictates a lot of this.

I know it's a big ask, getting full size IX access in a microIX, but that's
what big government projects are for.  Get these carriers that are crossing
various jurisdictions to drop transport services, waves or dark viber etc,
into something useful like a school, courthouse, town hall, whatever, and
in that build out link to the two IX's that fiber crossing was going
between.  Just put in the deal that they put in optics aligned with the
population.  Frankly, 40G to most of these areas would be plenty for a
decade or more and 40G optics long distance modules are only a few grand
each.  Maybe $10-15,000 for redundant 40G and they've already run the fiber
as part of that delivery to the facility (double that for really long
runs...).  Schools would be my #1 pick here because it solves a lot of
issues.  Gov pulls in at least 1x 40G to every single incorporated school
and builds access facilities for that (conduits to edges of property etc)
and at some threshold that's 1x40G with 2 providers then 3 providers for
bigger populations and as populations grow.  Standard prices on port and
they are all just a vlan or equiv on the pipe back to the IX.  Basically
these would be like IX extension sites with layer2 ports between provided
by long-haul providers.



On Sun, Oct 15, 2023 at 8:19 AM Tim Burke via LibreQoS <
libreqos@lists.bufferbloat.net> wrote:

> I’ve found that most of the CDNs that matter are in one facility in
> Houston, the Databank West (formerly Cyrus One) campus. We are about to
> light up a POP there so we’ll at least be able to get PNIs to them. There
> is even an IX in the facility, but it’s relatively small (likely because
> the operator wants near-transit pricing to get on it) so we’ll just PNI
> what we can for now.
>
> On Oct 15, 2023, at 08:50, Mike Hammett <nanog@ics-il.net> wrote:
>
> 
> Houston is tricky as due to it's geographic scope, it's quite expensive to
> build an IX that goes into enough facilities to achieve meaningful scale.
> CDN 1 is in facility A. CDN 2 in facility B. CDN 3 is in facility C. When I
> last looked, it was about 80 driving miles to have a dark fiber ring that
> encompassed all of the facilities one would need to be in.
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
> ------------------------------
> *From: *"Tim Burke" <tim@mid.net>
> *To: *"Dave Taht" <dave.taht@gmail.com>
> *Cc: *"Network Neutrality is back! Let´s make the technical aspects heard
> this time!" <nnagain@lists.bufferbloat.net>, "libreqos" <
> libreqos@lists.bufferbloat.net>, "NANOG" <nanog@nanog.org>
> *Sent: *Saturday, October 14, 2023 10:45:47 PM
> *Subject: *Re: transit and peering costs projections
>
> I would say that a 1Gbit IP transit in a carrier neutral DC can be had for
> a good bit less than $900 on the wholesale market.
>
> Sadly, IXP’s are seemingly turning into a pay to play game, with rates
> almost costing as much as transit in many cases after you factor in loop
> costs.
>
> For example, in the Houston market (one of the largest and fastest growing
> regions in the US!), we do not have a major IX, so to get up to Dallas it’s
> several thousand for a 100g wave, plus several thousand for a 100g port on
> one of those major IXes. Or, a better option, we can get a 100g flat
> internet transit for just a little bit more.
>
> Fortunately, for us as an eyeball network, there are a good number of
> major content networks that are allowing for private peering in markets
> like Houston for just the cost of a cross connect and a QSFP if you’re in
> the right DC, with Google and some others being the outliers.
>
> So for now, we'll keep paying for transit to get to the others (since it’s
> about as much as transporting IXP from Dallas), and hoping someone at
> Google finally sees Houston as more than a third rate city hanging off of
> Dallas. Or… someone finally brings a worthwhile IX to Houston that gets us
> more than peering to Kansas City. Yeah, I think the former is more likely.
> 😊
>
> See y’all in San Diego this week,
> Tim
>
> On Oct 14, 2023, at 18:04, Dave Taht <dave.taht@gmail.com> wrote:
> >
> > This set of trendlines was very interesting. Unfortunately the data
> > stops in 2015. Does anyone have more recent data?
> >
> >
> https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php
> >
> > I believe a gbit circuit that an ISP can resell still runs at about
> > $900 - $1.4k (?) in the usa? How about elsewhere?
> >
> > ...
> >
> > I am under the impression that many IXPs remain very successful,
> > states without them suffer, and I also find the concept of doing micro
> > IXPs at the city level, appealing, and now achievable with cheap gear.
> > Finer grained cross connects between telco and ISP and IXP would lower
> > latencies across town quite hugely...
> >
> > PS I hear ARIN is planning on dropping the price for, and bundling 3
> > BGP AS numbers at a time, as of the end of this year, also.
> >
> >
> >
> > --
> > Oct 30:
> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> > Dave Täht CSO, LibreQos
>
> _______________________________________________
> LibreQoS mailing list
> LibreQoS@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/libreqos
>

[-- Attachment #2: Type: text/html, Size: 8304 bytes --]

Re: [LibreQoS] transit and peering costs projections

[Tom Beecher]

[-- Attachment #1: Type: text/plain, Size: 3966 bytes --]

>
> So for now, we'll keep paying for transit to get to the others (since it’s
> about as much as transporting IXP from Dallas), and hoping someone at
> Google finally sees Houston as more than a third rate city hanging off of
> Dallas. Or… someone finally brings a worthwhile IX to Houston that gets us
> more than peering to Kansas City. Yeah, I think the former is more
> likely. 😊
>

There is often a chicken/egg scenario here with the economics. As an
eyeball network, your costs to build out and connect to Dallas are greater
than your transit cost, so you do that. Totally fair.

However think about it from the content side. Say I want to build into to
Houston. I have to put routers in, and a bunch of cache servers, so I have
capital outlay , plus opex for space, power, IX/backhaul/transit costs.
That's not cheap, so there's a lot of calculations that go into it. Is
there enough total eyeball traffic there to make it worth it? Is saving
8-10ms enough of a performance boost to justify the spend? What are the
long term trends in that market? These answers are of course different for
a company running their own CDN vs the commercial CDNs.

I don't work for Google and obviously don't speak for them, but I would
suspect that they're happy to eat a 8-10ms performance hit to serve from
Dallas , versus the amount of capital outlay to build out there right now.

On Sat, Oct 14, 2023 at 11:47 PM Tim Burke <tim@mid.net> wrote:

> I would say that a 1Gbit IP transit in a carrier neutral DC can be had for
> a good bit less than $900 on the wholesale market.
>
> Sadly, IXP’s are seemingly turning into a pay to play game, with rates
> almost costing as much as transit in many cases after you factor in loop
> costs.
>
> For example, in the Houston market (one of the largest and fastest growing
> regions in the US!), we do not have a major IX, so to get up to Dallas it’s
> several thousand for a 100g wave, plus several thousand for a 100g port on
> one of those major IXes. Or, a better option, we can get a 100g flat
> internet transit for just a little bit more.
>
> Fortunately, for us as an eyeball network, there are a good number of
> major content networks that are allowing for private peering in markets
> like Houston for just the cost of a cross connect and a QSFP if you’re in
> the right DC, with Google and some others being the outliers.
>
> So for now, we'll keep paying for transit to get to the others (since it’s
> about as much as transporting IXP from Dallas), and hoping someone at
> Google finally sees Houston as more than a third rate city hanging off of
> Dallas. Or… someone finally brings a worthwhile IX to Houston that gets us
> more than peering to Kansas City. Yeah, I think the former is more likely.
> 😊
>
> See y’all in San Diego this week,
> Tim
>
> On Oct 14, 2023, at 18:04, Dave Taht <dave.taht@gmail.com> wrote:
> >
> > This set of trendlines was very interesting. Unfortunately the data
> > stops in 2015. Does anyone have more recent data?
> >
> >
> https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php
> >
> > I believe a gbit circuit that an ISP can resell still runs at about
> > $900 - $1.4k (?) in the usa? How about elsewhere?
> >
> > ...
> >
> > I am under the impression that many IXPs remain very successful,
> > states without them suffer, and I also find the concept of doing micro
> > IXPs at the city level, appealing, and now achievable with cheap gear.
> > Finer grained cross connects between telco and ISP and IXP would lower
> > latencies across town quite hugely...
> >
> > PS I hear ARIN is planning on dropping the price for, and bundling 3
> > BGP AS numbers at a time, as of the end of this year, also.
> >
> >
> >
> > --
> > Oct 30:
> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> > Dave Täht CSO, LibreQos
>

[-- Attachment #2: Type: text/html, Size: 4870 bytes --]

Re: [LibreQoS] transit and peering costs projections

[Tim Burke]

[-- Attachment #1: Type: text/plain, Size: 4523 bytes --]

I agree, but there are fortunately several large content networks that have had the forethought to put their stuff in Houston - Meta, Fastly, Akamai, AWS just to name a few… There is enough of a need to warrant those other networks having a presence, so hopefully it’s just a matter of time before other content networks jump in too.

Those 4 (plus Google cache fills) make up a huge majority of our transit usage, so at least we’ll get a majority of it peered off after we get these PNI’s stood up. And yes, I will continue to push for Google to light something up in Houston. 🤣

On Oct 15, 2023, at 11:33, Tom Beecher <beecher@beecher.cc> wrote:


So for now, we'll keep paying for transit to get to the others (since it’s about as much as transporting IXP from Dallas), and hoping someone at Google finally sees Houston as more than a third rate city hanging off of Dallas. Or… someone finally brings a worthwhile IX to Houston that gets us more than peering to Kansas City. Yeah, I think the former is more likely. 😊

There is often a chicken/egg scenario here with the economics. As an eyeball network, your costs to build out and connect to Dallas are greater than your transit cost, so you do that. Totally fair.

However think about it from the content side. Say I want to build into to Houston. I have to put routers in, and a bunch of cache servers, so I have capital outlay , plus opex for space, power, IX/backhaul/transit costs. That's not cheap, so there's a lot of calculations that go into it. Is there enough total eyeball traffic there to make it worth it? Is saving 8-10ms enough of a performance boost to justify the spend? What are the long term trends in that market? These answers are of course different for a company running their own CDN vs the commercial CDNs.

I don't work for Google and obviously don't speak for them, but I would suspect that they're happy to eat a 8-10ms performance hit to serve from Dallas , versus the amount of capital outlay to build out there right now.

On Sat, Oct 14, 2023 at 11:47 PM Tim Burke <tim@mid.net<mailto:tim@mid.net>> wrote:
I would say that a 1Gbit IP transit in a carrier neutral DC can be had for a good bit less than $900 on the wholesale market.

Sadly, IXP’s are seemingly turning into a pay to play game, with rates almost costing as much as transit in many cases after you factor in loop costs.

For example, in the Houston market (one of the largest and fastest growing regions in the US!), we do not have a major IX, so to get up to Dallas it’s several thousand for a 100g wave, plus several thousand for a 100g port on one of those major IXes. Or, a better option, we can get a 100g flat internet transit for just a little bit more.

Fortunately, for us as an eyeball network, there are a good number of major content networks that are allowing for private peering in markets like Houston for just the cost of a cross connect and a QSFP if you’re in the right DC, with Google and some others being the outliers.

So for now, we'll keep paying for transit to get to the others (since it’s about as much as transporting IXP from Dallas), and hoping someone at Google finally sees Houston as more than a third rate city hanging off of Dallas. Or… someone finally brings a worthwhile IX to Houston that gets us more than peering to Kansas City. Yeah, I think the former is more likely. 😊

See y’all in San Diego this week,
Tim

On Oct 14, 2023, at 18:04, Dave Taht <dave.taht@gmail.com<mailto:dave.taht@gmail.com>> wrote:
>
> This set of trendlines was very interesting. Unfortunately the data
> stops in 2015. Does anyone have more recent data?
>
> https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php
>
> I believe a gbit circuit that an ISP can resell still runs at about
> $900 - $1.4k (?) in the usa? How about elsewhere?
>
> ...
>
> I am under the impression that many IXPs remain very successful,
> states without them suffer, and I also find the concept of doing micro
> IXPs at the city level, appealing, and now achievable with cheap gear.
> Finer grained cross connects between telco and ISP and IXP would lower
> latencies across town quite hugely...
>
> PS I hear ARIN is planning on dropping the price for, and bundling 3
> BGP AS numbers at a time, as of the end of this year, also.
>
>
>
> --
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos

[-- Attachment #2: Type: text/html, Size: 5947 bytes --]

Re: [LibreQoS] transit and peering costs projections

[Bill Woodcock]

[-- Attachment #1: Type: text/plain, Size: 2375 bytes --]

> On Oct 15, 2023, at 01:01, Dave Taht <dave.taht@gmail.com> wrote:
> I am under the impression that many IXPs remain very successful,

I know of 760 active IXPs, out of 1,148 total, so, over 31 years, two-thirds are still successful now.  Obviously they didn’t all start 31 years ago, they started on a gradually-accelerating curve.  I guess we could do the visualization to plot range of lifespans versus start dates, but we haven’t done that as yet.

> states without them suffer

Any populated area without one or more of them suffers by comparison with areas that do have them.  States, countries, cities, etc.  There are still a surprising number of whole countries that don’t yet have one.  We try to prioritize those in our work:

https://www.pch.net/ixp/summary

> I also find the concept of doing micro IXPs at the city level, appealing, and now achievable with cheap gear.

This has always, by definition, been achievable, since it’s the only way any IXP has ever succeeded, really.  I mean, big sample set, bell curve, you can always find a few things out at the fringes to argue about, but the thing that allows an IXP to succeed is good APBDC, and the thing that most frequently kills IXPs is over-investment.  An expensive switch at the outset is a huge liability, and one of the things most likely to tank a startup IXP.  Notably, that doesn’t mean a switch that costs the IXP a lot of money: you can tank an IXP by donating an expensive switch for free.  Expensive switches have expensive maintenance, whether you’re paying for it or not.  Maintenance means down-time, and down-time raises APBDC, regardless of whether you’ve laid out cash in parallel with it.

> Finer grained cross connects between telco and ISP and IXP would lower latencies across town quite hugely...

Of course, and that requires that they show up in the same building, ideally with an MMR.  The same places that work well for IXPs.  Interconnection basically just requires a lot of networks be present close to a population center.  Which always presents a little tension vis-a-vis datacenters, which profit immensely if there’s a successful IXP in them, but can never afford to locate themselves where IXPs would be most valuable, and don’t like to have to provide free backhaul to better IXP locations.

                                -Bill


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [LibreQoS] transit and peering costs projections

[Jim Troutman]

[-- Attachment #1: Type: text/plain, Size: 2053 bytes --]

Transit 1G wholesale in the right DCs is below $500 per port.  10gigE full
port can be had around $1k-1.5k month on long term deals from multiple
sources.   100g IP transit ports start around $4k.

The cost of transport (dark or wavelength) is generally at least as much as
the IP transit cost, and usually more in underserved markets.  In the
northeast it is very hard to get 10GigE wavelengths below $2k/month to any
location, and is generally closer to $3k.  100g waves are starting around
$4k and go up a lot.

Pricing has come down somewhat over time, but not as fast as transit
prices.   6 years ago a 10Gig wave to Boston from Maine would be about
$5k/month. Today about $2800.

With the cost of XCs in data centers and transport costs, you generally
don’t want to go beyond 2x10gigE before jumping to 100.

On Sat, Oct 14, 2023 at 19:02 Dave Taht via LibreQoS <
libreqos@lists.bufferbloat.net> wrote:

> This set of trendlines was very interesting. Unfortunately the data
> stops in 2015. Does anyone have more recent data?
>
>
> https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php
>
> I believe a gbit circuit that an ISP can resell still runs at about
> $900 - $1.4k (?) in the usa? How about elsewhere?
>
> ...
>
> I am under the impression that many IXPs remain very successful,
> states without them suffer, and I also find the concept of doing micro
> IXPs at the city level, appealing, and now achievable with cheap gear.
> Finer grained cross connects between telco and ISP and IXP would lower
> latencies across town quite hugely...
>
> PS I hear ARIN is planning on dropping the price for, and bundling 3
> BGP AS numbers at a time, as of the end of this year, also.
>
>
>
> --
> Oct 30:
> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos
> _______________________________________________
> LibreQoS mailing list
> LibreQoS@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/libreqos
>

[-- Attachment #2: Type: text/html, Size: 3039 bytes --]

Re: [LibreQoS] transit and peering costs projections

[Tim Burke]

[-- Attachment #1: Type: text/plain, Size: 2299 bytes --]

Man, I wanna know where you’re getting 100g transit for $4500 a month! Even someone as fly by night as Cogent wants almost double that, unfortunately.

On Oct 15, 2023, at 07:43, Jim Troutman <jamesltroutman@gmail.com> wrote:


Transit 1G wholesale in the right DCs is below $500 per port.  10gigE full port can be had around $1k-1.5k month on long term deals from multiple sources.   100g IP transit ports start around $4k.

The cost of transport (dark or wavelength) is generally at least as much as the IP transit cost, and usually more in underserved markets.  In the northeast it is very hard to get 10GigE wavelengths below $2k/month to any location, and is generally closer to $3k.  100g waves are starting around $4k and go up a lot.

Pricing has come down somewhat over time, but not as fast as transit prices.   6 years ago a 10Gig wave to Boston from Maine would be about $5k/month. Today about $2800.

With the cost of XCs in data centers and transport costs, you generally don’t want to go beyond 2x10gigE before jumping to 100.

On Sat, Oct 14, 2023 at 19:02 Dave Taht via LibreQoS <libreqos@lists.bufferbloat.net<mailto:libreqos@lists.bufferbloat.net>> wrote:
This set of trendlines was very interesting. Unfortunately the data
stops in 2015. Does anyone have more recent data?

https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php

I believe a gbit circuit that an ISP can resell still runs at about
$900 - $1.4k (?) in the usa? How about elsewhere?

...

I am under the impression that many IXPs remain very successful,
states without them suffer, and I also find the concept of doing micro
IXPs at the city level, appealing, and now achievable with cheap gear.
Finer grained cross connects between telco and ISP and IXP would lower
latencies across town quite hugely...

PS I hear ARIN is planning on dropping the price for, and bundling 3
BGP AS numbers at a time, as of the end of this year, also.



--
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos
_______________________________________________
LibreQoS mailing list
LibreQoS@lists.bufferbloat.net<mailto:LibreQoS@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/libreqos

[-- Attachment #2: Type: text/html, Size: 3684 bytes --]

Re: [LibreQoS] transit and peering costs projections

[Mike Hammett]

[-- Attachment #1: Type: text/plain, Size: 1801 bytes --]

I've seen some attempts to put an IX at every corner, but I don't think those efforts will be overly successful. 

It's still difficult to gain sufficient scale in NFL-sized cities. Big content won't join without big eyeballs (well, not the national-level guys because they almost never will). Big eyeballs just can't be bothered. Small guys don't move the needle enough. 




----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

----- Original Message -----

From: "Dave Taht" <dave.taht@gmail.com> 
To: "Network Neutrality is back! Let´s make the technical aspects heard this time!" <nnagain@lists.bufferbloat.net>, "libreqos" <libreqos@lists.bufferbloat.net>, "NANOG" <nanog@nanog.org> 
Sent: Saturday, October 14, 2023 6:01:54 PM 
Subject: transit and peering costs projections 

This set of trendlines was very interesting. Unfortunately the data 
stops in 2015. Does anyone have more recent data? 

https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php 

I believe a gbit circuit that an ISP can resell still runs at about 
$900 - $1.4k (?) in the usa? How about elsewhere? 

... 

I am under the impression that many IXPs remain very successful, 
states without them suffer, and I also find the concept of doing micro 
IXPs at the city level, appealing, and now achievable with cheap gear. 
Finer grained cross connects between telco and ISP and IXP would lower 
latencies across town quite hugely... 

PS I hear ARIN is planning on dropping the price for, and bundling 3 
BGP AS numbers at a time, as of the end of this year, also. 



-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html 
Dave Täht CSO, LibreQos 


[-- Attachment #2: Type: text/html, Size: 2271 bytes --]