From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yw1-x1129.google.com (mail-yw1-x1129.google.com [IPv6:2607:f8b0:4864:20::1129]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id DB3853CB39; Sun, 15 Oct 2023 12:44:41 -0400 (EDT) Received: by mail-yw1-x1129.google.com with SMTP id 00721157ae682-59b5484fbe6so46451297b3.1; Sun, 15 Oct 2023 09:44:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697388281; x=1697993081; darn=lists.bufferbloat.net; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=6C9m4/UHZXrfQtZ2aPJxjPfblrVvdsQ5oDbIj1AmZBk=; b=RUCsdCDzTrNIXvUf8jWzTz7R2PUtUTgTLqMpuQjXmiWTswmSolr3RRguNTBZY/CwFo F3CUi4N2Swn/djuZoGttRKFwBdIIEZUXYc3yYttSyCGpO/iNY2eNwTz1PwfKlRWe8RWU 1JOASzz/EQKFnizfoQ6kdXzjEMT2bSWcshsJo33IrPUSD+8mWduHqT3lyMZ5F18TiNzu D34COACSVnSs3v5boBP4W6e3H6G3KCG0oohPXwrIdeProBgip3uV5zYiE9vjT4lDhFKc boO6+X+WF+73yDnBf+CgiETnm1nyVgFpTugbGT3d9vhUHmEpOdVWPT8UwzchXJKs03mU TeYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697388281; x=1697993081; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=6C9m4/UHZXrfQtZ2aPJxjPfblrVvdsQ5oDbIj1AmZBk=; b=Uugb7ViXXAsc5lIrUJk/KNErl6fiSh78datsct/4Wzos2H5d/CTJYoBdU5mDN0eZPf /mT2/1uCfNk3kDVkQ1zaXmZr7dp6PX5J1dkSDJ81DXkhG1hn6wR+YtWUp4S959JY/x2Z c51Gejj8RtMq4Q9SD1rXUhC35YPL2Jhs8OUwoU7RnDx4zAMRd3li0gL5512iB07C3VVv M+ph/dnLbfGyLOMYoEZ2cJVb3Wa1vLFBFshe7UUwn1ubdkrjVeB5Ls5oqStQbK56fC2k tbu8+VTTQymNgdHxSvi4R5o+aK22i231suOxBryhYU8VOWxcBf/GBGr6L4/pR+jQouwu a0ag== X-Gm-Message-State: AOJu0YzPs+mxDAKle53kOFH24ifuAuiYPMuUPUtw9Fy6HjXoMDneA0Sp tU6jjy34vEqtkMstoSgT4T+6I18TO6uVW2GPpHg= X-Google-Smtp-Source: AGHT+IF2phEykGD4+kLUsLWi/E//PI0UO9yMG1tiCsOVia+Rx+U4mjzBUr2724iNhppvBUOOHhtzQhXx3+/iBNegQHI= X-Received: by 2002:a81:8882:0:b0:59c:678:7a32 with SMTP id y124-20020a818882000000b0059c06787a32mr36503491ywf.29.1697388281069; Sun, 15 Oct 2023 09:44:41 -0700 (PDT) MIME-Version: 1.0 References: <1526053658.959.1697377301597.JavaMail.mhammett@Thunderfuck2> In-Reply-To: From: dan Date: Sun, 15 Oct 2023 10:44:29 -0600 Message-ID: To: Tim Burke Cc: Mike Hammett , NANOG , =?UTF-8?Q?Network_Neutrality_is_back=21_Let=C2=B4s_make_the_technical_aspect?= =?UTF-8?Q?s_heard_this_time=21?= , libreqos Content-Type: multipart/alternative; boundary="0000000000008a03ed0607c401bb" Subject: Re: [NNagain] [LibreQoS] transit and peering costs projections X-BeenThere: nnagain@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: =?utf-8?q?Network_Neutrality_is_back!_Let=C2=B4s_make_the_technical_aspects_heard_this_time!?= List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 15 Oct 2023 16:44:41 -0000 --0000000000008a03ed0607c401bb Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Just want to rewind back to the IX map above. The problem is that it's really misleading. Dive down on a number of those (a big number) and they are registered as an IX but they have few tier1 providers in them. The closest one to me is essentially just fed by Zayo. Not much of an IX when there's only one path out on one provider... If big providers, or at least multiple providers linking to other IX' aren't participating, then the purpose isn't met. Zayo isn't offering IX rates in these for lack of competition so the incentive to build out from there is very low, ie I can get Zayo in a road-side hut for basically the same price and not have to share access. I also realize that getting 2-4 providers into a shack in the middle of no-where doesn't make sense either, but population dictates a lot of this. I know it's a big ask, getting full size IX access in a microIX, but that's what big government projects are for. Get these carriers that are crossing various jurisdictions to drop transport services, waves or dark viber etc, into something useful like a school, courthouse, town hall, whatever, and in that build out link to the two IX's that fiber crossing was going between. Just put in the deal that they put in optics aligned with the population. Frankly, 40G to most of these areas would be plenty for a decade or more and 40G optics long distance modules are only a few grand each. Maybe $10-15,000 for redundant 40G and they've already run the fiber as part of that delivery to the facility (double that for really long runs...). Schools would be my #1 pick here because it solves a lot of issues. Gov pulls in at least 1x 40G to every single incorporated school and builds access facilities for that (conduits to edges of property etc) and at some threshold that's 1x40G with 2 providers then 3 providers for bigger populations and as populations grow. Standard prices on port and they are all just a vlan or equiv on the pipe back to the IX. Basically these would be like IX extension sites with layer2 ports between provided by long-haul providers. On Sun, Oct 15, 2023 at 8:19=E2=80=AFAM Tim Burke via LibreQoS < libreqos@lists.bufferbloat.net> wrote: > I=E2=80=99ve found that most of the CDNs that matter are in one facility = in > Houston, the Databank West (formerly Cyrus One) campus. We are about to > light up a POP there so we=E2=80=99ll at least be able to get PNIs to the= m. There > is even an IX in the facility, but it=E2=80=99s relatively small (likely = because > the operator wants near-transit pricing to get on it) so we=E2=80=99ll ju= st PNI > what we can for now. > > On Oct 15, 2023, at 08:50, Mike Hammett wrote: > > =EF=BB=BF > Houston is tricky as due to it's geographic scope, it's quite expensive t= o > build an IX that goes into enough facilities to achieve meaningful scale. > CDN 1 is in facility A. CDN 2 in facility B. CDN 3 is in facility C. When= I > last looked, it was about 80 driving miles to have a dark fiber ring that > encompassed all of the facilities one would need to be in. > > > > ----- > Mike Hammett > Intelligent Computing Solutions > http://www.ics-il.com > > Midwest-IX > http://www.midwest-ix.com > > ------------------------------ > *From: *"Tim Burke" > *To: *"Dave Taht" > *Cc: *"Network Neutrality is back! Let=C2=B4s make the technical aspects = heard > this time!" , "libreqos" < > libreqos@lists.bufferbloat.net>, "NANOG" > *Sent: *Saturday, October 14, 2023 10:45:47 PM > *Subject: *Re: transit and peering costs projections > > I would say that a 1Gbit IP transit in a carrier neutral DC can be had fo= r > a good bit less than $900 on the wholesale market. > > Sadly, IXP=E2=80=99s are seemingly turning into a pay to play game, with = rates > almost costing as much as transit in many cases after you factor in loop > costs. > > For example, in the Houston market (one of the largest and fastest growin= g > regions in the US!), we do not have a major IX, so to get up to Dallas it= =E2=80=99s > several thousand for a 100g wave, plus several thousand for a 100g port o= n > one of those major IXes. Or, a better option, we can get a 100g flat > internet transit for just a little bit more. > > Fortunately, for us as an eyeball network, there are a good number of > major content networks that are allowing for private peering in markets > like Houston for just the cost of a cross connect and a QSFP if you=E2=80= =99re in > the right DC, with Google and some others being the outliers. > > So for now, we'll keep paying for transit to get to the others (since it= =E2=80=99s > about as much as transporting IXP from Dallas), and hoping someone at > Google finally sees Houston as more than a third rate city hanging off of > Dallas. Or=E2=80=A6 someone finally brings a worthwhile IX to Houston tha= t gets us > more than peering to Kansas City. Yeah, I think the former is more likely= . > =F0=9F=98=8A > > See y=E2=80=99all in San Diego this week, > Tim > > On Oct 14, 2023, at 18:04, Dave Taht wrote: > > > > =EF=BB=BFThis set of trendlines was very interesting. Unfortunately the= data > > stops in 2015. Does anyone have more recent data? > > > > > https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-An= d-Projected.php > > > > I believe a gbit circuit that an ISP can resell still runs at about > > $900 - $1.4k (?) in the usa? How about elsewhere? > > > > ... > > > > I am under the impression that many IXPs remain very successful, > > states without them suffer, and I also find the concept of doing micro > > IXPs at the city level, appealing, and now achievable with cheap gear. > > Finer grained cross connects between telco and ISP and IXP would lower > > latencies across town quite hugely... > > > > PS I hear ARIN is planning on dropping the price for, and bundling 3 > > BGP AS numbers at a time, as of the end of this year, also. > > > > > > > > -- > > Oct 30: > https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html > > Dave T=C3=A4ht CSO, LibreQos > > _______________________________________________ > LibreQoS mailing list > LibreQoS@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/libreqos > --0000000000008a03ed0607c401bb Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Just want to rewind back to the IX map above.=C2=A0 The pr= oblem is that it's really misleading.=C2=A0 Dive down on a number of th= ose (a big number) and they are registered as an IX but they=C2=A0have few = tier1 providers in them.=C2=A0 The closest one to me is essentially just fe= d by Zayo.=C2=A0 =C2=A0Not much of an IX when there's only one path out= on one provider...=C2=A0

If big providers, or at least multiple pro= viders linking to other IX' aren't participating, then the purpose = isn't met.=C2=A0 Zayo isn't offering IX rates in these for lack of = competition so the incentive to build out from there is very low, ie I can = get Zayo in a road-side hut for basically the same price and not have to sh= are access.=C2=A0 I also realize that getting 2-4 providers into a shack in= the middle of no-where=C2=A0doesn't make sense either, but population = dictates a lot of this.

I know it's a big ask, getting full size= IX access in a microIX, but that's what big government projects are fo= r.=C2=A0 Get these carriers that are crossing various jurisdictions to drop= transport services, waves or dark viber etc, into something useful like a = school, courthouse, town hall, whatever, and in that build out link to the = two IX's that fiber crossing was going between.=C2=A0 Just put in the d= eal that they put in optics aligned with the population.=C2=A0 Frankly, 40G= to most of these areas would be plenty for a decade or more and 40G optics= long distance modules are only a few grand each.=C2=A0 Maybe $10-15,000 fo= r redundant 40G and they've already run the fiber as part of that deliv= ery to the facility (double that for really long runs...).=C2=A0 Schools wo= uld be my #1 pick here because it solves a lot of issues.=C2=A0 Gov pulls i= n at least 1x 40G to every single incorporated school and builds access fac= ilities for that (conduits to edges of property etc) and at some threshold = that's 1x40G with 2 providers then 3 providers for bigger populations a= nd as populations grow.=C2=A0 Standard prices on port and they are all just= a vlan or equiv on the pipe back to the IX.=C2=A0 Basically these would be= like IX extension sites with layer2 ports between provided by long-haul pr= oviders.



On Sun, Oct 15, 2023 at 8:19=E2=80=AFAM Tim Burk= e via LibreQoS <libreq= os@lists.bufferbloat.net> wrote:
I=E2=80=99ve found that most of the CDNs that matter are in one facility in= Houston, the Databank West (formerly Cyrus One) campus. We are about to li= ght up a POP there so we=E2=80=99ll at least be able to get PNIs to them. T= here is even an IX in the facility, but it=E2=80=99s relatively small (likely because the operator wants near-transit pricing to get on it= ) so we=E2=80=99ll just PNI what we can for now.=C2=A0

On Oct 15, 2023, at 08:50, Mike Hammett <nanog@ics-il.net> w= rote:

=EF=BB=BF
Houston is tricky as due to it's geographic scope, it's quite expen= sive to build an IX that goes into enough facilities to achieve meaningful = scale. CDN 1 is in facility A. CDN 2 in facility B. CDN 3 is in facility C.= When I last looked, it was about 80 driving miles to have a dark fiber ring that encompassed all of the facilities one= would need to be in.



-----
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-= ix.com

From: "Tim Burke" <tim@mid.net>
To: "Dave Taht" <dave.taht@gmail.com>
Cc: "Network Neutrality is back! Let=C2=B4s make the technical = aspects heard this time!" <nnagain@lists.bufferbloat.net>, "lib= reqos" <libreqos@lists.bufferbloat.net>, "NANOG" <nanog@nanog.org> Sent: Saturday, October 14, 2023 10:45:47 PM
Subject: Re: transit and peering costs projections

I would say that a 1Gbit IP transit in a carrier neutral DC can be had for = a good bit less than $900 on the wholesale market.

Sadly, IXP=E2=80=99s are seemingly turning into a pay to play game, with ra= tes almost costing as much as transit in many cases after you factor in loo= p costs.

For example, in the Houston market (one of the largest and fastest growing = regions in the US!), we do not have a major IX, so to get up to Dallas it= =E2=80=99s several thousand for a 100g wave, plus several thousand for a 10= 0g port on one of those major IXes. Or, a better option, we can get a 100g flat internet transit for just a little b= it more.

Fortunately, for us as an eyeball network, there are a good number of major= content networks that are allowing for private peering in markets like Hou= ston for just the cost of a cross connect and a QSFP if you=E2=80=99re in t= he right DC, with Google and some others being the outliers.

So for now, we'll keep paying for transit to get to the others (since i= t=E2=80=99s about as much as transporting IXP from Dallas), and hoping some= one at Google finally sees Houston as more than a third rate city hanging o= ff of Dallas. Or=E2=80=A6 someone finally brings a worthwhile IX to Houston that gets us more than peering to Kansas City. Yeah, I think= the former is more likely. =F0=9F=98=8A

See y=E2=80=99all in San Diego this week,
Tim

On Oct 14, 2023, at 18:04, Dave Taht <dave.taht@gmail.com> wrote:
>
> =EF=BB=BFThis set of trendlines was very interesting. Unfortunately th= e data
> stops in 2015. Does anyone have more recent data?
>
> https://drpeering.net/whit= e-papers/Internet-Transit-Pricing-Historical-And-Projected.php
>
> I believe a gbit circuit that an ISP can resell still runs at about > $900 - $1.4k (?) in the usa? How about elsewhere?
>
> ...
>
> I am under the impression that many IXPs remain very successful,
> states without them suffer, and I also find the concept of doing micro=
> IXPs at the city level, appealing, and now achievable with cheap gear.=
> Finer grained cross connects between telco and ISP and IXP would lower=
> latencies across town quite hugely...
>
> PS I hear ARIN is planning on dropping the price for, and bundling 3 > BGP AS numbers at a time, as of the end of this year, also.
>
>
>
> --
> Oct 30: https://netdevconf.info/0x17/news/the-= maestro-and-the-music-bof.html
> Dave T=C3=A4ht CSO, LibreQos

_______________________________________________
LibreQoS mailing list
LibreQo= S@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/libreqos
--0000000000008a03ed0607c401bb--