From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-vs1-xe33.google.com (mail-vs1-xe33.google.com [IPv6:2607:f8b0:4864:20::e33]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id BE3843CB39 for ; Sun, 15 Oct 2023 12:32:59 -0400 (EDT) Received: by mail-vs1-xe33.google.com with SMTP id ada2fe7eead31-457ee0df54eso53314137.0 for ; Sun, 15 Oct 2023 09:32:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=beecher.cc; s=google; t=1697387579; x=1697992379; darn=lists.bufferbloat.net; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=kOwn0LDbfjLHpAYJjQiOLfcL66ot9pM0AdqJRq0WByg=; b=JPWqmFaZ/K5FGXFZPlkPHNMD8l7eFJXRX0s/Ihe13pjymPrFy553z77ZcPSalDzxux OhEZLE3llnKtpnzyhKxJByK+8Sn9RYwEKe4mlNX7fWfJlK8i02Lm+uYIJ3objEP2uukK 9Ea+Mhwf6GoOERXWVsh69j/dV7PFhvNXblTJt1PuPvlL0/pzpQoSAnhH9HAOWsz+bkzL nIvWUf6+0VrFyj/WDF0+a2d6kyEYrEBJ4U3aXer9c2sR/x+6t/VYG8OLJXZICXJPBe2M 0rx4w6fMyRDBXrSiDkPZveUxIiUdzb39ZdTkUgUDHFbeBR15PnNkLRCXJs9SVOC8UWRL pEWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697387579; x=1697992379; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=kOwn0LDbfjLHpAYJjQiOLfcL66ot9pM0AdqJRq0WByg=; b=p1ANkk8/HkaQXzaXIh+xwEliM9CyVc/Pg4IlPeVRbtFea0EqpRsjbiihZSxIS60Ct1 EZHsIVF3e4W0FhQyhU+x0Ao5lu02F9s1U8+btQ9CCdAiyoe7HIsXsd1kwaRv9BkXRjTn 0yJTNir6lw/ohDQz66Up6qL8m8Klpak3sW+AC6MW/xHXhdoL2dM4WD84SsS96t9VBC5Q rE4LRmx32HdkHjYLcrTVpfInOhzt98/tCSgG19zpQDlCCZJXLFkDJm1v/4vL+Cb1gbE6 CGNfYTkmcjws4nlWHONn2RMnTbSgTpb24fEk+pfREj4cYzCj7QJaKJO0C+WWb06Wwv5S JIYQ== X-Gm-Message-State: AOJu0YyiAxM5wtT05fiMFz0oC/cFeoLk8GwGa5GkY+A57QVbVaJipg+b Crrne0xfBA+qn+rPuz6GcUo79iNvENRtOIlNM8qklw== X-Google-Smtp-Source: AGHT+IHyoTmn/T+5OZz6aGgqxaU45FH4V2BZX/8nncOj8p93VaraPV2rlxCNSJgOm5Ryivk9iFvzxe3ZCkzskGZTJ/g= X-Received: by 2002:a05:6102:7c2:b0:44e:98d8:c62e with SMTP id y2-20020a05610207c200b0044e98d8c62emr29834025vsg.33.1697387579049; Sun, 15 Oct 2023 09:32:59 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Tom Beecher Date: Sun, 15 Oct 2023 12:32:48 -0400 Message-ID: To: Tim Burke Cc: Dave Taht , =?UTF-8?Q?Network_Neutrality_is_back=21_Let=C2=B4s_make_the_technical_aspect?= =?UTF-8?Q?s_heard_this_time=21?= , libreqos , NANOG Content-Type: multipart/alternative; boundary="000000000000b22d560607c3d792" X-Mailman-Approved-At: Mon, 16 Oct 2023 03:16:32 -0400 Subject: Re: [NNagain] transit and peering costs projections X-BeenThere: nnagain@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: =?utf-8?q?Network_Neutrality_is_back!_Let=C2=B4s_make_the_technical_aspects_heard_this_time!?= List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 15 Oct 2023 16:32:59 -0000 --000000000000b22d560607c3d792 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable > > So for now, we'll keep paying for transit to get to the others (since it= =E2=80=99s > about as much as transporting IXP from Dallas), and hoping someone at > Google finally sees Houston as more than a third rate city hanging off of > Dallas. Or=E2=80=A6 someone finally brings a worthwhile IX to Houston tha= t gets us > more than peering to Kansas City. Yeah, I think the former is more > likely. =F0=9F=98=8A > There is often a chicken/egg scenario here with the economics. As an eyeball network, your costs to build out and connect to Dallas are greater than your transit cost, so you do that. Totally fair. However think about it from the content side. Say I want to build into to Houston. I have to put routers in, and a bunch of cache servers, so I have capital outlay , plus opex for space, power, IX/backhaul/transit costs. That's not cheap, so there's a lot of calculations that go into it. Is there enough total eyeball traffic there to make it worth it? Is saving 8-10ms enough of a performance boost to justify the spend? What are the long term trends in that market? These answers are of course different for a company running their own CDN vs the commercial CDNs. I don't work for Google and obviously don't speak for them, but I would suspect that they're happy to eat a 8-10ms performance hit to serve from Dallas , versus the amount of capital outlay to build out there right now. On Sat, Oct 14, 2023 at 11:47=E2=80=AFPM Tim Burke wrote: > I would say that a 1Gbit IP transit in a carrier neutral DC can be had fo= r > a good bit less than $900 on the wholesale market. > > Sadly, IXP=E2=80=99s are seemingly turning into a pay to play game, with = rates > almost costing as much as transit in many cases after you factor in loop > costs. > > For example, in the Houston market (one of the largest and fastest growin= g > regions in the US!), we do not have a major IX, so to get up to Dallas it= =E2=80=99s > several thousand for a 100g wave, plus several thousand for a 100g port o= n > one of those major IXes. Or, a better option, we can get a 100g flat > internet transit for just a little bit more. > > Fortunately, for us as an eyeball network, there are a good number of > major content networks that are allowing for private peering in markets > like Houston for just the cost of a cross connect and a QSFP if you=E2=80= =99re in > the right DC, with Google and some others being the outliers. > > So for now, we'll keep paying for transit to get to the others (since it= =E2=80=99s > about as much as transporting IXP from Dallas), and hoping someone at > Google finally sees Houston as more than a third rate city hanging off of > Dallas. Or=E2=80=A6 someone finally brings a worthwhile IX to Houston tha= t gets us > more than peering to Kansas City. Yeah, I think the former is more likely= . > =F0=9F=98=8A > > See y=E2=80=99all in San Diego this week, > Tim > > On Oct 14, 2023, at 18:04, Dave Taht wrote: > > > > =EF=BB=BFThis set of trendlines was very interesting. Unfortunately the= data > > stops in 2015. Does anyone have more recent data? > > > > > https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-An= d-Projected.php > > > > I believe a gbit circuit that an ISP can resell still runs at about > > $900 - $1.4k (?) in the usa? How about elsewhere? > > > > ... > > > > I am under the impression that many IXPs remain very successful, > > states without them suffer, and I also find the concept of doing micro > > IXPs at the city level, appealing, and now achievable with cheap gear. > > Finer grained cross connects between telco and ISP and IXP would lower > > latencies across town quite hugely... > > > > PS I hear ARIN is planning on dropping the price for, and bundling 3 > > BGP AS numbers at a time, as of the end of this year, also. > > > > > > > > -- > > Oct 30: > https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html > > Dave T=C3=A4ht CSO, LibreQos > --000000000000b22d560607c3d792 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
So for n= ow, we'll keep paying for transit to get to the others (since it=E2=80= =99s about as much as transporting IXP from Dallas), and hoping someone at = Google finally sees Houston as more than a third rate city hanging off of D= allas. Or=E2=80=A6 someone finally brings a worthwhile IX to Houston that g= ets us more than peering to Kansas City. Yeah, I think the former is more l= ikely.=C2=A0=F0=9F=98=8A

There is often= a chicken/egg scenario here with the economics. As an eyeball network, you= r costs to build out and connect to Dallas are greater than your transit co= st, so you do that. Totally fair.=C2=A0

However th= ink about it from the content side. Say I want to build into to Houston. I = have to put routers in, and a bunch of cache servers, so I have capital out= lay=C2=A0, plus opex for space, power, IX/backhaul/transit costs. That'= s not cheap, so there's a lot of calculations that go into it. Is there= enough total eyeball traffic there to make it worth it? Is saving 8-10ms e= nough of a performance boost to justify the spend? What are the long term t= rends in that market? These answers are of course different for a company r= unning their own CDN vs the commercial CDNs.=C2=A0

I don't work for Google and obviously don't speak for them, but I = would suspect that they're happy to eat a 8-10ms performance hit to ser= ve from Dallas , versus the amount of capital outlay to build out there rig= ht now.=C2=A0

On Sat, Oct 14, 2023 at 11:47=E2=80=AFPM Tim Burke <<= a href=3D"mailto:tim@mid.net">tim@mid.net> wrote:
I would say that a 1Gbit IP transi= t in a carrier neutral DC can be had for a good bit less than $900 on the w= holesale market.

Sadly, IXP=E2=80=99s are seemingly turning into a pay to play game, with ra= tes almost costing as much as transit in many cases after you factor in loo= p costs.

For example, in the Houston market (one of the largest and fastest growing = regions in the US!), we do not have a major IX, so to get up to Dallas it= =E2=80=99s several thousand for a 100g wave, plus several thousand for a 10= 0g port on one of those major IXes. Or, a better option, we can get a 100g = flat internet transit for just a little bit more.

Fortunately, for us as an eyeball network, there are a good number of major= content networks that are allowing for private peering in markets like Hou= ston for just the cost of a cross connect and a QSFP if you=E2=80=99re in t= he right DC, with Google and some others being the outliers.

So for now, we'll keep paying for transit to get to the others (since i= t=E2=80=99s about as much as transporting IXP from Dallas), and hoping some= one at Google finally sees Houston as more than a third rate city hanging o= ff of Dallas. Or=E2=80=A6 someone finally brings a worthwhile IX to Houston= that gets us more than peering to Kansas City. Yeah, I think the former is= more likely. =F0=9F=98=8A

See y=E2=80=99all in San Diego this week,
Tim

On Oct 14, 2023, at 18:04, Dave Taht <dave.taht@gmail.com> wrote:
>
> =EF=BB=BFThis set of trendlines was very interesting. Unfortunately th= e data
> stops in 2015. Does anyone have more recent data?
>
> https:/= /drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Project= ed.php
>
> I believe a gbit circuit that an ISP can resell still runs at about > $900 - $1.4k (?) in the usa? How about elsewhere?
>
> ...
>
> I am under the impression that many IXPs remain very successful,
> states without them suffer, and I also find the concept of doing micro=
> IXPs at the city level, appealing, and now achievable with cheap gear.=
> Finer grained cross connects between telco and ISP and IXP would lower=
> latencies across town quite hugely...
>
> PS I hear ARIN is planning on dropping the price for, and bundling 3 > BGP AS numbers at a time, as of the end of this year, also.
>
>
>
> --
> Oct 30: https://netdevconf.= info/0x17/news/the-maestro-and-the-music-bof.html
> Dave T=C3=A4ht CSO, LibreQos
--000000000000b22d560607c3d792--