From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yb1-xb2d.google.com (mail-yb1-xb2d.google.com [IPv6:2607:f8b0:4864:20::b2d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 6550C3CB37; Sun, 15 Oct 2023 00:26:37 -0400 (EDT) Received: by mail-yb1-xb2d.google.com with SMTP id 3f1490d57ef6-d9b2ca542e5so1975578276.3; Sat, 14 Oct 2023 21:26:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697343996; x=1697948796; darn=lists.bufferbloat.net; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=lEmu4GQHDV/puY52VYX+MCbI4ZtlQUvjRSBxZmuLQ+s=; b=K01+OaUTLdzhyg4197Pcal8zgYAH5QYAevVyW70HbTtdSaqv09dNIHNSlncWBwejr4 sweWpZc3Ij6joP19BVvtmtgy5ObaUoEzglZo+mF0UUWQVnJLFYi97EO4iWsS1l0szJQW EKTL3PVyEy2KuIPcr1b2SUjNOm+GwmPv3ZRguGANRKd8WPBaE6gIf0AqPHRRLrH6ibt8 LBiKT4rGYlLtED/cQa36QfyJ8ZyAni7rh+oXiJk7eVjN+UXnj5EAMzVuwA0M9r908mI2 1Iq1hNUovys9nyhIHe3uuKYfpKIyGHGNbmy8EtclNtoOmLbvshVqRwqn9lk7rNnZFjAz tA4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697343996; x=1697948796; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=lEmu4GQHDV/puY52VYX+MCbI4ZtlQUvjRSBxZmuLQ+s=; b=H+ChnxDqdxaJAKdqe2xAdPJASfp5Fi4XU8KoHvgyomIfucy+Ae7ILMdvShSXLRK2y/ 0Vque2h+eMlq4B7vXU9NKODOSbaDmMSaGhEnkW+y3h8Cp/BqZW9rWeNhKcUOfJoIV/A+ ZRuADU6evLn8EoSzayx8Og8r4zoSjbAf5NXvOHu8sY/eUn7Nird1Y4N16IPZ0asovYGO T5xpktdwj2CUQDau+cKU2I212MgWYpaMJKEZFz7jdcdIFEcgJheekoPBuiGeBG3QarSt koLBh1dzjhjt5mG8p2KM1lH0To5f3WqPlcCaMzhqN0hXVLQG6OAwATJ7dya3ZYTI17Jg ncgQ== X-Gm-Message-State: AOJu0Ywyg3iOxIf+voQax00WBqtJLqMbfRKs/nblKSTNKSHXRLLAOqjs hBequXlurSUmHrjvIyDlnzes6WTL4Z3F8vL8wfE= X-Google-Smtp-Source: AGHT+IHyLR/JYaWc3rXGmRJo5BItAkMfUgDt+sgqvQGPoDT7WVrAToKMRKfZ+FIjNoKRZx+MRUiL3f8nb6DtnmcsvG8= X-Received: by 2002:a05:690c:f05:b0:5a7:e726:13f7 with SMTP id dc5-20020a05690c0f0500b005a7e72613f7mr15053325ywb.8.1697343996250; Sat, 14 Oct 2023 21:26:36 -0700 (PDT) MIME-Version: 1.0 References: <78D8577D-148B-4EB1-993C-62D42521791A@mid.net> In-Reply-To: From: dan Date: Sat, 14 Oct 2023 22:26:24 -0600 Message-ID: To: Dave Taht Cc: Tim Burke , Ryan Hamel , NANOG , =?UTF-8?Q?Network_Neutrality_is_back=21_Let=C2=B4s_make_the_technical_aspect?= =?UTF-8?Q?s_heard_this_time=21?= , libreqos Content-Type: multipart/alternative; boundary="000000000000f54e8a0607b9b11d" Subject: Re: [LibreQoS] transit and peering costs projections X-BeenThere: libreqos@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Many ISPs need the kinds of quality shaping cake can do List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 15 Oct 2023 04:26:37 -0000 --000000000000f54e8a0607b9b11d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable the 900-1400 for 1G is right at what I'm seeing in the rockies region. price scales decently at 10G. transit costs just as much as DIA or more because of port costs on each side. Also, with zayo and lumen, traversing their MPLS networks a few hundred miles probably costs you 10-15ms latency. That's what I'm seeing. So unless you're doing a wave or something that isn't getting battered in their neglected networks centralizing could be worse overall. On Sat, Oct 14, 2023 at 10:19=E2=80=AFPM Dave Taht via LibreQoS < libreqos@lists.bufferbloat.net> wrote: > On Sat, Oct 14, 2023 at 9:12=E2=80=AFPM Tim Burke wrote: > > > > It=E2=80=99s better for customer experience to keep it local instead of= adding > 200 miles to the route. All of the competition hauls all of their traffic > up to Dallas, so we easily have a nice 8-10ms latency advantage by keepin= g > transit and peering as close to the customer as possible. > > > > Plus, you can=E2=80=99t forget to mention another ~$10k MRC per pair in= DF costs > to get up to Dallas, not including colo, that we can spend on more transi= t > or better gear! > > Texas's BEAD funding and broadband offices are looking for proposals > and seem to have dollars to spend. I have spent much of the past few > years attempting to convince these entities that what was often more > needed was better, more local IXPs. Have you reached out to them? > > > > On Oct 14, 2023, at 23:03, Ryan Hamel wrote: > > > > =EF=BB=BF > > Why not place the routers in Dallas, aggregate the transit, IXP, and > PNI's there, and backhaul it over redundant dark fiber with DWDM waves or > 400G OpenZR? > > > > Ryan > > > > ________________________________ > > From: NANOG on behalf of T= im > Burke > > Sent: Saturday, October 14, 2023 8:45 PM > > To: Dave Taht > > Cc: Network Neutrality is back! Let=C2=B4s make the technical aspects h= eard > this time! ; libreqos < > libreqos@lists.bufferbloat.net>; NANOG > > Subject: Re: transit and peering costs projections > > > > Caution: This is an external email and may be malicious. Please take > care when clicking links or opening attachments. > > > > > > I would say that a 1Gbit IP transit in a carrier neutral DC can be had > for a good bit less than $900 on the wholesale market. > > > > Sadly, IXP=E2=80=99s are seemingly turning into a pay to play game, wit= h rates > almost costing as much as transit in many cases after you factor in loop > costs. > > > > For example, in the Houston market (one of the largest and fastest > growing regions in the US!), we do not have a major IX, so to get up to > Dallas it=E2=80=99s several thousand for a 100g wave, plus several thousa= nd for a > 100g port on one of those major IXes. Or, a better option, we can get a > 100g flat internet transit for just a little bit more. > > > > Fortunately, for us as an eyeball network, there are a good number of > major content networks that are allowing for private peering in markets > like Houston for just the cost of a cross connect and a QSFP if you=E2=80= =99re in > the right DC, with Google and some others being the outliers. > > > > So for now, we'll keep paying for transit to get to the others (since > it=E2=80=99s about as much as transporting IXP from Dallas), and hoping s= omeone at > Google finally sees Houston as more than a third rate city hanging off of > Dallas. Or=E2=80=A6 someone finally brings a worthwhile IX to Houston tha= t gets us > more than peering to Kansas City. Yeah, I think the former is more likely= . > =F0=9F=98=8A > > > > See y=E2=80=99all in San Diego this week, > > Tim > > > > On Oct 14, 2023, at 18:04, Dave Taht wrote: > > > > > > =EF=BB=BFThis set of trendlines was very interesting. Unfortunately t= he data > > > stops in 2015. Does anyone have more recent data? > > > > > > > https://nam12.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fdrpee= ring.net%2Fwhite-papers%2FInternet-Transit-Pricing-Historical-And-Projected= .php&data=3D05%7C01%7Cryan%40rkhtech.org%7Cc8ebae9f0ecd4b368dcb08dbcd319880= %7C81c24bb4f9ec4739ba4d25c42594d996%7C0%7C0%7C638329385118876648%7CUnknown%= 7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6M= n0%3D%7C3000%7C%7C%7C&sdata=3DnQeWrGi%2BblMmtiG9u7SdF3JOi1h9Fni7xXo%2FusZRo= pA%3D&reserved=3D0 > > > > > > I believe a gbit circuit that an ISP can resell still runs at about > > > $900 - $1.4k (?) in the usa? How about elsewhere? > > > > > > ... > > > > > > I am under the impression that many IXPs remain very successful, > > > states without them suffer, and I also find the concept of doing micr= o > > > IXPs at the city level, appealing, and now achievable with cheap gear= . > > > Finer grained cross connects between telco and ISP and IXP would lowe= r > > > latencies across town quite hugely... > > > > > > PS I hear ARIN is planning on dropping the price for, and bundling 3 > > > BGP AS numbers at a time, as of the end of this year, also. > > > > > > > > > > > > -- > > > Oct 30: > https://nam12.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fnetde= vconf.info%2F0x17%2Fnews%2Fthe-maestro-and-the-music-bof.html&data=3D05%7C0= 1%7Cryan%40rkhtech.org%7Cc8ebae9f0ecd4b368dcb08dbcd319880%7C81c24bb4f9ec473= 9ba4d25c42594d996%7C0%7C0%7C638329385118876648%7CUnknown%7CTWFpbGZsb3d8eyJW= IjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C= %7C&sdata=3DROLgtoeiBgfAG40UZqS8Zd8vMK%2B0HQB7RV%2FhQRvIcFM%3D&reserved=3D0 > > > Dave T=C3=A4ht CSO, LibreQos > > > > -- > Oct 30: > https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html > Dave T=C3=A4ht CSO, LibreQos > _______________________________________________ > LibreQoS mailing list > LibreQoS@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/libreqos > --000000000000f54e8a0607b9b11d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
the 900-1400 for 1G is right at what I'm seeing in the= rockies region.=C2=A0 price scales decently at 10G.=C2=A0 =C2=A0transit co= sts just as much as DIA or more because of port costs on each side.

= Also, with zayo and lumen, traversing their MPLS networks a few hundred mil= es probably costs you 10-15ms latency.=C2=A0 That's what I'm seeing= .=C2=A0 So unless you're doing a wave or something that isn't getti= ng battered in their neglected networks centralizing could be worse overall= .



On Sat, Oct 14, 2023 at 10:19=E2=80=AFPM Dave Taht via LibreQo= S <libreqos@lists.buff= erbloat.net> wrote:
On Sat, Oct 14, 2023 at 9:12=E2=80=AFPM Tim Burke <tim@mid.net> wrote:
>
> It=E2=80=99s better for customer experience to keep it local instead o= f adding 200 miles to the route. All of the competition hauls all of their = traffic up to Dallas, so we easily have a nice 8-10ms latency advantage by = keeping transit and peering as close to the customer as possible.
>
> Plus, you can=E2=80=99t forget to mention another ~$10k MRC per pair i= n DF costs to get up to Dallas, not including colo, that we can spend on mo= re transit or better gear!

Texas's BEAD funding and broadband offices are looking for proposals and seem to have dollars to spend. I have spent much of the past few
years attempting to convince these entities that what was often more
needed was better, more local IXPs. Have you reached out to them?


> On Oct 14, 2023, at 23:03, Ryan Hamel <ryan@rkhtech.org> wrote:
>
> =EF=BB=BF
> Why not place the routers in Dallas, aggregate the transit, IXP, and P= NI's there, and backhaul it over redundant dark fiber with DWDM waves o= r 400G OpenZR?
>
> Ryan
>
> ________________________________
> From: NANOG <nanog-bounces+ryan=3Drkhtech.org@nanog.org> on behalf of Tim Bu= rke <tim@mid.net>= ;
> Sent: Saturday, October 14, 2023 8:45 PM
> To: Dave Taht <dave.taht@gmail.com>
> Cc: Network Neutrality is back! Let=C2=B4s make the technical aspects = heard this time! <nnagain@lists.bufferbloat.net>; libreqos <libreqos@lists.bu= fferbloat.net>; NANOG <nanog@nanog.org>
> Subject: Re: transit and peering costs projections
>
> Caution: This is an external email and may be malicious. Please take c= are when clicking links or opening attachments.
>
>
> I would say that a 1Gbit IP transit in a carrier neutral DC can be had= for a good bit less than $900 on the wholesale market.
>
> Sadly, IXP=E2=80=99s are seemingly turning into a pay to play game, wi= th rates almost costing as much as transit in many cases after you factor i= n loop costs.
>
> For example, in the Houston market (one of the largest and fastest gro= wing regions in the US!), we do not have a major IX, so to get up to Dallas= it=E2=80=99s several thousand for a 100g wave, plus several thousand for a= 100g port on one of those major IXes. Or, a better option, we can get a 10= 0g flat internet transit for just a little bit more.
>
> Fortunately, for us as an eyeball network, there are a good number of = major content networks that are allowing for private peering in markets lik= e Houston for just the cost of a cross connect and a QSFP if you=E2=80=99re= in the right DC, with Google and some others being the outliers.
>
> So for now, we'll keep paying for transit to get to the others (si= nce it=E2=80=99s about as much as transporting IXP from Dallas), and hoping= someone at Google finally sees Houston as more than a third rate city hang= ing off of Dallas. Or=E2=80=A6 someone finally brings a worthwhile IX to Ho= uston that gets us more than peering to Kansas City. Yeah, I think the form= er is more likely. =F0=9F=98=8A
>
> See y=E2=80=99all in San Diego this week,
> Tim
>
> On Oct 14, 2023, at 18:04, Dave Taht <dave.taht@gmail.com> wrote:
> >
> > =EF=BB=BFThis set of trendlines was very interesting. Unfortunate= ly the data
> > stops in 2015. Does anyone have more recent data?
> >
> > https://nam12.safelinks.protection.outlook.com/?url=3Dhttps%3A= %2F%2Fdrpeering.net%2Fwhite-papers%2FInternet-Transit-Pricing-Historical-An= d-Projected.php&data=3D05%7C01%7Cryan%40rkhtech.org%7Cc8ebae9f0ecd4b368= dcb08dbcd319880%7C81c24bb4f9ec4739ba4d25c42594d996%7C0%7C0%7C63832938511887= 6648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik= 1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3DnQeWrGi%2BblMmtiG9u7SdF3JO= i1h9Fni7xXo%2FusZRopA%3D&reserved=3D0
> >
> > I believe a gbit circuit that an ISP can resell still runs at abo= ut
> > $900 - $1.4k (?) in the usa? How about elsewhere?
> >
> > ...
> >
> > I am under the impression that many IXPs remain very successful,<= br> > > states without them suffer, and I also find the concept of doing = micro
> > IXPs at the city level, appealing, and now achievable with cheap = gear.
> > Finer grained cross connects between telco and ISP and IXP would = lower
> > latencies across town quite hugely...
> >
> > PS I hear ARIN is planning on dropping the price for, and bundlin= g 3
> > BGP AS numbers at a time, as of the end of this year, also.
> >
> >
> >
> > --
> > Oct 30: https://nam12.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fnetd= evconf.info%2F0x17%2Fnews%2Fthe-maestro-and-the-music-bof.html&data=3D0= 5%7C01%7Cryan%40rkhtech.org%7Cc8ebae9f0ecd4b368dcb08dbcd319880%7C81c24bb4f9= ec4739ba4d25c42594d996%7C0%7C0%7C638329385118876648%7CUnknown%7CTWFpbGZsb3d= 8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%= 7C%7C%7C&sdata=3DROLgtoeiBgfAG40UZqS8Zd8vMK%2B0HQB7RV%2FhQRvIcFM%3D&= ;reserved=3D0
> > Dave T=C3=A4ht CSO, LibreQos



--
Oct 30: https://netdevconf.info/= 0x17/news/the-maestro-and-the-music-bof.html
Dave T=C3=A4ht CSO, LibreQos
_______________________________________________
LibreQoS mailing list
LibreQo= S@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/libreqos
--000000000000f54e8a0607b9b11d--