From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 52E5F3B2A4 for ; Wed, 26 Oct 2022 15:39:44 -0400 (EDT) Received: by mail-ed1-x52b.google.com with SMTP id 21so9361166edv.3 for ; Wed, 26 Oct 2022 12:39:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jackrabbitwireless.com; s=google; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=gCmIn1HJ/khr7G4+BKPLqTlBrc0cb0UvrqcLIkEIzaM=; b=H4eQrXVsJe1F0Ez1p8oHPxG0HvWmgS28MwIE4lMkhAtkzzo+o/DuUYeIk+MSqOI42z KQIvGWXMT7jP6wPt7EyIzdwJzLg1nGdqs2JZa1zaSC4Aq0x4XDEoXx8FHIZxQ+1L/KFD GmIHFsVqNezi3Uo08Bd5jKgsmv+xGxwhgPlpQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=gCmIn1HJ/khr7G4+BKPLqTlBrc0cb0UvrqcLIkEIzaM=; b=neZfx0meygnc9y55LeNH1jhRmN7NXNj7cqc0CjxGTVG6cm0N1v2RXaR2DEzYMOP59g waO0ULrmScqYtbkmJyXlus+5iqBjlf/80QgDqvHxixVCUt5ogxYVCfmDbBCFoAPlCLkv 6MlIAwIiOPO80pFaFPvNgC6NRyxfy4H/nGAtk8X6c6Vgyn7AY3Vik5VlYE/gLZuDMsnO 3iCXyYWsv4mGGFJIiV8/IVd+N2iSuhIYqBTBVH1Xc0KPsGVrjrIBrJKwH1WXulyum6ac 7j75eI61UhVXJ8o5ni1yIcyPk6VoQI7i3R9c0oWMxMeOSGNsmV7O3iOCz77zF2RSyclh iesg== X-Gm-Message-State: ACrzQf0hJDL5uAqGAwUnZw0LATRKtQk1zkRVeHJ4arrvoxDIVbnWZ3z2 QAJgSRKx5P3gJHj+VJabUmHvvIFV6JyZSq5onLKl3g== X-Google-Smtp-Source: AMsMyM6LNkU4m2sNyfSqv1jUAZs0BguZsDx6RlHwoKsygIeCoiDA0t7+cW0xCnYKj/VHYRvc/uupA9WdgZpBe6yFnSQ= X-Received: by 2002:aa7:ccd7:0:b0:461:c6e9:8b0 with SMTP id y23-20020aa7ccd7000000b00461c6e908b0mr16209476edt.287.1666813183214; Wed, 26 Oct 2022 12:39:43 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: =?UTF-8?Q?Robert_Chac=C3=B3n?= Date: Wed, 26 Oct 2022 13:39:32 -0600 Message-ID: To: Herbert Wolverson Cc: libreqos@lists.bufferbloat.net Content-Type: multipart/alternative; boundary="000000000000b1621505ebf52ffa" Subject: Re: [LibreQoS] how are you doing on ipv4 address supply? X-BeenThere: libreqos@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Many ISPs need the kinds of quality shaping cake can do List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Oct 2022 19:39:44 -0000 --000000000000b1621505ebf52ffa Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I think it could definitely be helpful. Cambium's Bequant-based QoE handles MPLS parsing. Like you said MPLS tends to be a holdup for a lot of people looking at Preseem. And it would make shaping at sites possible. You said it looks reasonably straightforward to implement within cpumap-pping/xdp-cpumap-tc? If you'd need more time to dedicate to it to explore that more, let us know what a fair feature bounty would be for it. =3D) I can see it helping a lot of operators. On Wed, Oct 26, 2022 at 1:33 PM Herbert Wolverson via LibreQoS < libreqos@lists.bufferbloat.net> wrote: > The main reason I was asking about MPLS is that it looks pretty easy to > extract the data, and I've seen a bunch of people on the WISP Talk FB gro= up > complaining about having to de-encapsulate before they pass things throug= h > Preseem. Where I could see that being most useful is if you have a shaper > at a tower (rather than just the one), and want to shape your backhaul > directly. I'll keep it on the "things to play with" list. :-) > > We ran MPLS/VPLS for a while; we were pretty surprised when our network > sped up a lot when it was removed! (Routers got really good at routing, a= nd > it's often faster than parsing a label these days). It also made it easie= r > for the other techs to diagnose faults, MPLS never seemed to "click" for > them. > > On Wed, Oct 26, 2022 at 2:15 PM Robert Chac=C3=B3n < > robert.chacon@jackrabbitwireless.com> wrote: > >> We use MPLS (VPLS tunnels back to core) with L2 isolation and DHCP >> snooping everywhere. >> It decapsulates at our core so we don't need MPLS parsing for LibreQoS i= n >> our case. >> It works well and minimizes the number of RFC6598 CG-NAT clients behind >> public addresses. >> I'd prefer simple routing, but with IPv4 exhaustion - that would lead to >> a lot of unused RFC6598 addresses (and hence high CG-NAT ratios). >> Many clients on one CG-NAT address leads to a lot of problems with >> services like Amazon Prime. >> I'm hopeful we can switch to VXLAN as Dan suggested. It could be a great >> alternative to VPLS/MPLS. >> >> On Wed, Oct 26, 2022 at 10:49 AM dan via LibreQoS < >> libreqos@lists.bufferbloat.net> wrote: >> >>> MPLS is great for those that have dedicated engineering staff or hire a >>> consultant to help out. It's a real PITA when the business is primaril= y >>> techs and they have no idea how to solve an issue. I've been pushing = for >>> vxlan/evpn or srv6 for a while as a way to simplify the model. 'wisp' >>> focused gear just isn't there yet though. high hopes for mikrotik's >>> support for vxlan to get hardware acceleration. >>> >>> On Wed, Oct 26, 2022 at 10:05 AM Herbert Wolverson via LibreQoS < >>> libreqos@lists.bufferbloat.net> wrote: >>> >>>> Actually had a conversation with our primary upstream provider >>>> yesterday about IPv6 (and why >>>> they won't provide it to us). One of their lead engineers, once plied >>>> with beer, said that every >>>> time they've deployed it they get 99% of it working well and 1% of >>>> things mysteriously >>>> stop working, or go wonky. That was similar to our experience a few >>>> years ago. It mostly >>>> helped a lot, but chasing down the "hey, this advertises a v6 address >>>> and doesn't actually >>>> support it" issues drove us crazy. >>>> >>>> Right now, we don't have enough IPv4 addresses, but that's being >>>> rectified. We mostly >>>> do CGNAT and 10.64 addresses in the meantime, with public IPs assigned >>>> where they >>>> are needed (mostly through a tunnel setup to avoid subnetting waste). >>>> Tunnels are a pain, >>>> but they work (once you chase down all of the MTU issues). >>>> >>>> Which reminds me, I have "can we support MPLS?" on my crazy notes list= . >>>> I know that >>>> Preseem and similar don't try, but we're already reading deeply enough >>>> into the ethernet >>>> header that saying "this is an MPLS label, advance 4 bytes", "this is = a >>>> VPLS label, advance >>>> X (I forget) bytes" looks do-able. [Note, I personally don't enjoy >>>> MPLS. It's handy when >>>> you want to pretend to have a flat network on top of a large routed >>>> network - and some >>>> WISP consultants absolutely swear by it - but my experience is that yo= u >>>> are adding >>>> complexity for the sake of it. Routing works remarkably well.] >>>> >>>> On Tue, Oct 25, 2022 at 10:30 PM Dave Taht via LibreQoS < >>>> libreqos@lists.bufferbloat.net> wrote: >>>> >>>>> in my continued rip-van-winkle, living in the third world (california= ) >>>>> way, I am curious as to how y'all are managing your >>>>> ipv4 address supply and if you are deploying ipv6 to any extent? >>>>> >>>>> In all this discussion of multi-gbit fiber, my own direct experience >>>>> is that AT&T's fiber rollout had very flaky ipv6, and more and more >>>>> services (like starlink) are appearing behind cgnats, which have thei= r >>>>> own capex and opex costs. >>>>> >>>>> I see a lot of rfc1918 being used as the operational overlay >>>>> elsewhere, tons of tunnels, also. >>>>> >>>>> -- >>>>> This song goes out to all the folk that thought Stadia would work: >>>>> >>>>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-69813= 66665607352320-FXtz >>>>> Dave T=C3=A4ht CEO, TekLibre, LLC >>>>> _______________________________________________ >>>>> LibreQoS mailing list >>>>> LibreQoS@lists.bufferbloat.net >>>>> https://lists.bufferbloat.net/listinfo/libreqos >>>>> >>>> _______________________________________________ >>>> LibreQoS mailing list >>>> LibreQoS@lists.bufferbloat.net >>>> https://lists.bufferbloat.net/listinfo/libreqos >>>> >>> _______________________________________________ >>> LibreQoS mailing list >>> LibreQoS@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/libreqos >>> >> >> >> -- >> Robert Chac=C3=B3n >> CEO | JackRabbit Wireless LLC >> > _______________________________________________ > LibreQoS mailing list > LibreQoS@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/libreqos > --=20 Robert Chac=C3=B3n CEO | JackRabbit Wireless LLC --000000000000b1621505ebf52ffa Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I think it could definitely be helpful. Cambium's= Bequant-based QoE handles MPLS parsing.
Like you said MPLS tends= to be a holdup for a lot of people looking at Preseem.
And it wo= uld make shaping at sites possible.
You said it looks reasona= bly straightforward to implement within cpumap-pping/xdp-cpumap-tc?
If you'd need more time to dedicate to it to explore that more, let = us know what a fair feature bounty would be for it. =3D) I can see it helpi= ng a lot of operators.

On Wed, Oct 26, 2022 at 1:33 PM Herbert Wol= verson via LibreQoS <l= ibreqos@lists.bufferbloat.net> wrote:
The main reason I was as= king about MPLS is that it looks pretty easy to extract the data, and I'= ;ve seen a bunch of people on the WISP Talk FB group complaining about havi= ng to de-encapsulate before they pass things through Preseem. Where I could= see that being most useful is if you have a shaper at a tower (rather than= just the one), and want to shape your backhaul directly. I'll keep it = on the "things to play with" list. :-)

W= e ran MPLS/VPLS for a while; we were pretty surprised when our network sped= up a lot when it was removed! (Routers got really good at routing, and it&= #39;s often faster than parsing a label these days). It also made it easier= for the other techs to diagnose faults, MPLS never seemed to "click&q= uot; for them.

On Wed, Oct 26, 2022 at 2:15 PM Robert Chac=C3=B3n <= r= obert.chacon@jackrabbitwireless.com> wrote:
We use MPLS (VPLS = tunnels back to core) with L2 isolation and DHCP snooping everywhere.
=
It decapsulates at our core so we don't need MPLS parsing for Libr= eQoS in our case.
It works well and minimizes the number of R= FC6598 CG-NAT clients behind public addresses.
I'd prefer sim= ple routing, but with IPv4 exhaustion - that would lead to a lot of unused = RFC6598 addresses (and hence high CG-NAT ratios).
Many clients on= one CG-NAT address leads to a lot of problems with services like Amazon Pr= ime.
I'm hopeful we can switch to=C2=A0VXLAN as Dan suggested= . It could be a great alternative to VPLS/MPLS.

On Wed, Oct 26, 20= 22 at 10:49 AM dan via LibreQoS <libreqos@lists.bufferbloat.net> wrote:<= br>
MPLS is great for those that have dedicated engineering staff or hire a co= nsultant to help out.=C2=A0 It's a real PITA when the business is prima= rily techs and they have no idea how to solve an issue.=C2=A0 =C2=A0I'v= e been pushing for vxlan/evpn or srv6 for a while as a way to simplify the = model.=C2=A0 'wisp' focused gear just isn't there yet though.= =C2=A0 high hopes for mikrotik's support for vxlan to get hardware acce= leration.=C2=A0 =C2=A0

On Wed, Oct 26, 2022 at 10:05 AM Herbert Wolverson vi= a LibreQoS <libreqos@lists.bufferbloat.net> wrote:
Actually had a = conversation with our primary upstream provider yesterday about IPv6 (and w= hy
they won't provide it to us). One of their lead engineers,= once plied with beer, said that every
time they've deployed = it they get 99% of it working well and 1% of things mysteriously
= stop working, or go wonky. That was similar to our experience a few years a= go. It mostly
helped a lot, but chasing down the "hey, this = advertises a v6 address and doesn't actually
support it"= issues drove us crazy.

Right now, we don't ha= ve enough IPv4 addresses, but that's being rectified. We mostly
do CGNAT and 10.64 addresses in the meantime, with public IPs assigned w= here they
are needed (mostly through a tunnel setup to avoid subn= etting waste). Tunnels are a pain,
but they work (once you chase = down all of the MTU issues).

Which reminds me, I h= ave "can we support MPLS?" on my crazy notes list. I know that
Preseem and similar don't try, but we're already reading de= eply enough into the ethernet
header that saying "this is an= MPLS label, advance 4 bytes", "this is a VPLS label, advance
X (I forget) bytes" looks do-able. [Note, I personally don'= t enjoy MPLS. It's handy when
you want to pretend to have a f= lat network on top of a large routed network - and some
WISP cons= ultants absolutely swear by it - but my experience is that you are adding
complexity for the sake of it. Routing works remarkably well.]
=

On Tue, Oct 25, 2022 at 10:30 PM Dave Taht via LibreQoS <libreqos@lists= .bufferbloat.net> wrote:
in my continued rip-van-winkle, living in the third world (= california)
way, I am curious as to how y'all are managing your
ipv4 address supply and if you are deploying ipv6 to any extent?

In all this discussion of multi-gbit fiber, my own direct experience
is that AT&T's fiber rollout had very flaky ipv6, and more and more=
services (like starlink) are appearing behind cgnats, which have their
own capex and opex costs.

I see a lot of rfc1918 being used as the operational overlay
elsewhere, tons of tunnels, also.

--
This song goes out to all the folk that thought Stadia would work:
https://www.= linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXt= z
Dave T=C3=A4ht CEO, TekLibre, LLC
_______________________________________________
LibreQoS mailing list
LibreQo= S@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/libreqos
_______________________________________________
LibreQoS mailing list
LibreQo= S@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/libreqos
_______________________________________________
LibreQoS mailing list
LibreQo= S@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/libreqos


--
Robert Chac=C3=B3n
CEO | JackRabbit Wireless LLC
_______________________________________________
LibreQoS mailing list
LibreQo= S@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/libreqos


--
Robert Chac=C3=B3n
CEO | JackRabbit Wireless LLC
--000000000000b1621505ebf52ffa--