[Starlink] Starlink tidbits from NANOG

[Starlink] Starlink tidbits from NANOG

[Darrell Budic]

I was at NANOG in Minneapolis, and got a chance to ask a couple question of a Starlink Network Engineer who’s attending. I was already talking to him about Starlink’s network efforts (see below) but it was nice to meet in person. Don’t quote me on any of this, but here’s a few tidbits this list may appreciate:

- Starlink is expanding their own network operations, and is connecting to more IXPs. They were already on SIX in Seattle, have connected to DECIX NY, and are in the process of connecting to ChIX in Chicago. As I run ChIX, I had a good excuse to talk to them about other things. :) IXPs and their own networks are in the works for Europe and other areas as well.
- They have been obtaining more v4 addresses, but I don’t know if they have enough to not do CGNAT. I don't think they do yet, but it seems like it may be a long term target.
- v6 is deliberately not fully functional, but they know some of use are using it and it will eventually be fully activated. May be waiting on the regional connectivity, so will be intersting to see if changes for some areas and not others as they roll it out.
- They hate Google's outsourced NOC as much as the rest of us
- New ground stations with more capacity are coming (and will be upgrades). They are using waves back to regional DCs now, but will be moving to dark fiber over the next year or two
- the new satellites have more than 2 lasers, and there is enough capacity on them to do routing. no details on how or what protocols, alas
- new birds also have 2-3x more ku bandwidth than first gen
- new dishes are in the works, v4 coming with lower power use, more capacity, not round any more
- larger dishes coming for commercial apps
- as we know, they aren’t doing any AQM yet, but it sounds like it may be in the works and we may see it in new code in 4-6 months. Not my guys department, so no more details.
- it’s encrypted up and down. I didn’t know that yet, but I may have just missed it.

  -Darrell

Re: [Starlink] Starlink tidbits from NANOG

[Dave Taht]

Good to hear there was some horizontal communication! I was enthused
to hear they are opening up india, in part because
our simulations team is at nitk. I wish there was a way to get
starlink in touch with mohit, cc'd. They've been simulating transport
various congestion controls vs AQM and FQ technologies at the rtts
common to starlink and viasat of late, and could use better numbers to
plug in at the very least.

On Thu, Nov 4, 2021 at 8:26 AM Darrell Budic <budic@onholyground.com> wrote:
>
> I was at NANOG in Minneapolis, and got a chance to ask a couple question of a Starlink Network Engineer who’s attending. I was already talking to him about Starlink’s network efforts (see below) but it was nice to meet in person. Don’t quote me on any of this, but here’s a few tidbits this list may appreciate:
>
> - Starlink is expanding their own network operations, and is connecting to more IXPs. They were already on SIX in Seattle, have connected to DECIX NY, and are in the process of connecting to ChIX in Chicago. As I run ChIX, I had a good excuse to talk to them about other things. :) IXPs and their own networks are in the works for Europe and other areas as well.
> - They have been obtaining more v4 addresses, but I don’t know if they have enough to not do CGNAT. I don't think they do yet, but it seems like it may be a long term target.
> - v6 is deliberately not fully functional, but they know some of use are using it and it will eventually be fully activated. May be waiting on the regional connectivity, so will be intersting to see if changes for some areas and not others as they roll it out.
> - They hate Google's outsourced NOC as much as the rest of us
> - New ground stations with more capacity are coming (and will be upgrades). They are using waves back to regional DCs now, but will be moving to dark fiber over the next year or two
> - the new satellites have more than 2 lasers, and there is enough capacity on them to do routing. no details on how or what protocols, alas
> - new birds also have 2-3x more ku bandwidth than first gen
> - new dishes are in the works, v4 coming with lower power use, more capacity, not round any more
> - larger dishes coming for commercial apps
> - as we know, they aren’t doing any AQM yet, but it sounds like it may be in the works and we may see it in new code in 4-6 months. Not my guys department, so no more details.
> - it’s encrypted up and down. I didn’t know that yet, but I may have just missed it.
>
>   -Darrell
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink



-- 
I tried to build a better future, a few times:
https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org

Dave Täht CEO, TekLibre, LLC

Re: [Starlink] Starlink tidbits from NANOG

[Dave Taht]

I have an ipv4/23 lying around since the 90s. I don't want to sell it,
but my co-owner and I would really like a dishy and a static IPv4/IPv6
address... and service for life... and whatever else we could
negotiate. :)

On Thu, Nov 4, 2021 at 8:26 AM Darrell Budic <budic@onholyground.com> wrote:
>
> I was at NANOG in Minneapolis, and got a chance to ask a couple question of a Starlink Network Engineer who’s attending. I was already talking to him about Starlink’s network efforts (see below) but it was nice to meet in person. Don’t quote me on any of this, but here’s a few tidbits this list may appreciate:
>
> - Starlink is expanding their own network operations, and is connecting to more IXPs. They were already on SIX in Seattle, have connected to DECIX NY, and are in the process of connecting to ChIX in Chicago. As I run ChIX, I had a good excuse to talk to them about other things. :) IXPs and their own networks are in the works for Europe and other areas as well.

It was my dream they would realize the huge number of small isps that
would like to peer with starlink and leverage those. I'd like to
restore
the original, routable internet back out to the edges where it
belongs. But that's me. The whole "Terminal" language bothers me, when
there's
so much more that could be created locally along the edge if only
stuff could move there. Email back to the edge, videoconfernening
services across villiages....

> - They have been obtaining more v4 addresses, but I don’t know if they have enough to not do CGNAT. I don't think they do yet, but it seems like it may be a long term target.

I keep hoping for IETF action politically on opening up 0/8 and 240/4.
There are of course a vast swath of military/8 assignments... I've
lost track did the post office ever sell any off? There's space left
in 44/8 too if they are willing to talk to ardc but that would have to
go to an open bid.

> - v6 is deliberately not fully functional, but they know some of use are using it and it will eventually be fully activated. May be waiting on the regional connectivity, so will be intersting to see if changes for some areas and not others as they roll it out.

Deliberate... well, if you ship 10 year old openwrt software to users
(we'd made a big push for ipv6 there before ipv6 launch day in 2013),
and
don't keep up...  I guess you could call that deliberate. I'm pretty
happy with openwrt 20.2.1. IMHO: ipv6 really requires a modern kernel
and tools, not less than 4 years old, to deploy well. Maybe there's a
worthwhile SDR stack, I don't know...

Lately it seems like ipv6 things have been moving backwards with flow
offloads in certain chipsets being very limited or very buggy with
ipv6. Offloads in general have been cropping up as an increasing
problem - lot's of enthusiasm for putting in RED into nvidia's cards
apparently.

> - They hate Google's outsourced NOC as much as the rest of us

Do say more. :) I'm told google at least have a very nice set of tools
for looking at the characteristics of interchange traffic.

> - New ground stations with more capacity are coming (and will be upgrades).

Would so love universities to get in on some of those. I remember the IMP.

> They are using waves back to regional DCs now, but will be moving to dark fiber over the next year or two

"waves"?

> - the new satellites have more than 2 lasers, and there is enough capacity on them to do routing. no details on how or what protocols, alas

Still on a custom mac, though, I suppose. Thx for all the teasers,
this is the most info I've seen in months. Way better than hitting
reload on reddit. :P

> - new birds also have 2-3x more ku bandwidth than first gen

up and down?

My take on the up problem was that it was regulatory. ?

(and they really need ack-filtering)

> - new dishes are in the works, v4 coming with lower power use, more capacity, not round any more
> - larger dishes coming for commercial apps

> - as we know, they aren’t doing any AQM yet, but it sounds like it may be in the works and we may see it in new code in 4-6 months. Not my guys department, so no more details.

fq is a better start.

Just someone telling me under pain of death, "dave, you can't talk for
X months, but we're going to do cake/fq-codel/pie/something" would be
comforting. There's a whole internet elsewhere left to fix, starlink
getting it right and a little publicity around it would do wonders...
and certainly wifi is highest on my list. As it is, I got annoyed
enough last week to try and get the autorate sensing code to work well
on starlink. There's a prototype now that seems to be working well on
lte, see here: https://forum.openwrt.org/t/cakes-autorate-ingress-testing-needed/108848/186

Testers wanted.

 Fixing fixed wireless has been a pain point far, far, far greater
than the disappointment I felt at starlink so totally missing the
bufferbloat problem initially. It will take a decade to sort out 5g,
4-6 months more for starlink oh! yes! yes! yes!

More news on that as it happens.

> - it’s encrypted up and down. I didn’t know that yet, but I may have just missed it.

I did. But it's really hard to trust that black box and the world has
otherwise shifted to e2e encryption.

>
>   -Darrell
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink



-- 
I tried to build a better future, a few times:
https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org

Dave Täht CEO, TekLibre, LLC

Re: [Starlink] Starlink tidbits from NANOG

[Dave Taht]

On Thu, Nov 4, 2021 at 9:46 AM Dave Taht <dave.taht@gmail.com> wrote:
>
> I have an ipv4/23 lying around since the 90s. I don't want to sell it,
> but my co-owner and I would really like a dishy and a static IPv4/IPv6
> address... and service for life... and whatever else we could
> negotiate. :)

Perhaps 2 zeroth addresses in exchange, also. Getting 0 back is order
millions more real ipv4s for the world. code's in linux and bsd now.

https://tools.ietf.org/id/draft-schoen-intarea-lowest-address-00.txt

(there's a preso nov 9th in intarea on this)

> On Thu, Nov 4, 2021 at 8:26 AM Darrell Budic <budic@onholyground.com> wrote:
> >
> > I was at NANOG in Minneapolis, and got a chance to ask a couple question of a Starlink Network Engineer who’s attending. I was already talking to him about Starlink’s network efforts (see below) but it was nice to meet in person. Don’t quote me on any of this, but here’s a few tidbits this list may appreciate:
> >
> > - Starlink is expanding their own network operations, and is connecting to more IXPs. They were already on SIX in Seattle, have connected to DECIX NY, and are in the process of connecting to ChIX in Chicago. As I run ChIX, I had a good excuse to talk to them about other things. :) IXPs and their own networks are in the works for Europe and other areas as well.
>
> It was my dream they would realize the huge number of small isps that
> would like to peer with starlink and leverage those. I'd like to
> restore
> the original, routable internet back out to the edges where it
> belongs. But that's me. The whole "Terminal" language bothers me, when
> there's
> so much more that could be created locally along the edge if only
> stuff could move there. Email back to the edge, videoconfernening
> services across villiages....
>
> > - They have been obtaining more v4 addresses, but I don’t know if they have enough to not do CGNAT. I don't think they do yet, but it seems like it may be a long term target.
>
> I keep hoping for IETF action politically on opening up 0/8 and 240/4.
> There are of course a vast swath of military/8 assignments... I've
> lost track did the post office ever sell any off? There's space left
> in 44/8 too if they are willing to talk to ardc but that would have to
> go to an open bid.
>
> > - v6 is deliberately not fully functional, but they know some of use are using it and it will eventually be fully activated. May be waiting on the regional connectivity, so will be intersting to see if changes for some areas and not others as they roll it out.
>
> Deliberate... well, if you ship 10 year old openwrt software to users
> (we'd made a big push for ipv6 there before ipv6 launch day in 2013),
> and
> don't keep up...  I guess you could call that deliberate. I'm pretty
> happy with openwrt 20.2.1. IMHO: ipv6 really requires a modern kernel
> and tools, not less than 4 years old, to deploy well. Maybe there's a
> worthwhile SDR stack, I don't know...
>
> Lately it seems like ipv6 things have been moving backwards with flow
> offloads in certain chipsets being very limited or very buggy with
> ipv6. Offloads in general have been cropping up as an increasing
> problem - lot's of enthusiasm for putting in RED into nvidia's cards
> apparently.
>
> > - They hate Google's outsourced NOC as much as the rest of us
>
> Do say more. :) I'm told google at least have a very nice set of tools
> for looking at the characteristics of interchange traffic.
>
> > - New ground stations with more capacity are coming (and will be upgrades).
>
> Would so love universities to get in on some of those. I remember the IMP.
>
> > They are using waves back to regional DCs now, but will be moving to dark fiber over the next year or two
>
> "waves"?
>
> > - the new satellites have more than 2 lasers, and there is enough capacity on them to do routing. no details on how or what protocols, alas
>
> Still on a custom mac, though, I suppose. Thx for all the teasers,
> this is the most info I've seen in months. Way better than hitting
> reload on reddit. :P
>
> > - new birds also have 2-3x more ku bandwidth than first gen
>
> up and down?
>
> My take on the up problem was that it was regulatory. ?
>
> (and they really need ack-filtering)
>
> > - new dishes are in the works, v4 coming with lower power use, more capacity, not round any more
> > - larger dishes coming for commercial apps
>
> > - as we know, they aren’t doing any AQM yet, but it sounds like it may be in the works and we may see it in new code in 4-6 months. Not my guys department, so no more details.
>
> fq is a better start.
>
> Just someone telling me under pain of death, "dave, you can't talk for
> X months, but we're going to do cake/fq-codel/pie/something" would be
> comforting. There's a whole internet elsewhere left to fix, starlink
> getting it right and a little publicity around it would do wonders...
> and certainly wifi is highest on my list. As it is, I got annoyed
> enough last week to try and get the autorate sensing code to work well
> on starlink. There's a prototype now that seems to be working well on
> lte, see here: https://forum.openwrt.org/t/cakes-autorate-ingress-testing-needed/108848/186
>
> Testers wanted.
>
>  Fixing fixed wireless has been a pain point far, far, far greater
> than the disappointment I felt at starlink so totally missing the
> bufferbloat problem initially. It will take a decade to sort out 5g,
> 4-6 months more for starlink oh! yes! yes! yes!
>
> More news on that as it happens.
>
> > - it’s encrypted up and down. I didn’t know that yet, but I may have just missed it.
>
> I did. But it's really hard to trust that black box and the world has
> otherwise shifted to e2e encryption.
>
> >
> >   -Darrell
> > _______________________________________________
> > Starlink mailing list
> > Starlink@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/starlink
>
>
>
> --
> I tried to build a better future, a few times:
> https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
>
> Dave Täht CEO, TekLibre, LLC



-- 
I tried to build a better future, a few times:
https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org

Dave Täht CEO, TekLibre, LLC

Re: [Starlink] Starlink tidbits from NANOG

[Michael Richardson]

[-- Attachment #1: Type: text/plain, Size: 835 bytes --]


Thanks for bringing us this view..

Darrell Budic <budic@onholyground.com> wrote:
    > - v6 is deliberately
    > not fully functional, but they know some of use are using it and it
    > will eventually be fully activated. May be waiting on the regional
    > connectivity, so will be intersting to see if changes for some areas
    > and not others as they roll it out.

Even if they used IPv6 for the "last mile" and ran NAT64, that would still be
a major improvement over trying to run dual stack.   That's what the smarter
mobile operators are already doing. (jio, some US carriers...)

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        | network architect  [
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [Starlink] Starlink tidbits from NANOG

[Inemesit Affia]

[-- Attachment #1: Type: text/plain, Size: 1353 bytes --]

Waves are a unit of optical transport. Similar to PON, you can get several
wavelengths on the same fiber and sell them to different users, Split them
at both ends. Thats the whole 'point of DWDM. It's not just faster speeds
or less cable

On Thu, Nov 4, 2021 at 7:16 PM Michael Richardson <mcr@sandelman.ca> wrote:

>
> Thanks for bringing us this view..
>
> Darrell Budic <budic@onholyground.com> wrote:
>     > - v6 is deliberately
>     > not fully functional, but they know some of use are using it and it
>     > will eventually be fully activated. May be waiting on the regional
>     > connectivity, so will be intersting to see if changes for some areas
>     > and not others as they roll it out.
>
> Even if they used IPv6 for the "last mile" and ran NAT64, that would still
> be
> a major improvement over trying to run dual stack.   That's what the
> smarter
> mobile operators are already doing. (jio, some US carriers...)
>
> --
> ]               Never tell me the odds!                 | ipv6 mesh
> networks [
> ]   Michael Richardson, Sandelman Software Works        | network
> architect  [
> ]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on
> rails    [
>
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
>

[-- Attachment #2: Type: text/html, Size: 2133 bytes --]

Re: [Starlink] Starlink tidbits from NANOG

[Dave Taht]

On Thu, Nov 4, 2021 at 11:16 AM Michael Richardson <mcr@sandelman.ca> wrote:
>
>
> Thanks for bringing us this view..
>
> Darrell Budic <budic@onholyground.com> wrote:
>     > - v6 is deliberately
>     > not fully functional, but they know some of use are using it and it
>     > will eventually be fully activated. May be waiting on the regional
>     > connectivity, so will be intersting to see if changes for some areas
>     > and not others as they roll it out.
>
> Even if they used IPv6 for the "last mile" and ran NAT64, that would still be
> a major improvement over trying to run dual stack.   That's what the smarter
> mobile operators are already doing. (jio, some US carriers...)

have these operators have solved the mobility problem ipv6 has always had?

if not then I tend towards a custom l2 that supports anything on top
of it, which might
even include cool things like icn.
> --
> ]               Never tell me the odds!                 | ipv6 mesh networks [
> ]   Michael Richardson, Sandelman Software Works        | network architect  [
> ]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [
>
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink



-- 
I tried to build a better future, a few times:
https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org

Dave Täht CEO, TekLibre, LLC

Re: [Starlink] Starlink tidbits from NANOG

[Michael Richardson]

<#secure method=pgpmime mode=sign>

    >> Even if they used IPv6 for the "last mile" and ran NAT64, that would still be
    >> a major improvement over trying to run dual stack.   That's what the smarter
    >> mobile operators are already doing. (jio, some US carriers...)

    > have these operators have solved the mobility problem ipv6 has always
    > had?

If you are speaking about the issue that Enterprises have that they'd like to
continue to run their internal network when changing ISPs (using NAT44 at the
edge), then the answer is that mostly, it's a problem caused by IPv4-think.
(The fact that public /48s are not yet free is also an issue)

I don't expect that Starlink will have this problem, and in the home, the
ULAs generated by 7204 compliant CPE devices solves most issues.

(We still have a source address selection problem: but that problem won't go
away until a few dozen core Google/Microsoft/Apple developers are forced to
live in an IPv6-only home network.)

And JIO (India) and other LTE operators' deal mostly with handsets, which
don't have permanently up internal networks (alas).

    > if not then I tend towards a custom l2 that supports anything on top
    > of it, which might
    > even include cool things like icn.

More L2 tricks will not save the day. It just makes the network invisible and
thus undebuggable.   It's time to acknowledge that ethernet is not a fat Yellow COAX.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [

Re: [Starlink] Starlink tidbits from NANOG

[Ulrich Speidel]

Thanks for that Darrell - that's really interesting! A few comments on 
that front:

On 5/11/2021 4:26 am, Darrell Budic wrote:
> I was at NANOG in Minneapolis, and got a chance to ask a couple 
> question of a Starlink Network Engineer who’s attending. I was already 
> talking to him about Starlink’s network efforts (see below) but it was 
> nice to meet in person. Don’t quote me on any of this, but here’s a 
> few tidbits this list may appreciate:
>
> - Starlink is expanding their own network operations, and is 
> connecting to more IXPs. They were already on SIX in Seattle, have 
> connected to DECIX NY, and are in the process of connecting to ChIX in 
> Chicago. As I run ChIX, I had a good excuse to talk to them about 
> other things. :) IXPs and their own networks are in the works for 
> Europe and other areas as well.
Makes sense.
> - They have been obtaining more v4 addresses, but I don’t know if they 
> have enough to not do CGNAT. I don't think they do yet, but it seems 
> like it may be a long term target.
> - v6 is deliberately not fully functional, but they know some of use 
> are using it and it will eventually be fully activated. May be waiting 
> on the regional connectivity, so will be intersting to see if changes 
> for some areas and not others as they roll it out.

So I guess we need to distinguish between:

- IPv4 addresses for any CGNAT they might run
- IPv4 addresses as static addresses for (some of?) their customers
- IPv6 addresses as customer addresses
- IPv6 addresses to support geographic routing as discussed in earlier 
posts (subnet maps to cell / satellite)

There are quite a number of feasible configurations in this. E.g., they 
could be running a CGNAT setup with a v4 pool on the Internet side, use 
v6 to tunnel route from there to the satellite the end customer connects 
to, and then map that customer back to a (private) IPv4 address in a NAT 
on the satellite. One aspect that hasn't really been mentioned much here 
is that of PDU size on the link between end customer and satellite. 
Keeping Dishy and its successors small and cheap creates an incentive to 
operate at marginal SNR, and this favours smaller PDUs over larger ones 
as the probability of PDU checksum errors increases with PDU size. But 
having lots of small PDUs means having lots of headers, and as IPv4 
addresses are leaner than IPv6 ones, this saves bandwidth here. Probably 
not a biggie though.
> - New ground stations with more capacity are coming (and will be 
> upgrades). 
Any word on where? At the moment, most of the world can see Starlink 
satellites, but most Starlink satellites can't see a ground station.
> They are using waves back to regional DCs now, but will be moving to 
> dark fiber over the next year or two
If that means "radio" waves, then this goes a long way to explaining why 
there's already limited capacity even near the US-Canada border.
> - the new satellites have more than 2 lasers, and there is enough 
> capacity on them to do routing. no details on how or what protocols, alas
Any word on when we can expect to see routing in action?
> - new birds also have 2-3x more ku bandwidth than first gen
Hm. Sounds cool, but with 3 billion or so underserved on the planet & 
typical annual growth rates, that's still just a drop in the bucket.
> - new dishes are in the works, v4 coming with lower power use, more 
> capacity, not round any more
Trayee? Squary? Just joking ;-)
> - larger dishes coming for commercial apps
That's good news, as this will allow Starlink to be used in places where 
direct-to-site crashes into regulatory hurdles. If we can get the big 
CDN providers to come up with small (virtual?) appliances that can be 
put at the remote end of such links by local ISPs, then that'll also 
help to preserve space segment capacity.

-- 
****************************************************************
Dr. Ulrich Speidel

School of Computer Science

Room 303S.594 (City Campus)
Ph: (+64-9)-373-7599 ext. 85282

The University of Auckland
ulrich@cs.auckland.ac.nz
http://www.cs.auckland.ac.nz/~ulrich/
****************************************************************

Re: [Starlink] Starlink tidbits from NANOG

[Nathan Owens]

[-- Attachment #1: Type: text/plain, Size: 4954 bytes --]

> They are using waves back to regional DCs now, but will be moving to
> dark fiber over the next year or two
If that means "radio" waves, then this goes a long way to explaining why
there's already limited capacity even near the US-Canada border.

Waves in this case generally refers to 10G/100G leased optical circuit
capacity.

On Thu, Nov 4, 2021 at 5:35 PM Ulrich Speidel <ulrich@cs.auckland.ac.nz>
wrote:

> Thanks for that Darrell - that's really interesting! A few comments on
> that front:
>
> On 5/11/2021 4:26 am, Darrell Budic wrote:
> > I was at NANOG in Minneapolis, and got a chance to ask a couple
> > question of a Starlink Network Engineer who’s attending. I was already
> > talking to him about Starlink’s network efforts (see below) but it was
> > nice to meet in person. Don’t quote me on any of this, but here’s a
> > few tidbits this list may appreciate:
> >
> > - Starlink is expanding their own network operations, and is
> > connecting to more IXPs. They were already on SIX in Seattle, have
> > connected to DECIX NY, and are in the process of connecting to ChIX in
> > Chicago. As I run ChIX, I had a good excuse to talk to them about
> > other things. :) IXPs and their own networks are in the works for
> > Europe and other areas as well.
> Makes sense.
> > - They have been obtaining more v4 addresses, but I don’t know if they
> > have enough to not do CGNAT. I don't think they do yet, but it seems
> > like it may be a long term target.
> > - v6 is deliberately not fully functional, but they know some of use
> > are using it and it will eventually be fully activated. May be waiting
> > on the regional connectivity, so will be intersting to see if changes
> > for some areas and not others as they roll it out.
>
> So I guess we need to distinguish between:
>
> - IPv4 addresses for any CGNAT they might run
> - IPv4 addresses as static addresses for (some of?) their customers
> - IPv6 addresses as customer addresses
> - IPv6 addresses to support geographic routing as discussed in earlier
> posts (subnet maps to cell / satellite)
>
> There are quite a number of feasible configurations in this. E.g., they
> could be running a CGNAT setup with a v4 pool on the Internet side, use
> v6 to tunnel route from there to the satellite the end customer connects
> to, and then map that customer back to a (private) IPv4 address in a NAT
> on the satellite. One aspect that hasn't really been mentioned much here
> is that of PDU size on the link between end customer and satellite.
> Keeping Dishy and its successors small and cheap creates an incentive to
> operate at marginal SNR, and this favours smaller PDUs over larger ones
> as the probability of PDU checksum errors increases with PDU size. But
> having lots of small PDUs means having lots of headers, and as IPv4
> addresses are leaner than IPv6 ones, this saves bandwidth here. Probably
> not a biggie though.
> > - New ground stations with more capacity are coming (and will be
> > upgrades).
> Any word on where? At the moment, most of the world can see Starlink
> satellites, but most Starlink satellites can't see a ground station.
> > They are using waves back to regional DCs now, but will be moving to
> > dark fiber over the next year or two
> If that means "radio" waves, then this goes a long way to explaining why
> there's already limited capacity even near the US-Canada border.
> > - the new satellites have more than 2 lasers, and there is enough
> > capacity on them to do routing. no details on how or what protocols, alas
> Any word on when we can expect to see routing in action?
> > - new birds also have 2-3x more ku bandwidth than first gen
> Hm. Sounds cool, but with 3 billion or so underserved on the planet &
> typical annual growth rates, that's still just a drop in the bucket.
> > - new dishes are in the works, v4 coming with lower power use, more
> > capacity, not round any more
> Trayee? Squary? Just joking ;-)
> > - larger dishes coming for commercial apps
> That's good news, as this will allow Starlink to be used in places where
> direct-to-site crashes into regulatory hurdles. If we can get the big
> CDN providers to come up with small (virtual?) appliances that can be
> put at the remote end of such links by local ISPs, then that'll also
> help to preserve space segment capacity.
>
> --
> ****************************************************************
> Dr. Ulrich Speidel
>
> School of Computer Science
>
> Room 303S.594 (City Campus)
> Ph: (+64-9)-373-7599 ext. 85282
>
> The University of Auckland
> ulrich@cs.auckland.ac.nz
> http://www.cs.auckland.ac.nz/~ulrich/
> ****************************************************************
>
>
>
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
>

[-- Attachment #2: Type: text/html, Size: 6587 bytes --]

Re: [Starlink] Starlink tidbits from NANOG

[Darrell Budic]

[-- Attachment #1: Type: text/plain, Size: 1453 bytes --]


> On Nov 4, 2021, at 8:05 PM, Nathan Owens <nathan@nathan.io> wrote:
> 
> > They are using waves back to regional DCs now, but will be moving to 
> > dark fiber over the next year or two
> If that means "radio" waves, then this goes a long way to explaining why 
> there's already limited capacity even near the US-Canada border.
> 
> Waves in this case generally refers to 10G/100G leased optical circuit capacity. 

As has already been mentioned, I did mean optical waves. Comes from my perspective as a network guy who uses lots of optical transport and DWDM systems. Realized I should have been more specific about optical waves, especially in this forum. Even more specifically, “waves” are generally a product provided you by someone else who lights the fiber for you. In this case, probably Zayo, Lumen, Crown Castle, and maybe even Google operating the fiber. Generally 10G waves, or some multiple there of, although 100G waves are out there, and 400G are coming/testing/just entering production. 

In this case, I suspect it means 10G waves given the currently estimated capacity of the ground stations. Moving to dark lets them do Nx10G links more cost effectively, or 100G link if the distance is short enough (around 100km at the moment before you have to go to the much more expensive coherent optics). And you can do you own support and monitoring, which I got the impression was part of their problem with Google’s NOC.

[-- Attachment #2: Type: text/html, Size: 2675 bytes --]

Re: [Starlink] Starlink tidbits from NANOG

[Darrell Budic]

> On Nov 4, 2021, at 7:34 PM, Ulrich Speidel <ulrich@cs.auckland.ac.nz> wrote:
> 
> Thanks for that Darrell - that's really interesting! A few comments on that front:
> 
> On 5/11/2021 4:26 am, Darrell Budic wrote:
>> I was at NANOG in Minneapolis, and got a chance to ask a couple question of a Starlink Network Engineer who’s attending. I was already talking to him about Starlink’s network efforts (see below) but it was nice to meet in person. Don’t quote me on any of this, but here’s a few tidbits this list may appreciate:
>> 
>> - Starlink is expanding their own network operations, and is connecting to more IXPs. They were already on SIX in Seattle, have connected to DECIX NY, and are in the process of connecting to ChIX in Chicago. As I run ChIX, I had a good excuse to talk to them about other things. :) IXPs and their own networks are in the works for Europe and other areas as well.
> Makes sense.
>> - They have been obtaining more v4 addresses, but I don’t know if they have enough to not do CGNAT. I don't think they do yet, but it seems like it may be a long term target.
>> - v6 is deliberately not fully functional, but they know some of use are using it and it will eventually be fully activated. May be waiting on the regional connectivity, so will be intersting to see if changes for some areas and not others as they roll it out.
> 
> So I guess we need to distinguish between:
> 
> - IPv4 addresses for any CGNAT they might run
> - IPv4 addresses as static addresses for (some of?) their customers
> - IPv6 addresses as customer addresses
> - IPv6 addresses to support geographic routing as discussed in earlier posts (subnet maps to cell / satellite)
> 
> There are quite a number of feasible configurations in this. E.g., they could be running a CGNAT setup with a v4 pool on the Internet side, use v6 to tunnel route from there to the satellite the end customer connects to, and then map that customer back to a (private) IPv4 address in a NAT on the satellite. One aspect that hasn't really been mentioned much here is that of PDU size on the link between end customer and satellite. Keeping Dishy and its successors small and cheap creates an incentive to operate at marginal SNR, and this favours smaller PDUs over larger ones as the probability of PDU checksum errors increases with PDU size. But having lots of small PDUs means having lots of headers, and as IPv4 addresses are leaner than IPv6 ones, this saves bandwidth here. Probably not a biggie though.

“v6 deliberately non functional” as in they havn’t enabled radvd, not that they’ve broken it, btw. I’m thinking they’ll move to enable v6 fully, and continue to CGNAT the v4 for non-commercial customers. If they can enable routing or tunnels for commercial customers, the customers could likely bring their own v4 ranges. Effectively what I’m doing right now with some simple tunnels...

>> - New ground stations with more capacity are coming (and will be upgrades). 
> Any word on where? At the moment, most of the world can see Starlink satellites, but most Starlink satellites can't see a ground station.

I got the impression it will be new stations to increase density, especially at lower latitudes, but also would be ongoing upgrades for existing POPs to allow the new satellites to take full advantage of their upgrades as well.

>> They are using waves back to regional DCs now, but will be moving to dark fiber over the next year or two
> If that means "radio" waves, then this goes a long way to explaining why there's already limited capacity even near the US-Canada border.
>> - the new satellites have more than 2 lasers, and there is enough capacity on them to do routing. no details on how or what protocols, alas
> Any word on when we can expect to see routing in action?

Sounded like they expected to start testing after another launch or two (three?) of the upgraded sats, so 4 months maybe? Doesn’t seem like a set schedule as they will like test once they have enough capacity, then move to enable as they go.

>> - new birds also have 2-3x more ku bandwidth than first gen
> Hm. Sounds cool, but with 3 billion or so underserved on the planet & typical annual growth rates, that's still just a drop in the bucket.
>> - new dishes are in the works, v4 coming with lower power use, more capacity, not round any more
> Trayee? Squary? Just joking ;-)

Haha, Rectangly apparently. More in line with the size of their phased array antennas and also to lower cost. I imagine it will look a lot like some of the existing IPTV panel receivers.

>> - larger dishes coming for commercial apps
> That's good news, as this will allow Starlink to be used in places where direct-to-site crashes into regulatory hurdles. If we can get the big CDN providers to come up with small (virtual?) appliances that can be put at the remote end of such links by local ISPs, then that'll also help to preserve space segment capacity.

Sounds like they have some regional “super pops” where they can locate todays standard CDN cache services, but they want to try and peer as much as possible for it, both public and private. But emphasis on public up front, especially as they are still trying to get a good handle on the kind of traffic people are pushing over them.

> 
> -- 
> ****************************************************************
> Dr. Ulrich Speidel
> 
> School of Computer Science
> 
> Room 303S.594 (City Campus)
> Ph: (+64-9)-373-7599 ext. 85282
> 
> The University of Auckland
> ulrich@cs.auckland.ac.nz
> http://www.cs.auckland.ac.nz/~ulrich/
> ****************************************************************
> 
> 
> 
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink

Re: [Starlink] Starlink tidbits from NANOG

[Darrell Budic]

[-- Attachment #1: Type: text/plain, Size: 1253 bytes --]



> On Nov 4, 2021, at 12:11 PM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> On Thu, Nov 4, 2021 at 9:46 AM Dave Taht <dave.taht@gmail.com <mailto:dave.taht@gmail.com>> wrote:
>> 
>> I have an ipv4/23 lying around since the 90s. I don't want to sell it,
>> but my co-owner and I would really like a dishy and a static IPv4/IPv6
>> address... and service for life... and whatever else we could
>> negotiate. :)
> 
> Perhaps 2 zeroth addresses in exchange, also. Getting 0 back is order
> millions more real ipv4s for the world. code's in linux and bsd now.
> 
> https://tools.ietf.org/id/draft-schoen-intarea-lowest-address-00.txt <https://tools.ietf.org/id/draft-schoen-intarea-lowest-address-00.txt>
> 
> (there's a preso nov 9th in intarea on this)
> 

Haha, well, with v4 addresses now going for > $50 each, in the usual blocks of 256, it might be a fair trade.

If you want a static, I can tunnel you one for while… Works better the closer you are to Chicago, but I’m well connected there.

Although at ARIN 48 after NANOG, NIST laid out the details of the ongoing federal government push to IPv6-ONLY networking, with an 80% complete target by 2025. I expect we will see v6 pick up some steam over the next few years.


[-- Attachment #2: Type: text/html, Size: 8595 bytes --]

Re: [Starlink] Starlink tidbits from NANOG

[Darrell Budic]

> On Nov 4, 2021, at 11:46 AM, Dave Taht <dave.taht@gmail.com> wrote:
> 
>> - v6 is deliberately not fully functional, but they know some of use are using it and it will eventually be fully activated. May be waiting on the regional connectivity, so will be intersting to see if changes for some areas and not others as they roll it out.
> 
> Deliberate... well, if you ship 10 year old openwrt software to users
> (we'd made a big push for ipv6 there before ipv6 launch day in 2013),
> and
> don't keep up...  I guess you could call that deliberate. I'm pretty
> happy with openwrt 20.2.1. IMHO: ipv6 really requires a modern kernel
> and tools, not less than 4 years old, to deploy well. Maybe there's a
> worthwhile SDR stack, I don't know...
> 

Deliberately not enable radvd in this case, so hopefully easily solvable once they have a v6 backbone they are happy with.

>> - They hate Google's outsourced NOC as much as the rest of us
> 
> Do say more. :) I'm told google at least have a very nice set of tools
> for looking at the characteristics of interchange traffic.

If you’ve ever had to deal with them, it’s a lot of long delays on tickets, no explanation for problems and fix acknowledgements only if you’re lucky, and lots of “please do the needful and revert…” type non-answers. And they never give out data on your traffic once it’s left your box/boundary, even though you’re paying them for the traffic. One of the worst I’ve ever delt with, but fortunately offset by all the smart folk you never get to talk to that ensure most things work really well most of the time.


>> - New ground stations with more capacity are coming (and will be upgrades).
> 
> Would so love universities to get in on some of those. I remember the IMP.

They are emphasizing public peering, so I’m hopefully they’ll keep moving. Many of the university’s already peer at SIX and a couple are at ChIX, so their connectivity will open up a bit. Speaking of, I’ll give a 10G ChIX connection to any public educational institution that wants one and can get to me, so send ‘em my way if they’re in the midwest and interested!

> 
>> - new birds also have 2-3x more ku bandwidth than first gen
> 
> up and down?
> 
> My take on the up problem was that it was regulatory. ?
> 

It’s three beams “up/down” from the Dishys (one at a time in a TDM fashion, which we knew), and one locked on whatever ground station it’s tracking at the moment. Sounded like more capacity on all of them, unclear if it’s better antennas, more power, better tracking, or all of the above. It did sound like the ground stations will get bigger antennas, so that’s probably part of the downlink improvements.

> 
> Just someone telling me under pain of death, "dave, you can't talk for
> X months, but we're going to do cake/fq-codel/pie/something" would be
> comforting. There's a whole internet elsewhere left to fix, starlink
> getting it right and a little publicity around it would do wonders...
> and certainly wifi is highest on my list. As it is, I got annoyed
> enough last week to try and get the autorate sensing code to work well
> on starlink. There's a prototype now that seems to be working well on
> lte, see here: https://forum.openwrt.org/t/cakes-autorate-ingress-testing-needed/108848/186
> 
> Testers wanted.

I’ll get my cake install upgraded and test it out. To be honest, I haven’t even been running it for a while, so they’ve gotten better since you started talking to them even if it isn’t great yet.

> Fixing fixed wireless has been a pain point far, far, far greater
> than the disappointment I felt at starlink so totally missing the
> bufferbloat problem initially. It will take a decade to sort out 5g,
> 4-6 months more for starlink oh! yes! yes! yes!
> 
> More news on that as it happens.
> 
>> - it’s encrypted up and down. I didn’t know that yet, but I may have just missed it.
> 
> I did. But it's really hard to trust that black box and the world has
> otherwise shifted to e2e encryption.

My contact acknowledged that, but it’s nice to know it’s probably better encrypted than the LTE modems we all use everyday already. I imagine we’ll have to continue to depend on the application layer for that sort of e2e encryption, with the advantage that it won’t matter what our transport tech is then.

Re: [Starlink] Starlink tidbits from NANOG

[Darrell Budic]

> On Nov 4, 2021, at 11:46 AM, Dave Taht <dave.taht@gmail.com> wrote:
>> - new dishes are in the works, v4 coming with lower power use, more capacity, not round any more
>> - larger dishes coming for commercial apps
> 
>> - as we know, they aren’t doing any AQM yet, but it sounds like it may be in the works and we may see it in new code in 4-6 months. Not my guys department, so no more details.
> 
> fq is a better start.
> 
> Just someone telling me under pain of death, "dave, you can't talk for
> X months, but we're going to do cake/fq-codel/pie/something" would be
> comforting. There's a whole internet elsewhere left to fix, starlink
> getting it right and a little publicity around it would do wonders...
> and certainly wifi is highest on my list. As it is, I got annoyed
> enough last week to try and get the autorate sensing code to work well
> on starlink. There's a prototype now that seems to be working well on
> lte, see here: https://forum.openwrt.org/t/cakes-autorate-ingress-testing-needed/108848/186

Forgot to say that I did pitch the advantages of lowering latency and jumping over many other terrestrial providers for congestion control. My guy was ground networking though, but I can hope it will get passed over. Hoping I can stay in touch with them and continue to pipe things in and out of their black box...

[Starlink] data sovereignty

[Michael Richardson]

[-- Attachment #1: Type: text/plain, Size: 671 bytes --]


A number of Canadian IXs have pondered whether we could get Starlink to peer
via air.    I don't think this will be possible until they move to IPv6
so that they can do some geographic allocation of end-station addresses.

A question that was asked was: what is the altitude at which the data has
left the country...   Heinlein's _Man Who Sold the Moon_ is not true.
Countries have a limited altitude in which to claim soveignty.  But, what is
it?  (Would Alphabet Loon be within it? I suspect so)

Will this argument about data sovereignty be used by national governments (or
rather, the associated incumbent telco-ISPs) to forbid spending public money
on Starlink?




[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [Starlink] Starlink tidbits from NANOG

[Michael Richardson]

[-- Attachment #1: Type: text/plain, Size: 783 bytes --]


Darrell Budic <budic@onholyground.com> wrote:
    > Sounded like they expected to start testing after another launch or two
    > (three?) of the upgraded sats, so 4 months maybe? Doesn’t seem like a
    > set schedule as they will like test once they have enough capacity,
    > then move to enable as they go.

It seems like they will have a lot of generation-1, generation-2 birds in the
air, and that this will be annoying from an operational point of view.
I wonder if they will de-orbit the older satellites sooner.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [Starlink] data sovereignty

[Spencer Sevilla]

For what it’s worth, space is usually defined as either 50 miles (80km) or 62 miles (100 km) depending on the agency in question. I couldn’t find specific resources on how high a country’s “airspace” goes, but I do believe that “space" is considered international territory.

https://en.wikipedia.org/wiki/K%C3%A1rm%C3%A1n_line
https://en.wikipedia.org/wiki/Space_law

Spencer

> On Nov 5, 2021, at 08:00, Michael Richardson <mcr@sandelman.ca> wrote:
> 
> 
> A number of Canadian IXs have pondered whether we could get Starlink to peer
> via air.    I don't think this will be possible until they move to IPv6
> so that they can do some geographic allocation of end-station addresses.
> 
> A question that was asked was: what is the altitude at which the data has
> left the country...   Heinlein's _Man Who Sold the Moon_ is not true.
> Countries have a limited altitude in which to claim soveignty.  But, what is
> it?  (Would Alphabet Loon be within it? I suspect so)
> 
> Will this argument about data sovereignty be used by national governments (or
> rather, the associated incumbent telco-ISPs) to forbid spending public money
> on Starlink?
> 
> 
> 
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink

Re: [Starlink] data sovereignty

[Daniel AJ Sokolov]

On 2021-11-05 at 8:00 a.m., Michael Richardson wrote:
> (Would Alphabet Loon be within it? I suspect so)

Loon is done for. Closed.
Daniel AJ

Re: [Starlink] data sovereignty

[Daniel AJ Sokolov]

On 2021-11-05 at 8:00 a.m., Michael Richardson wrote:
> A question that was asked was: what is the altitude at which the data has
> left the country...

That's a good question.

The answer may not be in airspace vs. space. (That border is undefined, 
but if there is not enough air to get downlift, I would say it's not 
airspace anymore, but space.)

One answer may be that the vessel that is routing the data flies under a 
certain carrier flag. So one could argue that all SpaceX satellites are 
US flagged, as they obtained their license from the FCC. (Same for OneWeb.)

Could there be VPNs creating virtual Canadian territory in cyberspace? 
Why not, eh!
Daniel AJ

Re: [Starlink] Starlink tidbits from NANOG

[Ulrich Speidel]

[-- Attachment #1: Type: text/plain, Size: 1150 bytes --]

On 5/11/2021 2:05 pm, Nathan Owens wrote:
> > They are using waves back to regional DCs now, but will be moving to
> > dark fiber over the next year or two
> If that means "radio" waves, then this goes a long way to explaining why
> there's already limited capacity even near the US-Canada border.
>
> Waves in this case generally refers to 10G/100G leased optical circuit 
> capacity.
>
Thanks for clearing this up, that makes sense now!

Now I note that a single Starlink ground station is generally within 
sight of and set up to handle multiple satellite links (all known images 
show 6 or more radomes). As each bird has ~20 Gb/s capacity, I guess 
we'll probably be talking 100G circuits here (where they can get them). 
And the move to dark fibre is a logical step then.

-- 

****************************************************************
Dr. Ulrich Speidel

School of Computer Science

Room 303S.594 (City Campus)
Ph: (+64-9)-373-7599 ext. 85282

The University of Auckland
ulrich@cs.auckland.ac.nz  
http://www.cs.auckland.ac.nz/~ulrich/
****************************************************************



[-- Attachment #2: Type: text/html, Size: 2553 bytes --]

Re: [Starlink] Starlink tidbits from NANOG

[Ulrich Speidel]

[-- Attachment #1: Type: text/plain, Size: 1944 bytes --]

On 5/11/2021 2:18 pm, Darrell Budic wrote:
>
>>
>> Waves in this case generally refers to 10G/100G leased optical 
>> circuit capacity.
>
> As has already been mentioned, I did mean optical waves. Comes from my 
> perspective as a network guy who uses lots of optical transport and 
> DWDM systems. Realized I should have been more specific about optical 
> waves, especially in this forum. Even more specifically, “waves” are 
> generally a product provided you by someone else who lights the fiber 
> for you. In this case, probably Zayo, Lumen, Crown Castle, and maybe 
> even Google operating the fiber. Generally 10G waves, or some multiple 
> there of, although 100G waves are out there, and 400G are 
> coming/testing/just entering production.
>
> In this case, I suspect it means 10G waves given the currently 
> estimated capacity of the ground stations. Moving to dark lets them do 
> Nx10G links more cost effectively, or 100G link if the distance is 
> short enough (around 100km at the moment before you have to go to the 
> much more expensive coherent optics). And you can do you own support 
> and monitoring, which I got the impression was part of their problem 
> with Google’s NOC.
Thanks for that - I know them by a different term here but maybe I'm 
just not up with the play ;-) I know what you mean. That said, a single 
10G circuit wouldn't even feed a fully loaded bent pipe Gen 1 Starlink 
bird, so I could imagine that with several birds being trackable by each 
ground station, those 100G links and higher will be hot property.

-- 
****************************************************************
Dr. Ulrich Speidel

School of Computer Science

Room 303S.594 (City Campus)
Ph: (+64-9)-373-7599 ext. 85282

The University of Auckland
ulrich@cs.auckland.ac.nz  
http://www.cs.auckland.ac.nz/~ulrich/
****************************************************************



[-- Attachment #2: Type: text/html, Size: 2982 bytes --]