Re: [Cerowrt-devel] my thx to spacex (and kerbal space program) forcheering me up all year

Re: [Cerowrt-devel] my thx to spacex (and kerbal space program) forcheering me up all year

[David P. Reed]

It has bufferbloat? 

Why am I not surprised?

I can share that one stack hasn't had it from the start, by design. That is one implemented for trading at 10+ GB/sec, implemented in Verilog, and now apparently in production use at one of the largest NY trading intermediaries.

Why? Simply two reasons:

1. People who design parallel hardware systems are trained to focus on closing timing constraints. Which means never using FIFOs that are longer than absolute minimum. The designer is a VLSI designer by trade, not a networking guy.

2. Trading is all about managing delay. In this case, 100 msec packet delay is worst allowable case end to end.

Yet it is full TCP in hardware.

Can't share more, because I don't know more, it being all proprietary to the bank in question.

Now, one wonders: why can't Starlink get it right first time?

It's not like bufferbloat is hard on a single bent pipe hop, which is all Starlink does today.

-----Original Message-----
From: "Dave Taht" <dave.taht@gmail.com>
Sent: Thu, Dec 31, 2020 at 1:37 pm
To: "bloat" <bloat@lists.bufferbloat.net>, "cerowrt-devel" <cerowrt-devel@lists.bufferbloat.net>, "Scott Manley" <scottmanley1972@gmail.com>
Cc: "bloat" <bloat@lists.bufferbloat.net>, "cerowrt-devel" <cerowrt-devel@lists.bufferbloat.net>, "Scott Manley" <scottmanley1972@gmail.com>
Subject: [Cerowrt-devel] my thx to spacex (and kerbal space program) forcheering me up all year

If it wasn't for such a long list of wonderful accomplishments in
space, it would have been a sadder year. i just re-recorded my song
"one first landing" out on my dinghy:

https://www.youtube.com/watch?v=wjur0RG-v-I&feature=youtu.be

Maybe someday I'll get scott manley to do his verse on this. Try as I
might over the past few years, I still can't cop his accent.

Now if we can only fix starlink's bufferbloat! It looks to me like the
firmware is QCA's openwrt derivative...

-- 
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

dave@taht.net  CTO, TekLibre, LLC Tel: 1-831-435-0729
_______________________________________________
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] my thx to spacex (and kerbal space program) forcheering me up all year

[Jonathan Morton]

> On 2 Jan, 2021, at 1:31 am, David P. Reed <dpreed@deepplum.com> wrote:
> 
> Now, one wonders: why can't Starlink get it right first time?
> 
> It's not like bufferbloat is hard on a single bent pipe hop, which is all Starlink does today.

The bloat doesn't seem to be in Starlink itself, but in the consumer-end modem.  This is fixable, just as soon as Starlink put their minds to it, because it's based on the same Atheros SoCs as the consumer CPE we're already familiar with.

 - Jonathan Morton

Re: [Cerowrt-devel] my thx to spacex (and kerbal space program) forcheering me up all year

[Dave Taht]

Taking scott manley off the cc. As much as I would like him to sit in
on that song, I don't think he cares about bufferbloat as much as Id
like him to.... I do wish I knew someone that could just magically get
the bufferbloat issue raised within
starlink effectively....

On Fri, Jan 1, 2021 at 3:31 PM David P. Reed <dpreed@deepplum.com> wrote:
>
> It has bufferbloat?

Yep. See links below.

> Why am I not surprised?

I was surprised given the length of the talk I'd had with their VP,
eng, but I do figure that getting the thing working at all
was a greater challenge than tackling our issue. I've long worried of
course, that the mac layer on this thing was going to be very weird,
and since they were working with qca they'd end up buring everything
in the network offload processors, even though at present speeds the
cpu they are using is *loafing*, Im not as optimistic as jon is as to
how easy the port of cake or fq_codel would be to their hardware as it
is variable bandwidth and thus needs bql-like backpressure.


Since there doesn't seem to be a gpl drop yet we don't know a lot,
however there was a teardown of the hw and jim's posting and a start
at testing on reddit - the dslreports test was flawed in that ping did
not work at all....

https://www.reddit.com/r/Starlink/comments/k0dwon/starlink_and_bufferbloat_testing/

rate limiting with sqm works:

https://www.reddit.com/r/Starlink/comments/jxkef9/ahat_is_your_starlinks_bufferbloat_score/

and the starlink teardown was good:
https://www.reddit.com/r/hardware/comments/kchxl8/starlink_teardown_and_analysis/

We think that this hardware actually has fq_codel AND hw rate limiting
in it. But that's not the right thing for a starlink up or downlink
which is variable rate.

any I know at least 5 people on the beta list and maybe we can get
better testing done in the next month or two.

It would of course be great if somehow we could get loud enough for
musk to tweet back or hire one of us to help 'em get it right.... I
had a friend of mine send him a suitably inscribed copy of toke's
"bufferbloat and beyond" via interoffice mail... but finding someone
up there more focused on the terminal software itself would be better.
I mean low latency should be their bread and butter and while Im sad
that the early tests are dismal, I am pretty confident that ultimately
they will get it right.

> I can share that one stack hasn't had it from the start, by design. That is one implemented for trading at 10+ GB/sec, implemented in Verilog, and now apparently in production use at one of the largest NY trading intermediaries.

Yea!

>
> Why? Simply two reasons:
>
> 1. People who design parallel hardware systems are trained to focus on closing timing constraints. Which means never using FIFOs that are longer than absolute minimum. The designer is a VLSI designer by trade, not a networking guy.
>
> 2. Trading is all about managing delay. In this case, 100 msec packet delay is worst allowable case end to end.
>
> Yet it is full TCP in hardware.
>
> Can't share more, because I don't know more, it being all proprietary to the bank in question.
>
> Now, one wonders: why can't Starlink get it right first time?
>
> It's not like bufferbloat is hard on a single bent pipe hop, which is all Starlink does today.
>
> -----Original Message-----
> From: "Dave Taht" <dave.taht@gmail.com>
> Sent: Thu, Dec 31, 2020 at 1:37 pm
> To: "bloat" <bloat@lists.bufferbloat.net>, "cerowrt-devel" <cerowrt-devel@lists.bufferbloat.net>, "Scott Manley" <scottmanley1972@gmail.com>
> Cc: "bloat" <bloat@lists.bufferbloat.net>, "cerowrt-devel" <cerowrt-devel@lists.bufferbloat.net>, "Scott Manley" <scottmanley1972@gmail.com>
> Subject: [Cerowrt-devel] my thx to spacex (and kerbal space program) forcheering me up all year
>
> If it wasn't for such a long list of wonderful accomplishments in
> space, it would have been a sadder year. i just re-recorded my song
> "one first landing" out on my dinghy:
>
> https://www.youtube.com/watch?v=wjur0RG-v-I&feature=youtu.be
>
> Maybe someday I'll get scott manley to do his verse on this. Try as I
> might over the past few years, I still can't cop his accent.
>
> Now if we can only fix starlink's bufferbloat! It looks to me like the
> firmware is QCA's openwrt derivative...
>
> --
> "For a successful technology, reality must take precedence over public
> relations, for Mother Nature cannot be fooled" - Richard Feynman
>
> dave@taht.net  CTO, TekLibre, LLC Tel: 1-831-435-0729
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>


--
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729

Re: [Cerowrt-devel] my thx to spacex (and kerbal space program) forcheering me up all year

[Scott Manley]

[-- Attachment #1: Type: text/plain, Size: 5394 bytes --]

Oh but I'm fascinated by the discussion of Starlink's network performance.!

On Sat, Jan 2, 2021 at 9:17 AM Dave Taht <dave.taht@gmail.com> wrote:

> Taking scott manley off the cc. As much as I would like him to sit in
> on that song, I don't think he cares about bufferbloat as much as Id
> like him to.... I do wish I knew someone that could just magically get
> the bufferbloat issue raised within
> starlink effectively....
>
> On Fri, Jan 1, 2021 at 3:31 PM David P. Reed <dpreed@deepplum.com> wrote:
> >
> > It has bufferbloat?
>
> Yep. See links below.
>
> > Why am I not surprised?
>
> I was surprised given the length of the talk I'd had with their VP,
> eng, but I do figure that getting the thing working at all
> was a greater challenge than tackling our issue. I've long worried of
> course, that the mac layer on this thing was going to be very weird,
> and since they were working with qca they'd end up buring everything
> in the network offload processors, even though at present speeds the
> cpu they are using is *loafing*, Im not as optimistic as jon is as to
> how easy the port of cake or fq_codel would be to their hardware as it
> is variable bandwidth and thus needs bql-like backpressure.
>
>
> Since there doesn't seem to be a gpl drop yet we don't know a lot,
> however there was a teardown of the hw and jim's posting and a start
> at testing on reddit - the dslreports test was flawed in that ping did
> not work at all....
>
>
> https://www.reddit.com/r/Starlink/comments/k0dwon/starlink_and_bufferbloat_testing/
>
> rate limiting with sqm works:
>
>
> https://www.reddit.com/r/Starlink/comments/jxkef9/ahat_is_your_starlinks_bufferbloat_score/
>
> and the starlink teardown was good:
>
> https://www.reddit.com/r/hardware/comments/kchxl8/starlink_teardown_and_analysis/
>
> We think that this hardware actually has fq_codel AND hw rate limiting
> in it. But that's not the right thing for a starlink up or downlink
> which is variable rate.
>
> any I know at least 5 people on the beta list and maybe we can get
> better testing done in the next month or two.
>
> It would of course be great if somehow we could get loud enough for
> musk to tweet back or hire one of us to help 'em get it right.... I
> had a friend of mine send him a suitably inscribed copy of toke's
> "bufferbloat and beyond" via interoffice mail... but finding someone
> up there more focused on the terminal software itself would be better.
> I mean low latency should be their bread and butter and while Im sad
> that the early tests are dismal, I am pretty confident that ultimately
> they will get it right.
>
> > I can share that one stack hasn't had it from the start, by design. That
> is one implemented for trading at 10+ GB/sec, implemented in Verilog, and
> now apparently in production use at one of the largest NY trading
> intermediaries.
>
> Yea!
>
> >
> > Why? Simply two reasons:
> >
> > 1. People who design parallel hardware systems are trained to focus on
> closing timing constraints. Which means never using FIFOs that are longer
> than absolute minimum. The designer is a VLSI designer by trade, not a
> networking guy.
> >
> > 2. Trading is all about managing delay. In this case, 100 msec packet
> delay is worst allowable case end to end.
> >
> > Yet it is full TCP in hardware.
> >
> > Can't share more, because I don't know more, it being all proprietary to
> the bank in question.
> >
> > Now, one wonders: why can't Starlink get it right first time?
> >
> > It's not like bufferbloat is hard on a single bent pipe hop, which is
> all Starlink does today.
> >
> > -----Original Message-----
> > From: "Dave Taht" <dave.taht@gmail.com>
> > Sent: Thu, Dec 31, 2020 at 1:37 pm
> > To: "bloat" <bloat@lists.bufferbloat.net>, "cerowrt-devel" <
> cerowrt-devel@lists.bufferbloat.net>, "Scott Manley" <
> scottmanley1972@gmail.com>
> > Cc: "bloat" <bloat@lists.bufferbloat.net>, "cerowrt-devel" <
> cerowrt-devel@lists.bufferbloat.net>, "Scott Manley" <
> scottmanley1972@gmail.com>
> > Subject: [Cerowrt-devel] my thx to spacex (and kerbal space program)
> forcheering me up all year
> >
> > If it wasn't for such a long list of wonderful accomplishments in
> > space, it would have been a sadder year. i just re-recorded my song
> > "one first landing" out on my dinghy:
> >
> > https://www.youtube.com/watch?v=wjur0RG-v-I&feature=youtu.be
> >
> > Maybe someday I'll get scott manley to do his verse on this. Try as I
> > might over the past few years, I still can't cop his accent.
> >
> > Now if we can only fix starlink's bufferbloat! It looks to me like the
> > firmware is QCA's openwrt derivative...
> >
> > --
> > "For a successful technology, reality must take precedence over public
> > relations, for Mother Nature cannot be fooled" - Richard Feynman
> >
> > dave@taht.net  CTO, TekLibre, LLC Tel: 1-831-435-0729
> > _______________________________________________
> > Cerowrt-devel mailing list
> > Cerowrt-devel@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cerowrt-devel
> >
> >
>
>
> --
> "For a successful technology, reality must take precedence over public
> relations, for Mother Nature cannot be fooled" - Richard Feynman
>
> dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729
>

[-- Attachment #2: Type: text/html, Size: 7558 bytes --]

Re: [Cerowrt-devel] my thx to spacex (and kerbal space program) forcheering me up all year

[Dave Taht]

On Sat, Jan 2, 2021 at 1:39 PM Scott Manley <scottmanley1972@gmail.com> wrote:
>
> Oh but I'm fascinated by the discussion of Starlink's network performance.!

 I HAD figured that everyone at starlink has read about how NOT to
build a space-born
packet switched network. If you haven't read it, "Eccentric Orbits:
The Iridium Story" is a great read.

I have had quite the hobby trying to grok what starlink is up to, I
can go into it here if you like. I can't remember how much of the
bufferbloat problem I'd discussed with you at the last hackers
conference.

PS (can you pleeeeeze record you singing over your part in that song? :) )
>
> On Sat, Jan 2, 2021 at 9:17 AM Dave Taht <dave.taht@gmail.com> wrote:
>>
>> Taking scott manley off the cc. As much as I would like him to sit in
>> on that song, I don't think he cares about bufferbloat as much as Id
>> like him to.... I do wish I knew someone that could just magically get
>> the bufferbloat issue raised within
>> starlink effectively....
>>
>> On Fri, Jan 1, 2021 at 3:31 PM David P. Reed <dpreed@deepplum.com> wrote:
>> >
>> > It has bufferbloat?
>>
>> Yep. See links below.
>>
>> > Why am I not surprised?
>>
>> I was surprised given the length of the talk I'd had with their VP,
>> eng, but I do figure that getting the thing working at all
>> was a greater challenge than tackling our issue. I've long worried of
>> course, that the mac layer on this thing was going to be very weird,
>> and since they were working with qca they'd end up buring everything
>> in the network offload processors, even though at present speeds the
>> cpu they are using is *loafing*, Im not as optimistic as jon is as to
>> how easy the port of cake or fq_codel would be to their hardware as it
>> is variable bandwidth and thus needs bql-like backpressure.
>>
>>
>> Since there doesn't seem to be a gpl drop yet we don't know a lot,
>> however there was a teardown of the hw and jim's posting and a start
>> at testing on reddit - the dslreports test was flawed in that ping did
>> not work at all....
>>
>> https://www.reddit.com/r/Starlink/comments/k0dwon/starlink_and_bufferbloat_testing/
>>
>> rate limiting with sqm works:
>>
>> https://www.reddit.com/r/Starlink/comments/jxkef9/ahat_is_your_starlinks_bufferbloat_score/
>>
>> and the starlink teardown was good:
>> https://www.reddit.com/r/hardware/comments/kchxl8/starlink_teardown_and_analysis/
>>
>> We think that this hardware actually has fq_codel AND hw rate limiting
>> in it. But that's not the right thing for a starlink up or downlink
>> which is variable rate.
>>
>> any I know at least 5 people on the beta list and maybe we can get
>> better testing done in the next month or two.
>>
>> It would of course be great if somehow we could get loud enough for
>> musk to tweet back or hire one of us to help 'em get it right.... I
>> had a friend of mine send him a suitably inscribed copy of toke's
>> "bufferbloat and beyond" via interoffice mail... but finding someone
>> up there more focused on the terminal software itself would be better.
>> I mean low latency should be their bread and butter and while Im sad
>> that the early tests are dismal, I am pretty confident that ultimately
>> they will get it right.
>>
>> > I can share that one stack hasn't had it from the start, by design. That is one implemented for trading at 10+ GB/sec, implemented in Verilog, and now apparently in production use at one of the largest NY trading intermediaries.
>>
>> Yea!
>>
>> >
>> > Why? Simply two reasons:
>> >
>> > 1. People who design parallel hardware systems are trained to focus on closing timing constraints. Which means never using FIFOs that are longer than absolute minimum. The designer is a VLSI designer by trade, not a networking guy.
>> >
>> > 2. Trading is all about managing delay. In this case, 100 msec packet delay is worst allowable case end to end.
>> >
>> > Yet it is full TCP in hardware.
>> >
>> > Can't share more, because I don't know more, it being all proprietary to the bank in question.
>> >
>> > Now, one wonders: why can't Starlink get it right first time?
>> >
>> > It's not like bufferbloat is hard on a single bent pipe hop, which is all Starlink does today.
>> >
>> > -----Original Message-----
>> > From: "Dave Taht" <dave.taht@gmail.com>
>> > Sent: Thu, Dec 31, 2020 at 1:37 pm
>> > To: "bloat" <bloat@lists.bufferbloat.net>, "cerowrt-devel" <cerowrt-devel@lists.bufferbloat.net>, "Scott Manley" <scottmanley1972@gmail.com>
>> > Cc: "bloat" <bloat@lists.bufferbloat.net>, "cerowrt-devel" <cerowrt-devel@lists.bufferbloat.net>, "Scott Manley" <scottmanley1972@gmail.com>
>> > Subject: [Cerowrt-devel] my thx to spacex (and kerbal space program) forcheering me up all year
>> >
>> > If it wasn't for such a long list of wonderful accomplishments in
>> > space, it would have been a sadder year. i just re-recorded my song
>> > "one first landing" out on my dinghy:
>> >
>> > https://www.youtube.com/watch?v=wjur0RG-v-I&feature=youtu.be
>> >
>> > Maybe someday I'll get scott manley to do his verse on this. Try as I
>> > might over the past few years, I still can't cop his accent.
>> >
>> > Now if we can only fix starlink's bufferbloat! It looks to me like the
>> > firmware is QCA's openwrt derivative...
>> >
>> > --
>> > "For a successful technology, reality must take precedence over public
>> > relations, for Mother Nature cannot be fooled" - Richard Feynman
>> >
>> > dave@taht.net  CTO, TekLibre, LLC Tel: 1-831-435-0729
>> > _______________________________________________
>> > Cerowrt-devel mailing list
>> > Cerowrt-devel@lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> >
>> >
>>
>>
>> --
>> "For a successful technology, reality must take precedence over public
>> relations, for Mother Nature cannot be fooled" - Richard Feynman
>>
>> dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729



-- 
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729

Re: [Cerowrt-devel] [Bloat] my thx to spacex (and kerbal space program) forcheering me up all year

[Michael Richardson]

Well, it was supposed to be "simpler than IPv6", but my take away was really:
      "we never really understood IPv6"