[Starlink] Of interest: Comcast AQM Paper

[Starlink] Of interest: Comcast AQM Paper

[Livingood, Jason]

 FYI that I will be presenting a lightning talk at the IRTF MAPRG meeting today at 17:30 ET (https://datatracker.ietf.org/meeting/111/materials/agenda-111-maprg). The talk links to a just-published paper at https://arxiv.org/abs/2107.13968 (click PDF link in upper right of page) that will likely be of interest to these two lists.

High-level: turning on AQM in the cable modem (upstream queue) took working latency from around 250 ms to between 15-30 ms, which is actually kind of cool. ;-) AQM is turned on in all of our CMTSes (downstream queue) and in DOCSIS 3.1 modems (upstream queue).

Have a nice weekend,
Jason

Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

[Simon Barber]

Awesome to hear that you are turning this on both upstream and downstream. Do you know if the wifi stacks in your home routers also have AQM?

Simon


> On Jul 30, 2021, at 10:28 PM, Livingood, Jason via Bloat <bloat@lists.bufferbloat.net> wrote:
> 
> FYI that I will be presenting a lightning talk at the IRTF MAPRG meeting today at 17:30 ET (https://datatracker.ietf.org/meeting/111/materials/agenda-111-maprg). The talk links to a just-published paper at https://arxiv.org/abs/2107.13968 (click PDF link in upper right of page) that will likely be of interest to these two lists.
> 
> High-level: turning on AQM in the cable modem (upstream queue) took working latency from around 250 ms to between 15-30 ms, which is actually kind of cool. ;-) AQM is turned on in all of our CMTSes (downstream queue) and in DOCSIS 3.1 modems (upstream queue).
> 
> Have a nice weekend,
> Jason
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 1543 bytes --]

If we see that AQM appears to be not functioning as expected for upstream
connections on DOCSIS3.1, what's the right avenue for getting that
resolved?  (and does that only apply to the Comcast-owned, vs.
customer-owned, modems?)

On Sat, Jul 31, 2021 at 10:50 AM Simon Barber <simon@superduper.net> wrote:

> Awesome to hear that you are turning this on both upstream and downstream.
> Do you know if the wifi stacks in your home routers also have AQM?
>
> Simon
>
>
> > On Jul 30, 2021, at 10:28 PM, Livingood, Jason via Bloat <
> bloat@lists.bufferbloat.net> wrote:
> >
> > FYI that I will be presenting a lightning talk at the IRTF MAPRG meeting
> today at 17:30 ET (
> https://datatracker.ietf.org/meeting/111/materials/agenda-111-maprg). The
> talk links to a just-published paper at https://arxiv.org/abs/2107.13968
> (click PDF link in upper right of page) that will likely be of interest to
> these two lists.
> >
> > High-level: turning on AQM in the cable modem (upstream queue) took
> working latency from around 250 ms to between 15-30 ms, which is actually
> kind of cool. ;-) AQM is turned on in all of our CMTSes (downstream queue)
> and in DOCSIS 3.1 modems (upstream queue).
> >
> > Have a nice weekend,
> > Jason
> >
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>

[-- Attachment #2: Type: text/html, Size: 2542 bytes --]

Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

[Neal Cardwell]

On Sat, Jul 31, 2021 at 3:27 PM Aaron Wood <woody77@gmail.com> wrote:
>
> If we see that AQM appears to be not functioning as expected
>  for upstream connections on DOCSIS3.1, what's the right avenue
> for getting that resolved?  (and does that only apply to the
> Comcast-owned, vs. customer-owned, modems?)

FWIW, from the paper it sounds like not all Comcast cable modems
had/have PIE, which enabled the A/B experiment:

"10. Latency Measurement Results
As explained earlier, for two variants of XB6 cable modem gateway,
upstream DOCSIS-PIE AQM was enabled on the CGM4140COM (experiment)
variant but was not available on the TG3482G (control) variant during
the measurement period. The TG3482G variant used a buffer control
configuration that predated AQM in DOCSIS."

neal

>
> On Sat, Jul 31, 2021 at 10:50 AM Simon Barber <simon@superduper.net> wrote:
>>
>> Awesome to hear that you are turning this on both upstream and downstream. Do you know if the wifi stacks in your home routers also have AQM?
>>
>> Simon
>>
>>
>> > On Jul 30, 2021, at 10:28 PM, Livingood, Jason via Bloat <bloat@lists.bufferbloat.net> wrote:
>> >
>> > FYI that I will be presenting a lightning talk at the IRTF MAPRG meeting today at 17:30 ET (https://datatracker.ietf.org/meeting/111/materials/agenda-111-maprg). The talk links to a just-published paper at https://arxiv.org/abs/2107.13968 (click PDF link in upper right of page) that will likely be of interest to these two lists.
>> >
>> > High-level: turning on AQM in the cable modem (upstream queue) took working latency from around 250 ms to between 15-30 ms, which is actually kind of cool. ;-) AQM is turned on in all of our CMTSes (downstream queue) and in DOCSIS 3.1 modems (upstream queue).
>> >
>> > Have a nice weekend,
>> > Jason
>> >
>> > _______________________________________________
>> > Bloat mailing list
>> > Bloat@lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/bloat
>>
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

[Michael Richardson]

[-- Attachment #1: Type: text/plain, Size: 993 bytes --]


Neal Cardwell via Bloat <bloat@lists.bufferbloat.net> wrote:
    > On Sat, Jul 31, 2021 at 3:27 PM Aaron Wood <woody77@gmail.com> wrote:
    >>
    >> If we see that AQM appears to be not functioning as expected
    >> for upstream connections on DOCSIS3.1, what's the right avenue
    >> for getting that resolved?  (and does that only apply to the
    >> Comcast-owned, vs. customer-owned, modems?)

    > FWIW, from the paper it sounds like not all Comcast cable modems
    > had/have PIE, which enabled the A/B experiment:

This does bring up an interesting discovery question:

the presences of AQM (whether PIE or FQ_CODEL), and what the settings might
be (for some things need to be tuned), would be something that might be
interested to emit via LLDP.
And/or at a /.well-known URL (accessible via IPv6-LL only).

--
Michael Richardson <mcr+IETF@sandelman.ca>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide





[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

[Michael Richardson]

Frank Carmickle <frank@carmickle.com> wrote:
    > The Turris, unless I'm missing something, does not do 10g.

The PCIe links are 2.5Gb/s each, so no, it just can't physically move the bits.

Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

[Dave Taht]

On Sun, Aug 1, 2021 at 2:40 PM Dick Roy <dickroy@alum.mit.edu> wrote:
>
> I assume by WiFi what is really meant is devices that have at least one WiFi (layer 1/layer 2) interface. While there are queues in the MAC sublayer, there is really no queue management functionality ... yet ... AFAIK. I know IEEE P802.11bd in conjunction w/ IEEE 1609 is working on implementing a few rudimentary queue mgmt functions.

Thx for the steer to the relevant stds orgs. I dread discovering what
they are up to... However, there is plenty of queue management at the
txop layer.  The structure we put into the linux softmac layer is
described in "ending the anomaly":

The wifi management layer, however, is largely unmanaged, and in some
dense circumstances wifi mgmt frames have been observed to be eating
the most of the airtime available.
>
> That said, seems any AQM in such devices would more than likely be in layer 3 and above.
>
> RR
>
> -----Original Message-----
> From: Starlink [mailto:starlink-bounces@lists.bufferbloat.net] On Behalf Of Livingood, Jason
> Sent: Sunday, August 1, 2021 1:20 PM
> To: Simon Barber
> Cc: starlink@lists.bufferbloat.net; bloat
> Subject: Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper
>
> WiFi is a different challenge as you know. In this case it varies depending on the radio chipset vendor and is on my list of things to work on...
>
> JL
>
> On 7/31/21, 13:50, "Simon Barber" <simon@superduper.net> wrote:
>
>     Awesome to hear that you are turning this on both upstream and downstream. Do you know if the wifi stacks in your home routers also have AQM?
>
>     Simon
>
>
>     > On Jul 30, 2021, at 10:28 PM, Livingood, Jason via Bloat <bloat@lists.bufferbloat.net> wrote:
>     >
>     > FYI that I will be presenting a lightning talk at the IRTF MAPRG meeting today at 17:30 ET (https://urldefense.com/v3/__https://datatracker.ietf.org/meeting/111/materials/agenda-111-maprg__;!!CQl3mcHX2A!XLFMYPw-gnJgHzz_1nF-N7dNeIeT4QD-5wQny8vdAfYE6bzHtVQD3-lqiQI9YwAncrZUew$ ). The talk links to a just-published paper at https://urldefense.com/v3/__https://arxiv.org/abs/2107.13968__;!!CQl3mcHX2A!XLFMYPw-gnJgHzz_1nF-N7dNeIeT4QD-5wQny8vdAfYE6bzHtVQD3-lqiQI9YwCePfNyng$  (click PDF link in upper right of page) that will likely be of interest to these two lists.
>     >
>     > High-level: turning on AQM in the cable modem (upstream queue) took working latency from around 250 ms to between 15-30 ms, which is actually kind of cool. ;-) AQM is turned on in all of our CMTSes (downstream queue) and in DOCSIS 3.1 modems (upstream queue).
>     >
>     > Have a nice weekend,
>     > Jason
>     >
>     > _______________________________________________
>     > Bloat mailing list
>     > Bloat@lists.bufferbloat.net
>     > https://urldefense.com/v3/__https://lists.bufferbloat.net/listinfo/bloat__;!!CQl3mcHX2A!XLFMYPw-gnJgHzz_1nF-N7dNeIeT4QD-5wQny8vdAfYE6bzHtVQD3-lqiQI9YwCIw2ZGww$
>
>
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
>
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink



--
Fixing Starlink's Latencies: https://www.youtube.com/watch?v=c9gLo6Xrwgw

Dave Täht CEO, TekLibre, LLC

Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

[Frank Carmickle]

The Turris, unless I'm missing something, does not do 10g.

--FC

On Aug 4, 2021, at 8:24 PM, Michael Richardson <mcr@sandelman.ca> wrote:
> 
> 
> Jonathan Bennett <jonathanbennett@hackaday.com> wrote:
>> Isn't that what CZnic has done with the Turris router?
>> 
>> https://www.turris.com/en/mox/overview/
> 
>> I hadn't seen the Mox, that is clever. The downside is the price, and that
>> they're hard to get outside Europe.
> 
> Yup.
> The MOX is very nice, but maybe a bit too modular... :-)
> (but: they had their reasons)
> I found it difficult to figure out where to put the antennae.
> 
> CIRALabs got several dozen for the SecureHomeRouter project, but the pandemic
> kept us from getting them into alpha testing.   I think we bought direct.
> 
> --
> ]               Never tell me the odds!                 | ipv6 mesh networks [
> ]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
> ]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

[Michael Richardson]

[-- Attachment #1.1: Type: text/plain, Size: 602 bytes --]


Jonathan Bennett <jonathanbennett@hackaday.com> wrote:
    > Isn't that what CZnic has done with the Turris router?
    >
    > https://www.turris.com/en/mox/overview/

    > I hadn't seen the Mox, that is clever. The downside is the price, and that
    > they're hard to get outside Europe.

Yup.
The MOX is very nice, but maybe a bit too modular... :-)
(but: they had their reasons)
I found it difficult to figure out where to put the antennae.

CIRALabs got several dozen for the SecureHomeRouter project, but the pandemic
kept us from getting them into alpha testing.   I think we bought direct.


[-- Attachment #1.2: Signature --]
[-- Type: text/plain, Size: 241 bytes --]

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

[Sebastian Moeller]

Hi Jonathan,

> On Aug 4, 2021, at 20:28, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
> I firmly believe this is due to an I/O bottleneck in the SoC between
> the network complex and the CPU complex, not due to any limitation of
> the CPU itself.  It stems from the reliance on accelerated forwarding
> hardware to achieve full line-rate throughput.  Even so, I'd much
> rather have 40Mbps with Cake than 400Mbps with a dumb FIFO.  (Heck,
> 40Mbps would be a big upgrade from what I currently have.)  I think
> some of the newer Atheros chipsets are less constrained in this
> respect.
> 
> There are two reasonably good solutions to this problem in the hands
> of the SoC vendors:
> 
> 1: Relieve that I/O bottleneck, so that the CPU can handle packets at
> full line rate.  I assume this is not hugely complicated to implement,
> and just requires a sufficient degree of will to select the right
> option from the upstream fabless IP vendor's design library.
> 
> 2: Implement good shaping, FQ, and AQM within the network complex.  At
> consumer broadband/LAN speeds, this shouldn't be too difficult (unlike
> doing the same at 100+ Gbps), but it does require a significant amount
> of hardware design and validation, and that tends to have long lead
> times.
> 
> There is a third solution in the hands of us mere mortals:
> 
> 3: Leverage the Raspberry Pi ecosystem to build a CPE device that
> meets our needs.  This could be a Compute Module 4 (which has the
> necessary I/O throughput) mounted on a custom PCB that provides
> additional Ethernet ports and some reasonable Wifi AP.  It could
> alternatively be a standard Pi 4B with some USB Ethernet and Wifi
> hardware plugged into it.  Either will do the job withhout any
> Ethernet bottlenecks, although the capabilities of USB Wifi dongles
> are usually quite limited.

Have a look at https://www.dfrobot.com/product-2242.html, which is a small carrier for the raspberry pi4 compute module that offers two gigabit ethernet ports, the one from the CM and an additional RTL81111 one connected via the PCIe lanes.

At $45 it is a bit pricy, but it sure is small and elegant.

People on the OpenWrt Forum report traffic shaping at 1Gbps rates without overloading the CPUs (this needs minimal configuration for receive side packet steering, otherwise all network IRQ and qdisc processing sticks to CPU0, in which case shaping does not reach Gbps speeds, at least not for bi-directonal traffic).

That still leaves the need for an AP and a switch (one might be able to "abuse" an AP for switch ports if in a pinch).

Regards
	Sebastian




> 
> - Jonathan Morton

Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

[Jonathan Bennett]

[-- Attachment #1: Type: text/plain, Size: 1774 bytes --]

On Wed, Aug 4, 2021 at 3:18 PM Nathan Owens <nathan@nathan.io> wrote:

> Isn't that what CZnic has done with the Turris router?
>
> https://www.turris.com/en/mox/overview/
>
> I hadn't seen the Mox, that is clever. The downside is the price, and that
they're hard to get outside Europe.



> On Wed, Aug 4, 2021 at 1:08 PM Jonathan Bennett <
> jonathanbennett@hackaday.com> wrote:
>
>>
>>
>>
>>>
>>>
>>> The Compute Module 4 exposes the same integrated Ethernet port, and a
>>> PCIe lane in place of the USB 3 chipset (the latter being attached to
>>> the former in the standard Pi 4B).  This obviously allows attaching at
>>> least one real GigE port (with a free choice of PCIe-based chipset) at
>>> full line rate, without the intermediate step of USB.  I think it
>>> would be reasonable to include a small Ethernet switch downstream of
>>> this, matching the connectivity of typical CPE on the LAN side.  If a
>>> PCIe switch is inserted, then a choice of Mini-PCIe Wifi cards can be
>>> installed, with cables running to the normal array of external
>>> antennae, sidestepping the problem of USB Wifi dongles.
>>>
>>
>> I would hype the heck out of a router-style carrier board.  I'd even buy
>> a bunch myself, for that matter. I used to make good money by putting
>> OpenWRT on cheap routers for small businesses.
>>
>> I've had good success chatting with Eben about Hackaday articles. If this
>> were to become more than a pipe dream, we could reach out to him, to see if
>> the Pi Foundation had any interest in backing a Pi router carrier board
>> that beats bufferbloat.
>>
>> --Jonathan Bennett
>> _______________________________________________
>> Starlink mailing list
>> Starlink@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/starlink
>>
>

[-- Attachment #2: Type: text/html, Size: 3100 bytes --]

Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

[David Lang]

On Wed, 4 Aug 2021, Jonathan Morton wrote:

> 3: Leverage the Raspberry Pi ecosystem to build a CPE device that
> meets our needs.  This could be a Compute Module 4 (which has the
> necessary I/O throughput) mounted on a custom PCB that provides
> additional Ethernet ports and some reasonable Wifi AP.  It could
> alternatively be a standard Pi 4B with some USB Ethernet and Wifi
> hardware plugged into it.  Either will do the job withhout any
> Ethernet bottlenecks, although the capabilities of USB Wifi dongles
> are usually quite limited.

I'd buy it, you could also have a carrier board with some sockets that can take 
the wifi modules that laptops use.

David Lang

Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

[Nathan Owens]

[-- Attachment #1: Type: text/plain, Size: 1542 bytes --]

Isn't that what CZnic has done with the Turris router?

https://www.turris.com/en/mox/overview/

On Wed, Aug 4, 2021 at 1:08 PM Jonathan Bennett <
jonathanbennett@hackaday.com> wrote:

>
>
>
>>
>>
>> The Compute Module 4 exposes the same integrated Ethernet port, and a
>> PCIe lane in place of the USB 3 chipset (the latter being attached to
>> the former in the standard Pi 4B).  This obviously allows attaching at
>> least one real GigE port (with a free choice of PCIe-based chipset) at
>> full line rate, without the intermediate step of USB.  I think it
>> would be reasonable to include a small Ethernet switch downstream of
>> this, matching the connectivity of typical CPE on the LAN side.  If a
>> PCIe switch is inserted, then a choice of Mini-PCIe Wifi cards can be
>> installed, with cables running to the normal array of external
>> antennae, sidestepping the problem of USB Wifi dongles.
>>
>
> I would hype the heck out of a router-style carrier board.  I'd even buy a
> bunch myself, for that matter. I used to make good money by putting OpenWRT
> on cheap routers for small businesses.
>
> I've had good success chatting with Eben about Hackaday articles. If this
> were to become more than a pipe dream, we could reach out to him, to see if
> the Pi Foundation had any interest in backing a Pi router carrier board
> that beats bufferbloat.
>
> --Jonathan Bennett
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
>

[-- Attachment #2: Type: text/html, Size: 2403 bytes --]

Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

[Jonathan Bennett]

[-- Attachment #1: Type: text/plain, Size: 1160 bytes --]

>
>
> The Compute Module 4 exposes the same integrated Ethernet port, and a
> PCIe lane in place of the USB 3 chipset (the latter being attached to
> the former in the standard Pi 4B).  This obviously allows attaching at
> least one real GigE port (with a free choice of PCIe-based chipset) at
> full line rate, without the intermediate step of USB.  I think it
> would be reasonable to include a small Ethernet switch downstream of
> this, matching the connectivity of typical CPE on the LAN side.  If a
> PCIe switch is inserted, then a choice of Mini-PCIe Wifi cards can be
> installed, with cables running to the normal array of external
> antennae, sidestepping the problem of USB Wifi dongles.
>

I would hype the heck out of a router-style carrier board.  I'd even buy a
bunch myself, for that matter. I used to make good money by putting OpenWRT
on cheap routers for small businesses.

I've had good success chatting with Eben about Hackaday articles. If this
were to become more than a pipe dream, we could reach out to him, to see if
the Pi Foundation had any interest in backing a Pi router carrier board
that beats bufferbloat.

--Jonathan Bennett

[-- Attachment #2: Type: text/html, Size: 1501 bytes --]

Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

[Jonathan Morton]

On Wed, 4 Aug 2021 at 21:31, Juliusz Chroboczek <jch@irif.fr> wrote:
> A Cortex-A53 SoC at 1GHz with correctly designed Ethernet (i.e. not the
> Raspberry Pi) can push 1Gbit from userspace without breaking a sweat.

That was true of the earlier Raspberry Pis (eg. the Pi 3 uses a brace
of Cortex-A53s) which use Ethernet chipsets attached over USB 2, but
the Pi 4B has a directly integrated Ethernet port and two of the
external USB ports are USB 3, giving enough bandwidth to attach a
second GigE port.  We have tested this in practice, and got full line
rate throughput through Cake (though the CPU usage went up fairly
sharply after about halfway).

The Compute Module 4 exposes the same integrated Ethernet port, and a
PCIe lane in place of the USB 3 chipset (the latter being attached to
the former in the standard Pi 4B).  This obviously allows attaching at
least one real GigE port (with a free choice of PCIe-based chipset) at
full line rate, without the intermediate step of USB.  I think it
would be reasonable to include a small Ethernet switch downstream of
this, matching the connectivity of typical CPE on the LAN side.  If a
PCIe switch is inserted, then a choice of Mini-PCIe Wifi cards can be
installed, with cables running to the normal array of external
antennae, sidestepping the problem of USB Wifi dongles.

Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

[Juliusz Chroboczek]

Hi Mikael,

> If it's not hw accelerated, it sucks.

Fortunately, that hasn't been true in a long time.  Two data points.

The WND3700v2/WNDR3800, which is now over ten years old, can easily
forward 400Mbit/s NATed IPv4 (max-sized packets) in software.  To be fair,
it can saturate 1Gbit/s with hardware offload.

A Cortex-A53 SoC at 1GHz with correctly designed Ethernet (i.e. not the
Raspberry Pi) can push 1Gbit from userspace without breaking a sweat.

I agree with you that hardware acceleration has a number of advantages,
but given the speed of modern embedded chips, there is no longer much
reason to give up the flexibility and ease of development of pure software
solutions just to save a couple hundred mW, at least not until we get
10Gbit/s CPEs.

-- Juliusz

Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

[Jonathan Morton]

I firmly believe this is due to an I/O bottleneck in the SoC between
the network complex and the CPU complex, not due to any limitation of
the CPU itself.  It stems from the reliance on accelerated forwarding
hardware to achieve full line-rate throughput.  Even so, I'd much
rather have 40Mbps with Cake than 400Mbps with a dumb FIFO.  (Heck,
40Mbps would be a big upgrade from what I currently have.)  I think
some of the newer Atheros chipsets are less constrained in this
respect.

There are two reasonably good solutions to this problem in the hands
of the SoC vendors:

1: Relieve that I/O bottleneck, so that the CPU can handle packets at
full line rate.  I assume this is not hugely complicated to implement,
and just requires a sufficient degree of will to select the right
option from the upstream fabless IP vendor's design library.

2: Implement good shaping, FQ, and AQM within the network complex.  At
consumer broadband/LAN speeds, this shouldn't be too difficult (unlike
doing the same at 100+ Gbps), but it does require a significant amount
of hardware design and validation, and that tends to have long lead
times.

There is a third solution in the hands of us mere mortals:

3: Leverage the Raspberry Pi ecosystem to build a CPE device that
meets our needs.  This could be a Compute Module 4 (which has the
necessary I/O throughput) mounted on a custom PCB that provides
additional Ethernet ports and some reasonable Wifi AP.  It could
alternatively be a standard Pi 4B with some USB Ethernet and Wifi
hardware plugged into it.  Either will do the job withhout any
Ethernet bottlenecks, although the capabilities of USB Wifi dongles
are usually quite limited.

 - Jonathan Morton

Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

[Sebastian Moeller]

Hi Mikael,



> On Aug 4, 2021, at 15:06, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> 
> On Wed, 4 Aug 2021, Sebastian Moeller wrote:
> 
>> I guess the point is AQM is not really that expensive, even FQ AQM, traffic shaping however is expensive. But for wifi shaping is not required so AQM became feasible.
> 
> My point is that CPU based forwarding has very bad performance on some platforms, regardless if you're doing shaping, AQM or none of them (FIFO).
> 
> If it's not hw accelerated, it sucks.
> 
> When I did tests on MT7621 it did ~100 meg/s without flow-offload, and full gig with it.

	Reminds me of the joke of the gent sitting in a pub boasting the size of his property being so immense it takes a full week to ride around the perimeter...
To which another guest responds wryly, he used to have a horse that slow in the past as well....
	Point being a number of commercially sold CPE are not fit for the job of doing anything at Gbps link rates... sure accelerators can be nice, but they all come at the expense of reduced generality... Give that a raspberry Pi 4B has no trouble doing traffic shaping a 1Gbps (plus firewalling, NAT, PPPoE, ... and all of this in "software" with full generality) there is little excuse of still selling deploying CPE that were a decent fit for DSL link rates... But I live in a rich country where ISP take 2-5 EUR rent per month for CPE, so money is not a reasonable excuse for an ISP over here to cut corners.... 

Regards
	Sebastian



> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

[Mikael Abrahamsson]

On Wed, 4 Aug 2021, Sebastian Moeller wrote:

> I guess the point is AQM is not really that expensive, even FQ AQM, 
> traffic shaping however is expensive. But for wifi shaping is not 
> required so AQM became feasible.

My point is that CPU based forwarding has very bad performance on some 
platforms, regardless if you're doing shaping, AQM or none of them (FIFO).

If it's not hw accelerated, it sucks.

When I did tests on MT7621 it did ~100 meg/s without flow-offload, and 
full gig with it.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

[Sebastian Moeller]

[-- Attachment #1: Type: text/plain, Size: 1367 bytes --]

I guess the point is AQM is not really that expensive, even FQ AQM, traffic shaping however is expensive. But for wifi shaping is not required so AQM became feasible.

Regards
        Sebastian

On 4 August 2021 14:46:30 CEST, Mikael Abrahamsson via Bloat <bloat@lists.bufferbloat.net> wrote:
>On Wed, 4 Aug 2021, Jonathan Morton wrote:
>
>> Linux-based CPE devices have AQM functionality integrated into the Wifi 
>> stack.  The AQM itself operates at layer 3, but the Linux Wifi stack 
>> implementation uses information from layers 2 and 4 to improve 
>> scheduling decisions, eg. airtime-fairness and flow-isolation (FQ). This 
>> works best on soft-MAC Wifi hardware, such as ath9k/10k and MT76, where 
>> this information is most readily available to software.  In principle it 
>> could also be implemented in the MAC, but I don't know of any vendor 
>> that's done that yet.
>
>Does this work also with flowoffload enabled, or is that not accelerated 
>on for instance MT76? I'm surprised since MT76 can barely do 100 meg/s of 
>large packets using only CPU?
>
>-- 
>Mikael Abrahamsson    email: swmike@swm.pp.se
>_______________________________________________
>Bloat mailing list
>Bloat@lists.bufferbloat.net
>https://lists.bufferbloat.net/listinfo/bloat

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 1889 bytes --]

Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

[Mikael Abrahamsson]

On Wed, 4 Aug 2021, Jonathan Morton wrote:

> Linux-based CPE devices have AQM functionality integrated into the Wifi 
> stack.  The AQM itself operates at layer 3, but the Linux Wifi stack 
> implementation uses information from layers 2 and 4 to improve 
> scheduling decisions, eg. airtime-fairness and flow-isolation (FQ). This 
> works best on soft-MAC Wifi hardware, such as ath9k/10k and MT76, where 
> this information is most readily available to software.  In principle it 
> could also be implemented in the MAC, but I don't know of any vendor 
> that's done that yet.

Does this work also with flowoffload enabled, or is that not accelerated 
on for instance MT76? I'm surprised since MT76 can barely do 100 meg/s of 
large packets using only CPU?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

[Jonathan Morton]

> I assume by WiFi what is really meant is devices that have at least one WiFi (layer 1/layer 2) interface. While there are queues in the MAC sublayer, there is really no queue management functionality ... yet ... AFAIK. I know IEEE P802.11bd in conjunction w/ IEEE 1609 is working on implementing a few rudimentary queue mgmt functions.
>
> That said, seems any AQM in such devices would more than likely be in layer 3 and above.

Linux-based CPE devices have AQM functionality integrated into the
Wifi stack.  The AQM itself operates at layer 3, but the Linux Wifi
stack implementation uses information from layers 2 and 4 to improve
scheduling decisions, eg. airtime-fairness and flow-isolation (FQ).
This works best on soft-MAC Wifi hardware, such as ath9k/10k and MT76,
where this information is most readily available to software.  In
principle it could also be implemented in the MAC, but I don't know of
any vendor that's done that yet.

 - Jonathan Morton

Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

[Dick Roy]

I assume by WiFi what is really meant is devices that have at least one WiFi (layer 1/layer 2) interface. While there are queues in the MAC sublayer, there is really no queue management functionality ... yet ... AFAIK. I know IEEE P802.11bd in conjunction w/ IEEE 1609 is working on implementing a few rudimentary queue mgmt functions.  

That said, seems any AQM in such devices would more than likely be in layer 3 and above.  

RR 

-----Original Message-----
From: Starlink [mailto:starlink-bounces@lists.bufferbloat.net] On Behalf Of Livingood, Jason
Sent: Sunday, August 1, 2021 1:20 PM
To: Simon Barber
Cc: starlink@lists.bufferbloat.net; bloat
Subject: Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

WiFi is a different challenge as you know. In this case it varies depending on the radio chipset vendor and is on my list of things to work on...

JL

On 7/31/21, 13:50, "Simon Barber" <simon@superduper.net> wrote:

    Awesome to hear that you are turning this on both upstream and downstream. Do you know if the wifi stacks in your home routers also have AQM?

    Simon


    > On Jul 30, 2021, at 10:28 PM, Livingood, Jason via Bloat <bloat@lists.bufferbloat.net> wrote:
    >
    > FYI that I will be presenting a lightning talk at the IRTF MAPRG meeting today at 17:30 ET (https://urldefense.com/v3/__https://datatracker.ietf.org/meeting/111/materials/agenda-111-maprg__;!!CQl3mcHX2A!XLFMYPw-gnJgHzz_1nF-N7dNeIeT4QD-5wQny8vdAfYE6bzHtVQD3-lqiQI9YwAncrZUew$ ). The talk links to a just-published paper at https://urldefense.com/v3/__https://arxiv.org/abs/2107.13968__;!!CQl3mcHX2A!XLFMYPw-gnJgHzz_1nF-N7dNeIeT4QD-5wQny8vdAfYE6bzHtVQD3-lqiQI9YwCePfNyng$  (click PDF link in upper right of page) that will likely be of interest to these two lists.
    >
    > High-level: turning on AQM in the cable modem (upstream queue) took working latency from around 250 ms to between 15-30 ms, which is actually kind of cool. ;-) AQM is turned on in all of our CMTSes (downstream queue) and in DOCSIS 3.1 modems (upstream queue).
    >
    > Have a nice weekend,
    > Jason
    >
    > _______________________________________________
    > Bloat mailing list
    > Bloat@lists.bufferbloat.net
    > https://urldefense.com/v3/__https://lists.bufferbloat.net/listinfo/bloat__;!!CQl3mcHX2A!XLFMYPw-gnJgHzz_1nF-N7dNeIeT4QD-5wQny8vdAfYE6bzHtVQD3-lqiQI9YwCIw2ZGww$


_______________________________________________
Starlink mailing list
Starlink@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink

Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

[Livingood, Jason]

[-- Attachment #1: Type: text/plain, Size: 2828 bytes --]

Should be for COAM and leased. If you are having issues – you can email me 1:1 at this work address. Please include your modem make/model and account # so I can investigate. :-)

JL

From: Aaron Wood <woody77@gmail.com>
Date: Saturday, July 31, 2021 at 15:27
To: Simon Barber <simon@superduper.net>
Cc: "Livingood, Jason" <Jason_Livingood@cable.comcast.com>, "starlink@lists.bufferbloat.net" <starlink@lists.bufferbloat.net>, bloat <bloat@lists.bufferbloat.net>
Subject: [EXTERNAL] Re: [Bloat] Of interest: Comcast AQM Paper

If we see that AQM appears to be not functioning as expected for upstream connections on DOCSIS3.1, what's the right avenue for getting that resolved?  (and does that only apply to the Comcast-owned, vs. customer-owned, modems?)

On Sat, Jul 31, 2021 at 10:50 AM Simon Barber <simon@superduper.net<mailto:simon@superduper.net>> wrote:
Awesome to hear that you are turning this on both upstream and downstream. Do you know if the wifi stacks in your home routers also have AQM?

Simon


> On Jul 30, 2021, at 10:28 PM, Livingood, Jason via Bloat <bloat@lists.bufferbloat.net<mailto:bloat@lists.bufferbloat.net>> wrote:
>
> FYI that I will be presenting a lightning talk at the IRTF MAPRG meeting today at 17:30 ET (https://datatracker.ietf.org/meeting/111/materials/agenda-111-maprg<https://urldefense.com/v3/__https:/datatracker.ietf.org/meeting/111/materials/agenda-111-maprg__;!!CQl3mcHX2A!Rz15R4ZotXRWgIwMICwg0hP8rUn_zYmu7Xj_ujBrgBBYt5p7D8wGY4Toi7StQUQRf1LEmw$>). The talk links to a just-published paper at https://arxiv.org/abs/2107.13968<https://urldefense.com/v3/__https:/arxiv.org/abs/2107.13968__;!!CQl3mcHX2A!Rz15R4ZotXRWgIwMICwg0hP8rUn_zYmu7Xj_ujBrgBBYt5p7D8wGY4Toi7StQUSKQx-diA$> (click PDF link in upper right of page) that will likely be of interest to these two lists.
>
> High-level: turning on AQM in the cable modem (upstream queue) took working latency from around 250 ms to between 15-30 ms, which is actually kind of cool. ;-) AQM is turned on in all of our CMTSes (downstream queue) and in DOCSIS 3.1 modems (upstream queue).
>
> Have a nice weekend,
> Jason
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net<mailto:Bloat@lists.bufferbloat.net>
> https://lists.bufferbloat.net/listinfo/bloat<https://urldefense.com/v3/__https:/lists.bufferbloat.net/listinfo/bloat__;!!CQl3mcHX2A!Rz15R4ZotXRWgIwMICwg0hP8rUn_zYmu7Xj_ujBrgBBYt5p7D8wGY4Toi7StQUQ2nBiSiA$>

_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net<mailto:Bloat@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/bloat<https://urldefense.com/v3/__https:/lists.bufferbloat.net/listinfo/bloat__;!!CQl3mcHX2A!Rz15R4ZotXRWgIwMICwg0hP8rUn_zYmu7Xj_ujBrgBBYt5p7D8wGY4Toi7StQUQ2nBiSiA$>

[-- Attachment #2: Type: text/html, Size: 5542 bytes --]

Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper

[Livingood, Jason]

WiFi is a different challenge as you know. In this case it varies depending on the radio chipset vendor and is on my list of things to work on...

JL

On 7/31/21, 13:50, "Simon Barber" <simon@superduper.net> wrote:

    Awesome to hear that you are turning this on both upstream and downstream. Do you know if the wifi stacks in your home routers also have AQM?

    Simon


    > On Jul 30, 2021, at 10:28 PM, Livingood, Jason via Bloat <bloat@lists.bufferbloat.net> wrote:
    >
    > FYI that I will be presenting a lightning talk at the IRTF MAPRG meeting today at 17:30 ET (https://urldefense.com/v3/__https://datatracker.ietf.org/meeting/111/materials/agenda-111-maprg__;!!CQl3mcHX2A!XLFMYPw-gnJgHzz_1nF-N7dNeIeT4QD-5wQny8vdAfYE6bzHtVQD3-lqiQI9YwAncrZUew$ ). The talk links to a just-published paper at https://urldefense.com/v3/__https://arxiv.org/abs/2107.13968__;!!CQl3mcHX2A!XLFMYPw-gnJgHzz_1nF-N7dNeIeT4QD-5wQny8vdAfYE6bzHtVQD3-lqiQI9YwCePfNyng$  (click PDF link in upper right of page) that will likely be of interest to these two lists.
    >
    > High-level: turning on AQM in the cable modem (upstream queue) took working latency from around 250 ms to between 15-30 ms, which is actually kind of cool. ;-) AQM is turned on in all of our CMTSes (downstream queue) and in DOCSIS 3.1 modems (upstream queue).
    >
    > Have a nice weekend,
    > Jason
    >
    > _______________________________________________
    > Bloat mailing list
    > Bloat@lists.bufferbloat.net
    > https://urldefense.com/v3/__https://lists.bufferbloat.net/listinfo/bloat__;!!CQl3mcHX2A!XLFMYPw-gnJgHzz_1nF-N7dNeIeT4QD-5wQny8vdAfYE6bzHtVQD3-lqiQI9YwCIw2ZGww$