[Cerowrt-devel] fq_codel is SEVEN years old today...

[Cerowrt-devel] fq_codel is SEVEN years old today...

[Rich Brown]

Folks,

If it feels like we've been at this for a long time, you're right. My calendar tells me that we've been using fq_codel for the last seven years.

https://lists.bufferbloat.net/pipermail/bloat/2014-May/001934.html

Let's all pat ourselves on the back for this good work!

Rich

Re: [Cerowrt-devel] fq_codel is SEVEN years old today...

[Valdis Klētnieks]

[-- Attachment #1: Type: text/plain, Size: 311 bytes --]

On Tue, 14 May 2019 08:16:06 -0400, Rich Brown said:

> Let's all pat ourselves on the back for this good work!

Do we have an estimate of what percent of connected devices
are actually using fq_codel or other modern anti-bloat methods?
I'm reasonably sure my TV, my PS3, and my PS4 are still
behind the curve.

[-- Attachment #2: Type: application/pgp-signature, Size: 832 bytes --]

Re: [Cerowrt-devel] fq_codel is SEVEN years old today...

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 2069 bytes --]


Well, of all the devices in my house (maybe 100), only the router attached to the cable modem (which is a 2x GigE Intel Linux board based on Fedora 29 server with sch_cake configured) is running fq_codel. And setting that up was a labor of love. But it works a charm for my asymmetric Gigabit cable service.
 
My home's backbone is 10 GigE fiber, so I suppose fq_codel would be helpful for devices that run on 1 GigE subnets like my 2 802.11ac access points when talking to my NAS's.
However, the 802.11ac access point high speed functionality doesn't seem to be supportable by LEDE. So what can I do? 
 
I suppose I could stick some little custom Intel Linux 2x GigE devices between access points and the 10 GigE backbone, and put fq_codel in there.
 
My point is, to get the primary benefit of bufferbloat reduction, one has to stick little Linux boxes everywhere, because fq_codel is not supported except via DIY hacking.
 
And indeed, 10 GigE->1 GigE buffering does affect storage access latency in bad ways.
 
We see the same problem in datacenter networks that have excessive buffering - a famous switch company backed by Andy Bechtolsheim is really problematic because they claim building up huge buffers is a "feature" not a bug.
-----Original Message-----
From: "Valdis Klētnieks" <valdis.kletnieks@vt.edu>
Sent: Tuesday, May 14, 2019 1:57pm
To: "Rich Brown" <richb.hanover@gmail.com>
Cc: "cerowrt-devel" <cerowrt-devel@lists.bufferbloat.net>, "bloat" <bloat@lists.bufferbloat.net>
Subject: Re: [Cerowrt-devel] fq_codel is SEVEN years old today...



_______________________________________________
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel
On Tue, 14 May 2019 08:16:06 -0400, Rich Brown said:

> Let's all pat ourselves on the back for this good work!

Do we have an estimate of what percent of connected devices
are actually using fq_codel or other modern anti-bloat methods?
I'm reasonably sure my TV, my PS3, and my PS4 are still
behind the curve.

[-- Attachment #2: Type: text/html, Size: 3645 bytes --]

Re: [Cerowrt-devel] fq_codel is SEVEN years old today...

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 4033 bytes --]


I wonder if an interesting project to design and pitch for CrowdSupply to fund would be a little board that packages sch_cake or something in the minimal hardware package that could sit between a 1 GigE symmetric port and either an asymmetric GigE or a symmetric 1 GigE connection into a 10 GigE switch.
The key point is that it needs to support wire-rate forwarding with small packets of Gigabit throughput. Ideally, it also supports a dnsmasq NAT and wireguard optionally.
 
I know a Celeron with 2 GB of RAM can easily do it (because that is what I use). We know (well that's what you guys tell me) that the dinky MIPS processors are underpowered to handle sch_cake at such packet rates. The Linksys and Netgear and TP-link guys seem to see no market at all for any such thing. But I see it as a useful jellybean device if it could be cheap and simple.
 
Could maybe design, produce, and sell this for $100? No one else seems to want to make such a thing. I could just barely design and implement the board and get it made, but to be honest I'm better at spec'ing and prototyping than making manufacturable hardware designs. I suspect I could find someone to do the PCB design, layout and parts selection as a project.
 
The idea for this hardware "product" is to decouple this buffer management from the WiFi compatibility and driver mess, and make it easy for people, maybe to demonstrate that it could be a great product. Forget designing the packaging, negotiating a sales channel, etc. Just do what is needed to make a few thousand for the CrowdSupply market.
 
Thoughts?
 
-----Original Message-----
From: "David P. Reed" <dpreed@deepplum.com>
Sent: Tuesday, May 14, 2019 2:38pm
To: "Valdis Klētnieks" <valdis.kletnieks@vt.edu>
Cc: "Rich Brown" <richb.hanover@gmail.com>, "cerowrt-devel" <cerowrt-devel@lists.bufferbloat.net>, "bloat" <bloat@lists.bufferbloat.net>
Subject: Re: [Cerowrt-devel] fq_codel is SEVEN years old today...



Well, of all the devices in my house (maybe 100), only the router attached to the cable modem (which is a 2x GigE Intel Linux board based on Fedora 29 server with sch_cake configured) is running fq_codel. And setting that up was a labor of love. But it works a charm for my asymmetric Gigabit cable service.
 
My home's backbone is 10 GigE fiber, so I suppose fq_codel would be helpful for devices that run on 1 GigE subnets like my 2 802.11ac access points when talking to my NAS's.
However, the 802.11ac access point high speed functionality doesn't seem to be supportable by LEDE. So what can I do? 
 
I suppose I could stick some little custom Intel Linux 2x GigE devices between access points and the 10 GigE backbone, and put fq_codel in there.
 
My point is, to get the primary benefit of bufferbloat reduction, one has to stick little Linux boxes everywhere, because fq_codel is not supported except via DIY hacking.
 
And indeed, 10 GigE->1 GigE buffering does affect storage access latency in bad ways.
 
We see the same problem in datacenter networks that have excessive buffering - a famous switch company backed by Andy Bechtolsheim is really problematic because they claim building up huge buffers is a "feature" not a bug.
-----Original Message-----
From: "Valdis Klētnieks" <valdis.kletnieks@vt.edu>
Sent: Tuesday, May 14, 2019 1:57pm
To: "Rich Brown" <richb.hanover@gmail.com>
Cc: "cerowrt-devel" <cerowrt-devel@lists.bufferbloat.net>, "bloat" <bloat@lists.bufferbloat.net>
Subject: Re: [Cerowrt-devel] fq_codel is SEVEN years old today...



_______________________________________________
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel
On Tue, 14 May 2019 08:16:06 -0400, Rich Brown said:

> Let's all pat ourselves on the back for this good work!

Do we have an estimate of what percent of connected devices
are actually using fq_codel or other modern anti-bloat methods?
I'm reasonably sure my TV, my PS3, and my PS4 are still
behind the curve.

[-- Attachment #2: Type: text/html, Size: 7229 bytes --]

Re: [Cerowrt-devel] [Bloat] fq_codel is SEVEN years old today...

[Toke Høiland-Jørgensen]

"David P. Reed" <dpreed@deepplum.com> writes:

> I wonder if an interesting project to design and pitch for CrowdSupply
> to fund would be a little board that packages sch_cake or something in
> the minimal hardware package that could sit between a 1 GigE symmetric
> port and either an asymmetric GigE or a symmetric 1 GigE connection
> into a 10 GigE switch. The key point is that it needs to support
> wire-rate forwarding with small packets of Gigabit throughput.
> Ideally, it also supports a dnsmasq NAT and wireguard optionally.
>  
> I know a Celeron with 2 GB of RAM can easily do it (because that is
> what I use). We know (well that's what you guys tell me) that the
> dinky MIPS processors are underpowered to handle sch_cake at such
> packet rates. The Linksys and Netgear and TP-link guys seem to see no
> market at all for any such thing. But I see it as a useful jellybean
> device if it could be cheap and simple.
>  
> Could maybe design, produce, and sell this for $100? No one else seems
> to want to make such a thing. I could just barely design and implement
> the board and get it made, but to be honest I'm better at spec'ing and
> prototyping than making manufacturable hardware designs. I suspect I
> could find someone to do the PCB design, layout and parts selection as
> a project.
>  
> The idea for this hardware "product" is to decouple this buffer
> management from the WiFi compatibility and driver mess, and make it
> easy for people, maybe to demonstrate that it could be a great
> product. Forget designing the packaging, negotiating a sales channel,
> etc. Just do what is needed to make a few thousand for the CrowdSupply
> market.
>  
> Thoughts?

It's a cool idea, and I'd certainly buy a couple to help the
crowdfunding ;)

Ideally, it would need to be self-configuring, though... I.e., something
like the IQRouter auto-measuring of the upstream bandwidth to tune the
shaper.

For reference, the GL.iNet routers are tiny and nicely packaged, and run
OpenWrt; they do have one with Gbit ports[0], priced around $70. I very
much doubt it can actually push a gigabit, though, but I haven't had a
chance to test it. However, losing the WiFi, and getting a slightly
beefier SoC in there will probably be doable without the price going
over $100, no?

-Toke

[0] https://www.gl-inet.com/products/gl-ar750s/

Re: [Cerowrt-devel] [Bloat] fq_codel is SEVEN years old today...

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 1310 bytes --]


 

Ideally, it would need to be self-configuring, though... I.e., something
like the IQRouter auto-measuring of the upstream bandwidth to tune the
shaper.
 
Sure, seems like this is easy to code because there are exactly two ports to measure, they can even be labeled physically "up" and "down" to indicate their function.

For reference, the GL.iNet routers are tiny and nicely packaged, and run
OpenWrt; they do have one with Gbit ports[0], priced around $70. I very
much doubt it can actually push a gigabit, though, but I haven't had a
chance to test it. However, losing the WiFi, and getting a slightly
beefier SoC in there will probably be doable without the price going
over $100, no?
 
I assume the WiFi silicon is probably the most costly piece of intellectual property in the system. So yeah. Maybe with the right parts being available, one could aim at $50 or less, without sales channel markup. (Raspberry Pi ARM64 boards don't have GigE, and I think that might be because the GigE interfaces are a bit pricey. However, the ARM64 SoC's available are typically Celeron-class multicore systems. I don't know why there aren't more ARM64 systems on a chip with dual GigE, but I suspect searching for them would turn up some).

-Toke

[0] https://www.gl-inet.com/products/gl-ar750s/

[-- Attachment #2: Type: text/html, Size: 2171 bytes --]

[Cerowrt-devel] (no subject)

[Sebastian Moeller]

Hi All,


I believe the following to be relevant to this discussion: https://apenwarr.ca/log/20180808
Where he discusses a similar idea including implementation albeit aimed at lower bandwidth and sans the automatic bandwidth tracking.


> On May 15, 2019, at 01:34, David P. Reed <dpreed@deepplum.com> wrote:
> 
> 
> Ideally, it would need to be self-configuring, though... I.e., something
> like the IQRouter auto-measuring of the upstream bandwidth to tune the
> shaper.

@Jonathan from your experience how tricky is it to get reliable speedtest endpoints and how reliable are they in practice. And do you do any sanitization, like take another measure immediate if the measured rate differs from the last by more than XX% or something like that?


> 
> Sure, seems like this is easy to code because there are exactly two ports to measure, they can even be labeled physically "up" and "down" to indicate their function.

IMHO the real challenge is automated measurements over the internet at Gbps speeds. It is not hard to get some test going (by e.g. tapping into ookla's fast net of confederated measurement endpoints) but getting something where the servers can reliably saturate 1Gbps+ seems somewhat trickier (last time I looked one required a 1Gbps connection to the server to participate in speedtest.net, obviously not really suited for measuring Gbps speeds).
In the EU there exists a mandate for national regulators to establish and/or endorse an anointed "official" speedtests, untended to keep ISP marketing honest, that come with stricter guarantees (e.g. the official German speedtest, breitbandmessung.de will only admit tests if the servers are having sufficient bandwidth reserves to actually saturate the link; the enduser is required to select the speed-tier giving them a strong hint about the required rates I believe).
For my back-burner toy project "per-packet-overhead estimation on arbitrary link technology" I am currently facing the same problem, I need a traffic sink and source that can reliably saturate my link so I can measure maximum achievable goodput, so if anybody in the list has ideas, I am all ears/eyes.

> 
> For reference, the GL.iNet routers are tiny and nicely packaged, and run
> OpenWrt; they do have one with Gbit ports[0], priced around $70. I very
> much doubt it can actually push a gigabit, though, but I haven't had a
> chance to test it. However, losing the WiFi, and getting a slightly
> beefier SoC in there will probably be doable without the price going
> over $100, no?
> 
> I assume the WiFi silicon is probably the most costly piece of intellectual property in the system. So yeah. Maybe with the right parts being available, one could aim at $50 or less, without sales channel markup. (Raspberry Pi ARM64 boards don't have GigE, and I think that might be because the GigE interfaces are a bit pricey. However, the ARM64 SoC's available are typically Celeron-class multicore systems. I don't know why there aren't more ARM64 systems on a chip with dual GigE, but I suspect searching for them would turn up some).

The turris MOX (https://www.turris.cz/en/specification/) might be a decent startimg point as it comes with one Gbethernet port and both a SGMII and a PCIe signals routed on a connector, they also have a 4 and an 8 port switch module, but for our purposes it might be possible to just create a small single Gb ethernet port board to get started. 

Best Regards
	Sebastian

> 
> -Toke
> 
> [0] https://www.gl-inet.com/products/gl-ar750s/
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] (no subject)

[Dave Taht]

If it helps any: Nick Feamster and Jason Livingood just published "
Internet Speed Measurement: Current Challenges and Future
Recommendations " ( https://arxiv.org/pdf/1905.02334.pdf ) a few days
ago, and outlines quite a few problems going forward at higher speeds.
I do wish the document had pointed out more clearly that router based
measurements have problems also, with weaker cpus unable to source
enough traffic for an accurate measurement, but I do hope this
document has impact, and it's a good read, regardless.

Still, somehow getting it right at lower speeds is always on my mind.
I'd long ago hoped that DSL devices would adopt BQL, and that
cablemodems would also, thus moving packet processing a little higher
on the stack so more advanced algorithms like cake could take hold.

On Wed, May 15, 2019 at 9:32 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>
> Hi All,
>
>
> I believe the following to be relevant to this discussion: https://apenwarr.ca/log/20180808
> Where he discusses a similar idea including implementation albeit aimed at lower bandwidth and sans the automatic bandwidth tracking.
>
>
> > On May 15, 2019, at 01:34, David P. Reed <dpreed@deepplum.com> wrote:
> >
> >
> > Ideally, it would need to be self-configuring, though... I.e., something
> > like the IQRouter auto-measuring of the upstream bandwidth to tune the
> > shaper.
>
> @Jonathan from your experience how tricky is it to get reliable speedtest endpoints and how reliable are they in practice. And do you do any sanitization, like take another measure immediate if the measured rate differs from the last by more than XX% or something like that?
>
>
> >
> > Sure, seems like this is easy to code because there are exactly two ports to measure, they can even be labeled physically "up" and "down" to indicate their function.
>
> IMHO the real challenge is automated measurements over the internet at Gbps speeds. It is not hard to get some test going (by e.g. tapping into ookla's fast net of confederated measurement endpoints) but getting something where the servers can reliably saturate 1Gbps+ seems somewhat trickier (last time I looked one required a 1Gbps connection to the server to participate in speedtest.net, obviously not really suited for measuring Gbps speeds).
> In the EU there exists a mandate for national regulators to establish and/or endorse an anointed "official" speedtests, untended to keep ISP marketing honest, that come with stricter guarantees (e.g. the official German speedtest, breitbandmessung.de will only admit tests if the servers are having sufficient bandwidth reserves to actually saturate the link; the enduser is required to select the speed-tier giving them a strong hint about the required rates I believe).
> For my back-burner toy project "per-packet-overhead estimation on arbitrary link technology" I am currently facing the same problem, I need a traffic sink and source that can reliably saturate my link so I can measure maximum achievable goodput, so if anybody in the list has ideas, I am all ears/eyes.
>
> >
> > For reference, the GL.iNet routers are tiny and nicely packaged, and run
> > OpenWrt; they do have one with Gbit ports[0], priced around $70. I very
> > much doubt it can actually push a gigabit, though, but I haven't had a
> > chance to test it. However, losing the WiFi, and getting a slightly
> > beefier SoC in there will probably be doable without the price going
> > over $100, no?
> >
> > I assume the WiFi silicon is probably the most costly piece of intellectual property in the system. So yeah. Maybe with the right parts being available, one could aim at $50 or less, without sales channel markup. (Raspberry Pi ARM64 boards don't have GigE, and I think that might be because the GigE interfaces are a bit pricey. However, the ARM64 SoC's available are typically Celeron-class multicore systems. I don't know why there aren't more ARM64 systems on a chip with dual GigE, but I suspect searching for them would turn up some).
>
> The turris MOX (https://www.turris.cz/en/specification/) might be a decent startimg point as it comes with one Gbethernet port and both a SGMII and a PCIe signals routed on a connector, they also have a 4 and an 8 port switch module, but for our purposes it might be possible to just create a small single Gb ethernet port board to get started.
>
> Best Regards
>         Sebastian
>
> >
> > -Toke
> >
> > [0] https://www.gl-inet.com/products/gl-ar750s/
> > _______________________________________________
> > Cerowrt-devel mailing list
> > Cerowrt-devel@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Cerowrt-devel] [Bloat] (no subject)

[Sebastian Moeller]

Hi Dave,


> On May 15, 2019, at 09:58, Dave Taht <dave.taht@gmail.com> wrote:
> 
> If it helps any: Nick Feamster and Jason Livingood just published "
> Internet Speed Measurement: Current Challenges and Future
> Recommendations " ( https://arxiv.org/pdf/1905.02334.pdf ) a few days
> ago, and outlines quite a few problems going forward at higher speeds.

	Excellent, will read (once time permits).


> I do wish the document had pointed out more clearly that router based
> measurements have problems also,

	+1, especially newer SoCs which implement power features can run into situations where depending on the load characteristics the CPUs throttle back (in frequency) and perfomance becomes quite choppy, which might not show up in bandwidth measurement but can really affect the experienced bufferbloat. I currently think it better for the load generation to be done by  different hosts than the DUT/de-bloater...


> with weaker cpus unable to source
> enough traffic for an accurate measurement, but I do hope this
> document has impact, and it's a good read, regardless.
> 
> Still, somehow getting it right at lower speeds is always on my mind.

	At lower speeds measuring the bandwidth is not that hard, since overloading the CPU and the server;s bandwidth gets less likely, no?

> I'd long ago hoped that DSL devices would adopt BQL, and that
> cablemodems would also, thus moving packet processing a little higher
> on the stack so more advanced algorithms like cake could take hold.

	Well, to make this happen we need to talk to modem manufacturers, so mostly intel/lantiq and broadcom I believe. I happen to have exact zero insiders in both companies in my "rolodex", but with in a group as diverse and expert as this list is, someone here must know someone that could get the word out inside those companies, no?
	For BQL on docsis modems though, I believe cablelabs might be the best route (not that BQL on a docsis modem will help a lot, given that we have, AFAICT, no open source OS running on a docsis modem, same seems true for GPON).

Best Regards
	Sebastian

> 
> On Wed, May 15, 2019 at 9:32 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> Hi All,
>> 
>> 
>> I believe the following to be relevant to this discussion: https://apenwarr.ca/log/20180808
>> Where he discusses a similar idea including implementation albeit aimed at lower bandwidth and sans the automatic bandwidth tracking.
>> 
>> 
>>> On May 15, 2019, at 01:34, David P. Reed <dpreed@deepplum.com> wrote:
>>> 
>>> 
>>> Ideally, it would need to be self-configuring, though... I.e., something
>>> like the IQRouter auto-measuring of the upstream bandwidth to tune the
>>> shaper.
>> 
>> @Jonathan from your experience how tricky is it to get reliable speedtest endpoints and how reliable are they in practice. And do you do any sanitization, like take another measure immediate if the measured rate differs from the last by more than XX% or something like that?
>> 
>> 
>>> 
>>> Sure, seems like this is easy to code because there are exactly two ports to measure, they can even be labeled physically "up" and "down" to indicate their function.
>> 
>> IMHO the real challenge is automated measurements over the internet at Gbps speeds. It is not hard to get some test going (by e.g. tapping into ookla's fast net of confederated measurement endpoints) but getting something where the servers can reliably saturate 1Gbps+ seems somewhat trickier (last time I looked one required a 1Gbps connection to the server to participate in speedtest.net, obviously not really suited for measuring Gbps speeds).
>> In the EU there exists a mandate for national regulators to establish and/or endorse an anointed "official" speedtests, untended to keep ISP marketing honest, that come with stricter guarantees (e.g. the official German speedtest, breitbandmessung.de will only admit tests if the servers are having sufficient bandwidth reserves to actually saturate the link; the enduser is required to select the speed-tier giving them a strong hint about the required rates I believe).
>> For my back-burner toy project "per-packet-overhead estimation on arbitrary link technology" I am currently facing the same problem, I need a traffic sink and source that can reliably saturate my link so I can measure maximum achievable goodput, so if anybody in the list has ideas, I am all ears/eyes.
>> 
>>> 
>>> For reference, the GL.iNet routers are tiny and nicely packaged, and run
>>> OpenWrt; they do have one with Gbit ports[0], priced around $70. I very
>>> much doubt it can actually push a gigabit, though, but I haven't had a
>>> chance to test it. However, losing the WiFi, and getting a slightly
>>> beefier SoC in there will probably be doable without the price going
>>> over $100, no?
>>> 
>>> I assume the WiFi silicon is probably the most costly piece of intellectual property in the system. So yeah. Maybe with the right parts being available, one could aim at $50 or less, without sales channel markup. (Raspberry Pi ARM64 boards don't have GigE, and I think that might be because the GigE interfaces are a bit pricey. However, the ARM64 SoC's available are typically Celeron-class multicore systems. I don't know why there aren't more ARM64 systems on a chip with dual GigE, but I suspect searching for them would turn up some).
>> 
>> The turris MOX (https://www.turris.cz/en/specification/) might be a decent startimg point as it comes with one Gbethernet port and both a SGMII and a PCIe signals routed on a connector, they also have a 4 and an 8 port switch module, but for our purposes it might be possible to just create a small single Gb ethernet port board to get started.
>> 
>> Best Regards
>>        Sebastian
>> 
>>> 
>>> -Toke
>>> 
>>> [0] https://www.gl-inet.com/products/gl-ar750s/
>>> _______________________________________________
>>> Cerowrt-devel mailing list
>>> Cerowrt-devel@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> 
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
> 
> 
> 
> -- 
> 
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740

Re: [Cerowrt-devel] [Bloat] (no subject)

[Jonathan Foulkes]

Thanks for sharing Dave.

A good paper, but there are few gaps worthy of mentioning on this list:

Testing when there is an AQM present means the test must adapt to the challenge of smaller cwnd existing for any one stream, therefore it will take many more streams to saturate a line with cwnd = 30 than if the cwnd is allowed to grow to >1,000
In general, the impact of cwnd relative to saturation and impact on delay was not visited, and yet it’s critical. One of the reasons for spiky delays on high speed lines is the ginormous cwnds hogging the line with their 800ms+ RTT’s

Asymmetry of provisioned upload relative to download, at some point, the ack-stream can be held up by either lack of capacity or bloat in the uplink. So even though a link can deliver 300Mbps down, a bloated uplink of 5mbps might never allow that level to be reached.
There are ISPs provisioning truly crazy asymmetric service.

They do make a good point about the local network, WiFi specifically being the new bottleneck, which is why we included an iperf instance that can be started on the IQrouter to help run client to server tests that help spot local network capacity limits, typically on WiFi.

Regarding their point about ‘Cross traffic’ impact on measurements, Cake’s per-host / per-target fairness also complicates AQM-enabled testing from client devices. Which is why we make the built-in speed test the arbiter of true line capacity, as it factors for ALL traffic flowing through the router. But, as you mention, that is also a challenge from a CPU resource standpoint on higher speeds.

The biggest gap in this paper is not paying sufficient attention to latency as a critical metric, and one that is controllable by an AQM. Bufferbloat metrics have more impact on end-user experience than +/- 50Mbps on a 100mbps baseline. 
I was rather miffed they do not even mention the DSLreports.com speedtest, or the fast.com test, as those are the two that provide a bufferbloat metric.

The industry as whole MUST pay attention and socialize the relevancy of managed latencies as being critical to customer satisfaction and good application performance. And that starts with tests that clearly grade that critical aspect. 

Cheers,

Jonathan

> On May 15, 2019, at 3:58 AM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> If it helps any: Nick Feamster and Jason Livingood just published "
> Internet Speed Measurement: Current Challenges and Future
> Recommendations " ( https://arxiv.org/pdf/1905.02334.pdf ) a few days
> ago, and outlines quite a few problems going forward at higher speeds.
> I do wish the document had pointed out more clearly that router based
> measurements have problems also, with weaker cpus unable to source
> enough traffic for an accurate measurement, but I do hope this
> document has impact, and it's a good read, regardless.
> 
> Still, somehow getting it right at lower speeds is always on my mind.
> I'd long ago hoped that DSL devices would adopt BQL, and that
> cablemodems would also, thus moving packet processing a little higher
> on the stack so more advanced algorithms like cake could take hold.
> 
> On Wed, May 15, 2019 at 9:32 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> Hi All,
>> 
>> 
>> I believe the following to be relevant to this discussion: https://apenwarr.ca/log/20180808
>> Where he discusses a similar idea including implementation albeit aimed at lower bandwidth and sans the automatic bandwidth tracking.
>> 
>> 
>>> On May 15, 2019, at 01:34, David P. Reed <dpreed@deepplum.com> wrote:
>>> 
>>> 
>>> Ideally, it would need to be self-configuring, though... I.e., something
>>> like the IQRouter auto-measuring of the upstream bandwidth to tune the
>>> shaper.
>> 
>> @Jonathan from your experience how tricky is it to get reliable speedtest endpoints and how reliable are they in practice. And do you do any sanitization, like take another measure immediate if the measured rate differs from the last by more than XX% or something like that?
>> 
>> 
>>> 
>>> Sure, seems like this is easy to code because there are exactly two ports to measure, they can even be labeled physically "up" and "down" to indicate their function.
>> 
>> IMHO the real challenge is automated measurements over the internet at Gbps speeds. It is not hard to get some test going (by e.g. tapping into ookla's fast net of confederated measurement endpoints) but getting something where the servers can reliably saturate 1Gbps+ seems somewhat trickier (last time I looked one required a 1Gbps connection to the server to participate in speedtest.net, obviously not really suited for measuring Gbps speeds).
>> In the EU there exists a mandate for national regulators to establish and/or endorse an anointed "official" speedtests, untended to keep ISP marketing honest, that come with stricter guarantees (e.g. the official German speedtest, breitbandmessung.de will only admit tests if the servers are having sufficient bandwidth reserves to actually saturate the link; the enduser is required to select the speed-tier giving them a strong hint about the required rates I believe).
>> For my back-burner toy project "per-packet-overhead estimation on arbitrary link technology" I am currently facing the same problem, I need a traffic sink and source that can reliably saturate my link so I can measure maximum achievable goodput, so if anybody in the list has ideas, I am all ears/eyes.
>> 
>>> 
>>> For reference, the GL.iNet routers are tiny and nicely packaged, and run
>>> OpenWrt; they do have one with Gbit ports[0], priced around $70. I very
>>> much doubt it can actually push a gigabit, though, but I haven't had a
>>> chance to test it. However, losing the WiFi, and getting a slightly
>>> beefier SoC in there will probably be doable without the price going
>>> over $100, no?
>>> 
>>> I assume the WiFi silicon is probably the most costly piece of intellectual property in the system. So yeah. Maybe with the right parts being available, one could aim at $50 or less, without sales channel markup. (Raspberry Pi ARM64 boards don't have GigE, and I think that might be because the GigE interfaces are a bit pricey. However, the ARM64 SoC's available are typically Celeron-class multicore systems. I don't know why there aren't more ARM64 systems on a chip with dual GigE, but I suspect searching for them would turn up some).
>> 
>> The turris MOX (https://www.turris.cz/en/specification/) might be a decent startimg point as it comes with one Gbethernet port and both a SGMII and a PCIe signals routed on a connector, they also have a 4 and an 8 port switch module, but for our purposes it might be possible to just create a small single Gb ethernet port board to get started.
>> 
>> Best Regards
>>        Sebastian
>> 
>>> 
>>> -Toke
>>> 
>>> [0] https://www.gl-inet.com/products/gl-ar750s/
>>> _______________________________________________
>>> Cerowrt-devel mailing list
>>> Cerowrt-devel@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> 
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
> 
> 
> 
> -- 
> 
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Cerowrt-devel] [Bloat] (no subject)

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 9346 bytes --]


Pardon, but cwnd should NEVER be larger than the number of forwarding hops between source and destination.
Kleinrock and students recently proved that the optimum cwnd for both throughput and minimized latency is achieved when there is one packet or less in each outbound queue from source to destination (including cross traffic - meaning other flows sharing the same outbound queue.
 
Now the idea that cwnd should be in the 1000's of packets is totally absurd, unless the source or destination buffers (at the end hosts) are counted, and that would be needed if the TCP source and destination application might, for example, be "swapped out" and thus unable to actually send and acknowledge packets at the instant of receiving an ACK.
 
If cwnd is sort of compensating for "swapping out" the TCP endpoint processes so that they take milliseconds to provide or acknowledge receipt of a packet, then that's fine (if you want throughput and terrible latency), but that's not the congestion window. That's just cramming the operating system's scheduling delay into the TCP stack.
 
TCP is not supposed to be designed around slow OS process schedulers. Those buffers should never be allowed to build up in the transport network, where they kill latency for everyone. That's just terrible design, conflating OS scheduling with congestion management.
 
 
On Thursday, May 16, 2019 6:01pm, "Jonathan Foulkes" <jf@jonathanfoulkes.com> said:



> Thanks for sharing Dave.
> 
> A good paper, but there are few gaps worthy of mentioning on this list:
> 
> Testing when there is an AQM present means the test must adapt to the challenge of
> smaller cwnd existing for any one stream, therefore it will take many more streams
> to saturate a line with cwnd = 30 than if the cwnd is allowed to grow to
> >1,000
> In general, the impact of cwnd relative to saturation and impact on delay was not
> visited, and yet it’s critical. One of the reasons for spiky delays on high
> speed lines is the ginormous cwnds hogging the line with their 800ms+ RTT’s
> 
> Asymmetry of provisioned upload relative to download, at some point, the
> ack-stream can be held up by either lack of capacity or bloat in the uplink. So
> even though a link can deliver 300Mbps down, a bloated uplink of 5mbps might never
> allow that level to be reached.
> There are ISPs provisioning truly crazy asymmetric service.
> 
> They do make a good point about the local network, WiFi specifically being the new
> bottleneck, which is why we included an iperf instance that can be started on the
> IQrouter to help run client to server tests that help spot local network capacity
> limits, typically on WiFi.
> 
> Regarding their point about ‘Cross traffic’ impact on measurements,
> Cake’s per-host / per-target fairness also complicates AQM-enabled testing
> from client devices. Which is why we make the built-in speed test the arbiter of
> true line capacity, as it factors for ALL traffic flowing through the router. But,
> as you mention, that is also a challenge from a CPU resource standpoint on higher
> speeds.
> 
> The biggest gap in this paper is not paying sufficient attention to latency as a
> critical metric, and one that is controllable by an AQM. Bufferbloat metrics have
> more impact on end-user experience than +/- 50Mbps on a 100mbps baseline.
> I was rather miffed they do not even mention the DSLreports.com speedtest, or the
> fast.com test, as those are the two that provide a bufferbloat metric.
> 
> The industry as whole MUST pay attention and socialize the relevancy of managed
> latencies as being critical to customer satisfaction and good application
> performance. And that starts with tests that clearly grade that critical aspect.
> 
> Cheers,
> 
> Jonathan
> 
> > On May 15, 2019, at 3:58 AM, Dave Taht <dave.taht@gmail.com> wrote:
> >
> > If it helps any: Nick Feamster and Jason Livingood just published "
> > Internet Speed Measurement: Current Challenges and Future
> > Recommendations " ( https://arxiv.org/pdf/1905.02334.pdf ) a few days
> > ago, and outlines quite a few problems going forward at higher speeds.
> > I do wish the document had pointed out more clearly that router based
> > measurements have problems also, with weaker cpus unable to source
> > enough traffic for an accurate measurement, but I do hope this
> > document has impact, and it's a good read, regardless.
> >
> > Still, somehow getting it right at lower speeds is always on my mind.
> > I'd long ago hoped that DSL devices would adopt BQL, and that
> > cablemodems would also, thus moving packet processing a little higher
> > on the stack so more advanced algorithms like cake could take hold.
> >
> > On Wed, May 15, 2019 at 9:32 AM Sebastian Moeller <moeller0@gmx.de>
> wrote:
> >>
> >> Hi All,
> >>
> >>
> >> I believe the following to be relevant to this discussion:
> https://apenwarr.ca/log/20180808
> >> Where he discusses a similar idea including implementation albeit aimed
> at lower bandwidth and sans the automatic bandwidth tracking.
> >>
> >>
> >>> On May 15, 2019, at 01:34, David P. Reed <dpreed@deepplum.com>
> wrote:
> >>>
> >>>
> >>> Ideally, it would need to be self-configuring, though... I.e.,
> something
> >>> like the IQRouter auto-measuring of the upstream bandwidth to tune
> the
> >>> shaper.
> >>
> >> @Jonathan from your experience how tricky is it to get reliable speedtest
> endpoints and how reliable are they in practice. And do you do any sanitization,
> like take another measure immediate if the measured rate differs from the last by
> more than XX% or something like that?
> >>
> >>
> >>>
> >>> Sure, seems like this is easy to code because there are exactly two
> ports to measure, they can even be labeled physically "up" and "down" to indicate
> their function.
> >>
> >> IMHO the real challenge is automated measurements over the internet at
> Gbps speeds. It is not hard to get some test going (by e.g. tapping into ookla's
> fast net of confederated measurement endpoints) but getting something where the
> servers can reliably saturate 1Gbps+ seems somewhat trickier (last time I looked
> one required a 1Gbps connection to the server to participate in speedtest.net,
> obviously not really suited for measuring Gbps speeds).
> >> In the EU there exists a mandate for national regulators to establish
> and/or endorse an anointed "official" speedtests, untended to keep ISP marketing
> honest, that come with stricter guarantees (e.g. the official German speedtest,
> breitbandmessung.de will only admit tests if the servers are having sufficient
> bandwidth reserves to actually saturate the link; the enduser is required to
> select the speed-tier giving them a strong hint about the required rates I
> believe).
> >> For my back-burner toy project "per-packet-overhead estimation on
> arbitrary link technology" I am currently facing the same problem, I need a
> traffic sink and source that can reliably saturate my link so I can measure
> maximum achievable goodput, so if anybody in the list has ideas, I am all
> ears/eyes.
> >>
> >>>
> >>> For reference, the GL.iNet routers are tiny and nicely packaged, and
> run
> >>> OpenWrt; they do have one with Gbit ports[0], priced around $70. I
> very
> >>> much doubt it can actually push a gigabit, though, but I haven't had
> a
> >>> chance to test it. However, losing the WiFi, and getting a slightly
> >>> beefier SoC in there will probably be doable without the price going
> >>> over $100, no?
> >>>
> >>> I assume the WiFi silicon is probably the most costly piece of
> intellectual property in the system. So yeah. Maybe with the right parts being
> available, one could aim at $50 or less, without sales channel markup. (Raspberry
> Pi ARM64 boards don't have GigE, and I think that might be because the GigE
> interfaces are a bit pricey. However, the ARM64 SoC's available are typically
> Celeron-class multicore systems. I don't know why there aren't more ARM64 systems
> on a chip with dual GigE, but I suspect searching for them would turn up some).
> >>
> >> The turris MOX (https://www.turris.cz/en/specification/) might be a
> decent startimg point as it comes with one Gbethernet port and both a SGMII and a
> PCIe signals routed on a connector, they also have a 4 and an 8 port switch
> module, but for our purposes it might be possible to just create a small single Gb
> ethernet port board to get started.
> >>
> >> Best Regards
> >> Sebastian
> >>
> >>>
> >>> -Toke
> >>>
> >>> [0] https://www.gl-inet.com/products/gl-ar750s/
> >>> _______________________________________________
> >>> Cerowrt-devel mailing list
> >>> Cerowrt-devel@lists.bufferbloat.net
> >>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> >>
> >> _______________________________________________
> >> Bloat mailing list
> >> Bloat@lists.bufferbloat.net
> >> https://lists.bufferbloat.net/listinfo/bloat
> >
> >
> >
> > --
> >
> > Dave Täht
> > CTO, TekLibre, LLC
> > http://www.teklibre.com
> > Tel: 1-831-205-9740
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
> 
> 

[-- Attachment #2: Type: text/html, Size: 12309 bytes --]

Re: [Cerowrt-devel] [Bloat] (no subject)

[Jonathan Morton]

> On 19 May, 2019, at 1:36 am, David P. Reed <dpreed@deepplum.com> wrote:
> 
> Pardon, but cwnd should NEVER be larger than the number of forwarding hops between source and destination.
> Kleinrock and students recently proved that the optimum cwnd for both throughput and minimized latency is achieved when there is one packet or less in each outbound queue from source to destination (including cross traffic - meaning other flows sharing the same outbound queue.

This argument holds only if time-of-flight *between* nodes is negligible.  Trivially, a geosynchronous satellite hop adds only two nodes but approximately half a second to the one-way path delay, with potentially thousands of packets existing only as radio waves in the distance between, not in a queue.

 - Jonathan Morton

Re: [Cerowrt-devel] [Bloat] (no subject)

[Jonathan Morton]

>> Pardon, but cwnd should NEVER be larger than the number of forwarding hops between source and destination.
>> Kleinrock and students recently proved that the optimum cwnd for both throughput and minimized latency is achieved when there is one packet or less in each outbound queue from source to destination (including cross traffic - meaning other flows sharing the same outbound queue.
> 
> This argument holds only if time-of-flight *between* nodes is negligible.  Trivially, a geosynchronous satellite hop adds only two nodes but approximately half a second to the one-way path delay, with potentially thousands of packets existing only as radio waves in the distance between, not in a queue.

Continuing my train of thought, what Kleinrock really implies is that the cwnd should *exceed* the native BDP by at most the number of hops.

 - Jonathan Morton

Re: [Cerowrt-devel] [Bloat] (no subject)

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 2440 bytes --]


Correction accepted. Between the US east and west coasts, the time of flight of packets on fiber or cable is about 23 msec. (Boston-LA, driving route, over fiber, at 207 Mm/sec).
 
So, if all intermediate links are equal in rate, at say, 10 Gb/sec, that means that there should be no more than 10,000,000,000 * 0.023 bits actually in transit on the actual fiber, plus a packet in each router between's outbound queue. Let's say a packet is around 1500 bytes, or 12,000 bits, since that is the MTU we stupidly enforce even today, and there are 10 hops (typical today between Boston and LA.)
 
So we would expect the optimal window size sum, for all flows on any hop of that path to be 10 * 12000 bits + 230,000,000 bits:
 
230,120,000 bits in flight between BOS and LAX. Divide that by 12,000 bits/packet, and you get about 19,177 packets along that path. At most points along the path, you would expect about 10 different flows or more to be in flight, so there would be, optimally, about 1,918 1500 byte packets. Each flow would get 1 Gb/sec as its share.
 
If the connection is limited to < 1 Gb/sec at either endpoint, then there's no reason for any intermediate node to buffer that much of the flow.
 
This gives a reasonable understanding of where AQM should be ending up in terms of the cwnd needed to sustain max throughput and optimum end-to-end latency (which would be about 23 msec + 10 hops * 12000 / 1 Gb/sec.  = 23.012 msec. for a packet to get from one end to the other).
 
 
 
On Saturday, May 18, 2019 6:57pm, "Jonathan Morton" <chromatix99@gmail.com> said:



> > On 19 May, 2019, at 1:36 am, David P. Reed <dpreed@deepplum.com>
> wrote:
> >
> > Pardon, but cwnd should NEVER be larger than the number of forwarding hops
> between source and destination.
> > Kleinrock and students recently proved that the optimum cwnd for both
> throughput and minimized latency is achieved when there is one packet or less in
> each outbound queue from source to destination (including cross traffic - meaning
> other flows sharing the same outbound queue.
> 
> This argument holds only if time-of-flight *between* nodes is negligible. 
> Trivially, a geosynchronous satellite hop adds only two nodes but approximately
> half a second to the one-way path delay, with potentially thousands of packets
> existing only as radio waves in the distance between, not in a queue.
> 
> - Jonathan Morton
> 
> 

[-- Attachment #2: Type: text/html, Size: 4282 bytes --]

Re: [Cerowrt-devel] [Bloat] (no subject)

[Jonathan Foulkes]

> @Jonathan from your experience how tricky is it to get reliable speedtest endpoints and how reliable are they in practice. And do you do any sanitization, like take another measure immediate if the measured rate differs from the last by more than XX% or something like that?

It is tricky to get the profiling of a line right. For one, access to the test endpoints need to be managed so concurrency does not swamp the server/link capacity, otherwise the test results are suspect. You don’t know if measured capacity was limited due to link constraints or server capacity. We spent a good bit of time engineering our capacity testing infrastructure and process to the point of where it is quite reliable in terms of metrics.

Multiple measurements over time are required to correctly profile a line. Generally, early morning (4am or so) tests find the high-water mark on most lines.

As discussed elsewhere, another challenge as line capacity goes up, is where the source of the test is run from. On-router testing is fine for sub-100mbps, but becomes more and more of challenge as speeds go up. Testing a Gbps line using an in-router process takes a pretty beefy box in terms of CPU and NICs.

Best regards,

Jonathan

> On May 15, 2019, at 3:31 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> Hi All,
> 
> 
> I believe the following to be relevant to this discussion: https://apenwarr.ca/log/20180808
> Where he discusses a similar idea including implementation albeit aimed at lower bandwidth and sans the automatic bandwidth tracking.
> 
> 
>> On May 15, 2019, at 01:34, David P. Reed <dpreed@deepplum.com> wrote:
>> 
>> 
>> Ideally, it would need to be self-configuring, though... I.e., something
>> like the IQRouter auto-measuring of the upstream bandwidth to tune the
>> shaper.
> 
> @Jonathan from your experience how tricky is it to get reliable speedtest endpoints and how reliable are they in practice. And do you do any sanitization, like take another measure immediate if the measured rate differs from the last by more than XX% or something like that?
> 
> 
>> 
>> Sure, seems like this is easy to code because there are exactly two ports to measure, they can even be labeled physically "up" and "down" to indicate their function.
> 
> IMHO the real challenge is automated measurements over the internet at Gbps speeds. It is not hard to get some test going (by e.g. tapping into ookla's fast net of confederated measurement endpoints) but getting something where the servers can reliably saturate 1Gbps+ seems somewhat trickier (last time I looked one required a 1Gbps connection to the server to participate in speedtest.net, obviously not really suited for measuring Gbps speeds).
> In the EU there exists a mandate for national regulators to establish and/or endorse an anointed "official" speedtests, untended to keep ISP marketing honest, that come with stricter guarantees (e.g. the official German speedtest, breitbandmessung.de will only admit tests if the servers are having sufficient bandwidth reserves to actually saturate the link; the enduser is required to select the speed-tier giving them a strong hint about the required rates I believe).
> For my back-burner toy project "per-packet-overhead estimation on arbitrary link technology" I am currently facing the same problem, I need a traffic sink and source that can reliably saturate my link so I can measure maximum achievable goodput, so if anybody in the list has ideas, I am all ears/eyes.
> 
>> 
>> For reference, the GL.iNet routers are tiny and nicely packaged, and run
>> OpenWrt; they do have one with Gbit ports[0], priced around $70. I very
>> much doubt it can actually push a gigabit, though, but I haven't had a
>> chance to test it. However, losing the WiFi, and getting a slightly
>> beefier SoC in there will probably be doable without the price going
>> over $100, no?
>> 
>> I assume the WiFi silicon is probably the most costly piece of intellectual property in the system. So yeah. Maybe with the right parts being available, one could aim at $50 or less, without sales channel markup. (Raspberry Pi ARM64 boards don't have GigE, and I think that might be because the GigE interfaces are a bit pricey. However, the ARM64 SoC's available are typically Celeron-class multicore systems. I don't know why there aren't more ARM64 systems on a chip with dual GigE, but I suspect searching for them would turn up some).
> 
> The turris MOX (https://www.turris.cz/en/specification/) might be a decent startimg point as it comes with one Gbethernet port and both a SGMII and a PCIe signals routed on a connector, they also have a 4 and an 8 port switch module, but for our purposes it might be possible to just create a small single Gb ethernet port board to get started. 
> 
> Best Regards
> 	Sebastian
> 
>> 
>> -Toke
>> 
>> [0] https://www.gl-inet.com/products/gl-ar750s/
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Cerowrt-devel] [Bloat] (no subject)

[Sebastian Moeller]

Hi Jonathan,

thanks for your isights, it is always good to talk to folks (or would that be foulkes) that actually implemented an idea ;)

> On May 16, 2019, at 18:40, Jonathan Foulkes <jf@jonathanfoulkes.com> wrote:
> 
>> @Jonathan from your experience how tricky is it to get reliable speedtest endpoints and how reliable are they in practice. And do you do any sanitization, like take another measure immediate if the measured rate differs from the last by more than XX% or something like that?
> 
> It is tricky to get the profiling of a line right. For one, access to the test endpoints need to be managed so concurrency does not swamp the server/link capacity, otherwise the test results are suspect. You don’t know if measured capacity was limited due to link constraints or server capacity. We spent a good bit of time engineering our capacity testing infrastructure and process to the point of where it is quite reliable in terms of metrics.

	Ah, I was reading similar things for the German regulators official speedtest site and wondered how often that does cause issues in reality; thanks to your data point I understand this is not just a theoretical concern.

> 
> Multiple measurements over time are required to correctly profile a line. Generally, early morning (4am or so) tests find the high-water mark on most lines.

	Again, thanks for sharing, while obvious post-hoc, not something I would have looked for (I would have given way more importance to the most recent measurement; clearly demonstrating that I have not actually tried to implement something like this ;) ).

> 
> As discussed elsewhere, another challenge as line capacity goes up, is where the source of the test is run from. On-router testing is fine for sub-100mbps, but becomes more and more of challenge as speeds go up. Testing a Gbps line using an in-router process takes a pretty beefy box in terms of CPU and NICs.

	Great; that certainly puts a somewhat higher lower performance bound on devices to be useful for David's idea, the CPU(s) need to be beefy enough to both shape at Gbps speeds as well as actually generate enough traffic to saturate the link....

Again thanks a lot for sharing!

Best Regards
	Sebastian


> 
> Best regards,
> 
> Jonathan
> 
>> On May 15, 2019, at 3:31 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> Hi All,
>> 
>> 
>> I believe the following to be relevant to this discussion: https://apenwarr.ca/log/20180808
>> Where he discusses a similar idea including implementation albeit aimed at lower bandwidth and sans the automatic bandwidth tracking.
>> 
>> 
>>> On May 15, 2019, at 01:34, David P. Reed <dpreed@deepplum.com> wrote:
>>> 
>>> 
>>> Ideally, it would need to be self-configuring, though... I.e., something
>>> like the IQRouter auto-measuring of the upstream bandwidth to tune the
>>> shaper.
>> 
>> @Jonathan from your experience how tricky is it to get reliable speedtest endpoints and how reliable are they in practice. And do you do any sanitization, like take another measure immediate if the measured rate differs from the last by more than XX% or something like that?
>> 
>> 
>>> 
>>> Sure, seems like this is easy to code because there are exactly two ports to measure, they can even be labeled physically "up" and "down" to indicate their function.
>> 
>> IMHO the real challenge is automated measurements over the internet at Gbps speeds. It is not hard to get some test going (by e.g. tapping into ookla's fast net of confederated measurement endpoints) but getting something where the servers can reliably saturate 1Gbps+ seems somewhat trickier (last time I looked one required a 1Gbps connection to the server to participate in speedtest.net, obviously not really suited for measuring Gbps speeds).
>> In the EU there exists a mandate for national regulators to establish and/or endorse an anointed "official" speedtests, untended to keep ISP marketing honest, that come with stricter guarantees (e.g. the official German speedtest, breitbandmessung.de will only admit tests if the servers are having sufficient bandwidth reserves to actually saturate the link; the enduser is required to select the speed-tier giving them a strong hint about the required rates I believe).
>> For my back-burner toy project "per-packet-overhead estimation on arbitrary link technology" I am currently facing the same problem, I need a traffic sink and source that can reliably saturate my link so I can measure maximum achievable goodput, so if anybody in the list has ideas, I am all ears/eyes.
>> 
>>> 
>>> For reference, the GL.iNet routers are tiny and nicely packaged, and run
>>> OpenWrt; they do have one with Gbit ports[0], priced around $70. I very
>>> much doubt it can actually push a gigabit, though, but I haven't had a
>>> chance to test it. However, losing the WiFi, and getting a slightly
>>> beefier SoC in there will probably be doable without the price going
>>> over $100, no?
>>> 
>>> I assume the WiFi silicon is probably the most costly piece of intellectual property in the system. So yeah. Maybe with the right parts being available, one could aim at $50 or less, without sales channel markup. (Raspberry Pi ARM64 boards don't have GigE, and I think that might be because the GigE interfaces are a bit pricey. However, the ARM64 SoC's available are typically Celeron-class multicore systems. I don't know why there aren't more ARM64 systems on a chip with dual GigE, but I suspect searching for them would turn up some).
>> 
>> The turris MOX (https://www.turris.cz/en/specification/) might be a decent startimg point as it comes with one Gbethernet port and both a SGMII and a PCIe signals routed on a connector, they also have a 4 and an 8 port switch module, but for our purposes it might be possible to just create a small single Gb ethernet port board to get started. 
>> 
>> Best Regards
>> 	Sebastian
>> 
>>> 
>>> -Toke
>>> 
>>> [0] https://www.gl-inet.com/products/gl-ar750s/
>>> _______________________________________________
>>> Cerowrt-devel mailing list
>>> Cerowrt-devel@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> 
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
> 

Re: [Cerowrt-devel] [Bloat] fq_codel is SEVEN years old today...

[Dave Taht]

The future seemed so bright after free.fr deployed it so fast. And so
far as I can tell, fq_codelis ubiquitous across all of linux and linux
containers and vms, and we've headed off major trouble at the wifi
pass with the OSX and QCA (ath9k, ath10k) implementations. 10s of
millions deployed - but where it's most needed, it's not, and ISPs
mostly ship it not.

So while I do take heart in the enormous deployment and figure we'll
turn the corner in the next year or seven on these other devices, I
was grumpily looking over:

http://conferences.sigcomm.org/sigcomm/2014/doc/slides/137.pdf

again today, and wishing we had some way of addressing the structural
problems we have with academic and industry research in general.

On Wed, May 15, 2019 at 1:34 AM David P. Reed <dpreed@deepplum.com> wrote:
>
>
>
> Ideally, it would need to be self-configuring, though... I.e., something
> like the IQRouter auto-measuring of the upstream bandwidth to tune the
> shaper.
>
>
>
> Sure, seems like this is easy to code because there are exactly two ports to measure, they can even be labeled physically "up" and "down" to indicate their function.
>
>
> For reference, the GL.iNet routers are tiny and nicely packaged, and run
> OpenWrt; they do have one with Gbit ports[0], priced around $70. I very
> much doubt it can actually push a gigabit, though, but I haven't had a
> chance to test it. However, losing the WiFi, and getting a slightly
> beefier SoC in there will probably be doable without the price going
> over $100, no?
>
>
>
> I assume the WiFi silicon is probably the most costly piece of intellectual property in the system. So yeah. Maybe with the right parts being available, one could aim at $50 or less, without sales channel markup. (Raspberry Pi ARM64 boards don't have GigE, and I think that might be because the GigE interfaces are a bit pricey. However, the ARM64 SoC's available are typically Celeron-class multicore systems. I don't know why there aren't more ARM64 systems on a chip with dual GigE, but I suspect searching for them would turn up some).
>
> -Toke
>
> [0] https://www.gl-inet.com/products/gl-ar750s/
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

[Cerowrt-devel] Invisibility of bufferbloat and its remedies

[dpreed]

https://www.cordcuttersnews.com/3-easy-tips-to-fix-constant-buffering/

It's distressing how little the tech press understands the real problem.

Of course, cable companies like Charter and ATT who have mostly DOCSIS 2 gear deployed can't admit to their plant being bloat-causing.

In fact it protects their cable business against cord cutters.

And the solution is needed in the CMTS once neighbors all start becoming heavier users, because it is a shared buffering pool with no fairness or bloat protection.

Still, routers with queue management that reduce bloat would help a lot, if "buffering" is seen frequently under load.

So why isn't anyone talking about this problem after at least a decade of knowing it, and knowing how to fix it?

I blame IETF members, individually and collectively. If ietf exists for any reason other than as a boondoggle for world travel, it's for resolving issues like this one.

Re: [Cerowrt-devel] Invisibility of bufferbloat and its remedies

[Dave Taht]

On Mon, Jun 18, 2018 at 9:26 AM, dpreed@deepplum.com
<dpreed@deepplum.com> wrote:
> https://www.cordcuttersnews.com/3-easy-tips-to-fix-constant-buffering/
>
> It's distressing how little the tech press understands the real problem.

Yea, that one is pretty sad.

It still remains a field of active academic research:

https://scholar.google.com/scholar?as_ylo=2018&q=bufferbloat&hl=en&as_sdt=0,5

>
> Of course, cable companies like Charter and ATT who have mostly DOCSIS 2 gear deployed can't admit to their plant being bloat-causing.
>
> In fact it protects their cable business against cord cutters.

Lacking competition in general, doesn't help.

What I am actually more frustrated about is the network neutrality
advocates A) conflating "buffering" with malfeasance, rather than a
technical problem
and B) Using politics rather than technology to attempt to achieve
their goals. If *only* a few prominent members of that side of affairs
"got" that some better technology, deployed, might solve some of their
problems and make the internet better for everyone, we'd make more
progress.

fq_codel is a uniquely "american" algorithm, in that it gives the
"little guy" a little boost until it achieves parity with everyone
else.

>
> And the solution is needed in the CMTS once neighbors all start becoming heavier users, because it is a shared buffering pool with no fairness or bloat protection.

My principle observation is that with the changes in traffic patterns
in the last decade, and the predominance of application-rate limited
streaming, that most
folk are merely forced into a bandwidth tier that is less rarely
annoying. This does not of course solve the corporate gateway problems
very well, nor does it truly kill it dead, but until that day when
"the right stuff" is readily available, and more informed demand
exists.

I was sad to see recently a cisco white paper that even ignored their
own work on pie.

> Still, routers with queue management that reduce bloat would help a lot, if "buffering" is seen frequently under load.
>
> So why isn't anyone talking about this problem after at least a decade of knowing it, and knowing how to fix it?
>
> I blame IETF members, individually and collectively. If ietf exists for any reason other than as a boondoggle for world travel, it's for resolving issues like this one.

Heh. I have essentially abandoned the IETF as the inmates are running
the asylum, and trying to continue to make our points there was
seemingly fruitless
- and out of my budget. I'd rather stay home and get better code out
the door. Or come up with some other set of orgs to annoy into paying
attention.

I would not mind going to another IETF meeting to give a preso (on,
say, cake), but I'm unwilling to front the funds or time anymore.


>



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cerowrt-devel] Invisibility of bufferbloat and its remedies

[dpreed]

[-- Attachment #1: Type: text/plain, Size: 6294 bytes --]


I have been one of the most prominent advocates of network neutrality. I'm constantly informing my friends and the press about "buffering" not being related to neutrality at all.
 
I think that's the best we can do.
 
Packet neutrality is actually a key part of the Internet's design, pushing control mechanisms to the edge, with a minimum of "intelligence" in the routers/switches/gateways. In particular, "content-based" and "endpoint-address-based" targeted throttling was never how the Internet Protocol layer was supposed to work. That's a fundamental result of the "end-to-end argument" applied to congestion management. I've spent a lot of time on that issue technically. The original discussions going back before Van Jacobson's early work, up to RED and ECN, all are based on providing congestion signalling sufficient to cause endpoints competing for resources to adapt their behavior cooperatively in real time, while maintaining minimal latency under load.
 
Latency under load is the crucial metric across the entire IP layer, endpoints and routers. It's clear that minimizing latency under load has to be achieved by avoiding buffering insite the network, moving it to the source (and destination).
 
I've given this lecture to policy people a lot. In fact, deliberate creation of "bloat" is a technical strategy that has been used in the past to destroy VoIP and other real-time communicaitons. Microsoft was caught doing it decades ago, as were some other conflicted communicaitons providers. They could selectively delay small packets using DPI, while letting FTPs get full speed. That's one of the reasons we coined the idea of Network Neutrality.
 
But radical right wingers of the sort that blossom in the paranoid world of the dark net started arguing that the routers should have political freedom to do whatever they damn well pleased with packets, because routers are people just like corporations, and a "free market" is the solution to everything.
 
Well, technically, the Internet doesn't work if their is not some mechanism for eliminiating lag under load by eliminating queueing delay in bottleneck nodes.
 
That's ultimately what Network Neutrality is about.  There's a lot of other crap being pushed by folks who pile on to the Network Neutrality discussion. People want to "fix prices" for example, arguing that profits are bad. Those guys are not the problem.
 
The problem is that the vertically integrated monopolists want to claim that the Internet should be subject to Deep Packet Inspection at every router, designed to charge rents based on content of the packets and who is the original sender or destination of the packet - that is, charging Netflix or NBC Universal packets nothing, and charging IPFS packets 100x as much.
 
So, no, the Network Neutrality people are NOT the problem with Bufferbloat.
 
Comcast, on the other hand, has been slow-rolling DOCSIS 3.0, because their customers on DOCSIS 2.0 are just ordering faster service tiers to overcome the Bufferbloat in their DOCSIS 2 CMTS's.
-----Original Message-----
From: "Dave Taht" <dave.taht@gmail.com>
Sent: Monday, June 18, 2018 3:07pm
To: "dpreed@deepplum.com" <dpreed@deepplum.com>
Cc: cerowrt-devel@lists.bufferbloat.net, "bloat" <bloat@lists.bufferbloat.net>
Subject: Re: Invisibility of bufferbloat and its remedies



On Mon, Jun 18, 2018 at 9:26 AM, dpreed@deepplum.com
<dpreed@deepplum.com> wrote:
> https://www.cordcuttersnews.com/3-easy-tips-to-fix-constant-buffering/
>
> It's distressing how little the tech press understands the real problem.

Yea, that one is pretty sad.

It still remains a field of active academic research:

https://scholar.google.com/scholar?as_ylo=2018&q=bufferbloat&hl=en&as_sdt=0,5

>
> Of course, cable companies like Charter and ATT who have mostly DOCSIS 2 gear deployed can't admit to their plant being bloat-causing.
>
> In fact it protects their cable business against cord cutters.

Lacking competition in general, doesn't help.

What I am actually more frustrated about is the network neutrality
advocates A) conflating "buffering" with malfeasance, rather than a
technical problem
and B) Using politics rather than technology to attempt to achieve
their goals. If *only* a few prominent members of that side of affairs
"got" that some better technology, deployed, might solve some of their
problems and make the internet better for everyone, we'd make more
progress.

fq_codel is a uniquely "american" algorithm, in that it gives the
"little guy" a little boost until it achieves parity with everyone
else.

>
> And the solution is needed in the CMTS once neighbors all start becoming heavier users, because it is a shared buffering pool with no fairness or bloat protection.

My principle observation is that with the changes in traffic patterns
in the last decade, and the predominance of application-rate limited
streaming, that most
folk are merely forced into a bandwidth tier that is less rarely
annoying. This does not of course solve the corporate gateway problems
very well, nor does it truly kill it dead, but until that day when
"the right stuff" is readily available, and more informed demand
exists.

I was sad to see recently a cisco white paper that even ignored their
own work on pie.

> Still, routers with queue management that reduce bloat would help a lot, if "buffering" is seen frequently under load.
>
> So why isn't anyone talking about this problem after at least a decade of knowing it, and knowing how to fix it?
>
> I blame IETF members, individually and collectively. If ietf exists for any reason other than as a boondoggle for world travel, it's for resolving issues like this one.

Heh. I have essentially abandoned the IETF as the inmates are running
the asylum, and trying to continue to make our points there was
seemingly fruitless
- and out of my budget. I'd rather stay home and get better code out
the door. Or come up with some other set of orgs to annoy into paying
attention.

I would not mind going to another IETF meeting to give a preso (on,
say, cake), but I'm unwilling to front the funds or time anymore.


>



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

[-- Attachment #2: Type: text/html, Size: 8984 bytes --]

Re: [Cerowrt-devel] Invisibility of bufferbloat and its remedies

[Dave Taht]

I will be in D.C., presenting https://www.cs.kau.se/tohojo/cake/ - next week.

If there are people to see, asses to kick or kiss, I'm up for it, let
me know. Presently I just plan to give my talk and get the heck out of
dodge.

One of cake's "minor" features is the *perfect* defeat of the htb
based shaper in cable modems. If you know the set-rate on the modem,
you
just set it to the same thing and get vastly superior performance to
docsis 3.1, pie, or the sqm-scripts.

On Mon, Jun 18, 2018 at 3:43 PM, dpreed@deepplum.com
<dpreed@deepplum.com> wrote:
> I have been one of the most prominent advocates of network neutrality. I'm
> constantly informing my friends and the press about "buffering" not being
> related to neutrality at all.
>
>
>
> I think that's the best we can do.
>
>
>
> Packet neutrality is actually a key part of the Internet's design, pushing
> control mechanisms to the edge, with a minimum of "intelligence" in the
> routers/switches/gateways. In particular, "content-based" and
> "endpoint-address-based" targeted throttling was never how the Internet
> Protocol layer was supposed to work. That's a fundamental result of the
> "end-to-end argument" applied to congestion management. I've spent a lot of
> time on that issue technically. The original discussions going back before
> Van Jacobson's early work, up to RED and ECN, all are based on providing
> congestion signalling sufficient to cause endpoints competing for resources
> to adapt their behavior cooperatively in real time, while maintaining
> minimal latency under load.
>
>
>
> Latency under load is the crucial metric across the entire IP layer,
> endpoints and routers. It's clear that minimizing latency under load has to
> be achieved by avoiding buffering insite the network, moving it to the
> source (and destination).
>
>
>
> I've given this lecture to policy people a lot. In fact, deliberate creation
> of "bloat" is a technical strategy that has been used in the past to destroy
> VoIP and other real-time communicaitons. Microsoft was caught doing it
> decades ago, as were some other conflicted communicaitons providers. They
> could selectively delay small packets using DPI, while letting FTPs get full
> speed. That's one of the reasons we coined the idea of Network Neutrality.
>
>
>
> But radical right wingers of the sort that blossom in the paranoid world of
> the dark net started arguing that the routers should have political freedom
> to do whatever they damn well pleased with packets, because routers are
> people just like corporations, and a "free market" is the solution to
> everything.
>
>
>
> Well, technically, the Internet doesn't work if their is not some mechanism
> for eliminiating lag under load by eliminating queueing delay in bottleneck
> nodes.
>
>
>
> That's ultimately what Network Neutrality is about.  There's a lot of other
> crap being pushed by folks who pile on to the Network Neutrality discussion.
> People want to "fix prices" for example, arguing that profits are bad. Those
> guys are not the problem.
>
>
>
> The problem is that the vertically integrated monopolists want to claim that
> the Internet should be subject to Deep Packet Inspection at every router,
> designed to charge rents based on content of the packets and who is the
> original sender or destination of the packet - that is, charging Netflix or
> NBC Universal packets nothing, and charging IPFS packets 100x as much.
>
>
>
> So, no, the Network Neutrality people are NOT the problem with Bufferbloat.
>
>
>
> Comcast, on the other hand, has been slow-rolling DOCSIS 3.0, because their
> customers on DOCSIS 2.0 are just ordering faster service tiers to overcome
> the Bufferbloat in their DOCSIS 2 CMTS's.
>
> -----Original Message-----
> From: "Dave Taht" <dave.taht@gmail.com>
> Sent: Monday, June 18, 2018 3:07pm
> To: "dpreed@deepplum.com" <dpreed@deepplum.com>
> Cc: cerowrt-devel@lists.bufferbloat.net, "bloat"
> <bloat@lists.bufferbloat.net>
> Subject: Re: Invisibility of bufferbloat and its remedies
>
> On Mon, Jun 18, 2018 at 9:26 AM, dpreed@deepplum.com
> <dpreed@deepplum.com> wrote:
>> https://www.cordcuttersnews.com/3-easy-tips-to-fix-constant-buffering/
>>
>> It's distressing how little the tech press understands the real problem.
>
> Yea, that one is pretty sad.
>
> It still remains a field of active academic research:
>
> https://scholar.google.com/scholar?as_ylo=2018&q=bufferbloat&hl=en&as_sdt=0,5
>
>>
>> Of course, cable companies like Charter and ATT who have mostly DOCSIS 2
>> gear deployed can't admit to their plant being bloat-causing.
>>
>> In fact it protects their cable business against cord cutters.
>
> Lacking competition in general, doesn't help.
>
> What I am actually more frustrated about is the network neutrality
> advocates A) conflating "buffering" with malfeasance, rather than a
> technical problem
> and B) Using politics rather than technology to attempt to achieve
> their goals. If *only* a few prominent members of that side of affairs
> "got" that some better technology, deployed, might solve some of their
> problems and make the internet better for everyone, we'd make more
> progress.
>
> fq_codel is a uniquely "american" algorithm, in that it gives the
> "little guy" a little boost until it achieves parity with everyone
> else.
>
>>
>> And the solution is needed in the CMTS once neighbors all start becoming
>> heavier users, because it is a shared buffering pool with no fairness or
>> bloat protection.
>
> My principle observation is that with the changes in traffic patterns
> in the last decade, and the predominance of application-rate limited
> streaming, that most
> folk are merely forced into a bandwidth tier that is less rarely
> annoying. This does not of course solve the corporate gateway problems
> very well, nor does it truly kill it dead, but until that day when
> "the right stuff" is readily available, and more informed demand
> exists.
>
> I was sad to see recently a cisco white paper that even ignored their
> own work on pie.
>
>> Still, routers with queue management that reduce bloat would help a lot,
>> if "buffering" is seen frequently under load.
>>
>> So why isn't anyone talking about this problem after at least a decade of
>> knowing it, and knowing how to fix it?
>>
>> I blame IETF members, individually and collectively. If ietf exists for
>> any reason other than as a boondoggle for world travel, it's for resolving
>> issues like this one.
>
> Heh. I have essentially abandoned the IETF as the inmates are running
> the asylum, and trying to continue to make our points there was
> seemingly fruitless
> - and out of my budget. I'd rather stay home and get better code out
> the door. Or come up with some other set of orgs to annoy into paying
> attention.
>
> I would not mind going to another IETF meeting to give a preso (on,
> say, cake), but I'm unwilling to front the funds or time anymore.
>
>
>>
>
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cerowrt-devel] Invisibility of bufferbloat and its remedies

[valdis.kletnieks]

On Mon, 18 Jun 2018 18:46:18 -0700, Dave Taht said:

> One of cake's "minor" features is the *perfect* defeat of the htb
> based shaper in cable modems. If you know the set-rate on the modem,
> you just set it to the same thing and get vastly superior performance to
> docsis 3.1, pie, or the sqm-scripts.

Do we have a good cookbook on how to determine the set-rate?

Re: [Cerowrt-devel] Invisibility of bufferbloat and its remedies

[Sebastian Moeller]

Well,

On June 19, 2018 10:34:07 PM GMT+02:00, valdis.kletnieks@vt.edu wrote:
>On Mon, 18 Jun 2018 18:46:18 -0700, Dave Taht said:
>
>> One of cake's "minor" features is the *perfect* defeat of the htb
>> based shaper in cable modems. If you know the set-rate on the modem,
>> you just set it to the same thing and get vastly superior performance
>to
>> docsis 3.1, pie, or the sqm-scripts.
>
>Do we have a good cookbook on how to determine the set-rate?

With cable <DOCSIS 3.1 one can Snoop the management frames that will also carry the bandwidth definitions. Or with a compromised cable modem one could dump the DOCSIS config file that also contains that information. Or ask the ISP. Short of any of that one could run a umber of speedtests over a 24 hour period and simply extrapolate from the measured goodput to the required 'gross' DOCSIS shaper rate:

Measured TCP/ipv4 goodput * ((1518)/(1500-20-20)) = lower bound gross bandwidth estimate

DOCSIS employs a shaper to limit user's exceeding the contracted rates, that per DOCSIS standard assumes 1518 bytes of accountable raw frame size. Tangent that 1518 obviously is a 'lie' in that it does not include all per packet overhead in use on a DOCSIS carrier, but from an end-user perspective that difference is immaterial...

I realize that you probably wanted something simpler and more accurate, sorry to disappoint.

Best Regards
        Sebastian



>_______________________________________________
>Cerowrt-devel mailing list
>Cerowrt-devel@lists.bufferbloat.net
>https://lists.bufferbloat.net/listinfo/cerowrt-devel

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Cerowrt-devel] (no subject)

[Sebastian Moeller]

Hi all,


> On Jun 20, 2018, at 01:32, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> Well,
> 
> On June 19, 2018 10:34:07 PM GMT+02:00, valdis.kletnieks@vt.edu wrote:
>> On Mon, 18 Jun 2018 18:46:18 -0700, Dave Taht said:
>> 
>>> One of cake's "minor" features is the *perfect* defeat of the htb
>>> based shaper in cable modems. If you know the set-rate on the modem,
>>> you just set it to the same thing and get vastly superior performance
>> to
>>> docsis 3.1, pie, or the sqm-scripts.
>> 
>> Do we have a good cookbook on how to determine the set-rate?
> 
> With cable <DOCSIS 3.1 one can Snoop the management frames that will also carry the bandwidth definitions. Or with a compromised cable modem one could dump the DOCSIS config file that also contains that information. Or ask the ISP. Short of any of that one could run a umber of speedtests over a 24 hour period and simply extrapolate from the measured goodput to the required 'gross' DOCSIS shaper rate:
> 
> Measured TCP/ipv4 goodput * ((1518)/(1500-20-20)) = lower bound gross bandwidth estimate

Addendum: when running speedtests on cable for the purpose of estimating the "true" docsis shaper goodput one needs to take care to not be fooled by transient bandwidth allowances like powerboost but rather one needs to find the sustainable stable  maximum bandwidth; so fun all around. (And one needs to account for the correct overhead in the equation above so in the IPv6 case the (1500-20-20) needs to be replaced by (1500-40-20))

Best Regards
	Sebastian


> 
> DOCSIS employs a shaper to limit user's exceeding the contracted rates, that per DOCSIS standard assumes 1518 bytes of accountable raw frame size. Tangent that 1518 obviously is a 'lie' in that it does not include all per packet overhead in use on a DOCSIS carrier, but from an end-user perspective that difference is immaterial...
> 
> I realize that you probably wanted something simpler and more accurate, sorry to disappoint.
> 
> Best Regards
>        Sebastian
> 
> 
> 
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel