[Cerowrt-devel] fq_codel is SEVEN years old today...

[Cerowrt-devel] fq_codel is SEVEN years old today...

[Rich Brown]

Folks,

If it feels like we've been at this for a long time, you're right. My calendar tells me that we've been using fq_codel for the last seven years.

https://lists.bufferbloat.net/pipermail/bloat/2014-May/001934.html

Let's all pat ourselves on the back for this good work!

Rich

Re: [Cerowrt-devel] fq_codel is SEVEN years old today...

[Valdis Klētnieks]

[-- Attachment #1: Type: text/plain, Size: 311 bytes --]

On Tue, 14 May 2019 08:16:06 -0400, Rich Brown said:

> Let's all pat ourselves on the back for this good work!

Do we have an estimate of what percent of connected devices
are actually using fq_codel or other modern anti-bloat methods?
I'm reasonably sure my TV, my PS3, and my PS4 are still
behind the curve.

[-- Attachment #2: Type: application/pgp-signature, Size: 832 bytes --]

Re: [Cerowrt-devel] fq_codel is SEVEN years old today...

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 2069 bytes --]


Well, of all the devices in my house (maybe 100), only the router attached to the cable modem (which is a 2x GigE Intel Linux board based on Fedora 29 server with sch_cake configured) is running fq_codel. And setting that up was a labor of love. But it works a charm for my asymmetric Gigabit cable service.
 
My home's backbone is 10 GigE fiber, so I suppose fq_codel would be helpful for devices that run on 1 GigE subnets like my 2 802.11ac access points when talking to my NAS's.
However, the 802.11ac access point high speed functionality doesn't seem to be supportable by LEDE. So what can I do? 
 
I suppose I could stick some little custom Intel Linux 2x GigE devices between access points and the 10 GigE backbone, and put fq_codel in there.
 
My point is, to get the primary benefit of bufferbloat reduction, one has to stick little Linux boxes everywhere, because fq_codel is not supported except via DIY hacking.
 
And indeed, 10 GigE->1 GigE buffering does affect storage access latency in bad ways.
 
We see the same problem in datacenter networks that have excessive buffering - a famous switch company backed by Andy Bechtolsheim is really problematic because they claim building up huge buffers is a "feature" not a bug.
-----Original Message-----
From: "Valdis Klētnieks" <valdis.kletnieks@vt.edu>
Sent: Tuesday, May 14, 2019 1:57pm
To: "Rich Brown" <richb.hanover@gmail.com>
Cc: "cerowrt-devel" <cerowrt-devel@lists.bufferbloat.net>, "bloat" <bloat@lists.bufferbloat.net>
Subject: Re: [Cerowrt-devel] fq_codel is SEVEN years old today...



_______________________________________________
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel
On Tue, 14 May 2019 08:16:06 -0400, Rich Brown said:

> Let's all pat ourselves on the back for this good work!

Do we have an estimate of what percent of connected devices
are actually using fq_codel or other modern anti-bloat methods?
I'm reasonably sure my TV, my PS3, and my PS4 are still
behind the curve.

[-- Attachment #2: Type: text/html, Size: 3645 bytes --]

Re: [Cerowrt-devel] fq_codel is SEVEN years old today...

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 4033 bytes --]


I wonder if an interesting project to design and pitch for CrowdSupply to fund would be a little board that packages sch_cake or something in the minimal hardware package that could sit between a 1 GigE symmetric port and either an asymmetric GigE or a symmetric 1 GigE connection into a 10 GigE switch.
The key point is that it needs to support wire-rate forwarding with small packets of Gigabit throughput. Ideally, it also supports a dnsmasq NAT and wireguard optionally.
 
I know a Celeron with 2 GB of RAM can easily do it (because that is what I use). We know (well that's what you guys tell me) that the dinky MIPS processors are underpowered to handle sch_cake at such packet rates. The Linksys and Netgear and TP-link guys seem to see no market at all for any such thing. But I see it as a useful jellybean device if it could be cheap and simple.
 
Could maybe design, produce, and sell this for $100? No one else seems to want to make such a thing. I could just barely design and implement the board and get it made, but to be honest I'm better at spec'ing and prototyping than making manufacturable hardware designs. I suspect I could find someone to do the PCB design, layout and parts selection as a project.
 
The idea for this hardware "product" is to decouple this buffer management from the WiFi compatibility and driver mess, and make it easy for people, maybe to demonstrate that it could be a great product. Forget designing the packaging, negotiating a sales channel, etc. Just do what is needed to make a few thousand for the CrowdSupply market.
 
Thoughts?
 
-----Original Message-----
From: "David P. Reed" <dpreed@deepplum.com>
Sent: Tuesday, May 14, 2019 2:38pm
To: "Valdis Klētnieks" <valdis.kletnieks@vt.edu>
Cc: "Rich Brown" <richb.hanover@gmail.com>, "cerowrt-devel" <cerowrt-devel@lists.bufferbloat.net>, "bloat" <bloat@lists.bufferbloat.net>
Subject: Re: [Cerowrt-devel] fq_codel is SEVEN years old today...



Well, of all the devices in my house (maybe 100), only the router attached to the cable modem (which is a 2x GigE Intel Linux board based on Fedora 29 server with sch_cake configured) is running fq_codel. And setting that up was a labor of love. But it works a charm for my asymmetric Gigabit cable service.
 
My home's backbone is 10 GigE fiber, so I suppose fq_codel would be helpful for devices that run on 1 GigE subnets like my 2 802.11ac access points when talking to my NAS's.
However, the 802.11ac access point high speed functionality doesn't seem to be supportable by LEDE. So what can I do? 
 
I suppose I could stick some little custom Intel Linux 2x GigE devices between access points and the 10 GigE backbone, and put fq_codel in there.
 
My point is, to get the primary benefit of bufferbloat reduction, one has to stick little Linux boxes everywhere, because fq_codel is not supported except via DIY hacking.
 
And indeed, 10 GigE->1 GigE buffering does affect storage access latency in bad ways.
 
We see the same problem in datacenter networks that have excessive buffering - a famous switch company backed by Andy Bechtolsheim is really problematic because they claim building up huge buffers is a "feature" not a bug.
-----Original Message-----
From: "Valdis Klētnieks" <valdis.kletnieks@vt.edu>
Sent: Tuesday, May 14, 2019 1:57pm
To: "Rich Brown" <richb.hanover@gmail.com>
Cc: "cerowrt-devel" <cerowrt-devel@lists.bufferbloat.net>, "bloat" <bloat@lists.bufferbloat.net>
Subject: Re: [Cerowrt-devel] fq_codel is SEVEN years old today...



_______________________________________________
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel
On Tue, 14 May 2019 08:16:06 -0400, Rich Brown said:

> Let's all pat ourselves on the back for this good work!

Do we have an estimate of what percent of connected devices
are actually using fq_codel or other modern anti-bloat methods?
I'm reasonably sure my TV, my PS3, and my PS4 are still
behind the curve.

[-- Attachment #2: Type: text/html, Size: 7229 bytes --]

Re: [Cerowrt-devel] [Bloat] fq_codel is SEVEN years old today...

[Toke Høiland-Jørgensen]

"David P. Reed" <dpreed@deepplum.com> writes:

> I wonder if an interesting project to design and pitch for CrowdSupply
> to fund would be a little board that packages sch_cake or something in
> the minimal hardware package that could sit between a 1 GigE symmetric
> port and either an asymmetric GigE or a symmetric 1 GigE connection
> into a 10 GigE switch. The key point is that it needs to support
> wire-rate forwarding with small packets of Gigabit throughput.
> Ideally, it also supports a dnsmasq NAT and wireguard optionally.
>  
> I know a Celeron with 2 GB of RAM can easily do it (because that is
> what I use). We know (well that's what you guys tell me) that the
> dinky MIPS processors are underpowered to handle sch_cake at such
> packet rates. The Linksys and Netgear and TP-link guys seem to see no
> market at all for any such thing. But I see it as a useful jellybean
> device if it could be cheap and simple.
>  
> Could maybe design, produce, and sell this for $100? No one else seems
> to want to make such a thing. I could just barely design and implement
> the board and get it made, but to be honest I'm better at spec'ing and
> prototyping than making manufacturable hardware designs. I suspect I
> could find someone to do the PCB design, layout and parts selection as
> a project.
>  
> The idea for this hardware "product" is to decouple this buffer
> management from the WiFi compatibility and driver mess, and make it
> easy for people, maybe to demonstrate that it could be a great
> product. Forget designing the packaging, negotiating a sales channel,
> etc. Just do what is needed to make a few thousand for the CrowdSupply
> market.
>  
> Thoughts?

It's a cool idea, and I'd certainly buy a couple to help the
crowdfunding ;)

Ideally, it would need to be self-configuring, though... I.e., something
like the IQRouter auto-measuring of the upstream bandwidth to tune the
shaper.

For reference, the GL.iNet routers are tiny and nicely packaged, and run
OpenWrt; they do have one with Gbit ports[0], priced around $70. I very
much doubt it can actually push a gigabit, though, but I haven't had a
chance to test it. However, losing the WiFi, and getting a slightly
beefier SoC in there will probably be doable without the price going
over $100, no?

-Toke

[0] https://www.gl-inet.com/products/gl-ar750s/

Re: [Cerowrt-devel] [Bloat] fq_codel is SEVEN years old today...

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 1310 bytes --]


 

Ideally, it would need to be self-configuring, though... I.e., something
like the IQRouter auto-measuring of the upstream bandwidth to tune the
shaper.
 
Sure, seems like this is easy to code because there are exactly two ports to measure, they can even be labeled physically "up" and "down" to indicate their function.

For reference, the GL.iNet routers are tiny and nicely packaged, and run
OpenWrt; they do have one with Gbit ports[0], priced around $70. I very
much doubt it can actually push a gigabit, though, but I haven't had a
chance to test it. However, losing the WiFi, and getting a slightly
beefier SoC in there will probably be doable without the price going
over $100, no?
 
I assume the WiFi silicon is probably the most costly piece of intellectual property in the system. So yeah. Maybe with the right parts being available, one could aim at $50 or less, without sales channel markup. (Raspberry Pi ARM64 boards don't have GigE, and I think that might be because the GigE interfaces are a bit pricey. However, the ARM64 SoC's available are typically Celeron-class multicore systems. I don't know why there aren't more ARM64 systems on a chip with dual GigE, but I suspect searching for them would turn up some).

-Toke

[0] https://www.gl-inet.com/products/gl-ar750s/

[-- Attachment #2: Type: text/html, Size: 2171 bytes --]

[Cerowrt-devel] (no subject)

[Sebastian Moeller]

Hi All,


I believe the following to be relevant to this discussion: https://apenwarr.ca/log/20180808
Where he discusses a similar idea including implementation albeit aimed at lower bandwidth and sans the automatic bandwidth tracking.


> On May 15, 2019, at 01:34, David P. Reed <dpreed@deepplum.com> wrote:
> 
> 
> Ideally, it would need to be self-configuring, though... I.e., something
> like the IQRouter auto-measuring of the upstream bandwidth to tune the
> shaper.

@Jonathan from your experience how tricky is it to get reliable speedtest endpoints and how reliable are they in practice. And do you do any sanitization, like take another measure immediate if the measured rate differs from the last by more than XX% or something like that?


> 
> Sure, seems like this is easy to code because there are exactly two ports to measure, they can even be labeled physically "up" and "down" to indicate their function.

IMHO the real challenge is automated measurements over the internet at Gbps speeds. It is not hard to get some test going (by e.g. tapping into ookla's fast net of confederated measurement endpoints) but getting something where the servers can reliably saturate 1Gbps+ seems somewhat trickier (last time I looked one required a 1Gbps connection to the server to participate in speedtest.net, obviously not really suited for measuring Gbps speeds).
In the EU there exists a mandate for national regulators to establish and/or endorse an anointed "official" speedtests, untended to keep ISP marketing honest, that come with stricter guarantees (e.g. the official German speedtest, breitbandmessung.de will only admit tests if the servers are having sufficient bandwidth reserves to actually saturate the link; the enduser is required to select the speed-tier giving them a strong hint about the required rates I believe).
For my back-burner toy project "per-packet-overhead estimation on arbitrary link technology" I am currently facing the same problem, I need a traffic sink and source that can reliably saturate my link so I can measure maximum achievable goodput, so if anybody in the list has ideas, I am all ears/eyes.

> 
> For reference, the GL.iNet routers are tiny and nicely packaged, and run
> OpenWrt; they do have one with Gbit ports[0], priced around $70. I very
> much doubt it can actually push a gigabit, though, but I haven't had a
> chance to test it. However, losing the WiFi, and getting a slightly
> beefier SoC in there will probably be doable without the price going
> over $100, no?
> 
> I assume the WiFi silicon is probably the most costly piece of intellectual property in the system. So yeah. Maybe with the right parts being available, one could aim at $50 or less, without sales channel markup. (Raspberry Pi ARM64 boards don't have GigE, and I think that might be because the GigE interfaces are a bit pricey. However, the ARM64 SoC's available are typically Celeron-class multicore systems. I don't know why there aren't more ARM64 systems on a chip with dual GigE, but I suspect searching for them would turn up some).

The turris MOX (https://www.turris.cz/en/specification/) might be a decent startimg point as it comes with one Gbethernet port and both a SGMII and a PCIe signals routed on a connector, they also have a 4 and an 8 port switch module, but for our purposes it might be possible to just create a small single Gb ethernet port board to get started. 

Best Regards
	Sebastian

> 
> -Toke
> 
> [0] https://www.gl-inet.com/products/gl-ar750s/
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] (no subject)

[Dave Taht]

If it helps any: Nick Feamster and Jason Livingood just published "
Internet Speed Measurement: Current Challenges and Future
Recommendations " ( https://arxiv.org/pdf/1905.02334.pdf ) a few days
ago, and outlines quite a few problems going forward at higher speeds.
I do wish the document had pointed out more clearly that router based
measurements have problems also, with weaker cpus unable to source
enough traffic for an accurate measurement, but I do hope this
document has impact, and it's a good read, regardless.

Still, somehow getting it right at lower speeds is always on my mind.
I'd long ago hoped that DSL devices would adopt BQL, and that
cablemodems would also, thus moving packet processing a little higher
on the stack so more advanced algorithms like cake could take hold.

On Wed, May 15, 2019 at 9:32 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>
> Hi All,
>
>
> I believe the following to be relevant to this discussion: https://apenwarr.ca/log/20180808
> Where he discusses a similar idea including implementation albeit aimed at lower bandwidth and sans the automatic bandwidth tracking.
>
>
> > On May 15, 2019, at 01:34, David P. Reed <dpreed@deepplum.com> wrote:
> >
> >
> > Ideally, it would need to be self-configuring, though... I.e., something
> > like the IQRouter auto-measuring of the upstream bandwidth to tune the
> > shaper.
>
> @Jonathan from your experience how tricky is it to get reliable speedtest endpoints and how reliable are they in practice. And do you do any sanitization, like take another measure immediate if the measured rate differs from the last by more than XX% or something like that?
>
>
> >
> > Sure, seems like this is easy to code because there are exactly two ports to measure, they can even be labeled physically "up" and "down" to indicate their function.
>
> IMHO the real challenge is automated measurements over the internet at Gbps speeds. It is not hard to get some test going (by e.g. tapping into ookla's fast net of confederated measurement endpoints) but getting something where the servers can reliably saturate 1Gbps+ seems somewhat trickier (last time I looked one required a 1Gbps connection to the server to participate in speedtest.net, obviously not really suited for measuring Gbps speeds).
> In the EU there exists a mandate for national regulators to establish and/or endorse an anointed "official" speedtests, untended to keep ISP marketing honest, that come with stricter guarantees (e.g. the official German speedtest, breitbandmessung.de will only admit tests if the servers are having sufficient bandwidth reserves to actually saturate the link; the enduser is required to select the speed-tier giving them a strong hint about the required rates I believe).
> For my back-burner toy project "per-packet-overhead estimation on arbitrary link technology" I am currently facing the same problem, I need a traffic sink and source that can reliably saturate my link so I can measure maximum achievable goodput, so if anybody in the list has ideas, I am all ears/eyes.
>
> >
> > For reference, the GL.iNet routers are tiny and nicely packaged, and run
> > OpenWrt; they do have one with Gbit ports[0], priced around $70. I very
> > much doubt it can actually push a gigabit, though, but I haven't had a
> > chance to test it. However, losing the WiFi, and getting a slightly
> > beefier SoC in there will probably be doable without the price going
> > over $100, no?
> >
> > I assume the WiFi silicon is probably the most costly piece of intellectual property in the system. So yeah. Maybe with the right parts being available, one could aim at $50 or less, without sales channel markup. (Raspberry Pi ARM64 boards don't have GigE, and I think that might be because the GigE interfaces are a bit pricey. However, the ARM64 SoC's available are typically Celeron-class multicore systems. I don't know why there aren't more ARM64 systems on a chip with dual GigE, but I suspect searching for them would turn up some).
>
> The turris MOX (https://www.turris.cz/en/specification/) might be a decent startimg point as it comes with one Gbethernet port and both a SGMII and a PCIe signals routed on a connector, they also have a 4 and an 8 port switch module, but for our purposes it might be possible to just create a small single Gb ethernet port board to get started.
>
> Best Regards
>         Sebastian
>
> >
> > -Toke
> >
> > [0] https://www.gl-inet.com/products/gl-ar750s/
> > _______________________________________________
> > Cerowrt-devel mailing list
> > Cerowrt-devel@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Cerowrt-devel] [Bloat] (no subject)

[Sebastian Moeller]

Hi Dave,


> On May 15, 2019, at 09:58, Dave Taht <dave.taht@gmail.com> wrote:
> 
> If it helps any: Nick Feamster and Jason Livingood just published "
> Internet Speed Measurement: Current Challenges and Future
> Recommendations " ( https://arxiv.org/pdf/1905.02334.pdf ) a few days
> ago, and outlines quite a few problems going forward at higher speeds.

	Excellent, will read (once time permits).


> I do wish the document had pointed out more clearly that router based
> measurements have problems also,

	+1, especially newer SoCs which implement power features can run into situations where depending on the load characteristics the CPUs throttle back (in frequency) and perfomance becomes quite choppy, which might not show up in bandwidth measurement but can really affect the experienced bufferbloat. I currently think it better for the load generation to be done by  different hosts than the DUT/de-bloater...


> with weaker cpus unable to source
> enough traffic for an accurate measurement, but I do hope this
> document has impact, and it's a good read, regardless.
> 
> Still, somehow getting it right at lower speeds is always on my mind.

	At lower speeds measuring the bandwidth is not that hard, since overloading the CPU and the server;s bandwidth gets less likely, no?

> I'd long ago hoped that DSL devices would adopt BQL, and that
> cablemodems would also, thus moving packet processing a little higher
> on the stack so more advanced algorithms like cake could take hold.

	Well, to make this happen we need to talk to modem manufacturers, so mostly intel/lantiq and broadcom I believe. I happen to have exact zero insiders in both companies in my "rolodex", but with in a group as diverse and expert as this list is, someone here must know someone that could get the word out inside those companies, no?
	For BQL on docsis modems though, I believe cablelabs might be the best route (not that BQL on a docsis modem will help a lot, given that we have, AFAICT, no open source OS running on a docsis modem, same seems true for GPON).

Best Regards
	Sebastian

> 
> On Wed, May 15, 2019 at 9:32 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> Hi All,
>> 
>> 
>> I believe the following to be relevant to this discussion: https://apenwarr.ca/log/20180808
>> Where he discusses a similar idea including implementation albeit aimed at lower bandwidth and sans the automatic bandwidth tracking.
>> 
>> 
>>> On May 15, 2019, at 01:34, David P. Reed <dpreed@deepplum.com> wrote:
>>> 
>>> 
>>> Ideally, it would need to be self-configuring, though... I.e., something
>>> like the IQRouter auto-measuring of the upstream bandwidth to tune the
>>> shaper.
>> 
>> @Jonathan from your experience how tricky is it to get reliable speedtest endpoints and how reliable are they in practice. And do you do any sanitization, like take another measure immediate if the measured rate differs from the last by more than XX% or something like that?
>> 
>> 
>>> 
>>> Sure, seems like this is easy to code because there are exactly two ports to measure, they can even be labeled physically "up" and "down" to indicate their function.
>> 
>> IMHO the real challenge is automated measurements over the internet at Gbps speeds. It is not hard to get some test going (by e.g. tapping into ookla's fast net of confederated measurement endpoints) but getting something where the servers can reliably saturate 1Gbps+ seems somewhat trickier (last time I looked one required a 1Gbps connection to the server to participate in speedtest.net, obviously not really suited for measuring Gbps speeds).
>> In the EU there exists a mandate for national regulators to establish and/or endorse an anointed "official" speedtests, untended to keep ISP marketing honest, that come with stricter guarantees (e.g. the official German speedtest, breitbandmessung.de will only admit tests if the servers are having sufficient bandwidth reserves to actually saturate the link; the enduser is required to select the speed-tier giving them a strong hint about the required rates I believe).
>> For my back-burner toy project "per-packet-overhead estimation on arbitrary link technology" I am currently facing the same problem, I need a traffic sink and source that can reliably saturate my link so I can measure maximum achievable goodput, so if anybody in the list has ideas, I am all ears/eyes.
>> 
>>> 
>>> For reference, the GL.iNet routers are tiny and nicely packaged, and run
>>> OpenWrt; they do have one with Gbit ports[0], priced around $70. I very
>>> much doubt it can actually push a gigabit, though, but I haven't had a
>>> chance to test it. However, losing the WiFi, and getting a slightly
>>> beefier SoC in there will probably be doable without the price going
>>> over $100, no?
>>> 
>>> I assume the WiFi silicon is probably the most costly piece of intellectual property in the system. So yeah. Maybe with the right parts being available, one could aim at $50 or less, without sales channel markup. (Raspberry Pi ARM64 boards don't have GigE, and I think that might be because the GigE interfaces are a bit pricey. However, the ARM64 SoC's available are typically Celeron-class multicore systems. I don't know why there aren't more ARM64 systems on a chip with dual GigE, but I suspect searching for them would turn up some).
>> 
>> The turris MOX (https://www.turris.cz/en/specification/) might be a decent startimg point as it comes with one Gbethernet port and both a SGMII and a PCIe signals routed on a connector, they also have a 4 and an 8 port switch module, but for our purposes it might be possible to just create a small single Gb ethernet port board to get started.
>> 
>> Best Regards
>>        Sebastian
>> 
>>> 
>>> -Toke
>>> 
>>> [0] https://www.gl-inet.com/products/gl-ar750s/
>>> _______________________________________________
>>> Cerowrt-devel mailing list
>>> Cerowrt-devel@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> 
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
> 
> 
> 
> -- 
> 
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740

Re: [Cerowrt-devel] [Bloat] (no subject)

[Jonathan Foulkes]

Thanks for sharing Dave.

A good paper, but there are few gaps worthy of mentioning on this list:

Testing when there is an AQM present means the test must adapt to the challenge of smaller cwnd existing for any one stream, therefore it will take many more streams to saturate a line with cwnd = 30 than if the cwnd is allowed to grow to >1,000
In general, the impact of cwnd relative to saturation and impact on delay was not visited, and yet it’s critical. One of the reasons for spiky delays on high speed lines is the ginormous cwnds hogging the line with their 800ms+ RTT’s

Asymmetry of provisioned upload relative to download, at some point, the ack-stream can be held up by either lack of capacity or bloat in the uplink. So even though a link can deliver 300Mbps down, a bloated uplink of 5mbps might never allow that level to be reached.
There are ISPs provisioning truly crazy asymmetric service.

They do make a good point about the local network, WiFi specifically being the new bottleneck, which is why we included an iperf instance that can be started on the IQrouter to help run client to server tests that help spot local network capacity limits, typically on WiFi.

Regarding their point about ‘Cross traffic’ impact on measurements, Cake’s per-host / per-target fairness also complicates AQM-enabled testing from client devices. Which is why we make the built-in speed test the arbiter of true line capacity, as it factors for ALL traffic flowing through the router. But, as you mention, that is also a challenge from a CPU resource standpoint on higher speeds.

The biggest gap in this paper is not paying sufficient attention to latency as a critical metric, and one that is controllable by an AQM. Bufferbloat metrics have more impact on end-user experience than +/- 50Mbps on a 100mbps baseline. 
I was rather miffed they do not even mention the DSLreports.com speedtest, or the fast.com test, as those are the two that provide a bufferbloat metric.

The industry as whole MUST pay attention and socialize the relevancy of managed latencies as being critical to customer satisfaction and good application performance. And that starts with tests that clearly grade that critical aspect. 

Cheers,

Jonathan

> On May 15, 2019, at 3:58 AM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> If it helps any: Nick Feamster and Jason Livingood just published "
> Internet Speed Measurement: Current Challenges and Future
> Recommendations " ( https://arxiv.org/pdf/1905.02334.pdf ) a few days
> ago, and outlines quite a few problems going forward at higher speeds.
> I do wish the document had pointed out more clearly that router based
> measurements have problems also, with weaker cpus unable to source
> enough traffic for an accurate measurement, but I do hope this
> document has impact, and it's a good read, regardless.
> 
> Still, somehow getting it right at lower speeds is always on my mind.
> I'd long ago hoped that DSL devices would adopt BQL, and that
> cablemodems would also, thus moving packet processing a little higher
> on the stack so more advanced algorithms like cake could take hold.
> 
> On Wed, May 15, 2019 at 9:32 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> Hi All,
>> 
>> 
>> I believe the following to be relevant to this discussion: https://apenwarr.ca/log/20180808
>> Where he discusses a similar idea including implementation albeit aimed at lower bandwidth and sans the automatic bandwidth tracking.
>> 
>> 
>>> On May 15, 2019, at 01:34, David P. Reed <dpreed@deepplum.com> wrote:
>>> 
>>> 
>>> Ideally, it would need to be self-configuring, though... I.e., something
>>> like the IQRouter auto-measuring of the upstream bandwidth to tune the
>>> shaper.
>> 
>> @Jonathan from your experience how tricky is it to get reliable speedtest endpoints and how reliable are they in practice. And do you do any sanitization, like take another measure immediate if the measured rate differs from the last by more than XX% or something like that?
>> 
>> 
>>> 
>>> Sure, seems like this is easy to code because there are exactly two ports to measure, they can even be labeled physically "up" and "down" to indicate their function.
>> 
>> IMHO the real challenge is automated measurements over the internet at Gbps speeds. It is not hard to get some test going (by e.g. tapping into ookla's fast net of confederated measurement endpoints) but getting something where the servers can reliably saturate 1Gbps+ seems somewhat trickier (last time I looked one required a 1Gbps connection to the server to participate in speedtest.net, obviously not really suited for measuring Gbps speeds).
>> In the EU there exists a mandate for national regulators to establish and/or endorse an anointed "official" speedtests, untended to keep ISP marketing honest, that come with stricter guarantees (e.g. the official German speedtest, breitbandmessung.de will only admit tests if the servers are having sufficient bandwidth reserves to actually saturate the link; the enduser is required to select the speed-tier giving them a strong hint about the required rates I believe).
>> For my back-burner toy project "per-packet-overhead estimation on arbitrary link technology" I am currently facing the same problem, I need a traffic sink and source that can reliably saturate my link so I can measure maximum achievable goodput, so if anybody in the list has ideas, I am all ears/eyes.
>> 
>>> 
>>> For reference, the GL.iNet routers are tiny and nicely packaged, and run
>>> OpenWrt; they do have one with Gbit ports[0], priced around $70. I very
>>> much doubt it can actually push a gigabit, though, but I haven't had a
>>> chance to test it. However, losing the WiFi, and getting a slightly
>>> beefier SoC in there will probably be doable without the price going
>>> over $100, no?
>>> 
>>> I assume the WiFi silicon is probably the most costly piece of intellectual property in the system. So yeah. Maybe with the right parts being available, one could aim at $50 or less, without sales channel markup. (Raspberry Pi ARM64 boards don't have GigE, and I think that might be because the GigE interfaces are a bit pricey. However, the ARM64 SoC's available are typically Celeron-class multicore systems. I don't know why there aren't more ARM64 systems on a chip with dual GigE, but I suspect searching for them would turn up some).
>> 
>> The turris MOX (https://www.turris.cz/en/specification/) might be a decent startimg point as it comes with one Gbethernet port and both a SGMII and a PCIe signals routed on a connector, they also have a 4 and an 8 port switch module, but for our purposes it might be possible to just create a small single Gb ethernet port board to get started.
>> 
>> Best Regards
>>        Sebastian
>> 
>>> 
>>> -Toke
>>> 
>>> [0] https://www.gl-inet.com/products/gl-ar750s/
>>> _______________________________________________
>>> Cerowrt-devel mailing list
>>> Cerowrt-devel@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> 
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
> 
> 
> 
> -- 
> 
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Cerowrt-devel] [Bloat] (no subject)

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 9346 bytes --]


Pardon, but cwnd should NEVER be larger than the number of forwarding hops between source and destination.
Kleinrock and students recently proved that the optimum cwnd for both throughput and minimized latency is achieved when there is one packet or less in each outbound queue from source to destination (including cross traffic - meaning other flows sharing the same outbound queue.
 
Now the idea that cwnd should be in the 1000's of packets is totally absurd, unless the source or destination buffers (at the end hosts) are counted, and that would be needed if the TCP source and destination application might, for example, be "swapped out" and thus unable to actually send and acknowledge packets at the instant of receiving an ACK.
 
If cwnd is sort of compensating for "swapping out" the TCP endpoint processes so that they take milliseconds to provide or acknowledge receipt of a packet, then that's fine (if you want throughput and terrible latency), but that's not the congestion window. That's just cramming the operating system's scheduling delay into the TCP stack.
 
TCP is not supposed to be designed around slow OS process schedulers. Those buffers should never be allowed to build up in the transport network, where they kill latency for everyone. That's just terrible design, conflating OS scheduling with congestion management.
 
 
On Thursday, May 16, 2019 6:01pm, "Jonathan Foulkes" <jf@jonathanfoulkes.com> said:



> Thanks for sharing Dave.
> 
> A good paper, but there are few gaps worthy of mentioning on this list:
> 
> Testing when there is an AQM present means the test must adapt to the challenge of
> smaller cwnd existing for any one stream, therefore it will take many more streams
> to saturate a line with cwnd = 30 than if the cwnd is allowed to grow to
> >1,000
> In general, the impact of cwnd relative to saturation and impact on delay was not
> visited, and yet it’s critical. One of the reasons for spiky delays on high
> speed lines is the ginormous cwnds hogging the line with their 800ms+ RTT’s
> 
> Asymmetry of provisioned upload relative to download, at some point, the
> ack-stream can be held up by either lack of capacity or bloat in the uplink. So
> even though a link can deliver 300Mbps down, a bloated uplink of 5mbps might never
> allow that level to be reached.
> There are ISPs provisioning truly crazy asymmetric service.
> 
> They do make a good point about the local network, WiFi specifically being the new
> bottleneck, which is why we included an iperf instance that can be started on the
> IQrouter to help run client to server tests that help spot local network capacity
> limits, typically on WiFi.
> 
> Regarding their point about ‘Cross traffic’ impact on measurements,
> Cake’s per-host / per-target fairness also complicates AQM-enabled testing
> from client devices. Which is why we make the built-in speed test the arbiter of
> true line capacity, as it factors for ALL traffic flowing through the router. But,
> as you mention, that is also a challenge from a CPU resource standpoint on higher
> speeds.
> 
> The biggest gap in this paper is not paying sufficient attention to latency as a
> critical metric, and one that is controllable by an AQM. Bufferbloat metrics have
> more impact on end-user experience than +/- 50Mbps on a 100mbps baseline.
> I was rather miffed they do not even mention the DSLreports.com speedtest, or the
> fast.com test, as those are the two that provide a bufferbloat metric.
> 
> The industry as whole MUST pay attention and socialize the relevancy of managed
> latencies as being critical to customer satisfaction and good application
> performance. And that starts with tests that clearly grade that critical aspect.
> 
> Cheers,
> 
> Jonathan
> 
> > On May 15, 2019, at 3:58 AM, Dave Taht <dave.taht@gmail.com> wrote:
> >
> > If it helps any: Nick Feamster and Jason Livingood just published "
> > Internet Speed Measurement: Current Challenges and Future
> > Recommendations " ( https://arxiv.org/pdf/1905.02334.pdf ) a few days
> > ago, and outlines quite a few problems going forward at higher speeds.
> > I do wish the document had pointed out more clearly that router based
> > measurements have problems also, with weaker cpus unable to source
> > enough traffic for an accurate measurement, but I do hope this
> > document has impact, and it's a good read, regardless.
> >
> > Still, somehow getting it right at lower speeds is always on my mind.
> > I'd long ago hoped that DSL devices would adopt BQL, and that
> > cablemodems would also, thus moving packet processing a little higher
> > on the stack so more advanced algorithms like cake could take hold.
> >
> > On Wed, May 15, 2019 at 9:32 AM Sebastian Moeller <moeller0@gmx.de>
> wrote:
> >>
> >> Hi All,
> >>
> >>
> >> I believe the following to be relevant to this discussion:
> https://apenwarr.ca/log/20180808
> >> Where he discusses a similar idea including implementation albeit aimed
> at lower bandwidth and sans the automatic bandwidth tracking.
> >>
> >>
> >>> On May 15, 2019, at 01:34, David P. Reed <dpreed@deepplum.com>
> wrote:
> >>>
> >>>
> >>> Ideally, it would need to be self-configuring, though... I.e.,
> something
> >>> like the IQRouter auto-measuring of the upstream bandwidth to tune
> the
> >>> shaper.
> >>
> >> @Jonathan from your experience how tricky is it to get reliable speedtest
> endpoints and how reliable are they in practice. And do you do any sanitization,
> like take another measure immediate if the measured rate differs from the last by
> more than XX% or something like that?
> >>
> >>
> >>>
> >>> Sure, seems like this is easy to code because there are exactly two
> ports to measure, they can even be labeled physically "up" and "down" to indicate
> their function.
> >>
> >> IMHO the real challenge is automated measurements over the internet at
> Gbps speeds. It is not hard to get some test going (by e.g. tapping into ookla's
> fast net of confederated measurement endpoints) but getting something where the
> servers can reliably saturate 1Gbps+ seems somewhat trickier (last time I looked
> one required a 1Gbps connection to the server to participate in speedtest.net,
> obviously not really suited for measuring Gbps speeds).
> >> In the EU there exists a mandate for national regulators to establish
> and/or endorse an anointed "official" speedtests, untended to keep ISP marketing
> honest, that come with stricter guarantees (e.g. the official German speedtest,
> breitbandmessung.de will only admit tests if the servers are having sufficient
> bandwidth reserves to actually saturate the link; the enduser is required to
> select the speed-tier giving them a strong hint about the required rates I
> believe).
> >> For my back-burner toy project "per-packet-overhead estimation on
> arbitrary link technology" I am currently facing the same problem, I need a
> traffic sink and source that can reliably saturate my link so I can measure
> maximum achievable goodput, so if anybody in the list has ideas, I am all
> ears/eyes.
> >>
> >>>
> >>> For reference, the GL.iNet routers are tiny and nicely packaged, and
> run
> >>> OpenWrt; they do have one with Gbit ports[0], priced around $70. I
> very
> >>> much doubt it can actually push a gigabit, though, but I haven't had
> a
> >>> chance to test it. However, losing the WiFi, and getting a slightly
> >>> beefier SoC in there will probably be doable without the price going
> >>> over $100, no?
> >>>
> >>> I assume the WiFi silicon is probably the most costly piece of
> intellectual property in the system. So yeah. Maybe with the right parts being
> available, one could aim at $50 or less, without sales channel markup. (Raspberry
> Pi ARM64 boards don't have GigE, and I think that might be because the GigE
> interfaces are a bit pricey. However, the ARM64 SoC's available are typically
> Celeron-class multicore systems. I don't know why there aren't more ARM64 systems
> on a chip with dual GigE, but I suspect searching for them would turn up some).
> >>
> >> The turris MOX (https://www.turris.cz/en/specification/) might be a
> decent startimg point as it comes with one Gbethernet port and both a SGMII and a
> PCIe signals routed on a connector, they also have a 4 and an 8 port switch
> module, but for our purposes it might be possible to just create a small single Gb
> ethernet port board to get started.
> >>
> >> Best Regards
> >> Sebastian
> >>
> >>>
> >>> -Toke
> >>>
> >>> [0] https://www.gl-inet.com/products/gl-ar750s/
> >>> _______________________________________________
> >>> Cerowrt-devel mailing list
> >>> Cerowrt-devel@lists.bufferbloat.net
> >>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> >>
> >> _______________________________________________
> >> Bloat mailing list
> >> Bloat@lists.bufferbloat.net
> >> https://lists.bufferbloat.net/listinfo/bloat
> >
> >
> >
> > --
> >
> > Dave Täht
> > CTO, TekLibre, LLC
> > http://www.teklibre.com
> > Tel: 1-831-205-9740
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
> 
> 

[-- Attachment #2: Type: text/html, Size: 12309 bytes --]

Re: [Cerowrt-devel] [Bloat] (no subject)

[Jonathan Morton]

> On 19 May, 2019, at 1:36 am, David P. Reed <dpreed@deepplum.com> wrote:
> 
> Pardon, but cwnd should NEVER be larger than the number of forwarding hops between source and destination.
> Kleinrock and students recently proved that the optimum cwnd for both throughput and minimized latency is achieved when there is one packet or less in each outbound queue from source to destination (including cross traffic - meaning other flows sharing the same outbound queue.

This argument holds only if time-of-flight *between* nodes is negligible.  Trivially, a geosynchronous satellite hop adds only two nodes but approximately half a second to the one-way path delay, with potentially thousands of packets existing only as radio waves in the distance between, not in a queue.

 - Jonathan Morton

Re: [Cerowrt-devel] [Bloat] (no subject)

[Jonathan Morton]

>> Pardon, but cwnd should NEVER be larger than the number of forwarding hops between source and destination.
>> Kleinrock and students recently proved that the optimum cwnd for both throughput and minimized latency is achieved when there is one packet or less in each outbound queue from source to destination (including cross traffic - meaning other flows sharing the same outbound queue.
> 
> This argument holds only if time-of-flight *between* nodes is negligible.  Trivially, a geosynchronous satellite hop adds only two nodes but approximately half a second to the one-way path delay, with potentially thousands of packets existing only as radio waves in the distance between, not in a queue.

Continuing my train of thought, what Kleinrock really implies is that the cwnd should *exceed* the native BDP by at most the number of hops.

 - Jonathan Morton

Re: [Cerowrt-devel] [Bloat] (no subject)

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 2440 bytes --]


Correction accepted. Between the US east and west coasts, the time of flight of packets on fiber or cable is about 23 msec. (Boston-LA, driving route, over fiber, at 207 Mm/sec).
 
So, if all intermediate links are equal in rate, at say, 10 Gb/sec, that means that there should be no more than 10,000,000,000 * 0.023 bits actually in transit on the actual fiber, plus a packet in each router between's outbound queue. Let's say a packet is around 1500 bytes, or 12,000 bits, since that is the MTU we stupidly enforce even today, and there are 10 hops (typical today between Boston and LA.)
 
So we would expect the optimal window size sum, for all flows on any hop of that path to be 10 * 12000 bits + 230,000,000 bits:
 
230,120,000 bits in flight between BOS and LAX. Divide that by 12,000 bits/packet, and you get about 19,177 packets along that path. At most points along the path, you would expect about 10 different flows or more to be in flight, so there would be, optimally, about 1,918 1500 byte packets. Each flow would get 1 Gb/sec as its share.
 
If the connection is limited to < 1 Gb/sec at either endpoint, then there's no reason for any intermediate node to buffer that much of the flow.
 
This gives a reasonable understanding of where AQM should be ending up in terms of the cwnd needed to sustain max throughput and optimum end-to-end latency (which would be about 23 msec + 10 hops * 12000 / 1 Gb/sec.  = 23.012 msec. for a packet to get from one end to the other).
 
 
 
On Saturday, May 18, 2019 6:57pm, "Jonathan Morton" <chromatix99@gmail.com> said:



> > On 19 May, 2019, at 1:36 am, David P. Reed <dpreed@deepplum.com>
> wrote:
> >
> > Pardon, but cwnd should NEVER be larger than the number of forwarding hops
> between source and destination.
> > Kleinrock and students recently proved that the optimum cwnd for both
> throughput and minimized latency is achieved when there is one packet or less in
> each outbound queue from source to destination (including cross traffic - meaning
> other flows sharing the same outbound queue.
> 
> This argument holds only if time-of-flight *between* nodes is negligible. 
> Trivially, a geosynchronous satellite hop adds only two nodes but approximately
> half a second to the one-way path delay, with potentially thousands of packets
> existing only as radio waves in the distance between, not in a queue.
> 
> - Jonathan Morton
> 
> 

[-- Attachment #2: Type: text/html, Size: 4282 bytes --]

Re: [Cerowrt-devel] [Bloat] (no subject)

[Jonathan Foulkes]

> @Jonathan from your experience how tricky is it to get reliable speedtest endpoints and how reliable are they in practice. And do you do any sanitization, like take another measure immediate if the measured rate differs from the last by more than XX% or something like that?

It is tricky to get the profiling of a line right. For one, access to the test endpoints need to be managed so concurrency does not swamp the server/link capacity, otherwise the test results are suspect. You don’t know if measured capacity was limited due to link constraints or server capacity. We spent a good bit of time engineering our capacity testing infrastructure and process to the point of where it is quite reliable in terms of metrics.

Multiple measurements over time are required to correctly profile a line. Generally, early morning (4am or so) tests find the high-water mark on most lines.

As discussed elsewhere, another challenge as line capacity goes up, is where the source of the test is run from. On-router testing is fine for sub-100mbps, but becomes more and more of challenge as speeds go up. Testing a Gbps line using an in-router process takes a pretty beefy box in terms of CPU and NICs.

Best regards,

Jonathan

> On May 15, 2019, at 3:31 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> Hi All,
> 
> 
> I believe the following to be relevant to this discussion: https://apenwarr.ca/log/20180808
> Where he discusses a similar idea including implementation albeit aimed at lower bandwidth and sans the automatic bandwidth tracking.
> 
> 
>> On May 15, 2019, at 01:34, David P. Reed <dpreed@deepplum.com> wrote:
>> 
>> 
>> Ideally, it would need to be self-configuring, though... I.e., something
>> like the IQRouter auto-measuring of the upstream bandwidth to tune the
>> shaper.
> 
> @Jonathan from your experience how tricky is it to get reliable speedtest endpoints and how reliable are they in practice. And do you do any sanitization, like take another measure immediate if the measured rate differs from the last by more than XX% or something like that?
> 
> 
>> 
>> Sure, seems like this is easy to code because there are exactly two ports to measure, they can even be labeled physically "up" and "down" to indicate their function.
> 
> IMHO the real challenge is automated measurements over the internet at Gbps speeds. It is not hard to get some test going (by e.g. tapping into ookla's fast net of confederated measurement endpoints) but getting something where the servers can reliably saturate 1Gbps+ seems somewhat trickier (last time I looked one required a 1Gbps connection to the server to participate in speedtest.net, obviously not really suited for measuring Gbps speeds).
> In the EU there exists a mandate for national regulators to establish and/or endorse an anointed "official" speedtests, untended to keep ISP marketing honest, that come with stricter guarantees (e.g. the official German speedtest, breitbandmessung.de will only admit tests if the servers are having sufficient bandwidth reserves to actually saturate the link; the enduser is required to select the speed-tier giving them a strong hint about the required rates I believe).
> For my back-burner toy project "per-packet-overhead estimation on arbitrary link technology" I am currently facing the same problem, I need a traffic sink and source that can reliably saturate my link so I can measure maximum achievable goodput, so if anybody in the list has ideas, I am all ears/eyes.
> 
>> 
>> For reference, the GL.iNet routers are tiny and nicely packaged, and run
>> OpenWrt; they do have one with Gbit ports[0], priced around $70. I very
>> much doubt it can actually push a gigabit, though, but I haven't had a
>> chance to test it. However, losing the WiFi, and getting a slightly
>> beefier SoC in there will probably be doable without the price going
>> over $100, no?
>> 
>> I assume the WiFi silicon is probably the most costly piece of intellectual property in the system. So yeah. Maybe with the right parts being available, one could aim at $50 or less, without sales channel markup. (Raspberry Pi ARM64 boards don't have GigE, and I think that might be because the GigE interfaces are a bit pricey. However, the ARM64 SoC's available are typically Celeron-class multicore systems. I don't know why there aren't more ARM64 systems on a chip with dual GigE, but I suspect searching for them would turn up some).
> 
> The turris MOX (https://www.turris.cz/en/specification/) might be a decent startimg point as it comes with one Gbethernet port and both a SGMII and a PCIe signals routed on a connector, they also have a 4 and an 8 port switch module, but for our purposes it might be possible to just create a small single Gb ethernet port board to get started. 
> 
> Best Regards
> 	Sebastian
> 
>> 
>> -Toke
>> 
>> [0] https://www.gl-inet.com/products/gl-ar750s/
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Cerowrt-devel] [Bloat] (no subject)

[Sebastian Moeller]

Hi Jonathan,

thanks for your isights, it is always good to talk to folks (or would that be foulkes) that actually implemented an idea ;)

> On May 16, 2019, at 18:40, Jonathan Foulkes <jf@jonathanfoulkes.com> wrote:
> 
>> @Jonathan from your experience how tricky is it to get reliable speedtest endpoints and how reliable are they in practice. And do you do any sanitization, like take another measure immediate if the measured rate differs from the last by more than XX% or something like that?
> 
> It is tricky to get the profiling of a line right. For one, access to the test endpoints need to be managed so concurrency does not swamp the server/link capacity, otherwise the test results are suspect. You don’t know if measured capacity was limited due to link constraints or server capacity. We spent a good bit of time engineering our capacity testing infrastructure and process to the point of where it is quite reliable in terms of metrics.

	Ah, I was reading similar things for the German regulators official speedtest site and wondered how often that does cause issues in reality; thanks to your data point I understand this is not just a theoretical concern.

> 
> Multiple measurements over time are required to correctly profile a line. Generally, early morning (4am or so) tests find the high-water mark on most lines.

	Again, thanks for sharing, while obvious post-hoc, not something I would have looked for (I would have given way more importance to the most recent measurement; clearly demonstrating that I have not actually tried to implement something like this ;) ).

> 
> As discussed elsewhere, another challenge as line capacity goes up, is where the source of the test is run from. On-router testing is fine for sub-100mbps, but becomes more and more of challenge as speeds go up. Testing a Gbps line using an in-router process takes a pretty beefy box in terms of CPU and NICs.

	Great; that certainly puts a somewhat higher lower performance bound on devices to be useful for David's idea, the CPU(s) need to be beefy enough to both shape at Gbps speeds as well as actually generate enough traffic to saturate the link....

Again thanks a lot for sharing!

Best Regards
	Sebastian


> 
> Best regards,
> 
> Jonathan
> 
>> On May 15, 2019, at 3:31 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> Hi All,
>> 
>> 
>> I believe the following to be relevant to this discussion: https://apenwarr.ca/log/20180808
>> Where he discusses a similar idea including implementation albeit aimed at lower bandwidth and sans the automatic bandwidth tracking.
>> 
>> 
>>> On May 15, 2019, at 01:34, David P. Reed <dpreed@deepplum.com> wrote:
>>> 
>>> 
>>> Ideally, it would need to be self-configuring, though... I.e., something
>>> like the IQRouter auto-measuring of the upstream bandwidth to tune the
>>> shaper.
>> 
>> @Jonathan from your experience how tricky is it to get reliable speedtest endpoints and how reliable are they in practice. And do you do any sanitization, like take another measure immediate if the measured rate differs from the last by more than XX% or something like that?
>> 
>> 
>>> 
>>> Sure, seems like this is easy to code because there are exactly two ports to measure, they can even be labeled physically "up" and "down" to indicate their function.
>> 
>> IMHO the real challenge is automated measurements over the internet at Gbps speeds. It is not hard to get some test going (by e.g. tapping into ookla's fast net of confederated measurement endpoints) but getting something where the servers can reliably saturate 1Gbps+ seems somewhat trickier (last time I looked one required a 1Gbps connection to the server to participate in speedtest.net, obviously not really suited for measuring Gbps speeds).
>> In the EU there exists a mandate for national regulators to establish and/or endorse an anointed "official" speedtests, untended to keep ISP marketing honest, that come with stricter guarantees (e.g. the official German speedtest, breitbandmessung.de will only admit tests if the servers are having sufficient bandwidth reserves to actually saturate the link; the enduser is required to select the speed-tier giving them a strong hint about the required rates I believe).
>> For my back-burner toy project "per-packet-overhead estimation on arbitrary link technology" I am currently facing the same problem, I need a traffic sink and source that can reliably saturate my link so I can measure maximum achievable goodput, so if anybody in the list has ideas, I am all ears/eyes.
>> 
>>> 
>>> For reference, the GL.iNet routers are tiny and nicely packaged, and run
>>> OpenWrt; they do have one with Gbit ports[0], priced around $70. I very
>>> much doubt it can actually push a gigabit, though, but I haven't had a
>>> chance to test it. However, losing the WiFi, and getting a slightly
>>> beefier SoC in there will probably be doable without the price going
>>> over $100, no?
>>> 
>>> I assume the WiFi silicon is probably the most costly piece of intellectual property in the system. So yeah. Maybe with the right parts being available, one could aim at $50 or less, without sales channel markup. (Raspberry Pi ARM64 boards don't have GigE, and I think that might be because the GigE interfaces are a bit pricey. However, the ARM64 SoC's available are typically Celeron-class multicore systems. I don't know why there aren't more ARM64 systems on a chip with dual GigE, but I suspect searching for them would turn up some).
>> 
>> The turris MOX (https://www.turris.cz/en/specification/) might be a decent startimg point as it comes with one Gbethernet port and both a SGMII and a PCIe signals routed on a connector, they also have a 4 and an 8 port switch module, but for our purposes it might be possible to just create a small single Gb ethernet port board to get started. 
>> 
>> Best Regards
>> 	Sebastian
>> 
>>> 
>>> -Toke
>>> 
>>> [0] https://www.gl-inet.com/products/gl-ar750s/
>>> _______________________________________________
>>> Cerowrt-devel mailing list
>>> Cerowrt-devel@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> 
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
> 

Re: [Cerowrt-devel] [Bloat] fq_codel is SEVEN years old today...

[Dave Taht]

The future seemed so bright after free.fr deployed it so fast. And so
far as I can tell, fq_codelis ubiquitous across all of linux and linux
containers and vms, and we've headed off major trouble at the wifi
pass with the OSX and QCA (ath9k, ath10k) implementations. 10s of
millions deployed - but where it's most needed, it's not, and ISPs
mostly ship it not.

So while I do take heart in the enormous deployment and figure we'll
turn the corner in the next year or seven on these other devices, I
was grumpily looking over:

http://conferences.sigcomm.org/sigcomm/2014/doc/slides/137.pdf

again today, and wishing we had some way of addressing the structural
problems we have with academic and industry research in general.

On Wed, May 15, 2019 at 1:34 AM David P. Reed <dpreed@deepplum.com> wrote:
>
>
>
> Ideally, it would need to be self-configuring, though... I.e., something
> like the IQRouter auto-measuring of the upstream bandwidth to tune the
> shaper.
>
>
>
> Sure, seems like this is easy to code because there are exactly two ports to measure, they can even be labeled physically "up" and "down" to indicate their function.
>
>
> For reference, the GL.iNet routers are tiny and nicely packaged, and run
> OpenWrt; they do have one with Gbit ports[0], priced around $70. I very
> much doubt it can actually push a gigabit, though, but I haven't had a
> chance to test it. However, losing the WiFi, and getting a slightly
> beefier SoC in there will probably be doable without the price going
> over $100, no?
>
>
>
> I assume the WiFi silicon is probably the most costly piece of intellectual property in the system. So yeah. Maybe with the right parts being available, one could aim at $50 or less, without sales channel markup. (Raspberry Pi ARM64 boards don't have GigE, and I think that might be because the GigE interfaces are a bit pricey. However, the ARM64 SoC's available are typically Celeron-class multicore systems. I don't know why there aren't more ARM64 systems on a chip with dual GigE, but I suspect searching for them would turn up some).
>
> -Toke
>
> [0] https://www.gl-inet.com/products/gl-ar750s/
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740