[Rpm] Seeking RPM Server package for OpenWrt

[Rpm] Seeking RPM Server package for OpenWrt

[Rich Brown]

The Apple "RPM Tool" is great for measuring network responsiveness.

High numbers from the tool (measured in round-trips per minute - or "RPM") show your network is responsive, even when it's heavily loaded with traffic. This also implies you have low "bufferbloat" - which is good.

There are several RPM clients available:

* `/usr/bin/networkQuality` on macOS Monterey
* an iOS 15 version described at https://support.apple.com/en-gb/HT212313
* a golang implementation at https://github.com/network-quality/goresponsiveness
* a Docker implementation in the same repository

BUT... These all test against servers "out on the internet". There's another interesting test to be had: testing against the local router. 

This is useful because it would help test the responsiveness of the Wi-Fi network/drivers of your router. It would allow you to measure whether in fact, you actually are too far from the router, and whether moving closer would help. 

My request... Is anyone interested in creating an OpenWrt package that implements an RPM server? 

Fundamentally, an RPM server is an HTTPS server that responds to the four URLs described on Page 12 of the Responsiveness spec at: https://github.com/network-quality/draft-ietf-ippm-responsiveness/blob/master/draft-ietf-ippm-responsiveness.pdf

Thanks.

Re: [Rpm] Seeking RPM Server package for OpenWrt

[Paul Spooren]

Hi Rich,

> On 23. Mar 2022, at 11:30, Rich Brown <richb.hanover@gmail.com> wrote:
> 
> The Apple "RPM Tool" is great for measuring network responsiveness.
> 
> High numbers from the tool (measured in round-trips per minute - or "RPM") show your network is responsive, even when it's heavily loaded with traffic. This also implies you have low "bufferbloat" - which is good.
> 
> There are several RPM clients available:
> 
> * `/usr/bin/networkQuality` on macOS Monterey
> * an iOS 15 version described at https://support.apple.com/en-gb/HT212313
> * a golang implementation at https://github.com/network-quality/goresponsiveness
> * a Docker implementation in the same repository
> 
> BUT... These all test against servers "out on the internet". There's another interesting test to be had: testing against the local router. 

The spec wants a 8GB file which seems a bit much for common home routers. We could look into reading from /dev/zero since the body content isn’t relevant but still the device is likely slower at offering the content than your laptop can chew. A dedicated device could be required.

Did you ask upstream about your idea? Maybe they have something in mind already.

> 
> This is useful because it would help test the responsiveness of the Wi-Fi network/drivers of your router. It would allow you to measure whether in fact, you actually are too far from the router, and whether moving closer would help. 
> 
> My request... Is anyone interested in creating an OpenWrt package that implements an RPM server? 
> 
> Fundamentally, an RPM server is an HTTPS server that responds to the four URLs described on Page 12 of the Responsiveness spec at: https://github.com/network-quality/draft-ietf-ippm-responsiveness/blob/master/draft-ietf-ippm-responsiveness.pdf
> 
> Thanks.
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel@lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-devel

Re: [Rpm] Seeking RPM Server package for OpenWrt

[Bjørn Mork]

Paul Spooren <mail@aparcar.org> writes:

> The spec wants a 8GB file which seems a bit much for common home
> routers. We could look into reading from /dev/zero since the body
> content isn’t relevant but still the device is likely slower at
> offering the content than your laptop can chew. A dedicated device
> could be required.

There is no need to read anything from a file or device.  You can just
serve the same memory buffer in a loop.

I did a quick look at the document and it seems under-specified.  Page
after page with blah-blah, but
- not defining Content-Type for any of the URLs
- not defining ciphers or any other TLS options, despite the rather
  restrictive TLSv1.3 requirment
- no config examples for common web servers
- no actual client algorithm

The last point is obviously the biggest problem.  You can do whatever
you want when implementng this, so the results from different clients
will not be comparable at all.

IMHO it's better let this soak for a while until they've reversed the
blah-blah to content ratio.  This doesn't look like a finished protocol.



Bjørn

Re: [Rpm] Seeking RPM Server package for OpenWrt

[Rich Brown]

Bjørn,

Thanks for these comments.

> On Mar 23, 2022, at 8:34 AM, Bjørn Mork <bjorn@mork.no> wrote:
> 
> There is no need to read anything from a file or device.  You can just
> serve the same memory buffer in a loop.

That might satisfy Paul Spooren's concern.

> I did a quick look at the document and it seems under-specified.  Page
> after page with blah-blah, but
> - not defining Content-Type for any of the URLs
> - not defining ciphers or any other TLS options, despite the rather
>  restrictive TLSv1.3 requirment
> - no config examples for common web servers
> - no actual client algorithm
> 
> The last point is obviously the biggest problem.  You can do whatever
> you want when implementng this, so the results from different clients
> will not be comparable at all.
> 
> IMHO it's better let this soak for a while until they've reversed the
> blah-blah to content ratio.  This doesn't look like a finished protocol.

Although I took an editorial pass over an earlier version of the RFC, I'm not in a position to address your questions. 

Let me propose this process for continuing the conversation. With this response, I'm cc'ing:

- openwrt-devel
- RPM mailing list
- named individuals

Not everyone is subscribed to both lists, but cc'ing everyone who responds should keep everyone included.

Thank you.

Rich

Re: [Rpm] Seeking RPM Server package for OpenWrt

[Paul Spooren]

> On 23. Mar 2022, at 12:57, Rich Brown <richb.hanover@gmail.com> wrote:
> 
> Bjørn,
> 
> Thanks for these comments.
> 
>> On Mar 23, 2022, at 8:34 AM, Bjørn Mork <bjorn@mork.no> wrote:
>> 
>> There is no need to read anything from a file or device.  You can just
>> serve the same memory buffer in a loop.

Thanks, I’ll check that in case I work on a server for OpenWrt.

> 
> That might satisfy Paul Spooren's concern.
> 
>> I did a quick look at the document and it seems under-specified.  Page
>> after page with blah-blah, but
>> - not defining Content-Type for any of the URLs

I guess it simply don’t matter, just like the content itself? In their demo setups they host either a single `x` (small file) or 4G times `x` (large file).

>> - not defining ciphers or any other TLS options, despite the rather
>> restrictive TLSv1.3 requirment
>> - no config examples for common web servers

https://github.com/network-quality/server

>> - no actual client algorithm

I found this implementation but din’t skim through the full code yet:
https://github.com/network-quality/goresponsiveness/blob/main/networkQuality.go

>> 
>> The last point is obviously the biggest problem.  You can do whatever
>> you want when implementng this, so the results from different clients
>> will not be comparable at all.
>> 
>> IMHO it's better let this soak for a while until they've reversed the
>> blah-blah to content ratio.  This doesn't look like a finished protocol.
> 
> Although I took an editorial pass over an earlier version of the RFC, I'm not in a position to address your questions. 
> 
> Let me propose this process for continuing the conversation. With this response, I'm cc'ing:
> 
> - openwrt-devel
> - RPM mailing list
> - named individuals
> 
> Not everyone is subscribed to both lists, but cc'ing everyone who responds should keep everyone included.
> 
> Thank you.
> 
> Rich

Re: [Rpm] Seeking RPM Server package for OpenWrt

[Christoph Paasch]

[-- Attachment #1: Type: text/plain, Size: 3129 bytes --]

Hello Bjorn,

Thanks for taking a look at this! Please see inline:

> On Mar 23, 2022, at 5:34 AM, Bjørn Mork via Rpm <rpm@lists.bufferbloat.net> wrote:
> 
> Paul Spooren <mail@aparcar.org> writes:
> 
>> The spec wants a 8GB file which seems a bit much for common home
>> routers. We could look into reading from /dev/zero since the body
>> content isn’t relevant but still the device is likely slower at
>> offering the content than your laptop can chew. A dedicated device
>> could be required.
> 
> There is no need to read anything from a file or device.  You can just
> serve the same memory buffer in a loop.

That's right! It does not really need to be a file. Some webserver implementations have such a capability to generate random content in memory. (e.g., https://docs.trafficserver.apache.org/en/9.0.x/admin-guide/plugins/generator.en.html <https://docs.trafficserver.apache.org/en/9.0.x/admin-guide/plugins/generator.en.html>)

> I did a quick look at the document and it seems under-specified.  Page
> after page with blah-blah, but
> - not defining Content-Type for any of the URLs

In what way is the content-type relevant for the responsiveness measurement ?

> - not defining ciphers or any other TLS options, despite the rather
>  restrictive TLSv1.3 requirment

I'm not sure in what way the cipher-suites are relevant to the responsiveness measurement itself. In terms of deployment, it is the same as for any other webservice. It is something that is usually not specified in an IETF-draft as cipher-suites come and go.

The TLSv1.3 requirement comes from the fact that we want to measure TLS handshake latency, and by requiring TLSv1.3 we know that the handshake is exactly 1 round-trip. Probably something to clarify in the draft! I filed https://github.com/network-quality/draft-ietf-ippm-responsiveness/issues/37 <https://github.com/network-quality/draft-ietf-ippm-responsiveness/issues/37>.

> - no config examples for common web servers

It is uncommon for an IETF-draft to provide such kind of configurations, because IETF-drafts are aiming to be implementation independent as implementations change, but standards don't.
We have several configurations (and two implementations - one in Go and one in Swift) available at https://github.com/network-quality/server/ <https://github.com/network-quality/server/>.

> - no actual client algorithm

Section 4 of the draft tries to explain the client algorithm. With specifically Section 4.1.4 formalizing the "working conditions" generation.  Can you explain a bit more what parts are unclear to you?

> The last point is obviously the biggest problem.  You can do whatever
> you want when implementng this, so the results from different clients
> will not be comparable at all.
> 
> IMHO it's better let this soak for a while until they've reversed the
> blah-blah to content ratio.  This doesn't look like a finished protocol.

We are actively looking for feedback like yours. Please explain more in detail what exactly is unclear, especially regarding the client algorithm.


Thanks,
Christoph



[-- Attachment #2: Type: text/html, Size: 4828 bytes --]

Re: [Rpm] Seeking RPM Server package for OpenWrt

[Bjørn Mork]

Christoph Paasch <cpaasch@apple.com> writes:

> Hello Bjorn,
>
> Thanks for taking a look at this! Please see inline:
>
>> On Mar 23, 2022, at 5:34 AM, Bjørn Mork via Rpm <rpm@lists.bufferbloat.net> wrote:
>> 
>> Paul Spooren <mail@aparcar.org> writes:
>> 
>>> The spec wants a 8GB file which seems a bit much for common home
>>> routers. We could look into reading from /dev/zero since the body
>>> content isn’t relevant but still the device is likely slower at
>>> offering the content than your laptop can chew. A dedicated device
>>> could be required.
>> 
>> There is no need to read anything from a file or device.  You can just
>> serve the same memory buffer in a loop.
>
> That's right! It does not really need to be a file. Some webserver
> implementations have such a capability to generate random content in
> memory. (e.g.,
> https://docs.trafficserver.apache.org/en/9.0.x/admin-guide/plugins/generator.en.html
> <https://docs.trafficserver.apache.org/en/9.0.x/admin-guide/plugins/generator.en.html>)
>
>> I did a quick look at the document and it seems under-specified.  Page
>> after page with blah-blah, but
>> - not defining Content-Type for any of the URLs
>
> In what way is the content-type relevant for the responsiveness measurement ?

It becomes relevant once one of the client or server implementations
start making assumptions about it.  Worst case is that you have two
implementations making different assumptions. So you specify strict
requirments to avoid that.

This is pretty basic for any API.  Maybe use OpenAPI or similar to
for clarity instead of the home-grown API spec?  


>> - not defining ciphers or any other TLS options, despite the rather
>>  restrictive TLSv1.3 requirment
>
> I'm not sure in what way the cipher-suites are relevant to the
> responsiveness measurement itself. In terms of deployment, it is the
> same as for any other webservice. It is something that is usually not
> specified in an IETF-draft as cipher-suites come and go.

They're relevant the same way the Content-Type is.  If you don't say
anything then you might end up with all sorts of incompatible
configurations.


> The TLSv1.3 requirement comes from the fact that we want to measure
> TLS handshake latency, and by requiring TLSv1.3 we know that the
> handshake is exactly 1 round-trip. Probably something to clarify in
> the draft! I filed
> https://github.com/network-quality/draft-ietf-ippm-responsiveness/issues/37
> <https://github.com/network-quality/draft-ietf-ippm-responsiveness/issues/37>.

Yes, that makes sense.  Thanks for explaining.  And I believe you should
include the explanation in the draft as well.

>> - no config examples for common web servers
>
> It is uncommon for an IETF-draft to provide such kind of
> configurations, because IETF-drafts are aiming to be implementation
> independent as implementations change, but standards don't.  We have
> several configurations (and two implementations - one in Go and one in
> Swift) available at https://github.com/network-quality/server/
> <https://github.com/network-quality/server/>.

I believe it's common to include a reference implementation if it's
semi-trivial, like the server side of this spec is.

And it's not unheard of that this reference implementation is given as
configuration examples, in cases where the documenent can be implemented
by configuring existing software.  For example:
https://datatracker.ietf.org/doc/html/rfc8806

Now I must admit that I haven't actually tried.  But I assume it's
possible to use most web servers for this purpose.  Or a pretty simple
python script, maybe.  

>> - no actual client algorithm
>
> Section 4 of the draft tries to explain the client algorithm. With
> specifically Section 4.1.4 formalizing the "working conditions"
> generation.  Can you explain a bit more what parts are unclear to you?

Re-reading this, I realize that I went out to harsh here.  Sorry.

I think it can be improved by replacing things like

   "It is left to the implementation what to do when saturation is not
   reached within that time-frame."

with a precise description of what to do.

But overall, you're right.  The algorithm is good enough.



Bjørn

Re: [Rpm] Seeking RPM Server package for OpenWrt

[Will Hawkins]

Thank you for all the conversation around RPM. I am one of the
developers of the Go's client that is hosted on Github and I am glad
to answer any questions. It seems like there is already some great
conversation going. I am a little under water with the day job over
the next few days, but please don't assume that I am not interested or
engaged.

Thanks again, everyone! Also, sorry about the top posting -- I just
didn't have anything context sensitive and thought it made sense to
just go at the top!

Will

On Wed, Mar 23, 2022 at 3:23 PM Bjørn Mork via Rpm
<rpm@lists.bufferbloat.net> wrote:
>
> Christoph Paasch <cpaasch@apple.com> writes:
>
> > Hello Bjorn,
> >
> > Thanks for taking a look at this! Please see inline:
> >
> >> On Mar 23, 2022, at 5:34 AM, Bjørn Mork via Rpm <rpm@lists.bufferbloat.net> wrote:
> >>
> >> Paul Spooren <mail@aparcar.org> writes:
> >>
> >>> The spec wants a 8GB file which seems a bit much for common home
> >>> routers. We could look into reading from /dev/zero since the body
> >>> content isn’t relevant but still the device is likely slower at
> >>> offering the content than your laptop can chew. A dedicated device
> >>> could be required.
> >>
> >> There is no need to read anything from a file or device.  You can just
> >> serve the same memory buffer in a loop.
> >
> > That's right! It does not really need to be a file. Some webserver
> > implementations have such a capability to generate random content in
> > memory. (e.g.,
> > https://docs.trafficserver.apache.org/en/9.0.x/admin-guide/plugins/generator.en.html
> > <https://docs.trafficserver.apache.org/en/9.0.x/admin-guide/plugins/generator.en.html>)
> >
> >> I did a quick look at the document and it seems under-specified.  Page
> >> after page with blah-blah, but
> >> - not defining Content-Type for any of the URLs
> >
> > In what way is the content-type relevant for the responsiveness measurement ?
>
> It becomes relevant once one of the client or server implementations
> start making assumptions about it.  Worst case is that you have two
> implementations making different assumptions. So you specify strict
> requirments to avoid that.
>
> This is pretty basic for any API.  Maybe use OpenAPI or similar to
> for clarity instead of the home-grown API spec?
>
>
> >> - not defining ciphers or any other TLS options, despite the rather
> >>  restrictive TLSv1.3 requirment
> >
> > I'm not sure in what way the cipher-suites are relevant to the
> > responsiveness measurement itself. In terms of deployment, it is the
> > same as for any other webservice. It is something that is usually not
> > specified in an IETF-draft as cipher-suites come and go.
>
> They're relevant the same way the Content-Type is.  If you don't say
> anything then you might end up with all sorts of incompatible
> configurations.
>
>
> > The TLSv1.3 requirement comes from the fact that we want to measure
> > TLS handshake latency, and by requiring TLSv1.3 we know that the
> > handshake is exactly 1 round-trip. Probably something to clarify in
> > the draft! I filed
> > https://github.com/network-quality/draft-ietf-ippm-responsiveness/issues/37
> > <https://github.com/network-quality/draft-ietf-ippm-responsiveness/issues/37>.
>
> Yes, that makes sense.  Thanks for explaining.  And I believe you should
> include the explanation in the draft as well.
>
> >> - no config examples for common web servers
> >
> > It is uncommon for an IETF-draft to provide such kind of
> > configurations, because IETF-drafts are aiming to be implementation
> > independent as implementations change, but standards don't.  We have
> > several configurations (and two implementations - one in Go and one in
> > Swift) available at https://github.com/network-quality/server/
> > <https://github.com/network-quality/server/>.
>
> I believe it's common to include a reference implementation if it's
> semi-trivial, like the server side of this spec is.
>
> And it's not unheard of that this reference implementation is given as
> configuration examples, in cases where the documenent can be implemented
> by configuring existing software.  For example:
> https://datatracker.ietf.org/doc/html/rfc8806
>
> Now I must admit that I haven't actually tried.  But I assume it's
> possible to use most web servers for this purpose.  Or a pretty simple
> python script, maybe.
>
> >> - no actual client algorithm
> >
> > Section 4 of the draft tries to explain the client algorithm. With
> > specifically Section 4.1.4 formalizing the "working conditions"
> > generation.  Can you explain a bit more what parts are unclear to you?
>
> Re-reading this, I realize that I went out to harsh here.  Sorry.
>
> I think it can be improved by replacing things like
>
>    "It is left to the implementation what to do when saturation is not
>    reached within that time-frame."
>
> with a precise description of what to do.
>
> But overall, you're right.  The algorithm is good enough.
>
>
>
> Bjørn
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm

Re: [Rpm] Seeking RPM Server package for OpenWrt

[Christoph Paasch]

> On Mar 23, 2022, at 12:23 PM, Bjørn Mork <bjorn@mork.no> wrote:
> 
> Christoph Paasch <cpaasch@apple.com> writes:
> 
>> Hello Bjorn,
>> 
>> Thanks for taking a look at this! Please see inline:
>> 
>>> On Mar 23, 2022, at 5:34 AM, Bjørn Mork via Rpm <rpm@lists.bufferbloat.net> wrote:
>>> 
>>> Paul Spooren <mail@aparcar.org> writes:
>>> 
>>>> The spec wants a 8GB file which seems a bit much for common home
>>>> routers. We could look into reading from /dev/zero since the body
>>>> content isn’t relevant but still the device is likely slower at
>>>> offering the content than your laptop can chew. A dedicated device
>>>> could be required.
>>> 
>>> There is no need to read anything from a file or device. You can just
>>> serve the same memory buffer in a loop.
>> 
>> That's right! It does not really need to be a file. Some webserver
>> implementations have such a capability to generate random content in
>> memory. (e.g.,
>> https://docs.trafficserver.apache.org/en/9.0.x/admin-guide/plugins/generator.en.html
>> <https://docs.trafficserver.apache.org/en/9.0.x/admin-guide/plugins/generator.en.html>)
>> 
>>> I did a quick look at the document and it seems under-specified. Page
>>> after page with blah-blah, but
>>> - not defining Content-Type for any of the URLs
>> 
>> In what way is the content-type relevant for the responsiveness measurement ?
> 
> It becomes relevant once one of the client or server implementations
> start making assumptions about it. Worst case is that you have two
> implementations making different assumptions. So you specify strict
> requirments to avoid that.

We could add a line saying that the content-type does not matter and must be ignored.

> This is pretty basic for any API. Maybe use OpenAPI or similar to
> for clarity instead of the home-grown API spec? 

I feel like OpenAPI is quite a different beast compared to what we are doing here.

>>> - not defining ciphers or any other TLS options, despite the rather
>>> restrictive TLSv1.3 requirment
>> 
>> I'm not sure in what way the cipher-suites are relevant to the
>> responsiveness measurement itself. In terms of deployment, it is the
>> same as for any other webservice. It is something that is usually not
>> specified in an IETF-draft as cipher-suites come and go.
> 
> They're relevant the same way the Content-Type is. If you don't say
> anything then you might end up with all sorts of incompatible
> configurations.

This is a deployment problem, not a specification-problem for the particular purpose of measuring responsiveness.

For example, we don't specify whether IPv4 or IPv6 should be used.
Also, if we list cipher-suites and one of them gets deprecated our RFC would need to be updated as well.

>> The TLSv1.3 requirement comes from the fact that we want to measure
>> TLS handshake latency, and by requiring TLSv1.3 we know that the
>> handshake is exactly 1 round-trip. Probably something to clarify in
>> the draft! I filed
>> https://github.com/network-quality/draft-ietf-ippm-responsiveness/issues/37
>> <https://github.com/network-quality/draft-ietf-ippm-responsiveness/issues/37>.
> 
> Yes, that makes sense. Thanks for explaining. And I believe you should
> include the explanation in the draft as well.
> 
>>> - no config examples for common web servers
>> 
>> It is uncommon for an IETF-draft to provide such kind of
>> configurations, because IETF-drafts are aiming to be implementation
>> independent as implementations change, but standards don't. We have
>> several configurations (and two implementations - one in Go and one in
>> Swift) available at https://github.com/network-quality/server/
>> <https://github.com/network-quality/server/>.
> 
> I believe it's common to include a reference implementation if it's
> semi-trivial, like the server side of this spec is.
> 
> And it's not unheard of that this reference implementation is given as
> configuration examples, in cases where the documenent can be implemented
> by configuring existing software. For example:
> https://datatracker.ietf.org/doc/html/rfc8806

Yeah, it could make sense to put something in the appendix (https://github.com/network-quality/draft-ietf-ippm-responsiveness/issues/39)

> 
> Now I must admit that I haven't actually tried. But I assume it's
> possible to use most web servers for this purpose. Or a pretty simple
> python script, maybe. 
> 
>>> - no actual client algorithm
>> 
>> Section 4 of the draft tries to explain the client algorithm. With
>> specifically Section 4.1.4 formalizing the "working conditions"
>> generation. Can you explain a bit more what parts are unclear to you?
> 
> Re-reading this, I realize that I went out to harsh here. Sorry.
> 
> I think it can be improved by replacing things like
> 
> "It is left to the implementation what to do when saturation is not
> reached within that time-frame."
> 
> with a precise description of what to do.

There are two approaches here:

Either, the implementation aborts and errors out.
Or, the implementation nevertheless measures the responsiveness and either presents the result as a valid result or as a result with a low confidence score.

We can probably outline the options that an implementation has. (https://github.com/network-quality/draft-ietf-ippm-responsiveness/issues/40)


Thanks for your feedback,
Christoph

Re: [Rpm] Seeking RPM Server package for OpenWrt

[Rich Brown]

[-- Attachment #1: Type: text/plain, Size: 716 bytes --]

Hi Paul,

> On Mar 23, 2022, at 8:05 AM, Paul Spooren <mail@aparcar.org> wrote:
> 
> The spec wants a 8GB file which seems a bit much for common home routers. We could look into reading from /dev/zero since the body content isn’t relevant but still the device is likely slower at offering the content than your laptop can chew. A dedicated device could be required.
> 
> Did you ask upstream about your idea? Maybe they have something in mind already.

Good point. But it's worse than that :-)

It's also possible low-powered CPUs won't be able to keep up with the data stream, so won't achieve saturation...

Still, I figured I would ask for help to see if it might be possible to try. Thanks!

Rich

[-- Attachment #2: Type: text/html, Size: 3069 bytes --]