[Codel] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Codel] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Dave Taht]

The program committee members are *amazing*. Perhaps, finally, we can
move the bar for the internet's quality metrics past endless, blind
repetitions of speedtest.

For complete details, please see:
https://www.iab.org/activities/workshops/network-quality/

Submissions Due: Monday 2nd August 2021, midnight AOE (Anywhere On Earth)
Invitations Issued by: Monday 16th August 2021

Workshop Date: This will be a virtual workshop, spread over three days:

1400-1800 UTC Tue 14th September 2021
1400-1800 UTC Wed 15th September 2021
1400-1800 UTC Thu 16th September 2021

Workshop co-chairs: Wes Hardaker, Evgeny Khorov, Omer Shapira

The Program Committee members:

Jari Arkko, Olivier Bonaventure, Vint Cerf, Stuart Cheshire, Sam
Crowford, Nick Feamster, Jim Gettys, Toke Hoiland-Jorgensen, Geoff
Huston, Cullen Jennings, Katarzyna Kosek-Szott, Mirja Kuehlewind,
Jason Livingood, Matt Mathias, Randall Meyer, Kathleen Nichols,
Christoph Paasch, Tommy Pauly, Greg White, Keith Winstein.

Send Submissions to: network-quality-workshop-pc@iab.org.

Position papers from academia, industry, the open source community and
others that focus on measurements, experiences, observations and
advice for the future are welcome. Papers that reflect experience
based on deployed services are especially welcome. The organizers
understand that specific actions taken by operators are unlikely to be
discussed in detail, so papers discussing general categories of
actions and issues without naming specific technologies, products, or
other players in the ecosystem are expected. Papers should not focus
on specific protocol solutions.

The workshop will be by invitation only. Those wishing to attend
should submit a position paper to the address above; it may take the
form of an Internet-Draft.

All inputs submitted and considered relevant will be published on the
workshop website. The organisers will decide whom to invite based on
the submissions received. Sessions will be organized according to
content, and not every accepted submission or invited attendee will
have an opportunity to present as the intent is to foster discussion
and not simply to have a sequence of presentations.

Position papers from those not planning to attend the virtual sessions
themselves are also encouraged. A workshop report will be published
afterwards.

Overview:

"We believe that one of the major factors behind this lack of progress
is the popular perception that throughput is the often sole measure of
the quality of Internet connectivity. With such narrow focus, people
don’t consider questions such as:

What is the latency under typical working conditions?
How reliable is the connectivity across longer time periods?
Does the network allow the use of a broad range of protocols?
What services can be run by clients of the network?
What kind of IPv4, NAT or IPv6 connectivity is offered, and are there firewalls?
What security mechanisms are available for local services, such as DNS?
To what degree are the privacy, confidentiality, integrity and
authenticity of user communications guarded?

Improving these aspects of network quality will likely depend on
measurement and exposing metrics to all involved parties, including to
end users in a meaningful way. Such measurements and exposure of the
right metrics will allow service providers and network operators to
focus on the aspects that impacts the users’ experience most and at
the same time empowers users to choose the Internet service that will
give them the best experience."


-- 
Latest Podcast:
https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/

Dave Täht CTO, TekLibre, LLC

Re: [Codel] [Cerowrt-devel] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Dave Taht]

In terms of trying to find "Quality" I have tried to encourage folk to
both read "zen and the art of motorcycle maintenance"[0], and Deming's
work on "total quality management".

My own slice at this network, computer and lifestyle "issue" is aiming
for "imperceptible latency" in all things. [1]. There's a lot of
fallout from that in terms of not just addressing queuing delay, but
caching, prefetching, and learning more about what a user really needs
(as opposed to wants) to know via intelligent agents.

[0] If you want to get depressed, read Pirsig's successor to "zen...",
lila, which is in part about what happens when an engineer hits an
insoluble problem.
[1] https://www.internetsociety.org/events/latency2013/



On Thu, Jul 1, 2021 at 6:16 PM David P. Reed <dpreed@deepplum.com> wrote:
>
> Well, nice that the folks doing the conference  are willing to consider that quality of user experience has little to do with signalling rate at the physical layer or throughput of FTP transfers.
>
>
>
> But honestly, the fact that they call the problem "network quality" suggests that they REALLY, REALLY don't understand the Internet isn't the hardware or the routers or even the routing algorithms *to its users*.
>
>
>
> By ignoring the diversity of applications now and in the future, and the fact that we DON'T KNOW what will be coming up, this conference will likely fall into the usual trap that net-heads fall into - optimizing for some imaginary reality that doesn't exist, and in fact will probably never be what users actually will do given the chance.
>
>
>
> I saw this issue in 1976 in the group developing the original Internet protocols - a desire to put *into the network* special tricks to optimize ASR33 logins to remote computers from terminal concentrators (aka remote login), bulk file transfers between file systems on different time-sharing systems, and "sessions" (virtual circuits) that required logins. And then trying to exploit underlying "multicast" by building it into the IP layer, because someone thought that TV broadcast would be the dominant application.
>
>
>
> Frankly, to think of "quality" as something that can be "provided" by "the network" misses the entire point of "end-to-end argument in system design". Quality is not a property defined or created by The Network. If you want to talk about Quality, you need to talk about users - all the users at all times, now and into the future, and that's something you can't do if you don't bother to include current and future users talking about what they might expect to experience that they don't experience.
>
>
>
> There was much fighting back in 1976 that basically involved "network experts" saying that the network was the place to "solve" such issues as quality, so applications could avoid having to solve such issues.
>
>
>
> What some of us managed to do was to argue that you can't "solve" such issues. All you can do is provide a framework that enables different uses to *cooperate* in some way.
>
>
>
> Which is why the Internet drops packets rather than queueing them, and why diffserv cannot work.
>
> (I know the latter is conftroversial, but at the moment, ALL of diffserv attempts to talk about end-to-end applicaiton specific metrics, but never, ever explains what the diffserv control points actually do w.r.t. what the IP layer can actually control. So it is meaningless - another violation of the so-called end-to-end principle).
>
>
>
> Networks are about getting packets from here to there, multiplexing the underlying resources. That's it. Quality is a whole different thing. Quality can be improved by end-to-end approaches, if the underlying network provides some kind of thing that actually creates a way for end-to-end applications to affect queueing and routing decisions, and more importantly getting "telemetry" from the network regarding what is actually going on with the other end-to-end users sharing the infrastructure.
>
>
>
> This conference won't talk about it this way. So don't waste your time.
>
>
>
>
>
>
>
> On Wednesday, June 30, 2021 8:12pm, "Dave Taht" <dave.taht@gmail.com> said:
>
> > The program committee members are *amazing*. Perhaps, finally, we can
> > move the bar for the internet's quality metrics past endless, blind
> > repetitions of speedtest.
> >
> > For complete details, please see:
> > https://www.iab.org/activities/workshops/network-quality/
> >
> > Submissions Due: Monday 2nd August 2021, midnight AOE (Anywhere On Earth)
> > Invitations Issued by: Monday 16th August 2021
> >
> > Workshop Date: This will be a virtual workshop, spread over three days:
> >
> > 1400-1800 UTC Tue 14th September 2021
> > 1400-1800 UTC Wed 15th September 2021
> > 1400-1800 UTC Thu 16th September 2021
> >
> > Workshop co-chairs: Wes Hardaker, Evgeny Khorov, Omer Shapira
> >
> > The Program Committee members:
> >
> > Jari Arkko, Olivier Bonaventure, Vint Cerf, Stuart Cheshire, Sam
> > Crowford, Nick Feamster, Jim Gettys, Toke Hoiland-Jorgensen, Geoff
> > Huston, Cullen Jennings, Katarzyna Kosek-Szott, Mirja Kuehlewind,
> > Jason Livingood, Matt Mathias, Randall Meyer, Kathleen Nichols,
> > Christoph Paasch, Tommy Pauly, Greg White, Keith Winstein.
> >
> > Send Submissions to: network-quality-workshop-pc@iab.org.
> >
> > Position papers from academia, industry, the open source community and
> > others that focus on measurements, experiences, observations and
> > advice for the future are welcome. Papers that reflect experience
> > based on deployed services are especially welcome. The organizers
> > understand that specific actions taken by operators are unlikely to be
> > discussed in detail, so papers discussing general categories of
> > actions and issues without naming specific technologies, products, or
> > other players in the ecosystem are expected. Papers should not focus
> > on specific protocol solutions.
> >
> > The workshop will be by invitation only. Those wishing to attend
> > should submit a position paper to the address above; it may take the
> > form of an Internet-Draft.
> >
> > All inputs submitted and considered relevant will be published on the
> > workshop website. The organisers will decide whom to invite based on
> > the submissions received. Sessions will be organized according to
> > content, and not every accepted submission or invited attendee will
> > have an opportunity to present as the intent is to foster discussion
> > and not simply to have a sequence of presentations.
> >
> > Position papers from those not planning to attend the virtual sessions
> > themselves are also encouraged. A workshop report will be published
> > afterwards.
> >
> > Overview:
> >
> > "We believe that one of the major factors behind this lack of progress
> > is the popular perception that throughput is the often sole measure of
> > the quality of Internet connectivity. With such narrow focus, people
> > don’t consider questions such as:
> >
> > What is the latency under typical working conditions?
> > How reliable is the connectivity across longer time periods?
> > Does the network allow the use of a broad range of protocols?
> > What services can be run by clients of the network?
> > What kind of IPv4, NAT or IPv6 connectivity is offered, and are there firewalls?
> > What security mechanisms are available for local services, such as DNS?
> > To what degree are the privacy, confidentiality, integrity and
> > authenticity of user communications guarded?
> >
> > Improving these aspects of network quality will likely depend on
> > measurement and exposing metrics to all involved parties, including to
> > end users in a meaningful way. Such measurements and exposure of the
> > right metrics will allow service providers and network operators to
> > focus on the aspects that impacts the users’ experience most and at
> > the same time empowers users to choose the Internet service that will
> > give them the best experience."
> >
> >
> > --
> > Latest Podcast:
> > https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/
> >
> > Dave Täht CTO, TekLibre, LLC
> > _______________________________________________
> > Cerowrt-devel mailing list
> > Cerowrt-devel@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cerowrt-devel
> >



--
Latest Podcast:
https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/

Dave Täht CTO, TekLibre, LLC

Re: [Codel] [Starlink] [Make-wifi-fast] [Cerowrt-devel] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Ben Greear]

Hello,

I am interested to hear wish lists for network testing features.  We make test equipment, supporting lots
of wifi stations and a distributed architecture, with built-in udp, tcp, ipv6, http, ... protocols,
and open to creating/improving some of our automated tests.

I know Dave has some test scripts already, so I'm not necessarily looking to reimplement that,
but more fishing for other/new ideas.

Thanks,
Ben

On 7/2/21 4:28 PM, Bob McMahon wrote:
> I think we need the language of math here. It seems like the network power metric, introduced by Kleinrock and Jaffe in the late 70s, is something useful. 
> Effective end/end queue depths per Little's law also seems useful. Both are available in iperf 2 from a test perspective. Repurposing test techniques to actual 
> traffic could be useful. Hence the question around what exact telemetry is useful to apps making socket write() and read() calls.
> 
> Bob
> 
> On Fri, Jul 2, 2021 at 10:07 AM Dave Taht <dave.taht@gmail.com <mailto:dave.taht@gmail.com>> wrote:
> 
>     In terms of trying to find "Quality" I have tried to encourage folk to
>     both read "zen and the art of motorcycle maintenance"[0], and Deming's
>     work on "total quality management".
> 
>     My own slice at this network, computer and lifestyle "issue" is aiming
>     for "imperceptible latency" in all things. [1]. There's a lot of
>     fallout from that in terms of not just addressing queuing delay, but
>     caching, prefetching, and learning more about what a user really needs
>     (as opposed to wants) to know via intelligent agents.
> 
>     [0] If you want to get depressed, read Pirsig's successor to "zen...",
>     lila, which is in part about what happens when an engineer hits an
>     insoluble problem.
>     [1] https://www.internetsociety.org/events/latency2013/ <https://www.internetsociety.org/events/latency2013/>
> 
> 
> 
>     On Thu, Jul 1, 2021 at 6:16 PM David P. Reed <dpreed@deepplum.com <mailto:dpreed@deepplum.com>> wrote:
>      >
>      > Well, nice that the folks doing the conference  are willing to consider that quality of user experience has little to do with signalling rate at the
>     physical layer or throughput of FTP transfers.
>      >
>      >
>      >
>      > But honestly, the fact that they call the problem "network quality" suggests that they REALLY, REALLY don't understand the Internet isn't the hardware or
>     the routers or even the routing algorithms *to its users*.
>      >
>      >
>      >
>      > By ignoring the diversity of applications now and in the future, and the fact that we DON'T KNOW what will be coming up, this conference will likely fall
>     into the usual trap that net-heads fall into - optimizing for some imaginary reality that doesn't exist, and in fact will probably never be what users
>     actually will do given the chance.
>      >
>      >
>      >
>      > I saw this issue in 1976 in the group developing the original Internet protocols - a desire to put *into the network* special tricks to optimize ASR33
>     logins to remote computers from terminal concentrators (aka remote login), bulk file transfers between file systems on different time-sharing systems, and
>     "sessions" (virtual circuits) that required logins. And then trying to exploit underlying "multicast" by building it into the IP layer, because someone
>     thought that TV broadcast would be the dominant application.
>      >
>      >
>      >
>      > Frankly, to think of "quality" as something that can be "provided" by "the network" misses the entire point of "end-to-end argument in system design".
>     Quality is not a property defined or created by The Network. If you want to talk about Quality, you need to talk about users - all the users at all times,
>     now and into the future, and that's something you can't do if you don't bother to include current and future users talking about what they might expect to
>     experience that they don't experience.
>      >
>      >
>      >
>      > There was much fighting back in 1976 that basically involved "network experts" saying that the network was the place to "solve" such issues as quality,
>     so applications could avoid having to solve such issues.
>      >
>      >
>      >
>      > What some of us managed to do was to argue that you can't "solve" such issues. All you can do is provide a framework that enables different uses to
>     *cooperate* in some way.
>      >
>      >
>      >
>      > Which is why the Internet drops packets rather than queueing them, and why diffserv cannot work.
>      >
>      > (I know the latter is conftroversial, but at the moment, ALL of diffserv attempts to talk about end-to-end applicaiton specific metrics, but never, ever
>     explains what the diffserv control points actually do w.r.t. what the IP layer can actually control. So it is meaningless - another violation of the
>     so-called end-to-end principle).
>      >
>      >
>      >
>      > Networks are about getting packets from here to there, multiplexing the underlying resources. That's it. Quality is a whole different thing. Quality can
>     be improved by end-to-end approaches, if the underlying network provides some kind of thing that actually creates a way for end-to-end applications to
>     affect queueing and routing decisions, and more importantly getting "telemetry" from the network regarding what is actually going on with the other
>     end-to-end users sharing the infrastructure.
>      >
>      >
>      >
>      > This conference won't talk about it this way. So don't waste your time.
>      >
>      >
>      >
>      >
>      >
>      >
>      >
>      > On Wednesday, June 30, 2021 8:12pm, "Dave Taht" <dave.taht@gmail.com <mailto:dave.taht@gmail.com>> said:
>      >
>      > > The program committee members are *amazing*. Perhaps, finally, we can
>      > > move the bar for the internet's quality metrics past endless, blind
>      > > repetitions of speedtest.
>      > >
>      > > For complete details, please see:
>      > > https://www.iab.org/activities/workshops/network-quality/ <https://www.iab.org/activities/workshops/network-quality/>
>      > >
>      > > Submissions Due: Monday 2nd August 2021, midnight AOE (Anywhere On Earth)
>      > > Invitations Issued by: Monday 16th August 2021
>      > >
>      > > Workshop Date: This will be a virtual workshop, spread over three days:
>      > >
>      > > 1400-1800 UTC Tue 14th September 2021
>      > > 1400-1800 UTC Wed 15th September 2021
>      > > 1400-1800 UTC Thu 16th September 2021
>      > >
>      > > Workshop co-chairs: Wes Hardaker, Evgeny Khorov, Omer Shapira
>      > >
>      > > The Program Committee members:
>      > >
>      > > Jari Arkko, Olivier Bonaventure, Vint Cerf, Stuart Cheshire, Sam
>      > > Crowford, Nick Feamster, Jim Gettys, Toke Hoiland-Jorgensen, Geoff
>      > > Huston, Cullen Jennings, Katarzyna Kosek-Szott, Mirja Kuehlewind,
>      > > Jason Livingood, Matt Mathias, Randall Meyer, Kathleen Nichols,
>      > > Christoph Paasch, Tommy Pauly, Greg White, Keith Winstein.
>      > >
>      > > Send Submissions to: network-quality-workshop-pc@iab.org <mailto:network-quality-workshop-pc@iab.org>.
>      > >
>      > > Position papers from academia, industry, the open source community and
>      > > others that focus on measurements, experiences, observations and
>      > > advice for the future are welcome. Papers that reflect experience
>      > > based on deployed services are especially welcome. The organizers
>      > > understand that specific actions taken by operators are unlikely to be
>      > > discussed in detail, so papers discussing general categories of
>      > > actions and issues without naming specific technologies, products, or
>      > > other players in the ecosystem are expected. Papers should not focus
>      > > on specific protocol solutions.
>      > >
>      > > The workshop will be by invitation only. Those wishing to attend
>      > > should submit a position paper to the address above; it may take the
>      > > form of an Internet-Draft.
>      > >
>      > > All inputs submitted and considered relevant will be published on the
>      > > workshop website. The organisers will decide whom to invite based on
>      > > the submissions received. Sessions will be organized according to
>      > > content, and not every accepted submission or invited attendee will
>      > > have an opportunity to present as the intent is to foster discussion
>      > > and not simply to have a sequence of presentations.
>      > >
>      > > Position papers from those not planning to attend the virtual sessions
>      > > themselves are also encouraged. A workshop report will be published
>      > > afterwards.
>      > >
>      > > Overview:
>      > >
>      > > "We believe that one of the major factors behind this lack of progress
>      > > is the popular perception that throughput is the often sole measure of
>      > > the quality of Internet connectivity. With such narrow focus, people
>      > > don’t consider questions such as:
>      > >
>      > > What is the latency under typical working conditions?
>      > > How reliable is the connectivity across longer time periods?
>      > > Does the network allow the use of a broad range of protocols?
>      > > What services can be run by clients of the network?
>      > > What kind of IPv4, NAT or IPv6 connectivity is offered, and are there firewalls?
>      > > What security mechanisms are available for local services, such as DNS?
>      > > To what degree are the privacy, confidentiality, integrity and
>      > > authenticity of user communications guarded?
>      > >
>      > > Improving these aspects of network quality will likely depend on
>      > > measurement and exposing metrics to all involved parties, including to
>      > > end users in a meaningful way. Such measurements and exposure of the
>      > > right metrics will allow service providers and network operators to
>      > > focus on the aspects that impacts the users’ experience most and at
>      > > the same time empowers users to choose the Internet service that will
>      > > give them the best experience."
>      > >
>      > >
>      > > --
>      > > Latest Podcast:
>      > > https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/ <https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/>
>      > >
>      > > Dave Täht CTO, TekLibre, LLC
>      > > _______________________________________________
>      > > Cerowrt-devel mailing list
>      > > Cerowrt-devel@lists.bufferbloat.net <mailto:Cerowrt-devel@lists.bufferbloat.net>
>      > > https://lists.bufferbloat.net/listinfo/cerowrt-devel <https://lists.bufferbloat.net/listinfo/cerowrt-devel>
>      > >
> 
> 
> 
>     --
>     Latest Podcast:
>     https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/ <https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/>
> 
>     Dave Täht CTO, TekLibre, LLC
>     _______________________________________________
>     Make-wifi-fast mailing list
>     Make-wifi-fast@lists.bufferbloat.net <mailto:Make-wifi-fast@lists.bufferbloat.net>
>     https://lists.bufferbloat.net/listinfo/make-wifi-fast <https://lists.bufferbloat.net/listinfo/make-wifi-fast>
> 
> 
> This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of 
> the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise 
> restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, 
> you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you 
> received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.
> 
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
> 


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: [Codel] [Starlink] [Make-wifi-fast] [Cerowrt-devel] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Ben Greear]

We tried adding in an external butler matrix in the past, but could not notice any useful difference.  Possibly
we didn't have the right use case.

Typically we are competitive on price for full testing solutions, but you can get stand-alone attenuators
cheaper from specialized vendors.  Happy to discuss pricing offlist if you wish.

Thanks,
Ben

On 7/6/21 1:43 PM, Bob McMahon wrote:
> The four part attenuator part would be more interesting to me if it also had a solid state phase shifters.  This allows for testing 2x2 MIMO testing per 
> affecting the spatial stream eigen vectors/values.
> 
> Bob
> 
> PS. The price per port isn't competitive. Probably a good idea to survey the market competition.
> 
> On Tue, Jul 6, 2021 at 6:46 AM Ben Greear <greearb@candelatech.com <mailto:greearb@candelatech.com>> wrote:
> 
>     Hello,
> 
>     I am interested to hear wish lists for network testing features.  We make test equipment, supporting lots
>     of wifi stations and a distributed architecture, with built-in udp, tcp, ipv6, http, ... protocols,
>     and open to creating/improving some of our automated tests.
> 
>     I know Dave has some test scripts already, so I'm not necessarily looking to reimplement that,
>     but more fishing for other/new ideas.
> 
>     Thanks,
>     Ben
> 
>     On 7/2/21 4:28 PM, Bob McMahon wrote:
>      > I think we need the language of math here. It seems like the network power metric, introduced by Kleinrock and Jaffe in the late 70s, is something useful.
>      > Effective end/end queue depths per Little's law also seems useful. Both are available in iperf 2 from a test perspective. Repurposing test techniques to
>     actual
>      > traffic could be useful. Hence the question around what exact telemetry is useful to apps making socket write() and read() calls.
>      >
>      > Bob
>      >
>      > On Fri, Jul 2, 2021 at 10:07 AM Dave Taht <dave.taht@gmail.com <mailto:dave.taht@gmail.com> <mailto:dave.taht@gmail.com <mailto:dave.taht@gmail.com>>> wrote:
>      >
>      >     In terms of trying to find "Quality" I have tried to encourage folk to
>      >     both read "zen and the art of motorcycle maintenance"[0], and Deming's
>      >     work on "total quality management".
>      >
>      >     My own slice at this network, computer and lifestyle "issue" is aiming
>      >     for "imperceptible latency" in all things. [1]. There's a lot of
>      >     fallout from that in terms of not just addressing queuing delay, but
>      >     caching, prefetching, and learning more about what a user really needs
>      >     (as opposed to wants) to know via intelligent agents.
>      >
>      >     [0] If you want to get depressed, read Pirsig's successor to "zen...",
>      >     lila, which is in part about what happens when an engineer hits an
>      >     insoluble problem.
>      >     [1] https://www.internetsociety.org/events/latency2013/ <https://www.internetsociety.org/events/latency2013/>
>      >
>      >
>      >
>      >     On Thu, Jul 1, 2021 at 6:16 PM David P. Reed <dpreed@deepplum.com <mailto:dpreed@deepplum.com> <mailto:dpreed@deepplum.com
>     <mailto:dpreed@deepplum.com>>> wrote:
>      >      >
>      >      > Well, nice that the folks doing the conference  are willing to consider that quality of user experience has little to do with signalling rate at the
>      >     physical layer or throughput of FTP transfers.
>      >      >
>      >      >
>      >      >
>      >      > But honestly, the fact that they call the problem "network quality" suggests that they REALLY, REALLY don't understand the Internet isn't the
>     hardware or
>      >     the routers or even the routing algorithms *to its users*.
>      >      >
>      >      >
>      >      >
>      >      > By ignoring the diversity of applications now and in the future, and the fact that we DON'T KNOW what will be coming up, this conference will
>     likely fall
>      >     into the usual trap that net-heads fall into - optimizing for some imaginary reality that doesn't exist, and in fact will probably never be what users
>      >     actually will do given the chance.
>      >      >
>      >      >
>      >      >
>      >      > I saw this issue in 1976 in the group developing the original Internet protocols - a desire to put *into the network* special tricks to optimize ASR33
>      >     logins to remote computers from terminal concentrators (aka remote login), bulk file transfers between file systems on different time-sharing
>     systems, and
>      >     "sessions" (virtual circuits) that required logins. And then trying to exploit underlying "multicast" by building it into the IP layer, because someone
>      >     thought that TV broadcast would be the dominant application.
>      >      >
>      >      >
>      >      >
>      >      > Frankly, to think of "quality" as something that can be "provided" by "the network" misses the entire point of "end-to-end argument in system design".
>      >     Quality is not a property defined or created by The Network. If you want to talk about Quality, you need to talk about users - all the users at all
>     times,
>      >     now and into the future, and that's something you can't do if you don't bother to include current and future users talking about what they might
>     expect to
>      >     experience that they don't experience.
>      >      >
>      >      >
>      >      >
>      >      > There was much fighting back in 1976 that basically involved "network experts" saying that the network was the place to "solve" such issues as
>     quality,
>      >     so applications could avoid having to solve such issues.
>      >      >
>      >      >
>      >      >
>      >      > What some of us managed to do was to argue that you can't "solve" such issues. All you can do is provide a framework that enables different uses to
>      >     *cooperate* in some way.
>      >      >
>      >      >
>      >      >
>      >      > Which is why the Internet drops packets rather than queueing them, and why diffserv cannot work.
>      >      >
>      >      > (I know the latter is conftroversial, but at the moment, ALL of diffserv attempts to talk about end-to-end applicaiton specific metrics, but
>     never, ever
>      >     explains what the diffserv control points actually do w.r.t. what the IP layer can actually control. So it is meaningless - another violation of the
>      >     so-called end-to-end principle).
>      >      >
>      >      >
>      >      >
>      >      > Networks are about getting packets from here to there, multiplexing the underlying resources. That's it. Quality is a whole different thing.
>     Quality can
>      >     be improved by end-to-end approaches, if the underlying network provides some kind of thing that actually creates a way for end-to-end applications to
>      >     affect queueing and routing decisions, and more importantly getting "telemetry" from the network regarding what is actually going on with the other
>      >     end-to-end users sharing the infrastructure.
>      >      >
>      >      >
>      >      >
>      >      > This conference won't talk about it this way. So don't waste your time.
>      >      >
>      >      >
>      >      >
>      >      >
>      >      >
>      >      >
>      >      >
>      >      > On Wednesday, June 30, 2021 8:12pm, "Dave Taht" <dave.taht@gmail.com <mailto:dave.taht@gmail.com> <mailto:dave.taht@gmail.com
>     <mailto:dave.taht@gmail.com>>> said:
>      >      >
>      >      > > The program committee members are *amazing*. Perhaps, finally, we can
>      >      > > move the bar for the internet's quality metrics past endless, blind
>      >      > > repetitions of speedtest.
>      >      > >
>      >      > > For complete details, please see:
>      >      > > https://www.iab.org/activities/workshops/network-quality/ <https://www.iab.org/activities/workshops/network-quality/>
>      >      > >
>      >      > > Submissions Due: Monday 2nd August 2021, midnight AOE (Anywhere On Earth)
>      >      > > Invitations Issued by: Monday 16th August 2021
>      >      > >
>      >      > > Workshop Date: This will be a virtual workshop, spread over three days:
>      >      > >
>      >      > > 1400-1800 UTC Tue 14th September 2021
>      >      > > 1400-1800 UTC Wed 15th September 2021
>      >      > > 1400-1800 UTC Thu 16th September 2021
>      >      > >
>      >      > > Workshop co-chairs: Wes Hardaker, Evgeny Khorov, Omer Shapira
>      >      > >
>      >      > > The Program Committee members:
>      >      > >
>      >      > > Jari Arkko, Olivier Bonaventure, Vint Cerf, Stuart Cheshire, Sam
>      >      > > Crowford, Nick Feamster, Jim Gettys, Toke Hoiland-Jorgensen, Geoff
>      >      > > Huston, Cullen Jennings, Katarzyna Kosek-Szott, Mirja Kuehlewind,
>      >      > > Jason Livingood, Matt Mathias, Randall Meyer, Kathleen Nichols,
>      >      > > Christoph Paasch, Tommy Pauly, Greg White, Keith Winstein.
>      >      > >
>      >      > > Send Submissions to: network-quality-workshop-pc@iab.org <mailto:network-quality-workshop-pc@iab.org>
>     <mailto:network-quality-workshop-pc@iab.org <mailto:network-quality-workshop-pc@iab.org>>.
>      >      > >
>      >      > > Position papers from academia, industry, the open source community and
>      >      > > others that focus on measurements, experiences, observations and
>      >      > > advice for the future are welcome. Papers that reflect experience
>      >      > > based on deployed services are especially welcome. The organizers
>      >      > > understand that specific actions taken by operators are unlikely to be
>      >      > > discussed in detail, so papers discussing general categories of
>      >      > > actions and issues without naming specific technologies, products, or
>      >      > > other players in the ecosystem are expected. Papers should not focus
>      >      > > on specific protocol solutions.
>      >      > >
>      >      > > The workshop will be by invitation only. Those wishing to attend
>      >      > > should submit a position paper to the address above; it may take the
>      >      > > form of an Internet-Draft.
>      >      > >
>      >      > > All inputs submitted and considered relevant will be published on the
>      >      > > workshop website. The organisers will decide whom to invite based on
>      >      > > the submissions received. Sessions will be organized according to
>      >      > > content, and not every accepted submission or invited attendee will
>      >      > > have an opportunity to present as the intent is to foster discussion
>      >      > > and not simply to have a sequence of presentations.
>      >      > >
>      >      > > Position papers from those not planning to attend the virtual sessions
>      >      > > themselves are also encouraged. A workshop report will be published
>      >      > > afterwards.
>      >      > >
>      >      > > Overview:
>      >      > >
>      >      > > "We believe that one of the major factors behind this lack of progress
>      >      > > is the popular perception that throughput is the often sole measure of
>      >      > > the quality of Internet connectivity. With such narrow focus, people
>      >      > > don’t consider questions such as:
>      >      > >
>      >      > > What is the latency under typical working conditions?
>      >      > > How reliable is the connectivity across longer time periods?
>      >      > > Does the network allow the use of a broad range of protocols?
>      >      > > What services can be run by clients of the network?
>      >      > > What kind of IPv4, NAT or IPv6 connectivity is offered, and are there firewalls?
>      >      > > What security mechanisms are available for local services, such as DNS?
>      >      > > To what degree are the privacy, confidentiality, integrity and
>      >      > > authenticity of user communications guarded?
>      >      > >
>      >      > > Improving these aspects of network quality will likely depend on
>      >      > > measurement and exposing metrics to all involved parties, including to
>      >      > > end users in a meaningful way. Such measurements and exposure of the
>      >      > > right metrics will allow service providers and network operators to
>      >      > > focus on the aspects that impacts the users’ experience most and at
>      >      > > the same time empowers users to choose the Internet service that will
>      >      > > give them the best experience."
>      >      > >
>      >      > >
>      >      > > --
>      >      > > Latest Podcast:
>      >      > > https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/
>     <https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/>
>      >      > >
>      >      > > Dave Täht CTO, TekLibre, LLC
>      >      > > _______________________________________________
>      >      > > Cerowrt-devel mailing list
>      >      > > Cerowrt-devel@lists.bufferbloat.net <mailto:Cerowrt-devel@lists.bufferbloat.net> <mailto:Cerowrt-devel@lists.bufferbloat.net
>     <mailto:Cerowrt-devel@lists.bufferbloat.net>>
>      >      > > https://lists.bufferbloat.net/listinfo/cerowrt-devel <https://lists.bufferbloat.net/listinfo/cerowrt-devel>
>      >      > >
>      >
>      >
>      >
>      >     --
>      >     Latest Podcast:
>      > https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/ <https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/>
>      >
>      >     Dave Täht CTO, TekLibre, LLC
>      >     _______________________________________________
>      >     Make-wifi-fast mailing list
>      > Make-wifi-fast@lists.bufferbloat.net <mailto:Make-wifi-fast@lists.bufferbloat.net> <mailto:Make-wifi-fast@lists.bufferbloat.net
>     <mailto:Make-wifi-fast@lists.bufferbloat.net>>
>      > https://lists.bufferbloat.net/listinfo/make-wifi-fast <https://lists.bufferbloat.net/listinfo/make-wifi-fast>
>      >
>      >
>      > This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the
>     use of
>      > the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or
>     otherwise
>      > restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended
>     recipient,
>      > you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you
>      > received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.
>      >
>      > _______________________________________________
>      > Starlink mailing list
>      > Starlink@lists.bufferbloat.net <mailto:Starlink@lists.bufferbloat.net>
>      > https://lists.bufferbloat.net/listinfo/starlink
>      >
> 
> 
>     -- 
>     Ben Greear <greearb@candelatech.com <mailto:greearb@candelatech.com>>
>     Candela Technologies Inc http://www.candelatech.com
> 
> 
> This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of 
> the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise 
> restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, 
> you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you 
> received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: [Codel] [Starlink] [Make-wifi-fast] [Cerowrt-devel] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Ben Greear]

Thanks for the clarification.  There are vendors that make these, but we have not tried
integrating our software control with any of them.  To date, not many of our customers
have been interested in this, so it did not seem worth the effort.

Do you see this as being useful for normal AP vendors/users, or mostly just radio manufacturers such
as BCM?

In case someone has one of these that has a sane API, I'd consider adding automation support
to drive it while running throughput or RvR or whatever other types of tests seem interesting.

Thanks,
Ben

On 7/6/21 3:05 PM, Bob McMahon wrote:
> Sorry, I should have been more clear. Not a fixed butler matrix but a device with solid state, programmable, phase shifters, 0 - 360 degrees. It's a way to 
> create multiple phy channels and affect and vary the off diagonal elements of a MIMO H-matrix using conducted parts. Then automation software can have more 
> robust RF MIMO test scenarios that are reproducible.
> 
> https://web.stanford.edu/~dntse/Chapters_PDF/Fundamentals_Wireless_Communication_chapter7.pdf 
> <https://web.stanford.edu/~dntse/Chapters_PDF/Fundamentals_Wireless_Communication_chapter7.pdf>
> 
> Bob
> 
> On Tue, Jul 6, 2021 at 2:24 PM Ben Greear <greearb@candelatech.com <mailto:greearb@candelatech.com>> wrote:
> 
>     We tried adding in an external butler matrix in the past, but could not notice any useful difference.  Possibly
>     we didn't have the right use case.
> 
>     Typically we are competitive on price for full testing solutions, but you can get stand-alone attenuators
>     cheaper from specialized vendors.  Happy to discuss pricing offlist if you wish.
> 
>     Thanks,
>     Ben
> 
>     On 7/6/21 1:43 PM, Bob McMahon wrote:
>      > The four part attenuator part would be more interesting to me if it also had a solid state phase shifters.  This allows for testing 2x2 MIMO testing per
>      > affecting the spatial stream eigen vectors/values.
>      >
>      > Bob
>      >
>      > PS. The price per port isn't competitive. Probably a good idea to survey the market competition.
>      >
>      > On Tue, Jul 6, 2021 at 6:46 AM Ben Greear <greearb@candelatech.com <mailto:greearb@candelatech.com> <mailto:greearb@candelatech.com
>     <mailto:greearb@candelatech.com>>> wrote:
>      >
>      >     Hello,
>      >
>      >     I am interested to hear wish lists for network testing features.  We make test equipment, supporting lots
>      >     of wifi stations and a distributed architecture, with built-in udp, tcp, ipv6, http, ... protocols,
>      >     and open to creating/improving some of our automated tests.
>      >
>      >     I know Dave has some test scripts already, so I'm not necessarily looking to reimplement that,
>      >     but more fishing for other/new ideas.
>      >
>      >     Thanks,
>      >     Ben
>      >
>      >     On 7/2/21 4:28 PM, Bob McMahon wrote:
>      >      > I think we need the language of math here. It seems like the network power metric, introduced by Kleinrock and Jaffe in the late 70s, is something
>     useful.
>      >      > Effective end/end queue depths per Little's law also seems useful. Both are available in iperf 2 from a test perspective. Repurposing test
>     techniques to
>      >     actual
>      >      > traffic could be useful. Hence the question around what exact telemetry is useful to apps making socket write() and read() calls.
>      >      >
>      >      > Bob
>      >      >
>      >      > On Fri, Jul 2, 2021 at 10:07 AM Dave Taht <dave.taht@gmail.com <mailto:dave.taht@gmail.com> <mailto:dave.taht@gmail.com
>     <mailto:dave.taht@gmail.com>> <mailto:dave.taht@gmail.com <mailto:dave.taht@gmail.com> <mailto:dave.taht@gmail.com <mailto:dave.taht@gmail.com>>>> wrote:
>      >      >
>      >      >     In terms of trying to find "Quality" I have tried to encourage folk to
>      >      >     both read "zen and the art of motorcycle maintenance"[0], and Deming's
>      >      >     work on "total quality management".
>      >      >
>      >      >     My own slice at this network, computer and lifestyle "issue" is aiming
>      >      >     for "imperceptible latency" in all things. [1]. There's a lot of
>      >      >     fallout from that in terms of not just addressing queuing delay, but
>      >      >     caching, prefetching, and learning more about what a user really needs
>      >      >     (as opposed to wants) to know via intelligent agents.
>      >      >
>      >      >     [0] If you want to get depressed, read Pirsig's successor to "zen...",
>      >      >     lila, which is in part about what happens when an engineer hits an
>      >      >     insoluble problem.
>      >      >     [1] https://www.internetsociety.org/events/latency2013/ <https://www.internetsociety.org/events/latency2013/>
>     <https://www.internetsociety.org/events/latency2013/ <https://www.internetsociety.org/events/latency2013/>>
>      >      >
>      >      >
>      >      >
>      >      >     On Thu, Jul 1, 2021 at 6:16 PM David P. Reed <dpreed@deepplum.com <mailto:dpreed@deepplum.com> <mailto:dpreed@deepplum.com
>     <mailto:dpreed@deepplum.com>> <mailto:dpreed@deepplum.com <mailto:dpreed@deepplum.com>
>      >     <mailto:dpreed@deepplum.com <mailto:dpreed@deepplum.com>>>> wrote:
>      >      >      >
>      >      >      > Well, nice that the folks doing the conference  are willing to consider that quality of user experience has little to do with signalling
>     rate at the
>      >      >     physical layer or throughput of FTP transfers.
>      >      >      >
>      >      >      >
>      >      >      >
>      >      >      > But honestly, the fact that they call the problem "network quality" suggests that they REALLY, REALLY don't understand the Internet isn't the
>      >     hardware or
>      >      >     the routers or even the routing algorithms *to its users*.
>      >      >      >
>      >      >      >
>      >      >      >
>      >      >      > By ignoring the diversity of applications now and in the future, and the fact that we DON'T KNOW what will be coming up, this conference will
>      >     likely fall
>      >      >     into the usual trap that net-heads fall into - optimizing for some imaginary reality that doesn't exist, and in fact will probably never be
>     what users
>      >      >     actually will do given the chance.
>      >      >      >
>      >      >      >
>      >      >      >
>      >      >      > I saw this issue in 1976 in the group developing the original Internet protocols - a desire to put *into the network* special tricks to
>     optimize ASR33
>      >      >     logins to remote computers from terminal concentrators (aka remote login), bulk file transfers between file systems on different time-sharing
>      >     systems, and
>      >      >     "sessions" (virtual circuits) that required logins. And then trying to exploit underlying "multicast" by building it into the IP layer,
>     because someone
>      >      >     thought that TV broadcast would be the dominant application.
>      >      >      >
>      >      >      >
>      >      >      >
>      >      >      > Frankly, to think of "quality" as something that can be "provided" by "the network" misses the entire point of "end-to-end argument in
>     system design".
>      >      >     Quality is not a property defined or created by The Network. If you want to talk about Quality, you need to talk about users - all the users
>     at all
>      >     times,
>      >      >     now and into the future, and that's something you can't do if you don't bother to include current and future users talking about what they might
>      >     expect to
>      >      >     experience that they don't experience.
>      >      >      >
>      >      >      >
>      >      >      >
>      >      >      > There was much fighting back in 1976 that basically involved "network experts" saying that the network was the place to "solve" such issues as
>      >     quality,
>      >      >     so applications could avoid having to solve such issues.
>      >      >      >
>      >      >      >
>      >      >      >
>      >      >      > What some of us managed to do was to argue that you can't "solve" such issues. All you can do is provide a framework that enables different
>     uses to
>      >      >     *cooperate* in some way.
>      >      >      >
>      >      >      >
>      >      >      >
>      >      >      > Which is why the Internet drops packets rather than queueing them, and why diffserv cannot work.
>      >      >      >
>      >      >      > (I know the latter is conftroversial, but at the moment, ALL of diffserv attempts to talk about end-to-end applicaiton specific metrics, but
>      >     never, ever
>      >      >     explains what the diffserv control points actually do w.r.t. what the IP layer can actually control. So it is meaningless - another violation
>     of the
>      >      >     so-called end-to-end principle).
>      >      >      >
>      >      >      >
>      >      >      >
>      >      >      > Networks are about getting packets from here to there, multiplexing the underlying resources. That's it. Quality is a whole different thing.
>      >     Quality can
>      >      >     be improved by end-to-end approaches, if the underlying network provides some kind of thing that actually creates a way for end-to-end
>     applications to
>      >      >     affect queueing and routing decisions, and more importantly getting "telemetry" from the network regarding what is actually going on with the
>     other
>      >      >     end-to-end users sharing the infrastructure.
>      >      >      >
>      >      >      >
>      >      >      >
>      >      >      > This conference won't talk about it this way. So don't waste your time.
>      >      >      >
>      >      >      >
>      >      >      >
>      >      >      >
>      >      >      >
>      >      >      >
>      >      >      >
>      >      >      > On Wednesday, June 30, 2021 8:12pm, "Dave Taht" <dave.taht@gmail.com <mailto:dave.taht@gmail.com> <mailto:dave.taht@gmail.com
>     <mailto:dave.taht@gmail.com>> <mailto:dave.taht@gmail.com <mailto:dave.taht@gmail.com>
>      >     <mailto:dave.taht@gmail.com <mailto:dave.taht@gmail.com>>>> said:
>      >      >      >
>      >      >      > > The program committee members are *amazing*. Perhaps, finally, we can
>      >      >      > > move the bar for the internet's quality metrics past endless, blind
>      >      >      > > repetitions of speedtest.
>      >      >      > >
>      >      >      > > For complete details, please see:
>      >      >      > > https://www.iab.org/activities/workshops/network-quality/ <https://www.iab.org/activities/workshops/network-quality/>
>     <https://www.iab.org/activities/workshops/network-quality/ <https://www.iab.org/activities/workshops/network-quality/>>
>      >      >      > >
>      >      >      > > Submissions Due: Monday 2nd August 2021, midnight AOE (Anywhere On Earth)
>      >      >      > > Invitations Issued by: Monday 16th August 2021
>      >      >      > >
>      >      >      > > Workshop Date: This will be a virtual workshop, spread over three days:
>      >      >      > >
>      >      >      > > 1400-1800 UTC Tue 14th September 2021
>      >      >      > > 1400-1800 UTC Wed 15th September 2021
>      >      >      > > 1400-1800 UTC Thu 16th September 2021
>      >      >      > >
>      >      >      > > Workshop co-chairs: Wes Hardaker, Evgeny Khorov, Omer Shapira
>      >      >      > >
>      >      >      > > The Program Committee members:
>      >      >      > >
>      >      >      > > Jari Arkko, Olivier Bonaventure, Vint Cerf, Stuart Cheshire, Sam
>      >      >      > > Crowford, Nick Feamster, Jim Gettys, Toke Hoiland-Jorgensen, Geoff
>      >      >      > > Huston, Cullen Jennings, Katarzyna Kosek-Szott, Mirja Kuehlewind,
>      >      >      > > Jason Livingood, Matt Mathias, Randall Meyer, Kathleen Nichols,
>      >      >      > > Christoph Paasch, Tommy Pauly, Greg White, Keith Winstein.
>      >      >      > >
>      >      >      > > Send Submissions to: network-quality-workshop-pc@iab.org <mailto:network-quality-workshop-pc@iab.org>
>     <mailto:network-quality-workshop-pc@iab.org <mailto:network-quality-workshop-pc@iab.org>>
>      >     <mailto:network-quality-workshop-pc@iab.org <mailto:network-quality-workshop-pc@iab.org> <mailto:network-quality-workshop-pc@iab.org
>     <mailto:network-quality-workshop-pc@iab.org>>>.
>      >      >      > >
>      >      >      > > Position papers from academia, industry, the open source community and
>      >      >      > > others that focus on measurements, experiences, observations and
>      >      >      > > advice for the future are welcome. Papers that reflect experience
>      >      >      > > based on deployed services are especially welcome. The organizers
>      >      >      > > understand that specific actions taken by operators are unlikely to be
>      >      >      > > discussed in detail, so papers discussing general categories of
>      >      >      > > actions and issues without naming specific technologies, products, or
>      >      >      > > other players in the ecosystem are expected. Papers should not focus
>      >      >      > > on specific protocol solutions.
>      >      >      > >
>      >      >      > > The workshop will be by invitation only. Those wishing to attend
>      >      >      > > should submit a position paper to the address above; it may take the
>      >      >      > > form of an Internet-Draft.
>      >      >      > >
>      >      >      > > All inputs submitted and considered relevant will be published on the
>      >      >      > > workshop website. The organisers will decide whom to invite based on
>      >      >      > > the submissions received. Sessions will be organized according to
>      >      >      > > content, and not every accepted submission or invited attendee will
>      >      >      > > have an opportunity to present as the intent is to foster discussion
>      >      >      > > and not simply to have a sequence of presentations.
>      >      >      > >
>      >      >      > > Position papers from those not planning to attend the virtual sessions
>      >      >      > > themselves are also encouraged. A workshop report will be published
>      >      >      > > afterwards.
>      >      >      > >
>      >      >      > > Overview:
>      >      >      > >
>      >      >      > > "We believe that one of the major factors behind this lack of progress
>      >      >      > > is the popular perception that throughput is the often sole measure of
>      >      >      > > the quality of Internet connectivity. With such narrow focus, people
>      >      >      > > don’t consider questions such as:
>      >      >      > >
>      >      >      > > What is the latency under typical working conditions?
>      >      >      > > How reliable is the connectivity across longer time periods?
>      >      >      > > Does the network allow the use of a broad range of protocols?
>      >      >      > > What services can be run by clients of the network?
>      >      >      > > What kind of IPv4, NAT or IPv6 connectivity is offered, and are there firewalls?
>      >      >      > > What security mechanisms are available for local services, such as DNS?
>      >      >      > > To what degree are the privacy, confidentiality, integrity and
>      >      >      > > authenticity of user communications guarded?
>      >      >      > >
>      >      >      > > Improving these aspects of network quality will likely depend on
>      >      >      > > measurement and exposing metrics to all involved parties, including to
>      >      >      > > end users in a meaningful way. Such measurements and exposure of the
>      >      >      > > right metrics will allow service providers and network operators to
>      >      >      > > focus on the aspects that impacts the users’ experience most and at
>      >      >      > > the same time empowers users to choose the Internet service that will
>      >      >      > > give them the best experience."
>      >      >      > >
>      >      >      > >
>      >      >      > > --
>      >      >      > > Latest Podcast:
>      >      >      > > https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/
>     <https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/>
>      >     <https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/ <https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/>>
>      >      >      > >
>      >      >      > > Dave Täht CTO, TekLibre, LLC
>      >      >      > > _______________________________________________
>      >      >      > > Cerowrt-devel mailing list
>      >      >      > > Cerowrt-devel@lists.bufferbloat.net <mailto:Cerowrt-devel@lists.bufferbloat.net> <mailto:Cerowrt-devel@lists.bufferbloat.net
>     <mailto:Cerowrt-devel@lists.bufferbloat.net>> <mailto:Cerowrt-devel@lists.bufferbloat.net <mailto:Cerowrt-devel@lists.bufferbloat.net>
>      >     <mailto:Cerowrt-devel@lists.bufferbloat.net <mailto:Cerowrt-devel@lists.bufferbloat.net>>>
>      >      >      > > https://lists.bufferbloat.net/listinfo/cerowrt-devel <https://lists.bufferbloat.net/listinfo/cerowrt-devel>
>     <https://lists.bufferbloat.net/listinfo/cerowrt-devel <https://lists.bufferbloat.net/listinfo/cerowrt-devel>>
>      >      >      > >
>      >      >
>      >      >
>      >      >
>      >      >     --
>      >      >     Latest Podcast:
>      >      > https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/
>     <https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/> <https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/
>     <https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/>>
>      >      >
>      >      >     Dave Täht CTO, TekLibre, LLC
>      >      >     _______________________________________________
>      >      >     Make-wifi-fast mailing list
>      >      > Make-wifi-fast@lists.bufferbloat.net <mailto:Make-wifi-fast@lists.bufferbloat.net> <mailto:Make-wifi-fast@lists.bufferbloat.net
>     <mailto:Make-wifi-fast@lists.bufferbloat.net>> <mailto:Make-wifi-fast@lists.bufferbloat.net <mailto:Make-wifi-fast@lists.bufferbloat.net>
>      >     <mailto:Make-wifi-fast@lists.bufferbloat.net <mailto:Make-wifi-fast@lists.bufferbloat.net>>>
>      >      > https://lists.bufferbloat.net/listinfo/make-wifi-fast <https://lists.bufferbloat.net/listinfo/make-wifi-fast>
>     <https://lists.bufferbloat.net/listinfo/make-wifi-fast <https://lists.bufferbloat.net/listinfo/make-wifi-fast>>
>      >      >
>      >      >
>      >      > This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely
>     for the
>      >     use of
>      >      > the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or
>      >     otherwise
>      >      > restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended
>      >     recipient,
>      >      > you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly
>     prohibited. If you
>      >      > received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.
>      >      >
>      >      > _______________________________________________
>      >      > Starlink mailing list
>      >      > Starlink@lists.bufferbloat.net <mailto:Starlink@lists.bufferbloat.net> <mailto:Starlink@lists.bufferbloat.net <mailto:Starlink@lists.bufferbloat.net>>
>      >      > https://lists.bufferbloat.net/listinfo/starlink <https://lists.bufferbloat.net/listinfo/starlink>
>      >      >
>      >
>      >
>      >     --
>      >     Ben Greear <greearb@candelatech.com <mailto:greearb@candelatech.com> <mailto:greearb@candelatech.com <mailto:greearb@candelatech.com>>>
>      >     Candela Technologies Inc http://www.candelatech.com <http://www.candelatech.com>
>      >
>      >
>      > This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the
>     use of
>      > the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or
>     otherwise
>      > restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended
>     recipient,
>      > you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you
>      > received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.
> 
> 
>     -- 
>     Ben Greear <greearb@candelatech.com <mailto:greearb@candelatech.com>>
>     Candela Technologies Inc http://www.candelatech.com <http://www.candelatech.com>
> 
> 
> This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of 
> the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise 
> restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, 
> you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you 
> received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: [Codel] [Make-wifi-fast] [Starlink] [Cerowrt-devel] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Luca Muscariello]

[-- Attachment #1: Type: text/plain, Size: 17734 bytes --]

For those who might be interested in Little's law
there is a nice paper by John Little on the occasion
of the 50th anniversary  of the result.

https://www.informs.org/Blogs/Operations-Research-Forum/Little-s-Law-as-Viewed-on-its-50th-Anniversary

https://www.informs.org/content/download/255808/2414681/file/little_paper.pdf

Nice read.
Luca

P.S.
Who has not a copy of L. Kleinrock's books? I do have and am not ready to
lend them!

On Fri, Jul 9, 2021 at 11:01 AM Leonard Kleinrock <lk@cs.ucla.edu> wrote:

> David,
>
> I totally appreciate  your attention to when and when not analytical
> modeling works. Let me clarify a few things from your note.
>
> First, Little's law (also known as Little’s lemma or, as I use in my book,
> Little’s result) does not assume Poisson arrivals -  it is good for *any*
> arrival process and any service process and is an equality between time
> averages.  It states that the time average of the number in a system (for a
> sample path *w)* is equal to the average arrival rate to the system
> multiplied by the time-averaged time in the system for that sample path.
> This is often written as   NTimeAvg =λ·TTimeAvg .  Moreover, if the
> system is also ergodic, then the time average equals the ensemble average
> and we often write it as N ̄ = λ T ̄ .  In any case, this requires
> neither Poisson arrivals nor exponential service times.
>
> Queueing theorists often do study the case of Poisson arrivals.  True, it
> makes the analysis easier, yet there is a better reason it is often used,
> and that is because the sum of a large number of independent stationary
> renewal processes approaches a Poisson process.  So nature often gives us
> Poisson arrivals.
>
> Best,
> Len
>
>
>
> On Jul 8, 2021, at 12:38 PM, David P. Reed <dpreed@deepplum.com> wrote:
>
> I will tell you flat out that the arrival time distribution assumption
> made by Little's Lemma that allows "estimation of queue depth" is totally
> unreasonable on ANY Internet in practice.
>
>
> The assumption is a Poisson Arrival Process. In reality, traffic arrivals
> in real internet applications are extremely far from Poisson, and, of
> course, using TCP windowing, become highly intercorrelated with crossing
> traffic that shares the same queue.
>
>
> So, as I've tried to tell many, many net-heads (people who ignore
> applications layer behavior, like the people that think latency doesn't
> matter to end users, only throughput), end-to-end packet arrival times on a
> practical network are incredibly far from Poisson - and they are more like
> fractal probability distributions, very irregular at all scales of time.
>
>
> So, the idea that iperf can estimate queue depth by Little's Lemma by just
> measuring saturation of capacity of a path is bogus.The less Poisson, the
> worse the estimate gets, by a huge factor.
>
>
>
>
> Where does the Poisson assumption come from?  Well, like many theorems, it
> is the simplest tractable closed form solution - it creates a simplified
> view, by being a "single-parameter" distribution (the parameter is called
> lambda for a Poisson distribution).  And the analysis of a simple queue
> with poisson arrival distribution and a static, fixed service time is the
> first interesting Queueing Theory example in most textbooks. It is
> suggestive of an interesting phenomenon, but it does NOT characterize any
> real system.
>
>
> It's the queueing theory equivalent of "First, we assume a spherical
> cow...". in doing an example in a freshman physics class.
>
>
> Unfortunately, most networking engineers understand neither queuing theory
> nor application networking usage in interactive applications. Which makes
> them arrogant. They assume all distributions are poisson!
>
>
>
>
> On Tuesday, July 6, 2021 9:46am, "Ben Greear" <greearb@candelatech.com>
> said:
>
> > Hello,
> >
> > I am interested to hear wish lists for network testing features. We make
> test
> > equipment, supporting lots
> > of wifi stations and a distributed architecture, with built-in udp, tcp,
> ipv6,
> > http, ... protocols,
> > and open to creating/improving some of our automated tests.
> >
> > I know Dave has some test scripts already, so I'm not necessarily
> looking to
> > reimplement that,
> > but more fishing for other/new ideas.
> >
> > Thanks,
> > Ben
> >
> > On 7/2/21 4:28 PM, Bob McMahon wrote:
> > > I think we need the language of math here. It seems like the network
> > power metric, introduced by Kleinrock and Jaffe in the late 70s, is
> something
> > useful.
> > > Effective end/end queue depths per Little's law also seems useful.
> Both are
> > available in iperf 2 from a test perspective. Repurposing test
> techniques to
> > actual
> > > traffic could be useful. Hence the question around what exact telemetry
> > is useful to apps making socket write() and read() calls.
> > >
> > > Bob
> > >
> > > On Fri, Jul 2, 2021 at 10:07 AM Dave Taht <dave.taht@gmail.com
> > <mailto:dave.taht@gmail.com <dave.taht@gmail.com>>> wrote:
> > >
> > > In terms of trying to find "Quality" I have tried to encourage folk to
> > > both read "zen and the art of motorcycle maintenance"[0], and Deming's
> > > work on "total quality management".
> > >
> > > My own slice at this network, computer and lifestyle "issue" is aiming
> > > for "imperceptible latency" in all things. [1]. There's a lot of
> > > fallout from that in terms of not just addressing queuing delay, but
> > > caching, prefetching, and learning more about what a user really needs
> > > (as opposed to wants) to know via intelligent agents.
> > >
> > > [0] If you want to get depressed, read Pirsig's successor to "zen...",
> > > lila, which is in part about what happens when an engineer hits an
> > > insoluble problem.
> > > [1] https://www.internetsociety.org/events/latency2013/
> > <https://www.internetsociety.org/events/latency2013/>
> > >
> > >
> > >
> > > On Thu, Jul 1, 2021 at 6:16 PM David P. Reed <dpreed@deepplum.com
> > <mailto:dpreed@deepplum.com <dpreed@deepplum.com>>> wrote:
> > > >
> > > > Well, nice that the folks doing the conference  are willing to
> > consider that quality of user experience has little to do with
> signalling rate at
> > the
> > > physical layer or throughput of FTP transfers.
> > > >
> > > >
> > > >
> > > > But honestly, the fact that they call the problem "network quality"
> > suggests that they REALLY, REALLY don't understand the Internet isn't
> the hardware
> > or
> > > the routers or even the routing algorithms *to its users*.
> > > >
> > > >
> > > >
> > > > By ignoring the diversity of applications now and in the future,
> > and the fact that we DON'T KNOW what will be coming up, this conference
> will
> > likely fall
> > > into the usual trap that net-heads fall into - optimizing for some
> > imaginary reality that doesn't exist, and in fact will probably never be
> what
> > users
> > > actually will do given the chance.
> > > >
> > > >
> > > >
> > > > I saw this issue in 1976 in the group developing the original
> > Internet protocols - a desire to put *into the network* special tricks
> to optimize
> > ASR33
> > > logins to remote computers from terminal concentrators (aka remote
> > login), bulk file transfers between file systems on different
> time-sharing
> > systems, and
> > > "sessions" (virtual circuits) that required logins. And then trying to
> > exploit underlying "multicast" by building it into the IP layer, because
> someone
> > > thought that TV broadcast would be the dominant application.
> > > >
> > > >
> > > >
> > > > Frankly, to think of "quality" as something that can be "provided"
> > by "the network" misses the entire point of "end-to-end argument in
> system
> > design".
> > > Quality is not a property defined or created by The Network. If you
> want
> > to talk about Quality, you need to talk about users - all the users at
> all times,
> > > now and into the future, and that's something you can't do if you don't
> > bother to include current and future users talking about what they might
> expect
> > to
> > > experience that they don't experience.
> > > >
> > > >
> > > >
> > > > There was much fighting back in 1976 that basically involved
> > "network experts" saying that the network was the place to "solve" such
> issues as
> > quality,
> > > so applications could avoid having to solve such issues.
> > > >
> > > >
> > > >
> > > > What some of us managed to do was to argue that you can't "solve"
> > such issues. All you can do is provide a framework that enables
> different uses to
> > > *cooperate* in some way.
> > > >
> > > >
> > > >
> > > > Which is why the Internet drops packets rather than queueing them,
> > and why diffserv cannot work.
> > > >
> > > > (I know the latter is conftroversial, but at the moment, ALL of
> > diffserv attempts to talk about end-to-end applicaiton specific metrics,
> but
> > never, ever
> > > explains what the diffserv control points actually do w.r.t. what the
> IP
> > layer can actually control. So it is meaningless - another violation of
> the
> > > so-called end-to-end principle).
> > > >
> > > >
> > > >
> > > > Networks are about getting packets from here to there, multiplexing
> > the underlying resources. That's it. Quality is a whole different thing.
> Quality
> > can
> > > be improved by end-to-end approaches, if the underlying network
> provides
> > some kind of thing that actually creates a way for end-to-end
> applications to
> > > affect queueing and routing decisions, and more importantly getting
> > "telemetry" from the network regarding what is actually going on with
> the other
> > > end-to-end users sharing the infrastructure.
> > > >
> > > >
> > > >
> > > > This conference won't talk about it this way. So don't waste your
> > time.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wednesday, June 30, 2021 8:12pm, "Dave Taht"
> > <dave.taht@gmail.com <mailto:dave.taht@gmail.com <dave.taht@gmail.com>>>
> said:
> > > >
> > > > > The program committee members are *amazing*. Perhaps, finally,
> > we can
> > > > > move the bar for the internet's quality metrics past endless,
> > blind
> > > > > repetitions of speedtest.
> > > > >
> > > > > For complete details, please see:
> > > > > https://www.iab.org/activities/workshops/network-quality/
> > <https://www.iab.org/activities/workshops/network-quality/>
> > > > >
> > > > > Submissions Due: Monday 2nd August 2021, midnight AOE
> > (Anywhere On Earth)
> > > > > Invitations Issued by: Monday 16th August 2021
> > > > >
> > > > > Workshop Date: This will be a virtual workshop, spread over
> > three days:
> > > > >
> > > > > 1400-1800 UTC Tue 14th September 2021
> > > > > 1400-1800 UTC Wed 15th September 2021
> > > > > 1400-1800 UTC Thu 16th September 2021
> > > > >
> > > > > Workshop co-chairs: Wes Hardaker, Evgeny Khorov, Omer Shapira
> > > > >
> > > > > The Program Committee members:
> > > > >
> > > > > Jari Arkko, Olivier Bonaventure, Vint Cerf, Stuart Cheshire,
> > Sam
> > > > > Crowford, Nick Feamster, Jim Gettys, Toke Hoiland-Jorgensen,
> > Geoff
> > > > > Huston, Cullen Jennings, Katarzyna Kosek-Szott, Mirja
> > Kuehlewind,
> > > > > Jason Livingood, Matt Mathias, Randall Meyer, Kathleen
> > Nichols,
> > > > > Christoph Paasch, Tommy Pauly, Greg White, Keith Winstein.
> > > > >
> > > > > Send Submissions to: network-quality-workshop-pc@iab.org
> > <mailto:network-quality-workshop-pc@iab.org
> <network-quality-workshop-pc@iab.org>>.
> > > > >
> > > > > Position papers from academia, industry, the open source
> > community and
> > > > > others that focus on measurements, experiences, observations
> > and
> > > > > advice for the future are welcome. Papers that reflect
> > experience
> > > > > based on deployed services are especially welcome. The
> > organizers
> > > > > understand that specific actions taken by operators are
> > unlikely to be
> > > > > discussed in detail, so papers discussing general categories
> > of
> > > > > actions and issues without naming specific technologies,
> > products, or
> > > > > other players in the ecosystem are expected. Papers should not
> > focus
> > > > > on specific protocol solutions.
> > > > >
> > > > > The workshop will be by invitation only. Those wishing to
> > attend
> > > > > should submit a position paper to the address above; it may
> > take the
> > > > > form of an Internet-Draft.
> > > > >
> > > > > All inputs submitted and considered relevant will be published
> > on the
> > > > > workshop website. The organisers will decide whom to invite
> > based on
> > > > > the submissions received. Sessions will be organized according
> > to
> > > > > content, and not every accepted submission or invited attendee
> > will
> > > > > have an opportunity to present as the intent is to foster
> > discussion
> > > > > and not simply to have a sequence of presentations.
> > > > >
> > > > > Position papers from those not planning to attend the virtual
> > sessions
> > > > > themselves are also encouraged. A workshop report will be
> > published
> > > > > afterwards.
> > > > >
> > > > > Overview:
> > > > >
> > > > > "We believe that one of the major factors behind this lack of
> > progress
> > > > > is the popular perception that throughput is the often sole
> > measure of
> > > > > the quality of Internet connectivity. With such narrow focus,
> > people
> > > > > don’t consider questions such as:
> > > > >
> > > > > What is the latency under typical working conditions?
> > > > > How reliable is the connectivity across longer time periods?
> > > > > Does the network allow the use of a broad range of protocols?
> > > > > What services can be run by clients of the network?
> > > > > What kind of IPv4, NAT or IPv6 connectivity is offered, and
> > are there firewalls?
> > > > > What security mechanisms are available for local services,
> > such as DNS?
> > > > > To what degree are the privacy, confidentiality, integrity
> > and
> > > > > authenticity of user communications guarded?
> > > > >
> > > > > Improving these aspects of network quality will likely depend
> > on
> > > > > measurement and exposing metrics to all involved parties,
> > including to
> > > > > end users in a meaningful way. Such measurements and exposure
> > of the
> > > > > right metrics will allow service providers and network
> > operators to
> > > > > focus on the aspects that impacts the users’ experience
> > most and at
> > > > > the same time empowers users to choose the Internet service
> > that will
> > > > > give them the best experience."
> > > > >
> > > > >
> > > > > --
> > > > > Latest Podcast:
> > > > >
> >
> https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/
> > <
> https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/>
> > > > >
> > > > > Dave Täht CTO, TekLibre, LLC
> > > > > _______________________________________________
> > > > > Cerowrt-devel mailing list
> > > > > Cerowrt-devel@lists.bufferbloat.net
> > <mailto:Cerowrt-devel@lists.bufferbloat.net
> <Cerowrt-devel@lists.bufferbloat.net>>
> > > > > https://lists.bufferbloat.net/listinfo/cerowrt-devel
> > <https://lists.bufferbloat.net/listinfo/cerowrt-devel>
> > > > >
> > >
> > >
> > >
> > > --
> > > Latest Podcast:
> > >
> https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/
> > <
> https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/>
> > >
> > > Dave Täht CTO, TekLibre, LLC
> > > _______________________________________________
> > > Make-wifi-fast mailing list
> > > Make-wifi-fast@lists.bufferbloat.net
> > <mailto:Make-wifi-fast@lists.bufferbloat.net
> <Make-wifi-fast@lists.bufferbloat.net>>
> > > https://lists.bufferbloat.net/listinfo/make-wifi-fast
> > <https://lists.bufferbloat.net/listinfo/make-wifi-fast>
> > >
> > >
> > > This electronic communication and the information and any files
> transmitted
> > with it, or attached to it, are confidential and are intended solely for
> the use
> > of
> > > the individual or entity to whom it is addressed and may contain
> information
> > that is confidential, legally privileged, protected by privacy laws, or
> otherwise
> > > restricted from disclosure to anyone else. If you are not the intended
> > recipient or the person responsible for delivering the e-mail to the
> intended
> > recipient,
> > > you are hereby notified that any use, copying, distributing,
> dissemination,
> > forwarding, printing, or copying of this e-mail is strictly prohibited.
> If you
> > > received this e-mail in error, please return the e-mail to the sender,
> delete
> > it from your computer, and destroy any printed copy of it.
> > >
> > > _______________________________________________
> > > Starlink mailing list
> > > Starlink@lists.bufferbloat.net
> > > https://lists.bufferbloat.net/listinfo/starlink
> > >
> >
> >
> > --
> > Ben Greear <greearb@candelatech.com>
> > Candela Technologies Inc http://www.candelatech.com
> >
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
>
>
> _______________________________________________
> Make-wifi-fast mailing list
> Make-wifi-fast@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast

[-- Attachment #2: Type: text/html, Size: 26493 bytes --]

Re: [Codel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Toke Høiland-Jørgensen]

"Holland, Jake via Bloat" <bloat@lists.bufferbloat.net> writes:

> Hi David,
>
> That’s an interesting point, and I think you’re right that packet
> arrival is poorly modeled as a Poisson process, because in practice
> packet transmissions are very rarely unrelated to other packet
> transmissions.
>
> But now you’ve got me wondering what the right approach is. Do you
> have any advice for how to improve this kind of modeling?

I actually tried my hand at finding something better for my master's
thesis and came across something called a Markov-Modulated Poisson
Process (MMPP/D/1 queue)[0]. It looked promising, but unfortunately I
failed to make it produce any useful predictions. Most likely this was
as much a result of my own failings as a queueing theorist as it was the
fault of the model (I was in way over my head by the time I got to that
model); so I figured I'd mention it here in case anyone more qualified
would have any opinion on it.

I did manage to get the Linux kernel to produce queueing behaviour that
resembled that of a standard M/M/1 queue (if you squint a bit); all you
have to do is to use a traffic generator that emits packets with the
distribution the model assumes... :)

The full thesis is still available[1] for the perusal of morbidly curious.

-Toke

[0] https://www.sciencedirect.com/science/article/abs/pii/016653169390035S
[1] https://rucforsk.ruc.dk/ws/files/57613884/thesis-final.pdf

Re: [Codel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Jonathan Morton]

> On 10 Jul, 2021, at 2:01 am, Leonard Kleinrock <lk@cs.ucla.edu> wrote:
> 
> No question that non-stationarity and instability are what we often see in networks.  And, non-stationarity and instability are both topics that lead to very complex analytical problems in queueing theory.  You can find some results on the transient analysis in the queueing theory literature (including the second volume of my Queueing Systems book), but they are limited and hard. Nevertheless, the literature does contain some works on transient analysis of queueing systems as applied to network congestion control - again limited. On the other hand, as you said, control theory addresses stability head on and does offer some tools as well, but again, it is hairy. 

I was just about to mention control theory.

One basic characteristic of Poisson traffic is that it is inelastic, and assumes there is no control feedback whatsoever.  This means it can only be a valid model when the following are both true:

1: The offered load is *below* the link capacity, for all links, averaged over time.

2: A high degree of statistical multiplexing exists.

If 1: is not true and the traffic is truly inelastic, then the queues will inevitably fill up and congestion collapse will result, as shown from ARPANET experience in the 1980s; the solution was to introduce control feedback to the traffic, initially in the form of TCP Reno.  If 2: is not true then the traffic cannot be approximated as Poisson arrivals, regardless of load relative to capacity, because the degree of correlation is too high.

Taking the iPhone introduction anecdote as an illustrative example, measuring utilisation as very close to 100% is a clear warning sign that the Poisson model was inappropriate, and a control-theory approach was needed instead, to capture the feedback effects of congestion control.  The high degree of statistical multiplexing inherent to a major ISP backhaul is irrelevant to that determination.

Such a model would have found that the primary source of control feedback was human users giving up in disgust.  However, different humans have different levels of tolerance and persistence, so this feedback was not sufficient to reduce the load sufficiently to give the majority of users a good service; instead, *all* users received a poor service and many users received no usable service.  Introducing a technological control feedback, in the form of packet loss upon overflow of correctly-sized queues, improved service for everyone.

(BTW, DNS becomes significantly unreliable around 1-2 seconds RTT, due to protocol timeouts, which is inherited by all applications that rely on DNS lookups.  Merely reducing the delays consistently below that threshold would have improved perceived reliability markedly.)

Conversely, when talking about the traffic on a single ISP subscriber's last-mile link, the Poisson model has to be discarded due to criterion 2 being false.  The number of flows going to even a family household is probably in the low dozens at best.  A control-theory approach can also work here.

 - Jonathan Morton

Re: [Codel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Ben Greear]

Measuring one or a few links provides a bit of data, but seems like if someone is trying to understand
a large and real network, then the OWD between point A and B needs to just be input into something much
more grand.  Assuming real-time OWD data exists between 100 to 1000 endpoint pairs, has anyone found a way
to visualize this in a useful manner?

Also, considering something better than ntp may not really scale to 1000+ endpoints, maybe round-trip
time is only viable way to get this type of data.  In that case, maybe clever logic could use things
like trace-route to get some idea of how long it takes to get 'onto' the internet proper, and so estimate
the last-mile latency.  My assumption is that the last-mile latency is where most of the pervasive
assymetric network latencies would exist (or just ping 8.8.8.8 which is 20ms from everywhere due to
$magic).

Endpoints could also triangulate a bit if needed, using some anchor points in the network
under test.

Thanks,
Ben

On 7/12/21 11:21 AM, Bob McMahon wrote:
> iperf 2 supports OWD and gives full histograms for TCP write to read, TCP connect times, latency of packets (with UDP), latency of "frames" with 
> simulated video traffic (TCP and UDP), xfer times of bursts with low duty cycle traffic, and TCP RTT (sampling based.) It also has support for sampling (per 
> interval reports) down to 100 usecs if configured with --enable-fastsampling, otherwise the fastest sampling is 5 ms. We've released all this as open source.
> 
> OWD only works if the end realtime clocks are synchronized using a "machine level" protocol such as IEEE 1588 or PTP. Sadly, *most data centers don't provide 
> sufficient level of clock accuracy and the GPS pulse per second * to colo and vm customers.
> 
> https://iperf2.sourceforge.io/iperf-manpage.html
> 
> Bob
> 
> On Mon, Jul 12, 2021 at 10:40 AM David P. Reed <dpreed@deepplum.com <mailto:dpreed@deepplum.com>> wrote:
> 
> 
>     On Monday, July 12, 2021 9:46am, "Livingood, Jason" <Jason_Livingood@comcast.com <mailto:Jason_Livingood@comcast.com>> said:
> 
>      > I think latency/delay is becoming seen to be as important certainly, if not a more direct proxy for end user QoE. This is all still evolving and I have
>     to say is a super interesting & fun thing to work on. :-)
> 
>     If I could manage to sell one idea to the management hierarchy of communications industry CEOs (operators, vendors, ...) it is this one:
> 
>     "It's the end-to-end latency, stupid!"
> 
>     And I mean, by end-to-end, latency to complete a task at a relevant layer of abstraction.
> 
>     At the link level, it's packet send to packet receive completion.
> 
>     But at the transport level including retransmission buffers, it's datagram (or message) origination until the acknowledgement arrives for that message being
>     delivered after whatever number of retransmissions, freeing the retransmission buffer.
> 
>     At the WWW level, it's mouse click to display update corresponding to completion of the request.
> 
>     What should be noted is that lower level latencies don't directly predict the magnitude of higher-level latencies. But longer lower level latencies almost
>     always amplfify higher level latencies. Often non-linearly.
> 
>     Throughput is very, very weakly related to these latencies, in contrast.
> 
>     The amplification process has to do with the presence of queueing. Queueing is ALWAYS bad for latency, and throughput only helps if it is in exactly the
>     right place (the so-called input queue of the bottleneck process, which is often a link, but not always).
> 
>     Can we get that slogan into Harvard Business Review? Can we get it taught in Managerial Accounting at HBS? (which does address logistics/supply chain queueing).
> 
> 
> 
> 
> 
> 
> 
> This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of 
> the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise 
> restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, 
> you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you 
> received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: [Codel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Ben Greear]

UDP is better for getting actual packet latency, for sure.  TCP is typical-user-experience-latency though,
so it is also useful.

I'm interested in the test and visualization side of this.  If there were a way to give engineers
a good real-time look at a complex real-world network, then they have something to go on while trying
to tune various knobs in their network to improve it.

I'll let others try to figure out how build and tune the knobs, but the data acquisition and
visualization is something we might try to accomplish.  I have a feeling I'm not the
first person to think of this, however....probably someone already has done such
a thing.

Thanks,
Ben

On 7/12/21 1:04 PM, Bob McMahon wrote:
> I believe end host's TCP stats are insufficient as seen per the "failed" congested control mechanisms over the last decades. I think Jaffe pointed this out in 
> 1979 though he was using what's been deemed on this thread as "spherical cow queueing theory."
> 
> "Flow control in store-and-forward computer networks is appropriate for decentralized execution. A formal description of a class of "decentralized flow control 
> algorithms" is given. The feasibility of maximizing power with such algorithms is investigated. On the assumption that communication links behave like M/M/1 
> servers it is shown that no "decentralized flow control algorithm" can maximize network power. Power has been suggested in the literature as a network 
> performance objective. It is also shown that no objective based only on the users' throughputs and average delay is decentralizable. Finally, a restricted class 
> of algorithms cannot even approximate power."
> 
> https://ieeexplore.ieee.org/document/1095152
> 
> Did Jaffe make a mistake?
> 
> Also, it's been observed that latency is non-parametric in it's distributions and computing gaussians per the central limit theorem for OWD feedback loops 
> aren't effective. How does one design a control loop around things that are non-parametric? It also begs the question, what are the feed forward knobs that can 
> actually help?
> 
> Bob
> 
> On Mon, Jul 12, 2021 at 12:07 PM Ben Greear <greearb@candelatech.com <mailto:greearb@candelatech.com>> wrote:
> 
>     Measuring one or a few links provides a bit of data, but seems like if someone is trying to understand
>     a large and real network, then the OWD between point A and B needs to just be input into something much
>     more grand.  Assuming real-time OWD data exists between 100 to 1000 endpoint pairs, has anyone found a way
>     to visualize this in a useful manner?
> 
>     Also, considering something better than ntp may not really scale to 1000+ endpoints, maybe round-trip
>     time is only viable way to get this type of data.  In that case, maybe clever logic could use things
>     like trace-route to get some idea of how long it takes to get 'onto' the internet proper, and so estimate
>     the last-mile latency.  My assumption is that the last-mile latency is where most of the pervasive
>     assymetric network latencies would exist (or just ping 8.8.8.8 which is 20ms from everywhere due to
>     $magic).
> 
>     Endpoints could also triangulate a bit if needed, using some anchor points in the network
>     under test.
> 
>     Thanks,
>     Ben
> 
>     On 7/12/21 11:21 AM, Bob McMahon wrote:
>      > iperf 2 supports OWD and gives full histograms for TCP write to read, TCP connect times, latency of packets (with UDP), latency of "frames" with
>      > simulated video traffic (TCP and UDP), xfer times of bursts with low duty cycle traffic, and TCP RTT (sampling based.) It also has support for sampling (per
>      > interval reports) down to 100 usecs if configured with --enable-fastsampling, otherwise the fastest sampling is 5 ms. We've released all this as open source.
>      >
>      > OWD only works if the end realtime clocks are synchronized using a "machine level" protocol such as IEEE 1588 or PTP. Sadly, *most data centers don't
>     provide
>      > sufficient level of clock accuracy and the GPS pulse per second * to colo and vm customers.
>      >
>      > https://iperf2.sourceforge.io/iperf-manpage.html
>      >
>      > Bob
>      >
>      > On Mon, Jul 12, 2021 at 10:40 AM David P. Reed <dpreed@deepplum.com <mailto:dpreed@deepplum.com> <mailto:dpreed@deepplum.com
>     <mailto:dpreed@deepplum.com>>> wrote:
>      >
>      >
>      >     On Monday, July 12, 2021 9:46am, "Livingood, Jason" <Jason_Livingood@comcast.com <mailto:Jason_Livingood@comcast.com>
>     <mailto:Jason_Livingood@comcast.com <mailto:Jason_Livingood@comcast.com>>> said:
>      >
>      >      > I think latency/delay is becoming seen to be as important certainly, if not a more direct proxy for end user QoE. This is all still evolving and I
>     have
>      >     to say is a super interesting & fun thing to work on. :-)
>      >
>      >     If I could manage to sell one idea to the management hierarchy of communications industry CEOs (operators, vendors, ...) it is this one:
>      >
>      >     "It's the end-to-end latency, stupid!"
>      >
>      >     And I mean, by end-to-end, latency to complete a task at a relevant layer of abstraction.
>      >
>      >     At the link level, it's packet send to packet receive completion.
>      >
>      >     But at the transport level including retransmission buffers, it's datagram (or message) origination until the acknowledgement arrives for that
>     message being
>      >     delivered after whatever number of retransmissions, freeing the retransmission buffer.
>      >
>      >     At the WWW level, it's mouse click to display update corresponding to completion of the request.
>      >
>      >     What should be noted is that lower level latencies don't directly predict the magnitude of higher-level latencies. But longer lower level latencies
>     almost
>      >     always amplfify higher level latencies. Often non-linearly.
>      >
>      >     Throughput is very, very weakly related to these latencies, in contrast.
>      >
>      >     The amplification process has to do with the presence of queueing. Queueing is ALWAYS bad for latency, and throughput only helps if it is in exactly the
>      >     right place (the so-called input queue of the bottleneck process, which is often a link, but not always).
>      >
>      >     Can we get that slogan into Harvard Business Review? Can we get it taught in Managerial Accounting at HBS? (which does address logistics/supply chain
>     queueing).
>      >
>      >
>      >
>      >
>      >
>      >
>      >
>      > This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the
>     use of
>      > the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or
>     otherwise
>      > restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended
>     recipient,
>      > you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you
>      > received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.
> 
> 
>     -- 
>     Ben Greear <greearb@candelatech.com <mailto:greearb@candelatech.com>>
>     Candela Technologies Inc http://www.candelatech.com
> 
> 
> This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of 
> the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise 
> restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, 
> you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you 
> received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: [Codel] [Cake] [Bloat] Little's Law mea culpa, but not invalidating my main point

[David Lang]

[-- Attachment #1: Type: text/plain, Size: 9126 bytes --]

I have seen some performance tests that do explicit DNS timing tests separate 
from other throughput/latency tests.

Since DNS uses UDP (even if it then falls back to TCP in some cases), UDP 
performance (and especially probability of loss at congested links) is very 
important.

David Lang

On Mon, 12 Jul 2021, Ben Greear wrote:

> UDP is better for getting actual packet latency, for sure.  TCP is 
> typical-user-experience-latency though,
> so it is also useful.
>
> I'm interested in the test and visualization side of this.  If there were a 
> way to give engineers
> a good real-time look at a complex real-world network, then they have 
> something to go on while trying
> to tune various knobs in their network to improve it.
>
> I'll let others try to figure out how build and tune the knobs, but the data 
> acquisition and
> visualization is something we might try to accomplish.  I have a feeling I'm 
> not the
> first person to think of this, however....probably someone already has done 
> such
> a thing.
>
> Thanks,
> Ben
>
> On 7/12/21 1:04 PM, Bob McMahon wrote:
>> I believe end host's TCP stats are insufficient as seen per the "failed" 
> congested control mechanisms over the last decades. I think Jaffe pointed 
> this out in 
>> 1979 though he was using what's been deemed on this thread as "spherical 
> cow queueing theory."
>> 
>> "Flow control in store-and-forward computer networks is appropriate for 
> decentralized execution. A formal description of a class of "decentralized 
> flow control 
>> algorithms" is given. The feasibility of maximizing power with such 
> algorithms is investigated. On the assumption that communication links behave 
> like M/M/1 
>> servers it is shown that no "decentralized flow control algorithm" can 
> maximize network power. Power has been suggested in the literature as a 
> network 
>> performance objective. It is also shown that no objective based only on the 
> users' throughputs and average delay is decentralizable. Finally, a 
> restricted class 
>> of algorithms cannot even approximate power."
>> 
>> https://ieeexplore.ieee.org/document/1095152
>> 
>> Did Jaffe make a mistake?
>> 
>> Also, it's been observed that latency is non-parametric in it's 
> distributions and computing gaussians per the central limit theorem for OWD 
> feedback loops 
>> aren't effective. How does one design a control loop around things that are 
> non-parametric? It also begs the question, what are the feed forward knobs 
> that can 
>> actually help?
>> 
>> Bob
>> 
>> On Mon, Jul 12, 2021 at 12:07 PM Ben Greear <greearb@candelatech.com 
> <mailto:greearb@candelatech.com>> wrote:
>>
>>     Measuring one or a few links provides a bit of data, but seems like if 
> someone is trying to understand
>>     a large and real network, then the OWD between point A and B needs to 
> just be input into something much
>>     more grand.  Assuming real-time OWD data exists between 100 to 1000 
> endpoint pairs, has anyone found a way
>>     to visualize this in a useful manner?
>>
>>     Also, considering something better than ntp may not really scale to 
> 1000+ endpoints, maybe round-trip
>>     time is only viable way to get this type of data.  In that case, maybe 
> clever logic could use things
>>     like trace-route to get some idea of how long it takes to get 'onto' 
> the internet proper, and so estimate
>>     the last-mile latency.  My assumption is that the last-mile latency is 
> where most of the pervasive
>>     assymetric network latencies would exist (or just ping 8.8.8.8 which is 
> 20ms from everywhere due to
>>     $magic).
>>
>>     Endpoints could also triangulate a bit if needed, using some anchor 
> points in the network
>>     under test.
>>
>>     Thanks,
>>     Ben
>>
>>     On 7/12/21 11:21 AM, Bob McMahon wrote:
>>      > iperf 2 supports OWD and gives full histograms for TCP write to 
> read, TCP connect times, latency of packets (with UDP), latency of "frames" 
> with
>>      > simulated video traffic (TCP and UDP), xfer times of bursts with low 
> duty cycle traffic, and TCP RTT (sampling based.) It also has support for 
> sampling (per
>>      > interval reports) down to 100 usecs if configured with 
> --enable-fastsampling, otherwise the fastest sampling is 5 ms. We've released 
> all this as open source.
>>      >
>>      > OWD only works if the end realtime clocks are synchronized using a 
> "machine level" protocol such as IEEE 1588 or PTP. Sadly, *most data centers 
> don't
>>     provide
>>      > sufficient level of clock accuracy and the GPS pulse per second * to 
> colo and vm customers.
>>      >
>>      > https://iperf2.sourceforge.io/iperf-manpage.html
>>      >
>>      > Bob
>>      >
>>      > On Mon, Jul 12, 2021 at 10:40 AM David P. Reed <dpreed@deepplum.com 
> <mailto:dpreed@deepplum.com> <mailto:dpreed@deepplum.com
>>     <mailto:dpreed@deepplum.com>>> wrote:
>>      >
>>      >
>>      >     On Monday, July 12, 2021 9:46am, "Livingood, Jason" 
> <Jason_Livingood@comcast.com <mailto:Jason_Livingood@comcast.com>
>>     <mailto:Jason_Livingood@comcast.com 
> <mailto:Jason_Livingood@comcast.com>>> said:
>>      >
>>      >      > I think latency/delay is becoming seen to be as important 
> certainly, if not a more direct proxy for end user QoE. This is all still 
> evolving and I
>>     have
>>      >     to say is a super interesting & fun thing to work on. :-)
>>      >
>>      >     If I could manage to sell one idea to the management hierarchy 
> of communications industry CEOs (operators, vendors, ...) it is this one:
>>      >
>>      >     "It's the end-to-end latency, stupid!"
>>      >
>>      >     And I mean, by end-to-end, latency to complete a task at a 
> relevant layer of abstraction.
>>      >
>>      >     At the link level, it's packet send to packet receive 
> completion.
>>      >
>>      >     But at the transport level including retransmission buffers, 
> it's datagram (or message) origination until the acknowledgement arrives for 
> that
>>     message being
>>      >     delivered after whatever number of retransmissions, freeing the 
> retransmission buffer.
>>      >
>>      >     At the WWW level, it's mouse click to display update 
> corresponding to completion of the request.
>>      >
>>      >     What should be noted is that lower level latencies don't 
> directly predict the magnitude of higher-level latencies. But longer lower 
> level latencies
>>     almost
>>      >     always amplfify higher level latencies. Often non-linearly.
>>      >
>>      >     Throughput is very, very weakly related to these latencies, in 
> contrast.
>>      >
>>      >     The amplification process has to do with the presence of 
> queueing. Queueing is ALWAYS bad for latency, and throughput only helps if it 
> is in exactly the
>>      >     right place (the so-called input queue of the bottleneck 
> process, which is often a link, but not always).
>>      >
>>      >     Can we get that slogan into Harvard Business Review? Can we get 
> it taught in Managerial Accounting at HBS? (which does address 
> logistics/supply chain
>>     queueing).
>>      >
>>      >
>>      >
>>      >
>>      >
>>      >
>>      >
>>      > This electronic communication and the information and any files 
> transmitted with it, or attached to it, are confidential and are intended 
> solely for the
>>     use of
>>      > the individual or entity to whom it is addressed and may contain 
> information that is confidential, legally privileged, protected by privacy 
> laws, or
>>     otherwise
>>      > restricted from disclosure to anyone else. If you are not the 
> intended recipient or the person responsible for delivering the e-mail to the 
> intended
>>     recipient,
>>      > you are hereby notified that any use, copying, distributing, 
> dissemination, forwarding, printing, or copying of this e-mail is strictly 
> prohibited. If you
>>      > received this e-mail in error, please return the e-mail to the 
> sender, delete it from your computer, and destroy any printed copy of it.
>> 
>>
>>     --
>>     Ben Greear <greearb@candelatech.com <mailto:greearb@candelatech.com>>
>>     Candela Technologies Inc http://www.candelatech.com
>> 
>> 
>> This electronic communication and the information and any files transmitted 
> with it, or attached to it, are confidential and are intended solely for the 
> use of 
>> the individual or entity to whom it is addressed and may contain 
> information that is confidential, legally privileged, protected by privacy 
> laws, or otherwise 
>> restricted from disclosure to anyone else. If you are not the intended 
> recipient or the person responsible for delivering the e-mail to the intended 
> recipient, 
>> you are hereby notified that any use, copying, distributing, dissemination, 
> forwarding, printing, or copying of this e-mail is strictly prohibited. If 
> you 
>> received this e-mail in error, please return the e-mail to the sender, 
> delete it from your computer, and destroy any printed copy of it.
>
>
>

Re: [Codel] [Make-wifi-fast] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Jonathan Morton]

> On 12 Jul, 2021, at 11:04 pm, Bob McMahon via Make-wifi-fast <make-wifi-fast@lists.bufferbloat.net> wrote:
> 
> "Flow control in store-and-forward computer networks is appropriate for decentralized execution. A formal description of a class of "decentralized flow control algorithms" is given. The feasibility of maximizing power with such algorithms is investigated. On the assumption that communication links behave like M/M/1 servers it is shown that no "decentralized flow control algorithm" can maximize network power. Power has been suggested in the literature as a network performance objective. It is also shown that no objective based only on the users' throughputs and average delay is decentralizable. Finally, a restricted class of algorithms cannot even approximate power."
> 
> https://ieeexplore.ieee.org/document/1095152
> 
> Did Jaffe make a mistake?

I would suggest that if you model traffic as having no control feedback, you will inevitably find that no control occurs.  But real Internet traffic *does* have control feedback - though it was introduced some time *after* Jaffe's paper, so we can forgive him for a degree of ignorance on that point.  Perhaps Jaffe effectively predicted the ARPANET congestion collapse events with his analysis.

> Also, it's been observed that latency is non-parametric in it's distributions and computing gaussians per the central limit theorem for OWD feedback loops aren't effective. How does one design a control loop around things that are non-parametric? It also begs the question, what are the feed forward knobs that can actually help?

Control at endpoints benefits greatly from even small amounts of information supplied by the network about the degree of congestion present on the path.  This is the role played first by packets lost at queue overflow, then deliberately dropped by AQMs, then marked using the ECN mechanism rather than dropped.

AQM algorithms can be exceedingly simple, or they can be rather sophisticated.  Increased levels of sophistication in both the AQM and the endpoint's congestion control algorithm may be used to increase the "network power" actually obtained.  The required level of complexity for each, achieving reasonably good results, is however quite low.

 - Jonathan Morton

Re: [Codel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 11249 bytes --]

On Mon, Jul 12, 2021 at 1:32 PM Ben Greear <greearb@candelatech.com> wrote:

> UDP is better for getting actual packet latency, for sure.  TCP is
> typical-user-experience-latency though,
> so it is also useful.
>
> I'm interested in the test and visualization side of this.  If there were
> a way to give engineers
> a good real-time look at a complex real-world network, then they have
> something to go on while trying
> to tune various knobs in their network to improve it.
>

I've always liked the smoke-ping visualization, although a single graph is
only really useful for a single pair of endpoints (or a single segment,
maybe).  But I can see using a repeated set of graphs (Tufte has some
examples), that can represent an overview of pairwise collections of
latency+loss:
https://www.edwardtufte.com/bboard/images/0003Cs-8047.GIF
https://www.edwardtufte.com/tufte/psysvcs_p2

These work for understanding because the tiled graphs are all identically
constructed, and the reader first learns how to read a single tile, and
then learns the pattern of which tiles represent which measurements.

Further, they are opinionated.  In the second link above, the y axis is not
based on the measured data, but standardized expected values, which (I
think) is key to quick readability.  You never need to read the axes.  Much
like setting up gauges such that "nominal" is always at the same indicator
position for all graphs (e.g. straight up).  At a glance, you can see if
things are "correct" or not.

That tiling arrangement wouldn't be great for showing interrelationships
(although it may give you a good historical view of correlated behavior).
One thought is to overlay a network graph diagram (graph of all network
links) with small "sparkline" type graphs.

For a more physical-based network graph, I could see visualizing the queue
depth for each egress port (max value over a time of X, or percentage of
time at max depth).

Taken together, the timewise correlation could be useful (which peers are
having problems communicating, and which ports between them are impacted?).

I think getting good data about queue depth may be the hard part,
especially catching transients and the duty cycle / pulse-width of the load
(and then converting that to a number).  Back when I uncovered the iperf
application-level pacing granularity was too high 5 years ago, I called it
them "millibursts", and maybe dtaht pointed out that link utilization is
always 0% or 100%, and it's just a matter of the PWM of the packet rate
that makes it look like something in between.
https://burntchrome.blogspot.com/2016/09/iperf3-and-microbursts.html



I'll let others try to figure out how build and tune the knobs, but the
> data acquisition and
> visualization is something we might try to accomplish.  I have a feeling
> I'm not the
> first person to think of this, however....probably someone already has
> done such
> a thing.
>
> Thanks,
> Ben
>
> On 7/12/21 1:04 PM, Bob McMahon wrote:
> > I believe end host's TCP stats are insufficient as seen per the "failed"
> congested control mechanisms over the last decades. I think Jaffe pointed
> this out in
> > 1979 though he was using what's been deemed on this thread as "spherical
> cow queueing theory."
> >
> > "Flow control in store-and-forward computer networks is appropriate for
> decentralized execution. A formal description of a class of "decentralized
> flow control
> > algorithms" is given. The feasibility of maximizing power with such
> algorithms is investigated. On the assumption that communication links
> behave like M/M/1
> > servers it is shown that no "decentralized flow control algorithm" can
> maximize network power. Power has been suggested in the literature as a
> network
> > performance objective. It is also shown that no objective based only on
> the users' throughputs and average delay is decentralizable. Finally, a
> restricted class
> > of algorithms cannot even approximate power."
> >
> > https://ieeexplore.ieee.org/document/1095152
> >
> > Did Jaffe make a mistake?
> >
> > Also, it's been observed that latency is non-parametric in it's
> distributions and computing gaussians per the central limit theorem for OWD
> feedback loops
> > aren't effective. How does one design a control loop around things that
> are non-parametric? It also begs the question, what are the feed forward
> knobs that can
> > actually help?
> >
> > Bob
> >
> > On Mon, Jul 12, 2021 at 12:07 PM Ben Greear <greearb@candelatech.com
> <mailto:greearb@candelatech.com>> wrote:
> >
> >     Measuring one or a few links provides a bit of data, but seems like
> if someone is trying to understand
> >     a large and real network, then the OWD between point A and B needs
> to just be input into something much
> >     more grand.  Assuming real-time OWD data exists between 100 to 1000
> endpoint pairs, has anyone found a way
> >     to visualize this in a useful manner?
> >
> >     Also, considering something better than ntp may not really scale to
> 1000+ endpoints, maybe round-trip
> >     time is only viable way to get this type of data.  In that case,
> maybe clever logic could use things
> >     like trace-route to get some idea of how long it takes to get 'onto'
> the internet proper, and so estimate
> >     the last-mile latency.  My assumption is that the last-mile latency
> is where most of the pervasive
> >     assymetric network latencies would exist (or just ping 8.8.8.8 which
> is 20ms from everywhere due to
> >     $magic).
> >
> >     Endpoints could also triangulate a bit if needed, using some anchor
> points in the network
> >     under test.
> >
> >     Thanks,
> >     Ben
> >
> >     On 7/12/21 11:21 AM, Bob McMahon wrote:
> >      > iperf 2 supports OWD and gives full histograms for TCP write to
> read, TCP connect times, latency of packets (with UDP), latency of "frames"
> with
> >      > simulated video traffic (TCP and UDP), xfer times of bursts with
> low duty cycle traffic, and TCP RTT (sampling based.) It also has support
> for sampling (per
> >      > interval reports) down to 100 usecs if configured with
> --enable-fastsampling, otherwise the fastest sampling is 5 ms. We've
> released all this as open source.
> >      >
> >      > OWD only works if the end realtime clocks are synchronized using
> a "machine level" protocol such as IEEE 1588 or PTP. Sadly, *most data
> centers don't
> >     provide
> >      > sufficient level of clock accuracy and the GPS pulse per second *
> to colo and vm customers.
> >      >
> >      > https://iperf2.sourceforge.io/iperf-manpage.html
> >      >
> >      > Bob
> >      >
> >      > On Mon, Jul 12, 2021 at 10:40 AM David P. Reed <
> dpreed@deepplum.com <mailto:dpreed@deepplum.com> <mailto:
> dpreed@deepplum.com
> >     <mailto:dpreed@deepplum.com>>> wrote:
> >      >
> >      >
> >      >     On Monday, July 12, 2021 9:46am, "Livingood, Jason" <
> Jason_Livingood@comcast.com <mailto:Jason_Livingood@comcast.com>
> >     <mailto:Jason_Livingood@comcast.com <mailto:
> Jason_Livingood@comcast.com>>> said:
> >      >
> >      >      > I think latency/delay is becoming seen to be as important
> certainly, if not a more direct proxy for end user QoE. This is all still
> evolving and I
> >     have
> >      >     to say is a super interesting & fun thing to work on. :-)
> >      >
> >      >     If I could manage to sell one idea to the management
> hierarchy of communications industry CEOs (operators, vendors, ...) it is
> this one:
> >      >
> >      >     "It's the end-to-end latency, stupid!"
> >      >
> >      >     And I mean, by end-to-end, latency to complete a task at a
> relevant layer of abstraction.
> >      >
> >      >     At the link level, it's packet send to packet receive
> completion.
> >      >
> >      >     But at the transport level including retransmission buffers,
> it's datagram (or message) origination until the acknowledgement arrives
> for that
> >     message being
> >      >     delivered after whatever number of retransmissions, freeing
> the retransmission buffer.
> >      >
> >      >     At the WWW level, it's mouse click to display update
> corresponding to completion of the request.
> >      >
> >      >     What should be noted is that lower level latencies don't
> directly predict the magnitude of higher-level latencies. But longer lower
> level latencies
> >     almost
> >      >     always amplfify higher level latencies. Often non-linearly.
> >      >
> >      >     Throughput is very, very weakly related to these latencies,
> in contrast.
> >      >
> >      >     The amplification process has to do with the presence of
> queueing. Queueing is ALWAYS bad for latency, and throughput only helps if
> it is in exactly the
> >      >     right place (the so-called input queue of the bottleneck
> process, which is often a link, but not always).
> >      >
> >      >     Can we get that slogan into Harvard Business Review? Can we
> get it taught in Managerial Accounting at HBS? (which does address
> logistics/supply chain
> >     queueing).
> >      >
> >      >
> >      >
> >      >
> >      >
> >      >
> >      >
> >      > This electronic communication and the information and any files
> transmitted with it, or attached to it, are confidential and are intended
> solely for the
> >     use of
> >      > the individual or entity to whom it is addressed and may contain
> information that is confidential, legally privileged, protected by privacy
> laws, or
> >     otherwise
> >      > restricted from disclosure to anyone else. If you are not the
> intended recipient or the person responsible for delivering the e-mail to
> the intended
> >     recipient,
> >      > you are hereby notified that any use, copying, distributing,
> dissemination, forwarding, printing, or copying of this e-mail is strictly
> prohibited. If you
> >      > received this e-mail in error, please return the e-mail to the
> sender, delete it from your computer, and destroy any printed copy of it.
> >
> >
> >     --
> >     Ben Greear <greearb@candelatech.com <mailto:greearb@candelatech.com
> >>
> >     Candela Technologies Inc http://www.candelatech.com
> >
> >
> > This electronic communication and the information and any files
> transmitted with it, or attached to it, are confidential and are intended
> solely for the use of
> > the individual or entity to whom it is addressed and may contain
> information that is confidential, legally privileged, protected by privacy
> laws, or otherwise
> > restricted from disclosure to anyone else. If you are not the intended
> recipient or the person responsible for delivering the e-mail to the
> intended recipient,
> > you are hereby notified that any use, copying, distributing,
> dissemination, forwarding, printing, or copying of this e-mail is strictly
> prohibited. If you
> > received this e-mail in error, please return the e-mail to the sender,
> delete it from your computer, and destroy any printed copy of it.
>
>
> --
> Ben Greear <greearb@candelatech.com>
> Candela Technologies Inc  http://www.candelatech.com
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>

[-- Attachment #2: Type: text/html, Size: 14879 bytes --]

Re: [Codel] [Make-wifi-fast] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 4788 bytes --]

With the disclaimer that I'm not as strong in statistics and modelling as
I'd like to be....

I think it's not useful to attempt to stochastically model the behavior of
what are actually active (well, reactive) components.  The responses of
each piece are deterministic, but the inputs (users) are not.  So while you
could maybe measure the behavior of a network, and then build a hidden
markov model that can produce the same results, I don't see how it would be
useful for testing the behavior of either the reactive components (TCP CC
algs) or the layers below the reactive components (queues and links),
because the model needs to react to the behavior of the pieces it's sitting
on top of, not due to a stochastic process that's independent (in the
statistical sense) of the underlying queues and links.

Probably a "well duh..." thought for many here.  But I was _amazed_ when
working with very senior engineers for network hardware companies, who said
all testing was done with a static blend of "i-mix" traffic (in both
directions), even though they were looking at last-mile network usage which
was going to be primarily TCP download, just like a home, and nothing like
i-mix.  Or that the applications running on top of that gear were actually
reactive to their (mis-)management of their queues and loads.

On Fri, Jul 9, 2021 at 4:56 PM Jonathan Morton <chromatix99@gmail.com>
wrote:

> > On 10 Jul, 2021, at 2:01 am, Leonard Kleinrock <lk@cs.ucla.edu> wrote:
> >
> > No question that non-stationarity and instability are what we often see
> in networks.  And, non-stationarity and instability are both topics that
> lead to very complex analytical problems in queueing theory.  You can find
> some results on the transient analysis in the queueing theory literature
> (including the second volume of my Queueing Systems book), but they are
> limited and hard. Nevertheless, the literature does contain some works on
> transient analysis of queueing systems as applied to network congestion
> control - again limited. On the other hand, as you said, control theory
> addresses stability head on and does offer some tools as well, but again,
> it is hairy.
>
> I was just about to mention control theory.
>
> One basic characteristic of Poisson traffic is that it is inelastic, and
> assumes there is no control feedback whatsoever.  This means it can only be
> a valid model when the following are both true:
>
> 1: The offered load is *below* the link capacity, for all links, averaged
> over time.
>
> 2: A high degree of statistical multiplexing exists.
>
> If 1: is not true and the traffic is truly inelastic, then the queues will
> inevitably fill up and congestion collapse will result, as shown from
> ARPANET experience in the 1980s; the solution was to introduce control
> feedback to the traffic, initially in the form of TCP Reno.  If 2: is not
> true then the traffic cannot be approximated as Poisson arrivals,
> regardless of load relative to capacity, because the degree of correlation
> is too high.
>
> Taking the iPhone introduction anecdote as an illustrative example,
> measuring utilisation as very close to 100% is a clear warning sign that
> the Poisson model was inappropriate, and a control-theory approach was
> needed instead, to capture the feedback effects of congestion control.  The
> high degree of statistical multiplexing inherent to a major ISP backhaul is
> irrelevant to that determination.
>
> Such a model would have found that the primary source of control feedback
> was human users giving up in disgust.  However, different humans have
> different levels of tolerance and persistence, so this feedback was not
> sufficient to reduce the load sufficiently to give the majority of users a
> good service; instead, *all* users received a poor service and many users
> received no usable service.  Introducing a technological control feedback,
> in the form of packet loss upon overflow of correctly-sized queues,
> improved service for everyone.
>
> (BTW, DNS becomes significantly unreliable around 1-2 seconds RTT, due to
> protocol timeouts, which is inherited by all applications that rely on DNS
> lookups.  Merely reducing the delays consistently below that threshold
> would have improved perceived reliability markedly.)
>
> Conversely, when talking about the traffic on a single ISP subscriber's
> last-mile link, the Poisson model has to be discarded due to criterion 2
> being false.  The number of flows going to even a family household is
> probably in the low dozens at best.  A control-theory approach can also
> work here.
>
>  - Jonathan Morton
> _______________________________________________
> Make-wifi-fast mailing list
> Make-wifi-fast@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast

[-- Attachment #2: Type: text/html, Size: 5480 bytes --]

Re: [Codel] [Cake] [Make-wifi-fast] [Starlink] [Cerowrt-devel] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[David Lang]

If you are going to setup a test environment for wifi, you need to include the 
ability to make a fe cases that only happen with RF, not with wired networks and 
are commonly overlooked

1. station A can hear station B and C but they cannot hear each other
2. station A can hear station B but station B cannot hear station A 
3. station A can hear that station B is transmitting, but not with a strong 
enough signal to decode the signal (yes in theory you can work around 
interference, but in practice interference is still a real thing)

David Lang

Re: [Codel] [Cake] [Make-wifi-fast] [Starlink] [Cerowrt-devel] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Ben Greear]

On 8/2/21 4:16 PM, David Lang wrote:
> If you are going to setup a test environment for wifi, you need to include the ability to make a fe cases that only happen with RF, not with wired networks and 
> are commonly overlooked
> 
> 1. station A can hear station B and C but they cannot hear each other
> 2. station A can hear station B but station B cannot hear station A 3. station A can hear that station B is transmitting, but not with a strong enough signal to 
> decode the signal (yes in theory you can work around interference, but in practice interference is still a real thing)
> 
> David Lang
> 

To add to this, I think you need lots of different station devices, different capabilities (/n, /ac, /ax, etc)
different numbers of spatial streams, and different distances from the AP.  From download queueing perspective, changing
the capabilities may be sufficient while keeping all stations at same distance.  This assumes you are not
actually testing the wifi rate-ctrl alg. itself, so different throughput levels for different stations would be enough.

So, a good station emulator setup (and/or pile of real stations) and a few RF chambers and
programmable attenuators and you can test that setup...

 From upload perspective, I guess same setup would do the job.  Queuing/fairness might depend a bit more on the
station devices, emulated or otherwise, but I guess a clever AP could enforce fairness in upstream direction
too by implementing per-sta queues.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: [Codel] [Cake] [Make-wifi-fast] [Starlink] [Cerowrt-devel] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[David Lang]

that matrix cannot create asymmetric paths (at least, not unless you are also 
tinkering with power settings on the nodes), and will have trouble making hidden 
transmitters (station A can hear station B and C but B and C cannot tell the 
other exists) as a node can hear that something is transmitting at much lower 
power levels than it cn decode the signal.

David Lang

On Mon, 2 Aug 2021, Bob McMahon wrote:

> On Mon, Aug 2, 2021 at 4:16 PM David Lang <david@lang.hm> wrote:
>
>> If you are going to setup a test environment for wifi, you need to include
>> the
>> ability to make a fe cases that only happen with RF, not with wired
>> networks and
>> are commonly overlooked
>>
>> 1. station A can hear station B and C but they cannot hear each other
>> 2. station A can hear station B but station B cannot hear station A
>> 3. station A can hear that station B is transmitting, but not with a
>> strong
>> enough signal to decode the signal (yes in theory you can work around
>> interference, but in practice interference is still a real thing)
>>
>> David Lang
>>
>>
>
>

Re: [Codel] [Cake] [Make-wifi-fast] [Starlink] [Cerowrt-devel] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[David Lang]

I guess it depends on what you are intending to test. If you are not going to 
tinker with any of the over-the-air settings (including the number of packets 
transmitted in one aggregate), the details of what happen over the air don't 
matter much.

But if you are going to be doing any tinkering with what is getting sent, and 
you ignore the hidden transmitter type problems, you will create a solution that 
seems to work really well in the lab and falls on it's face out in the wild 
where spectrum overload and hidden transmitters are the norm (at least in urban 
areas), not rare corner cases.

you don't need to include them in every test, but you need to have a way to 
configure your lab to include them before you consider any settings/algorithm 
ready to try in the wild.

David Lang

On Mon, 2 Aug 2021, Bob McMahon wrote:

> We find four nodes, a primary BSS and an adjunct one quite good for lots of
> testing.  The six nodes allows for a primary BSS and two adjacent ones. We
> want to minimize complexity to necessary and sufficient.
>
> The challenge we find is having variability (e.g. montecarlos) that's
> reproducible and has relevant information. Basically, the distance matrices
> have h-matrices as their elements. Our chips can provide these h-matrices.
>
> The parts for solid state programmable attenuators and phase shifters
> aren't very expensive. A device that supports a five branch tree and 2x2
> MIMO seems a very good starting point.
>
> Bob
>
> On Mon, Aug 2, 2021 at 4:55 PM Ben Greear <greearb@candelatech.com> wrote:
>
>> On 8/2/21 4:16 PM, David Lang wrote:
>>> If you are going to setup a test environment for wifi, you need to
>> include the ability to make a fe cases that only happen with RF, not with
>> wired networks and
>>> are commonly overlooked
>>>
>>> 1. station A can hear station B and C but they cannot hear each other
>>> 2. station A can hear station B but station B cannot hear station A 3.
>> station A can hear that station B is transmitting, but not with a strong
>> enough signal to
>>> decode the signal (yes in theory you can work around interference, but
>> in practice interference is still a real thing)
>>>
>>> David Lang
>>>
>>
>> To add to this, I think you need lots of different station devices,
>> different capabilities (/n, /ac, /ax, etc)
>> different numbers of spatial streams, and different distances from the
>> AP.  From download queueing perspective, changing
>> the capabilities may be sufficient while keeping all stations at same
>> distance.  This assumes you are not
>> actually testing the wifi rate-ctrl alg. itself, so different throughput
>> levels for different stations would be enough.
>>
>> So, a good station emulator setup (and/or pile of real stations) and a few
>> RF chambers and
>> programmable attenuators and you can test that setup...
>>
>>  From upload perspective, I guess same setup would do the job.
>> Queuing/fairness might depend a bit more on the
>> station devices, emulated or otherwise, but I guess a clever AP could
>> enforce fairness in upstream direction
>> too by implementing per-sta queues.
>>
>> Thanks,
>> Ben
>>
>> --
>> Ben Greear <greearb@candelatech.com>
>> Candela Technologies Inc  http://www.candelatech.com
>>
>
>

Re: [Codel] [Cake] [Make-wifi-fast] [Starlink] [Cerowrt-devel] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[David Lang]

symmetry is not always (or usually) true. stations are commonly heard at much 
larger distances than they can talk, mobile devices have much less transmit 
power (becuase they are operating on batteries) than fixed stations, and when 
you adjust the transmit power on a station, you don't adjust it's receive 
sensitivity.

David Lang

  On Mon, 2 Aug 2021, Bob McMahon wrote:

> Date: Mon, 2 Aug 2021 20:23:06 -0700
> From: Bob McMahon <bob.mcmahon@broadcom.com>
> To: David Lang <david@lang.hm>
> Cc: Ben Greear <greearb@candelatech.com>,
>     Luca Muscariello <muscariello@ieee.org>,
>     Cake List <cake@lists.bufferbloat.net>,
>     Make-Wifi-fast <make-wifi-fast@lists.bufferbloat.net>,
>     Leonard Kleinrock <lk@cs.ucla.edu>, starlink@lists.bufferbloat.net,
>     codel@lists.bufferbloat.net,
>     cerowrt-devel <cerowrt-devel@lists.bufferbloat.net>,
>     bloat <bloat@lists.bufferbloat.net>
> Subject: Re: [Cake] [Make-wifi-fast] [Starlink] [Cerowrt-devel] Due Aug 2:
>     Internet Quality workshop CFP for the internet architecture board
> 
> The distance matrix defines signal attenuations/loss between pairs.  It's
> straightforward to create a distance matrix that has hidden nodes because
> all "signal  loss" between pairs is defined.  Let's say a 120dB attenuation
> path will cause a node to be hidden as an example.
>
>     A    B     C    D
> A   -   35   120   65
> B         -      65   65
> C               -       65
> D                         -
>
> So in the above, AC are hidden from each other but nobody else is. It does
> assume symmetry between pairs but that's typically true.
>
> The RF device takes these distance matrices as settings and calculates the
> five branch tree values (as demonstrated in the video). There are
> limitations to solutions though but I've found those not to be an issue to
> date. I've been able to produce hidden nodes quite readily. Add the phase
> shifters and spatial stream powers can also be affected, but this isn't
> shown in this simple example.
>
> Bob
>
> On Mon, Aug 2, 2021 at 8:12 PM David Lang <david@lang.hm> wrote:
>
>> I guess it depends on what you are intending to test. If you are not going
>> to
>> tinker with any of the over-the-air settings (including the number of
>> packets
>> transmitted in one aggregate), the details of what happen over the air
>> don't
>> matter much.
>>
>> But if you are going to be doing any tinkering with what is getting sent,
>> and
>> you ignore the hidden transmitter type problems, you will create a
>> solution that
>> seems to work really well in the lab and falls on it's face out in the
>> wild
>> where spectrum overload and hidden transmitters are the norm (at least in
>> urban
>> areas), not rare corner cases.
>>
>> you don't need to include them in every test, but you need to have a way
>> to
>> configure your lab to include them before you consider any
>> settings/algorithm
>> ready to try in the wild.
>>
>> David Lang
>>
>> On Mon, 2 Aug 2021, Bob McMahon wrote:
>>
>>> We find four nodes, a primary BSS and an adjunct one quite good for lots
>> of
>>> testing.  The six nodes allows for a primary BSS and two adjacent ones.
>> We
>>> want to minimize complexity to necessary and sufficient.
>>>
>>> The challenge we find is having variability (e.g. montecarlos) that's
>>> reproducible and has relevant information. Basically, the distance
>> matrices
>>> have h-matrices as their elements. Our chips can provide these
>> h-matrices.
>>>
>>> The parts for solid state programmable attenuators and phase shifters
>>> aren't very expensive. A device that supports a five branch tree and 2x2
>>> MIMO seems a very good starting point.
>>>
>>> Bob
>>>
>>> On Mon, Aug 2, 2021 at 4:55 PM Ben Greear <greearb@candelatech.com>
>> wrote:
>>>
>>>> On 8/2/21 4:16 PM, David Lang wrote:
>>>>> If you are going to setup a test environment for wifi, you need to
>>>> include the ability to make a fe cases that only happen with RF, not
>> with
>>>> wired networks and
>>>>> are commonly overlooked
>>>>>
>>>>> 1. station A can hear station B and C but they cannot hear each other
>>>>> 2. station A can hear station B but station B cannot hear station A 3.
>>>> station A can hear that station B is transmitting, but not with a strong
>>>> enough signal to
>>>>> decode the signal (yes in theory you can work around interference, but
>>>> in practice interference is still a real thing)
>>>>>
>>>>> David Lang
>>>>>
>>>>
>>>> To add to this, I think you need lots of different station devices,
>>>> different capabilities (/n, /ac, /ax, etc)
>>>> different numbers of spatial streams, and different distances from the
>>>> AP.  From download queueing perspective, changing
>>>> the capabilities may be sufficient while keeping all stations at same
>>>> distance.  This assumes you are not
>>>> actually testing the wifi rate-ctrl alg. itself, so different throughput
>>>> levels for different stations would be enough.
>>>>
>>>> So, a good station emulator setup (and/or pile of real stations) and a
>> few
>>>> RF chambers and
>>>> programmable attenuators and you can test that setup...
>>>>
>>>>  From upload perspective, I guess same setup would do the job.
>>>> Queuing/fairness might depend a bit more on the
>>>> station devices, emulated or otherwise, but I guess a clever AP could
>>>> enforce fairness in upstream direction
>>>> too by implementing per-sta queues.
>>>>
>>>> Thanks,
>>>> Ben
>>>>
>>>> --
>>>> Ben Greear <greearb@candelatech.com>
>>>> Candela Technologies Inc  http://www.candelatech.com
>>>>
>>>
>>>
>>
>
>

Re: [Codel] [Cake] [Make-wifi-fast] [Starlink] [Cerowrt-devel] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[David Lang]

I agree that we don't want to make perfect the enemy of better.

A lot of the issues I'm calling out can be simulated/enhanced with different 
power levels.

over wifi distances, I don't think time delays are going to be noticable (we're 
talking 10s to low 100s of feet, not miles)

David Lang

On Mon, 2 Aug 2021, Bob McMahon wrote:

> fair enough, but for this "RF emulator device" being able to support
> distance matrices, even hollow symmetric ones, is much better than what's
> typically done. The variable solid state phase shifters are 0-360 so don't
> provide real time delays either.
>
> This is another "something is better than nothing" type proposal. I think
> it can be deployed at a relatively low cost which allows for more
> standardized, automated test rigs and much less human interactions and
> human errors.
>
> Bob
>
> On Mon, Aug 2, 2021 at 9:30 PM David Lang <david@lang.hm> wrote:
>
>> symmetry is not always (or usually) true. stations are commonly heard at
>> much
>> larger distances than they can talk, mobile devices have much less
>> transmit
>> power (becuase they are operating on batteries) than fixed stations, and
>> when
>> you adjust the transmit power on a station, you don't adjust it's receive
>> sensitivity.
>>
>> David Lang
>>
>>   On Mon, 2 Aug 2021, Bob McMahon wrote:
>>
>>> Date: Mon, 2 Aug 2021 20:23:06 -0700
>>> From: Bob McMahon <bob.mcmahon@broadcom.com>
>>> To: David Lang <david@lang.hm>
>>> Cc: Ben Greear <greearb@candelatech.com>,
>>>     Luca Muscariello <muscariello@ieee.org>,
>>>     Cake List <cake@lists.bufferbloat.net>,
>>>     Make-Wifi-fast <make-wifi-fast@lists.bufferbloat.net>,
>>>     Leonard Kleinrock <lk@cs.ucla.edu>, starlink@lists.bufferbloat.net,
>>>     codel@lists.bufferbloat.net,
>>>     cerowrt-devel <cerowrt-devel@lists.bufferbloat.net>,
>>>     bloat <bloat@lists.bufferbloat.net>
>>> Subject: Re: [Cake] [Make-wifi-fast] [Starlink] [Cerowrt-devel] Due Aug
>> 2:
>>>     Internet Quality workshop CFP for the internet architecture board
>>>
>>> The distance matrix defines signal attenuations/loss between pairs.  It's
>>> straightforward to create a distance matrix that has hidden nodes because
>>> all "signal  loss" between pairs is defined.  Let's say a 120dB
>> attenuation
>>> path will cause a node to be hidden as an example.
>>>
>>>     A    B     C    D
>>> A   -   35   120   65
>>> B         -      65   65
>>> C               -       65
>>> D                         -
>>>
>>> So in the above, AC are hidden from each other but nobody else is. It
>> does
>>> assume symmetry between pairs but that's typically true.
>>>
>>> The RF device takes these distance matrices as settings and calculates
>> the
>>> five branch tree values (as demonstrated in the video). There are
>>> limitations to solutions though but I've found those not to be an issue
>> to
>>> date. I've been able to produce hidden nodes quite readily. Add the phase
>>> shifters and spatial stream powers can also be affected, but this isn't
>>> shown in this simple example.
>>>
>>> Bob
>>>
>>> On Mon, Aug 2, 2021 at 8:12 PM David Lang <david@lang.hm> wrote:
>>>
>>>> I guess it depends on what you are intending to test. If you are not
>> going
>>>> to
>>>> tinker with any of the over-the-air settings (including the number of
>>>> packets
>>>> transmitted in one aggregate), the details of what happen over the air
>>>> don't
>>>> matter much.
>>>>
>>>> But if you are going to be doing any tinkering with what is getting
>> sent,
>>>> and
>>>> you ignore the hidden transmitter type problems, you will create a
>>>> solution that
>>>> seems to work really well in the lab and falls on it's face out in the
>>>> wild
>>>> where spectrum overload and hidden transmitters are the norm (at least
>> in
>>>> urban
>>>> areas), not rare corner cases.
>>>>
>>>> you don't need to include them in every test, but you need to have a way
>>>> to
>>>> configure your lab to include them before you consider any
>>>> settings/algorithm
>>>> ready to try in the wild.
>>>>
>>>> David Lang
>>>>
>>>> On Mon, 2 Aug 2021, Bob McMahon wrote:
>>>>
>>>>> We find four nodes, a primary BSS and an adjunct one quite good for
>> lots
>>>> of
>>>>> testing.  The six nodes allows for a primary BSS and two adjacent ones.
>>>> We
>>>>> want to minimize complexity to necessary and sufficient.
>>>>>
>>>>> The challenge we find is having variability (e.g. montecarlos) that's
>>>>> reproducible and has relevant information. Basically, the distance
>>>> matrices
>>>>> have h-matrices as their elements. Our chips can provide these
>>>> h-matrices.
>>>>>
>>>>> The parts for solid state programmable attenuators and phase shifters
>>>>> aren't very expensive. A device that supports a five branch tree and
>> 2x2
>>>>> MIMO seems a very good starting point.
>>>>>
>>>>> Bob
>>>>>
>>>>> On Mon, Aug 2, 2021 at 4:55 PM Ben Greear <greearb@candelatech.com>
>>>> wrote:
>>>>>
>>>>>> On 8/2/21 4:16 PM, David Lang wrote:
>>>>>>> If you are going to setup a test environment for wifi, you need to
>>>>>> include the ability to make a fe cases that only happen with RF, not
>>>> with
>>>>>> wired networks and
>>>>>>> are commonly overlooked
>>>>>>>
>>>>>>> 1. station A can hear station B and C but they cannot hear each other
>>>>>>> 2. station A can hear station B but station B cannot hear station A
>> 3.
>>>>>> station A can hear that station B is transmitting, but not with a
>> strong
>>>>>> enough signal to
>>>>>>> decode the signal (yes in theory you can work around interference,
>> but
>>>>>> in practice interference is still a real thing)
>>>>>>>
>>>>>>> David Lang
>>>>>>>
>>>>>>
>>>>>> To add to this, I think you need lots of different station devices,
>>>>>> different capabilities (/n, /ac, /ax, etc)
>>>>>> different numbers of spatial streams, and different distances from the
>>>>>> AP.  From download queueing perspective, changing
>>>>>> the capabilities may be sufficient while keeping all stations at same
>>>>>> distance.  This assumes you are not
>>>>>> actually testing the wifi rate-ctrl alg. itself, so different
>> throughput
>>>>>> levels for different stations would be enough.
>>>>>>
>>>>>> So, a good station emulator setup (and/or pile of real stations) and a
>>>> few
>>>>>> RF chambers and
>>>>>> programmable attenuators and you can test that setup...
>>>>>>
>>>>>>  From upload perspective, I guess same setup would do the job.
>>>>>> Queuing/fairness might depend a bit more on the
>>>>>> station devices, emulated or otherwise, but I guess a clever AP could
>>>>>> enforce fairness in upstream direction
>>>>>> too by implementing per-sta queues.
>>>>>>
>>>>>> Thanks,
>>>>>> Ben
>>>>>>
>>>>>> --
>>>>>> Ben Greear <greearb@candelatech.com>
>>>>>> Candela Technologies Inc  http://www.candelatech.com
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>

Re: [Codel] [Make-wifi-fast] [Starlink] [Cake] [Cerowrt-devel] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 6221 bytes --]

My own experiments with this, in the past (5+ years ago), was that you
absolutely had to use cabled setups for repeatability, but then didn't have
enough randomness in the variability to really test anything that was
problematic.  We could create hidden nodes, or arbitrary meshes of devices,
but they were always static.

We used N-way RF splitters and either direct coax in lieu of antennas, or
isolation boxes with an antenna attached to a bulkhead fitting, with coax
on the outside.  One other problem we ran into was that unshielded radio
front-ends could "hear" each other without isolation boxes.

I really wanted both variable attenuators, and points where I could inject
RF noise, so that instead of broad-band attenuation, maybe we could just
swamp the communications with other noise (which is also a common thing we
were running into with both our 900Mhz (ZWave) and 2.4GHz (wifi) radios.

Less common, but something I still see, is that a moving station has
continual issues staying in proper MIMO phase(s) with the AP.  Or I think
that's what's happening.  Slow, continual movement of the two, relative to
each other, and the packet rate drops through the floor until they stop
having relative motion.  And I assume that also applies to time-varying
path-loss and path-distance (multipath reflections).

On Sat, Aug 7, 2021 at 10:15 PM Bob McMahon via Make-wifi-fast <
make-wifi-fast@lists.bufferbloat.net> wrote:

> We have hundreds of test rigs in multiple labs all over geography. Each
> rig is shielded from the others using things like RF enclosures. We want
> reproducibility in the RF paths/channels as well as variability. Most have
> built fixed rigs using conducted equipment. This is far from anything real.
> A butler matrix produces great condition numbers but that makes it too easy
> for MIMO rate selection algorithms.
>
> Our real world test is using a real house that has been rented. Not cheap
> nor scalable.
>
> There is quite a gap between the two. A RF path device that supports both
> variable range and variable mixing is a step towards closing the gap.
>
> Bob
>
> On Sat, Aug 7, 2021 at 10:07 PM Dick Roy <dickroy@alum.mit.edu> wrote:
>
>>
>>
>>
>> ------------------------------
>>
>> *From:* Starlink [mailto:starlink-bounces@lists.bufferbloat.net] *On
>> Behalf Of *Bob McMahon
>> *Sent:* Monday, August 2, 2021 6:24 PM
>> *To:* Leonard Kleinrock
>> *Cc:* starlink@lists.bufferbloat.net; Make-Wifi-fast; Cake List;
>> codel@lists.bufferbloat.net; cerowrt-devel; bloat
>> *Subject:* Re: [Starlink] [Cake] [Make-wifi-fast] [Cerowrt-devel] Due
>> Aug 2: Internet Quality workshop CFP for the internet architecture board
>>
>>
>>
>> I found the following talk relevant to distances between all the nodes.
>> https://www.youtube.com/watch?v=PNoUcQTCxiM
>>
>> Distance is an abstract idea but applies to energy into a node as well as
>> phylogenetic trees. It's the same problem, i.e. fitting a distance matrix
>> using some sort of tree. I've found the five branch tree works well for
>> four nodes.
>>
>> *[RR] These trees are means for approximating a higher dimensional
>> real-world problem with a lower dimensional structure.  You may be doing
>> this to save hardware when trying to cable up some complex test scenarios,
>> however I’m wondering why?  Why not just put the STAs in the lab and turn
>> them on rather than cabling them?*
>>
>>
>>
>> Bob
>>
>>
>>
>> On Mon, Aug 2, 2021 at 5:37 PM Leonard Kleinrock <lk@cs.ucla.edu> wrote:
>>
>> These cases are what my student, Fouad Tobagi and I called the Hidden
>> Terminal Problem (with the Busy Tone solution) back in 1975.
>>
>> Len
>>
>>
>> > On Aug 2, 2021, at 4:16 PM, David Lang <david@lang.hm> wrote:
>> >
>> > If you are going to setup a test environment for wifi, you need to
>> include the ability to make a fe cases that only happen with RF, not with
>> wired networks and are commonly overlooked
>> >
>> > 1. station A can hear station B and C but they cannot hear each other
>> > 2. station A can hear station B but station B cannot hear station A 3.
>> station A can hear that station B is transmitting, but not with a strong
>> enough signal to decode the signal (yes in theory you can work around
>> interference, but in practice interference is still a real thing)
>> >
>> > David Lang
>> >
>>
>>
>> This electronic communication and the information and any files
>> transmitted with it, or attached to it, are confidential and are intended
>> solely for the use of the individual or entity to whom it is addressed and
>> may contain information that is confidential, legally privileged, protected
>> by privacy laws, or otherwise restricted from disclosure to anyone else. If
>> you are not the intended recipient or the person responsible for delivering
>> the e-mail to the intended recipient, you are hereby notified that any use,
>> copying, distributing, dissemination, forwarding, printing, or copying of
>> this e-mail is strictly prohibited. If you received this e-mail in error,
>> please return the e-mail to the sender, delete it from your computer, and
>> destroy any printed copy of it.
>>
>
> This electronic communication and the information and any files
> transmitted with it, or attached to it, are confidential and are intended
> solely for the use of the individual or entity to whom it is addressed and
> may contain information that is confidential, legally privileged, protected
> by privacy laws, or otherwise restricted from disclosure to anyone else. If
> you are not the intended recipient or the person responsible for delivering
> the e-mail to the intended recipient, you are hereby notified that any use,
> copying, distributing, dissemination, forwarding, printing, or copying of
> this e-mail is strictly prohibited. If you received this e-mail in error,
> please return the e-mail to the sender, delete it from your computer, and
> destroy any printed copy of it.
> _______________________________________________
> Make-wifi-fast mailing list
> Make-wifi-fast@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast

[-- Attachment #2: Type: text/html, Size: 10068 bytes --]

Re: [Codel] [Bloat] [Make-wifi-fast] [Starlink] [Cake] [Cerowrt-devel] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Jonathan Morton]

> On 8 Aug, 2021, at 9:36 pm, Aaron Wood <woody77@gmail.com> wrote:
> 
> Less common, but something I still see, is that a moving station has continual issues staying in proper MIMO phase(s) with the AP.  Or I think that's what's happening.  Slow, continual movement of the two, relative to each other, and the packet rate drops through the floor until they stop having relative motion.  And I assume that also applies to time-varying path-loss and path-distance (multipath reflections).

So is it time to mount test stations on model railway wagons?

 - Jonathan Morton

Re: [Codel] [Starlink] Anhyone have a spare couple a hundred million ... Elon may need to start a go-fund-me page!

[David Lang]

[-- Attachment #1: Type: text/plain, Size: 1979 bytes --]

the biggest problem starlink faces is shipping enough devices (and launching the 
satellites to support them), not demand. There are enough people interested in 
paying full price that if the broadband subsities did not exist, it wouldn't 
reduce the demand noticably.

but if the feds are handing out money, SpaceX is foolish not to apply for it.

David Lang

On Tue, 10 Aug 2021, Jeremy Austin wrote:

> Date: Tue, 10 Aug 2021 12:33:11 -0800
> From: Jeremy Austin <jeremy@aterlo.com>
> To: dickroy@alum.mit.edu
> Cc: Cake List <cake@lists.bufferbloat.net>,
>     Make-Wifi-fast <make-wifi-fast@lists.bufferbloat.net>,
>     Bob McMahon <bob.mcmahon@broadcom.com>, starlink@lists.bufferbloat.net,
>     codel <codel@lists.bufferbloat.net>,
>     cerowrt-devel <cerowrt-devel@lists.bufferbloat.net>,
>     bloat <bloat@lists.bufferbloat.net>
> Subject: Re: [Starlink] Anhyone have a spare couple a hundred million ... Elon
>      may need to start a go-fund-me page!
> 
> A 5.7% reduction in funded locations for StarLink is… not dramatic. If the
> project falls on that basis, they've got bigger problems. Much of that
> discrepancy falls squarely on the shoulders of the FCC and incumbent ISPs
> filing form 477, as well as the RDOF auction being held before improving
> mapping — as Rosenworcel pointed out. The state of broadband mapping is
> still dire.
>
> If I felt like the reallocation of funds would be 100% guaranteed to
> benefit the end Internet user… I'd cheer too.
>
> If.
>
> JHA
>
> On Tue, Aug 10, 2021 at 12:16 PM Dick Roy <dickroy@alum.mit.edu> wrote:
>
>> You may find this of some relevance!
>>
>>
>>
>>
>> https://arstechnica.com/tech-policy/2021/07/ajit-pai-apparently-mismanaged-9-billion-fund-new-fcc-boss-starts-cleanup/
>>
>>
>>
>> Cheers (or whatever!),
>>
>>
>>
>> RR
>>
>>
>> _______________________________________________
>> Starlink mailing list
>> Starlink@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/starlink
>>
>
>
>

[-- Attachment #2: Type: text/plain, Size: 149 bytes --]

_______________________________________________
Starlink mailing list
Starlink@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink

Re: [Codel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Dave Taht]

I just wanted to comment on how awesome this thread was, and how few
people outside this group deeply grok what was discussed here. I would
so like
to somehow construct an educational TV series explaining "How the
Internet really works" to a wider, and new audience, consisting of
animations, anecdotes,
and interviews with the key figures of its evolution.

While I deeply understood Len Kleinrock's work in the period
2011-2015, and tried to pass on analogies and intuition without using
the math since, inspired by van jacobson's
analogies and Radia Perlman's poetry, it's hard for me now to follow
the argument. Queue theory in particular, is not well known or taught
anymore, despite its obvious applications to things like the Covid
crisis.

But that would be just one thing! The end to end argument, the side
effects of spitting postscript into a lego robot, what actually
happens during a web page load, how a cpu actually works, are all
things that are increasingly lost in multiple mental models, and in my
mind many could be taught in kindergarden, if we worked at explaining
it hard enough.

Re: [Codel] [Bloat] [Cerowrt-devel] Little's Law mea culpa, but not invalidating my main point

[David Lang]

[-- Attachment #1: Type: text/plain, Size: 421 bytes --]

On Mon, 20 Sep 2021, Valdis Klētnieks wrote:

> On Sun, 19 Sep 2021 18:21:56 -0700, Dave Taht said:
>> what actually happens during a web page load,
>
> I'm pretty sure that nobody actually understands that anymore, in any
> more than handwaving levels.

This is my favorite interview question, it's amazing and saddning at the answers 
that I get, even from supposedly senior security and networking people.

David Lang

[-- Attachment #2: Type: text/plain, Size: 140 bytes --]

_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Codel] [Bloat] [Make-wifi-fast] TCP_NOTSENT_LOWAT applied to e2e TCP msg latency

[Eric Dumazet]

On 10/25/21 8:11 PM, Stuart Cheshire via Bloat wrote:
> On 21 Oct 2021, at 17:51, Bob McMahon via Make-wifi-fast <make-wifi-fast@lists.bufferbloat.net> wrote:
> 
>> Hi All,
>>
>> Sorry for the spam. I'm trying to support a meaningful TCP message latency w/iperf 2 from the sender side w/o requiring e2e clock synchronization. I thought I'd try to use the TCP_NOTSENT_LOWAT event to help with this. It seems that this event goes off when the bytes are in flight vs have reached the destination network stack. If that's the case, then iperf 2 client (sender) may be able to produce the message latency by adding the drain time (write start to TCP_NOTSENT_LOWAT) and the sampled RTT.
>>
>> Does this seem reasonable?
> 
> I’m not 100% sure what you’re asking, but I will try to help.
> 
> When you set TCP_NOTSENT_LOWAT, the TCP implementation won’t report your endpoint as writable (e.g., via kqueue or epoll) until less than that threshold of data remains unsent. It won’t stop you writing more bytes if you want to, up to the socket send buffer size, but it won’t *ask* you for more data until the TCP_NOTSENT_LOWAT threshold is reached.


When I implemented TCP_NOTSENT_LOWAT back in 2013 [1], I made sure that sendmsg() would actually
stop feeding more bytes in TCP transmit queue if the current amount of unsent bytes
was above the threshold.

So it looks like Apple implementation is different, based on your description ?

[1] https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=c9bee3b7fdecb0c1d070c7b54113b3bdfb9a3d36

netperf does not use epoll(), but rather a loop over sendmsg().

One of the point of TCP_NOTSENT_LOWAT for Google was to be able to considerably increase
max number of bytes in transmit queues (3rd column of /proc/sys/net/ipv4/tcp_wmem)
by 10x, allowing for autotune to increase BDP for big RTT flows, this without
increasing memory needs for flows with small RTT.

 In other words, the TCP implementation attempts to keep BDP bytes in flight + TCP_NOTSENT_LOWAT bytes buffered and ready to go. The BDP of bytes in flight is necessary to fill the network pipe and get good throughput. The TCP_NOTSENT_LOWAT of bytes buffered and ready to go is provided to give the source software some advance notice that the TCP implementation will soon be looking for more bytes to send, so that the buffer doesn’t run dry, thereby lowering throughput. (The old SO_SNDBUF option conflates both “bytes in flight” and “bytes buffered and ready to go” into the same number.)
> 
> If you wait for the TCP_NOTSENT_LOWAT notification, write a chunk of n bytes of data, and then wait for the next TCP_NOTSENT_LOWAT notification, that will tell you roughly how long it took n bytes to depart the machine. You won’t know why, though. The bytes could depart the machine in response for acks indicating that the same number of bytes have been accepted at the receiver. But the bytes can also depart the machine because CWND is growing. Of course, both of those things are usually happening at the same time.
> 
> How to use TCP_NOTSENT_LOWAT is explained in this video:
> 
> <https://developer.apple.com/videos/play/wwdc2015/719/?time=2199>
> 
> Later in the same video is a two-minute demo (time offset 42:00 to time offset 44:00) showing a “before and after” demo illustrating the dramatic difference this makes for screen sharing responsiveness.
> 
> <https://developer.apple.com/videos/play/wwdc2015/719/?time=2520>
> 
> Stuart Cheshire
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
> 

Re: [Codel] [Make-wifi-fast] [Starlink] TCP_NOTSENT_LOWAT applied to e2e TCP msg latency

[Sebastian Moeller]

Hi Bob,

OWD != RTT/2 seems generically to be the rule on the internet not the exception, even with perfectly  symmetric access links. Routing between AS often is asymmetric in it self (hot potato routing, where each AS hands over packets destined to others as early as possible, means that forward and backward path are often noticeably different; or rather they are different but that is hard to notice unless one can get path measurements like traceroutes from both directions). That last point is what makes me believe that internet speedtests, should always also include traceroutes from server to client and from client to server so one at least has a rough idea where the packets are going, but I digress...

Regards
	Sebastian



> On Oct 26, 2021, at 19:23, Bob McMahon via Make-wifi-fast <make-wifi-fast@lists.bufferbloat.net> wrote:
> 
> Hi Bjørn,
> 
> I find, when possible, it's preferred to take telemetry data of actual traffic (or reads and writes) vs a proxy. We had a case where TCP BE was outperforming TCP w/VI because BE had the most engineering resources assigned to it and engineers did a better job with BE. Using a proxy protocol wouldn't have exercised the same logic paths (in this case it was in the L2 driver) as TCP did. Hence, measuring actual TCP traffic (or socket reads and socket writes) was needed to flush out the problem. Note: I also find that network engineers tend to focus on the stack but it's the e2e at the application level that impacts user experience. Send side bloat can drive the OWD while the TCP stack's RTT may look fine. For WiFi test & measurements, we've decided most testing should be using TCP_NOSENT_LOWAT because it helps mitigate send side bloat which WiFi engineering doesn't focus on per lack of ability to impact.
> 
> Also, I think OWD is under tested and two way based testing can give incomplete and inaccurate information, particularly with respect to things like an e2e transport's control loop.  A most obvious example is assuming 1/2 RTT is the same as OWD to/fro. For WiFi this assumption is most always false. It also false for many residential internet connections where OWD asymmetry is designed in.
> 
> Bob
> 
> 
> On Tue, Oct 26, 2021 at 3:04 AM Bjørn Ivar Teigen <bjorn@domos.no> wrote:
> Hi Bob,
> 
> My name is Bjørn Ivar Teigen and I'm working on modeling and measuring WiFi MAC-layer protocol performance for my PhD.
> 
> Is it necessary to measure the latency using the TCP stream itself? I had a similar problem in the past, and solved it by doing the latency measurements using TWAMP running alongside the TCP traffic. The requirement for this to work is that the TWAMP packets are placed in the same queue(s) as the TCP traffic, and that the impact of measurement traffic is small enough so as not to interfere too much with your TCP results.
> Just my two cents, hope it's helpful.
> 
> Bjørn
> 
> On Tue, 26 Oct 2021 at 06:32, Bob McMahon <bob.mcmahon@broadcom.com> wrote:
> Thanks Stuart this is helpful. I'm measuring the time just before the first write() (of potentially a burst of writes to achieve a burst size) per a socket fd's select event occurring when TCP_NOT_SENT_LOWAT being set to a small value, then sampling the RTT and CWND and providing histograms for all three, all on that event. I'm not sure the correctness of RTT and CWND at this sample point. This is a controlled test over 802.11ax and OFDMA where the TCP acks per the WiFi clients are being scheduled by the AP using 802.11ax trigger frames so the AP is affecting the end/end BDP per scheduling the transmits and the acks. The AP can grow the BDP or shrink it based on these scheduling decisions.  From there we're trying to maximize network power (throughput/delay) for elephant flows and just latency for mouse flows. (We also plan some RF frequency stuff to per OFDMA) Anyway, the AP based scheduling along with aggregation and OFDMA makes WiFi scheduling optimums non-obvious - at least to me - and I'm trying to provide insights into how an AP is affecting end/end performance.
> 
> The more direct approach for e2e TCP latency and network power has been to measure first write() to final read() and compute the e2e delay. This requires clock sync on the ends. (We're using ptp4l with GPS OCXO atomic references for that but this is typically only available in some labs.) 
> 
> Bob
>  
> 
> On Mon, Oct 25, 2021 at 8:11 PM Stuart Cheshire <cheshire@apple.com> wrote:
> On 21 Oct 2021, at 17:51, Bob McMahon via Make-wifi-fast <make-wifi-fast@lists.bufferbloat.net> wrote:
> 
> > Hi All,
> > 
> > Sorry for the spam. I'm trying to support a meaningful TCP message latency w/iperf 2 from the sender side w/o requiring e2e clock synchronization. I thought I'd try to use the TCP_NOTSENT_LOWAT event to help with this. It seems that this event goes off when the bytes are in flight vs have reached the destination network stack. If that's the case, then iperf 2 client (sender) may be able to produce the message latency by adding the drain time (write start to TCP_NOTSENT_LOWAT) and the sampled RTT.
> > 
> > Does this seem reasonable?
> 
> I’m not 100% sure what you’re asking, but I will try to help.
> 
> When you set TCP_NOTSENT_LOWAT, the TCP implementation won’t report your endpoint as writable (e.g., via kqueue or epoll) until less than that threshold of data remains unsent. It won’t stop you writing more bytes if you want to, up to the socket send buffer size, but it won’t *ask* you for more data until the TCP_NOTSENT_LOWAT threshold is reached. In other words, the TCP implementation attempts to keep BDP bytes in flight + TCP_NOTSENT_LOWAT bytes buffered and ready to go. The BDP of bytes in flight is necessary to fill the network pipe and get good throughput. The TCP_NOTSENT_LOWAT of bytes buffered and ready to go is provided to give the source software some advance notice that the TCP implementation will soon be looking for more bytes to send, so that the buffer doesn’t run dry, thereby lowering throughput. (The old SO_SNDBUF option conflates both “bytes in flight” and “bytes buffered and ready to go” into the same number.)
> 
> If you wait for the TCP_NOTSENT_LOWAT notification, write a chunk of n bytes of data, and then wait for the next TCP_NOTSENT_LOWAT notification, that will tell you roughly how long it took n bytes to depart the machine. You won’t know why, though. The bytes could depart the machine in response for acks indicating that the same number of bytes have been accepted at the receiver. But the bytes can also depart the machine because CWND is growing. Of course, both of those things are usually happening at the same time.
> 
> How to use TCP_NOTSENT_LOWAT is explained in this video:
> 
> <https://developer.apple.com/videos/play/wwdc2015/719/?time=2199>
> 
> Later in the same video is a two-minute demo (time offset 42:00 to time offset 44:00) showing a “before and after” demo illustrating the dramatic difference this makes for screen sharing responsiveness.
> 
> <https://developer.apple.com/videos/play/wwdc2015/719/?time=2520>
> 
> Stuart Cheshire
> 
> This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it._______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
> 
> 
> -- 
> Bjørn Ivar Teigen
> Head of Research
> +47 47335952 | bjorn@domos.no | www.domos.no
> WiFi Slicing by Domos
> 
> This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it._______________________________________________
> Make-wifi-fast mailing list
> Make-wifi-fast@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast

Re: [Codel] [Bloat] [Make-wifi-fast] TCP_NOTSENT_LOWAT applied to e2e TCP msg latency

[Eric Dumazet]

On 10/26/21 8:45 PM, Bob McMahon wrote:
> This is linux. The code flow is burst writes until the burst size, take a timestamp, call select(), take second timestamp and insert time delta into histogram, await clock_nanosleep() to schedule the next burst. (actually, the deltas, inserts into the histogram and user i/o are done in another thread, i.e. iperf 2's reporter thread.)
> 
> I still must be missing something.  Does anything else need to be set to reduce the skb size? Everything seems to be indicating 4K writes even when gso_max_size is 2000 (I assume these are units of bytes?) There are ten writes, ten reads and ten  RTTs for the bursts.  I don't see partial writes at the app level. 
> 
>     [root@localhost iperf2-code]# ip link set dev eth1 gso_max_size 2000

You could check with tcpdump on eth1, that outgoing packets are no longer 'TSO/GSO', but single MSS ones.

(Note: this device gso_max_size is only taken into account for flows established after the change)

> 
>     [root@localhost iperf2-code]# ip -d link sh dev eth1
>     9: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN mode DEFAULT group default qlen 1000
>         link/ether 00:90:4c:40:04:59 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 1500 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 2000 gso_max_segs 65535
>     [root@localhost iperf2-code]# uname -r
>     5.0.9-301.fc30.x86_64
> 
> 
> It looks like RTT is being driven by WiFi TXOPs as doubling the write size increases the aggregation by two but has no significant effect on the RTTs.
> 
>     4K writes: tot_mpdus 328 tot_ampdus 209 mpduperampdu 2
> 
> 
>     8k writes:  tot_mpdus 317 tot_ampdus 107 mpduperampdu 3
> 
> 
> [root@localhost iperf2-code]# src/iperf -c 192.168.1.1%eth1 --trip-times -i 1 -e --tcp-write-prefetch 4 -l 4K --burst-size=40K --histograms
> WARN: option of --burst-size without --burst-period defaults --burst-period to 1 second
> ------------------------------------------------------------
> Client connecting to 192.168.1.1, TCP port 5001 with pid 5145 via eth1 (1 flows)
> Write buffer size: 4096 Byte
> Bursting: 40.0 KByte every 1.00 seconds
> TCP window size: 85.0 KByte (default)
> Event based writes (pending queue watermark at 4 bytes)
> Enabled select histograms bin-width=0.100 ms, bins=10000
> ------------------------------------------------------------
> [  1] local 192.168.1.4%eth1 port 45680 connected with 192.168.1.1 port 5001 (MSS=1448) (prefetch=4) (trip-times) (sock=3) (ct=5.30 ms) on 2021-10-26 20:25:29 (PDT)
> [ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry     Cwnd/RTT        NetPwr
> [  1] 0.00-1.00 sec  40.1 KBytes   329 Kbits/sec  11/0          0       14K/10091 us  4
> [  1] 0.00-1.00 sec S8-PDF: bin(w=100us):cnt(10)=1:1,36:1,40:1,44:1,46:1,48:1,49:1,50:2,52:1 (5.00/95.00/99.7%=1/52/52,Outliers=0,obl/obu=0/0) (5.121 ms/1635305129.152339)
> [  1] 1.00-2.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/4990 us  8
> [  1] 1.00-2.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,39:1,45:1,49:5,50:1 (5.00/95.00/99.7%=1/50/50,Outliers=0,obl/obu=0/0) (4.991 ms/1635305130.153330)
> [  1] 2.00-3.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/4904 us  8
> [  1] 2.00-3.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,29:1,49:4,50:1,59:1,75:1 (5.00/95.00/99.7%=1/75/75,Outliers=0,obl/obu=0/0) (7.455 ms/1635305131.147353)
> [  1] 3.00-4.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/4964 us  8
> [  1] 3.00-4.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,49:4,50:2,59:1,65:1 (5.00/95.00/99.7%=1/65/65,Outliers=0,obl/obu=0/0) (6.460 ms/1635305132.146338)
> [  1] 4.00-5.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/4970 us  8
> [  1] 4.00-5.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,49:6,59:1,65:1 (5.00/95.00/99.7%=1/65/65,Outliers=0,obl/obu=0/0) (6.404 ms/1635305133.146335)
> [  1] 5.00-6.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/4986 us  8
> [  1] 5.00-6.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,48:1,49:1,50:4,59:1,64:1 (5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.395 ms/1635305134.146343)
> [  1] 6.00-7.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5059 us  8
> [  1] 6.00-7.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,39:1,49:3,50:2,60:1,85:1 (5.00/95.00/99.7%=1/85/85,Outliers=0,obl/obu=0/0) (8.417 ms/1635305135.148343)
> [  1] 7.00-8.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5407 us  8
> [  1] 7.00-8.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,40:1,49:4,50:1,59:1,75:1 (5.00/95.00/99.7%=1/75/75,Outliers=0,obl/obu=0/0) (7.428 ms/1635305136.147343)
> [  1] 8.00-9.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5188 us  8
> [  1] 8.00-9.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,40:1,49:3,50:3,64:1 (5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.388 ms/1635305137.146284)
> [  1] 9.00-10.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5306 us  8
> [  1] 9.00-10.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,39:1,49:2,50:2,51:1,60:1,65:1 (5.00/95.00/99.7%=1/65/65,Outliers=0,obl/obu=0/0) (6.422 ms/1635305138.146316)
> [  1] 0.00-10.01 sec   400 KBytes   327 Kbits/sec  102/0          0       14K/5939 us  7
> [  1] 0.00-10.01 sec S8(f)-PDF: bin(w=100us):cnt(100)=1:19,29:1,36:1,39:3,40:3,44:1,45:1,46:1,48:2,49:33,50:18,51:1,52:1,59:5,60:2,64:2,65:3,75:2,85:1 (5.00/95.00/99.7%=1/65/85,Outliers=0,obl/obu=0/0) (8.417 ms/1635305135.148343)
> 
> [root@localhost iperf2-code]# src/iperf -s -i 1 -e -B 192.168.1.1%eth1
> ------------------------------------------------------------
> Server listening on TCP port 5001 with pid 6287
> Binding to local address 192.168.1.1 and iface eth1
> Read buffer size:  128 KByte (Dist bin width=16.0 KByte)
> TCP window size:  128 KByte (default)
> ------------------------------------------------------------
> [  1] local 192.168.1.1%eth1 port 5001 connected with 192.168.1.4 port 45680 (MSS=1448) (burst-period=1.0000s) (trip-times) (sock=4) (peer 2.1.4-master) on 2021-10-26 20:25:29 (PDT)
> [ ID] Burst (start-end)  Transfer     Bandwidth       XferTime  (DC%)     Reads=Dist          NetPwr
> [  1] 0.0001-0.0500 sec  40.1 KBytes  6.59 Mbits/sec  49.848 ms (5%)    12=12:0:0:0:0:0:0:0  0
> [  1] 1.0002-1.0461 sec  40.0 KBytes  7.14 Mbits/sec  45.913 ms (4.6%)    10=10:0:0:0:0:0:0:0  0
> [  1] 2.0002-2.0491 sec  40.0 KBytes  6.70 Mbits/sec  48.876 ms (4.9%)    11=11:0:0:0:0:0:0:0  0
> [  1] 3.0002-3.0501 sec  40.0 KBytes  6.57 Mbits/sec  49.886 ms (5%)    10=10:0:0:0:0:0:0:0  0
> [  1] 4.0002-4.0501 sec  40.0 KBytes  6.57 Mbits/sec  49.887 ms (5%)    10=10:0:0:0:0:0:0:0  0
> [  1] 5.0002-5.0501 sec  40.0 KBytes  6.57 Mbits/sec  49.881 ms (5%)    10=10:0:0:0:0:0:0:0  0
> [  1] 6.0002-6.0511 sec  40.0 KBytes  6.44 Mbits/sec  50.895 ms (5.1%)    10=10:0:0:0:0:0:0:0  0
> [  1] 7.0002-7.0501 sec  40.0 KBytes  6.57 Mbits/sec  49.889 ms (5%)    10=10:0:0:0:0:0:0:0  0
> [  1] 8.0002-8.0481 sec  40.0 KBytes  6.84 Mbits/sec  47.901 ms (4.8%)    11=11:0:0:0:0:0:0:0  0
> [  1] 9.0002-9.0491 sec  40.0 KBytes  6.70 Mbits/sec  48.872 ms (4.9%)    10=10:0:0:0:0:0:0:0  0
> [  1] 0.0000-10.0031 sec   400 KBytes   328 Kbits/sec               104=104:0:0:0:0:0:0:0
> 
> Bob
> 
> On Tue, Oct 26, 2021 at 6:12 PM Eric Dumazet <eric.dumazet@gmail.com <mailto:eric.dumazet@gmail.com>> wrote:
> 
> 
> 
>     On 10/26/21 4:38 PM, Christoph Paasch wrote:
>     > Hi Bob,
>     >
>     >> On Oct 26, 2021, at 4:23 PM, Bob McMahon <bob.mcmahon@broadcom.com <mailto:bob.mcmahon@broadcom.com> <mailto:bob.mcmahon@broadcom.com <mailto:bob.mcmahon@broadcom.com>>> wrote:
>     >> I'm confused. I don't see any blocking nor partial writes per the write() at the app level with TCP_NOTSENT_LOWAT set at 4 bytes. The burst is 40K, the write size is 4K and the watermark is 4 bytes. There are ten writes per burst.
>     >
>     > You are on Linux here, right?
>     >
>     > AFAICS, Linux will still accept whatever fits in an skb. And that is likely more than 4K (with GSO on by default).
> 
>     This (max payload per skb) can be tuned at the driver level, at least for experimental purposes or dedicated devices.
> 
>     ip link set dev eth0 gso_max_size 8000
> 
>     To fetch current values :
> 
>     ip -d link sh dev eth0
> 
> 
>     >
>     > However, do you go back to select() after each write() or do you loop over the write() calls?
>     >
>     >
>     > Christoph
>     >
>     >> The S8 histograms are the times waiting on the select().  The first value is the bin number (multiplied by 100usec bin width) and second the bin count. The worst case time is at the end and is timestamped per unix epoch.
>     >>
>     >> The second run is over a controlled WiFi link where a 99.7% point of 4-8ms for a WiFi TX op arbitration win is in the ballpark. The first is 1G wired and is in the 600 usec range. (No media arbitration there.)
>     >>
>     >>  [root@localhost iperf2-code]# src/iperf -c 10.19.87.9 --trip-times -i 1 -e --tcp-write-prefetch 4 -l 4K --burst-size=40K --histograms
>     >> WARN: option of --burst-size without --burst-period defaults --burst-period to 1 second
>     >> ------------------------------------------------------------
>     >> Client connecting to 10.19.87.9, TCP port 5001 with pid 2124 (1 flows)
>     >> Write buffer size: 4096 Byte
>     >> Bursting: 40.0 KByte every 1.00 seconds
>     >> TCP window size: 85.0 KByte (default)
>     >> Event based writes (pending queue watermark at 4 bytes)
>     >> Enabled select histograms bin-width=0.100 ms, bins=10000
>     >> ------------------------------------------------------------
>     >> [  1] local 10.19.87.10%eth0 port 33166 connected with 10.19.87.9 port 5001 (MSS=1448) (prefetch=4) (trip-times) (sock=3) (ct=0.54 ms) on 2021-10-26 16:07:33 (PDT)
>     >> [ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry     Cwnd/RTT        NetPwr
>     >> [  1] 0.00-1.00 sec  40.1 KBytes   329 Kbits/sec  11/0          0       14K/5368 us  8
>     >> [  1] 0.00-1.00 sec S8-PDF: bin(w=100us):cnt(10)=1:1,2:5,3:2,4:1,11:1 (5.00/95.00/99.7%=1/11/11,Outliers=0,obl/obu=0/0) (1.089 ms/1635289653.928360)
>     >> [  1] 1.00-2.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/569 us  72
>     >> [  1] 1.00-2.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,2:1,3:4,4:1,7:1,8:1 (5.00/95.00/99.7%=1/8/8,Outliers=0,obl/obu=0/0) (0.736 ms/1635289654.928088)
>     >> [  1] 2.00-3.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/312 us  131
>     >> [  1] 2.00-3.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,2:2,3:2,5:2,6:1 (5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.548 ms/1635289655.927776)
>     >> [  1] 3.00-4.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/302 us  136
>     >> [  1] 3.00-4.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,2:2,3:5,6:1 (5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.584 ms/1635289656.927814)
>     >> [  1] 4.00-5.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/316 us  130
>     >> [  1] 4.00-5.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,3:2,4:2,5:2,6:1 (5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.572 ms/1635289657.927810)
>     >> [  1] 5.00-6.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/253 us  162
>     >> [  1] 5.00-6.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,2:2,3:4,5:1 (5.00/95.00/99.7%=1/5/5,Outliers=0,obl/obu=0/0) (0.417 ms/1635289658.927630)
>     >> [  1] 6.00-7.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/290 us  141
>     >> [  1] 6.00-7.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,3:3,4:3,6:1 (5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.573 ms/1635289659.927771)
>     >> [  1] 7.00-8.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/359 us  114
>     >> [  1] 7.00-8.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,3:4,4:3,6:1 (5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.570 ms/1635289660.927753)
>     >> [  1] 8.00-9.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/349 us  117
>     >> [  1] 8.00-9.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,3:5,4:1,7:1 (5.00/95.00/99.7%=1/7/7,Outliers=0,obl/obu=0/0) (0.608 ms/1635289661.927843)
>     >> [  1] 9.00-10.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/347 us  118
>     >> [  1] 9.00-10.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,2:1,3:5,8:1 (5.00/95.00/99.7%=1/8/8,Outliers=0,obl/obu=0/0) (0.725 ms/1635289662.927861)
>     >> [  1] 0.00-10.01 sec   400 KBytes   327 Kbits/sec  102/0          0       14K/1519 us  27
>     >> [  1] 0.00-10.01 sec S8(f)-PDF: bin(w=100us):cnt(100)=1:25,2:13,3:36,4:11,5:5,6:5,7:2,8:2,11:1 (5.00/95.00/99.7%=1/7/11,Outliers=0,obl/obu=0/0) (1.089 ms/1635289653.928360)
>     >>
>     >> [root@localhost iperf2-code]# src/iperf -c 192.168.1.1 --trip-times -i 1 -e --tcp-write-prefetch 4 -l 4K --burst-size=40K --histograms
>     >> WARN: option of --burst-size without --burst-period defaults --burst-period to 1 second
>     >> ------------------------------------------------------------
>     >> Client connecting to 192.168.1.1, TCP port 5001 with pid 2131 (1 flows)
>     >> Write buffer size: 4096 Byte
>     >> Bursting: 40.0 KByte every 1.00 seconds
>     >> TCP window size: 85.0 KByte (default)
>     >> Event based writes (pending queue watermark at 4 bytes)
>     >> Enabled select histograms bin-width=0.100 ms, bins=10000
>     >> ------------------------------------------------------------
>     >> [  1] local 192.168.1.4%eth1 port 45518 connected with 192.168.1.1 port 5001 (MSS=1448) (prefetch=4) (trip-times) (sock=3) (ct=5.48 ms) on 2021-10-26 16:07:56 (PDT)
>     >> [ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry     Cwnd/RTT        NetPwr
>     >> [  1] 0.00-1.00 sec  40.1 KBytes   329 Kbits/sec  11/0          0       14K/10339 us  4
>     >> [  1] 0.00-1.00 sec S8-PDF: bin(w=100us):cnt(10)=1:1,40:1,47:1,49:2,50:3,51:1,60:1 (5.00/95.00/99.7%=1/60/60,Outliers=0,obl/obu=0/0) (5.990 ms/1635289676.802143)
>     >> [  1] 1.00-2.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/4853 us  8
>     >> [  1] 1.00-2.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,38:1,39:1,44:1,45:1,49:1,51:1,52:1,60:1 (5.00/95.00/99.7%=1/60/60,Outliers=0,obl/obu=0/0) (5.937 ms/1635289677.802274)
>     >> [  1] 2.00-3.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/4991 us  8
>     >> [  1] 2.00-3.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,48:1,49:2,50:2,51:1,60:1,64:1 (5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.307 ms/1635289678.794326)
>     >> [  1] 3.00-4.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/4610 us  9
>     >> [  1] 3.00-4.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,49:3,50:3,56:1,64:1 (5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.362 ms/1635289679.794335)
>     >> [  1] 4.00-5.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5028 us  8
>     >> [  1] 4.00-5.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,49:6,59:1,64:1 (5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.367 ms/1635289680.794399)
>     >> [  1] 5.00-6.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5113 us  8
>     >> [  1] 5.00-6.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,49:3,50:2,58:1,60:1,65:1 (5.00/95.00/99.7%=1/65/65,Outliers=0,obl/obu=0/0) (6.442 ms/1635289681.794392)
>     >> [  1] 6.00-7.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5054 us  8
>     >> [  1] 6.00-7.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,39:1,49:3,51:1,60:2,64:1 (5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.374 ms/1635289682.794335)
>     >> [  1] 7.00-8.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5138 us  8
>     >> [  1] 7.00-8.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,39:2,40:1,49:2,50:1,60:1,64:1 (5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.396 ms/1635289683.794338)
>     >> [  1] 8.00-9.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5329 us  8
>     >> [  1] 8.00-9.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,38:1,45:2,49:1,50:3,63:1 (5.00/95.00/99.7%=1/63/63,Outliers=0,obl/obu=0/0) (6.292 ms/1635289684.794262)
>     >> [  1] 9.00-10.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5329 us  8
>     >> [  1] 9.00-10.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,39:1,49:3,50:3,84:1 (5.00/95.00/99.7%=1/84/84,Outliers=0,obl/obu=0/0) (8.306 ms/1635289685.796315)
>     >> [  1] 0.00-10.01 sec   400 KBytes   327 Kbits/sec  102/0          0       14K/6331 us  6
>     >> [  1] 0.00-10.01 sec S8(f)-PDF: bin(w=100us):cnt(100)=1:19,38:2,39:5,40:2,44:1,45:3,47:1,48:1,49:26,50:17,51:4,52:1,56:1,58:1,59:1,60:7,63:1,64:5,65:1,84:1 (5.00/95.00/99.7%=1/64/84,Outliers=0,obl/obu=0/0) (8.306 ms/1635289685.796315)
>     >>
>     >> Bob
>     >>
>     >> On Tue, Oct 26, 2021 at 11:45 AM Christoph Paasch <cpaasch@apple.com <mailto:cpaasch@apple.com> <mailto:cpaasch@apple.com <mailto:cpaasch@apple.com>>> wrote:
>     >>
>     >>     Hello,
>     >>
>     >>     > On Oct 25, 2021, at 9:24 PM, Eric Dumazet <eric.dumazet@gmail.com <mailto:eric.dumazet@gmail.com> <mailto:eric.dumazet@gmail.com <mailto:eric.dumazet@gmail.com>>> wrote:
>     >>     >
>     >>     >
>     >>     >
>     >>     > On 10/25/21 8:11 PM, Stuart Cheshire via Bloat wrote:
>     >>     >> On 21 Oct 2021, at 17:51, Bob McMahon via Make-wifi-fast <make-wifi-fast@lists.bufferbloat.net <mailto:make-wifi-fast@lists.bufferbloat.net> <mailto:make-wifi-fast@lists.bufferbloat.net <mailto:make-wifi-fast@lists.bufferbloat.net>>> wrote:
>     >>     >>
>     >>     >>> Hi All,
>     >>     >>>
>     >>     >>> Sorry for the spam. I'm trying to support a meaningful TCP message latency w/iperf 2 from the sender side w/o requiring e2e clock synchronization. I thought I'd try to use the TCP_NOTSENT_LOWAT event to help with this. It seems that this event goes off when the bytes are in flight vs have reached the destination network stack. If that's the case, then iperf 2 client (sender) may be able to produce the message latency by adding the drain time (write start to TCP_NOTSENT_LOWAT) and the sampled RTT.
>     >>     >>>
>     >>     >>> Does this seem reasonable?
>     >>     >>
>     >>     >> I’m not 100% sure what you’re asking, but I will try to help.
>     >>     >>
>     >>     >> When you set TCP_NOTSENT_LOWAT, the TCP implementation won’t report your endpoint as writable (e.g., via kqueue or epoll) until less than that threshold of data remains unsent. It won’t stop you writing more bytes if you want to, up to the socket send buffer size, but it won’t *ask* you for more data until the TCP_NOTSENT_LOWAT threshold is reached.
>     >>     >
>     >>     >
>     >>     > When I implemented TCP_NOTSENT_LOWAT back in 2013 [1], I made sure that sendmsg() would actually
>     >>     > stop feeding more bytes in TCP transmit queue if the current amount of unsent bytes
>     >>     > was above the threshold.
>     >>     >
>     >>     > So it looks like Apple implementation is different, based on your description ?
>     >>
>     >>     Yes, TCP_NOTSENT_LOWAT only impacts the wakeup on iOS/macOS/...
>     >>
>     >>     An app can still fill the send-buffer if it does a sendmsg() with a large buffer or does repeated calls to sendmsg().
>     >>
>     >>     Fur Apple, the goal of TCP_NOTSENT_LOWAT was to allow an app to quickly change the data it "scheduled" to send. And thus allow the app to write the smallest "logical unit" it has. If that unit is 512KB large, the app is allowed to send that.
>     >>     For example, in case of video-streaming one may want to skip ahead in the video. In that case the app still needs to transmit the remaining parts of the previous frame anyways, before it can send the new video frame.
>     >>     That's the reason why the Apple implementation allows one to write more than just the lowat threshold.
>     >>
>     >>
>     >>     That being said, I do think that Linux's way allows for an easier API because the app does not need to be careful at how much data it sends after an epoll/kqueue wakeup. So, the latency-benefits will be easier to get.
>     >>
>     >>
>     >>     Christoph
>     >>
>     >>
>     >>
>     >>     > [1] https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=c9bee3b7fdecb0c1d070c7b54113b3bdfb9a3d36 <https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=c9bee3b7fdecb0c1d070c7b54113b3bdfb9a3d36> <https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=c9bee3b7fdecb0c1d070c7b54113b3bdfb9a3d36 <https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=c9bee3b7fdecb0c1d070c7b54113b3bdfb9a3d36>>
>     >>     >
>     >>     > netperf does not use epoll(), but rather a loop over sendmsg().
>     >>     >
>     >>     > One of the point of TCP_NOTSENT_LOWAT for Google was to be able to considerably increase
>     >>     > max number of bytes in transmit queues (3rd column of /proc/sys/net/ipv4/tcp_wmem)
>     >>     > by 10x, allowing for autotune to increase BDP for big RTT flows, this without
>     >>     > increasing memory needs for flows with small RTT.
>     >>     >
>     >>     > In other words, the TCP implementation attempts to keep BDP bytes in flight + TCP_NOTSENT_LOWAT bytes buffered and ready to go. The BDP of bytes in flight is necessary to fill the network pipe and get good throughput. The TCP_NOTSENT_LOWAT of bytes buffered and ready to go is provided to give the source software some advance notice that the TCP implementation will soon be looking for more bytes to send, so that the buffer doesn’t run dry, thereby lowering throughput. (The old SO_SNDBUF option conflates both “bytes in flight” and “bytes buffered and ready to go” into the same number.)
>     >>     >>
>     >>     >> If you wait for the TCP_NOTSENT_LOWAT notification, write a chunk of n bytes of data, and then wait for the next TCP_NOTSENT_LOWAT notification, that will tell you roughly how long it took n bytes to depart the machine. You won’t know why, though. The bytes could depart the machine in response for acks indicating that the same number of bytes have been accepted at the receiver. But the bytes can also depart the machine because CWND is growing. Of course, both of those things are usually happening at the same time.
>     >>     >>
>     >>     >> How to use TCP_NOTSENT_LOWAT is explained in this video:
>     >>     >>
>     >>     >> <https://developer.apple.com/videos/play/wwdc2015/719/?time=2199 <https://developer.apple.com/videos/play/wwdc2015/719/?time=2199> <https://developer.apple.com/videos/play/wwdc2015/719/?time=2199 <https://developer.apple.com/videos/play/wwdc2015/719/?time=2199>>>
>     >>     >>
>     >>     >> Later in the same video is a two-minute demo (time offset 42:00 to time offset 44:00) showing a “before and after” demo illustrating the dramatic difference this makes for screen sharing responsiveness.
>     >>     >>
>     >>     >> <https://developer.apple.com/videos/play/wwdc2015/719/?time=2520 <https://developer.apple.com/videos/play/wwdc2015/719/?time=2520> <https://developer.apple.com/videos/play/wwdc2015/719/?time=2520 <https://developer.apple.com/videos/play/wwdc2015/719/?time=2520>>>
>     >>     >>
>     >>     >> Stuart Cheshire
>     >>     >> _______________________________________________
>     >>     >> Bloat mailing list
>     >>     >> Bloat@lists.bufferbloat.net <mailto:Bloat@lists.bufferbloat.net> <mailto:Bloat@lists.bufferbloat.net <mailto:Bloat@lists.bufferbloat.net>>
>     >>     >> https://lists.bufferbloat.net/listinfo/bloat <https://lists.bufferbloat.net/listinfo/bloat> <https://lists.bufferbloat.net/listinfo/bloat <https://lists.bufferbloat.net/listinfo/bloat>>
>     >>     >>
>     >>     > _______________________________________________
>     >>     > Bloat mailing list
>     >>     > Bloat@lists.bufferbloat.net <mailto:Bloat@lists.bufferbloat.net> <mailto:Bloat@lists.bufferbloat.net <mailto:Bloat@lists.bufferbloat.net>>
>     >>     > https://lists.bufferbloat.net/listinfo/bloat <https://lists.bufferbloat.net/listinfo/bloat> <https://lists.bufferbloat.net/listinfo/bloat <https://lists.bufferbloat.net/listinfo/bloat>>
>     >>
>     >>
>     >> This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.
> 
> 
> This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.

Re: [Codel] [Bloat] [Make-wifi-fast] TCP_NOTSENT_LOWAT applied to e2e TCP msg latency

[Eric Dumazet]

On 10/26/21 4:38 PM, Christoph Paasch wrote:
> Hi Bob,
> 
>> On Oct 26, 2021, at 4:23 PM, Bob McMahon <bob.mcmahon@broadcom.com <mailto:bob.mcmahon@broadcom.com>> wrote:
>> I'm confused. I don't see any blocking nor partial writes per the write() at the app level with TCP_NOTSENT_LOWAT set at 4 bytes. The burst is 40K, the write size is 4K and the watermark is 4 bytes. There are ten writes per burst.
> 
> You are on Linux here, right?
> 
> AFAICS, Linux will still accept whatever fits in an skb. And that is likely more than 4K (with GSO on by default).

This (max payload per skb) can be tuned at the driver level, at least for experimental purposes or dedicated devices.

ip link set dev eth0 gso_max_size 8000

To fetch current values :

ip -d link sh dev eth0


> 
> However, do you go back to select() after each write() or do you loop over the write() calls?
> 
> 
> Christoph
> 
>> The S8 histograms are the times waiting on the select().  The first value is the bin number (multiplied by 100usec bin width) and second the bin count. The worst case time is at the end and is timestamped per unix epoch.
>>
>> The second run is over a controlled WiFi link where a 99.7% point of 4-8ms for a WiFi TX op arbitration win is in the ballpark. The first is 1G wired and is in the 600 usec range. (No media arbitration there.)
>>
>>  [root@localhost iperf2-code]# src/iperf -c 10.19.87.9 --trip-times -i 1 -e --tcp-write-prefetch 4 -l 4K --burst-size=40K --histograms
>> WARN: option of --burst-size without --burst-period defaults --burst-period to 1 second
>> ------------------------------------------------------------
>> Client connecting to 10.19.87.9, TCP port 5001 with pid 2124 (1 flows)
>> Write buffer size: 4096 Byte
>> Bursting: 40.0 KByte every 1.00 seconds
>> TCP window size: 85.0 KByte (default)
>> Event based writes (pending queue watermark at 4 bytes)
>> Enabled select histograms bin-width=0.100 ms, bins=10000
>> ------------------------------------------------------------
>> [  1] local 10.19.87.10%eth0 port 33166 connected with 10.19.87.9 port 5001 (MSS=1448) (prefetch=4) (trip-times) (sock=3) (ct=0.54 ms) on 2021-10-26 16:07:33 (PDT)
>> [ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry     Cwnd/RTT        NetPwr
>> [  1] 0.00-1.00 sec  40.1 KBytes   329 Kbits/sec  11/0          0       14K/5368 us  8
>> [  1] 0.00-1.00 sec S8-PDF: bin(w=100us):cnt(10)=1:1,2:5,3:2,4:1,11:1 (5.00/95.00/99.7%=1/11/11,Outliers=0,obl/obu=0/0) (1.089 ms/1635289653.928360)
>> [  1] 1.00-2.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/569 us  72
>> [  1] 1.00-2.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,2:1,3:4,4:1,7:1,8:1 (5.00/95.00/99.7%=1/8/8,Outliers=0,obl/obu=0/0) (0.736 ms/1635289654.928088)
>> [  1] 2.00-3.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/312 us  131
>> [  1] 2.00-3.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,2:2,3:2,5:2,6:1 (5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.548 ms/1635289655.927776)
>> [  1] 3.00-4.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/302 us  136
>> [  1] 3.00-4.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,2:2,3:5,6:1 (5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.584 ms/1635289656.927814)
>> [  1] 4.00-5.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/316 us  130
>> [  1] 4.00-5.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,3:2,4:2,5:2,6:1 (5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.572 ms/1635289657.927810)
>> [  1] 5.00-6.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/253 us  162
>> [  1] 5.00-6.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,2:2,3:4,5:1 (5.00/95.00/99.7%=1/5/5,Outliers=0,obl/obu=0/0) (0.417 ms/1635289658.927630)
>> [  1] 6.00-7.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/290 us  141
>> [  1] 6.00-7.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,3:3,4:3,6:1 (5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.573 ms/1635289659.927771)
>> [  1] 7.00-8.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/359 us  114
>> [  1] 7.00-8.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,3:4,4:3,6:1 (5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.570 ms/1635289660.927753)
>> [  1] 8.00-9.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/349 us  117
>> [  1] 8.00-9.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,3:5,4:1,7:1 (5.00/95.00/99.7%=1/7/7,Outliers=0,obl/obu=0/0) (0.608 ms/1635289661.927843)
>> [  1] 9.00-10.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/347 us  118
>> [  1] 9.00-10.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,2:1,3:5,8:1 (5.00/95.00/99.7%=1/8/8,Outliers=0,obl/obu=0/0) (0.725 ms/1635289662.927861)
>> [  1] 0.00-10.01 sec   400 KBytes   327 Kbits/sec  102/0          0       14K/1519 us  27
>> [  1] 0.00-10.01 sec S8(f)-PDF: bin(w=100us):cnt(100)=1:25,2:13,3:36,4:11,5:5,6:5,7:2,8:2,11:1 (5.00/95.00/99.7%=1/7/11,Outliers=0,obl/obu=0/0) (1.089 ms/1635289653.928360)
>>
>> [root@localhost iperf2-code]# src/iperf -c 192.168.1.1 --trip-times -i 1 -e --tcp-write-prefetch 4 -l 4K --burst-size=40K --histograms
>> WARN: option of --burst-size without --burst-period defaults --burst-period to 1 second
>> ------------------------------------------------------------
>> Client connecting to 192.168.1.1, TCP port 5001 with pid 2131 (1 flows)
>> Write buffer size: 4096 Byte
>> Bursting: 40.0 KByte every 1.00 seconds
>> TCP window size: 85.0 KByte (default)
>> Event based writes (pending queue watermark at 4 bytes)
>> Enabled select histograms bin-width=0.100 ms, bins=10000
>> ------------------------------------------------------------
>> [  1] local 192.168.1.4%eth1 port 45518 connected with 192.168.1.1 port 5001 (MSS=1448) (prefetch=4) (trip-times) (sock=3) (ct=5.48 ms) on 2021-10-26 16:07:56 (PDT)
>> [ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry     Cwnd/RTT        NetPwr
>> [  1] 0.00-1.00 sec  40.1 KBytes   329 Kbits/sec  11/0          0       14K/10339 us  4
>> [  1] 0.00-1.00 sec S8-PDF: bin(w=100us):cnt(10)=1:1,40:1,47:1,49:2,50:3,51:1,60:1 (5.00/95.00/99.7%=1/60/60,Outliers=0,obl/obu=0/0) (5.990 ms/1635289676.802143)
>> [  1] 1.00-2.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/4853 us  8
>> [  1] 1.00-2.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,38:1,39:1,44:1,45:1,49:1,51:1,52:1,60:1 (5.00/95.00/99.7%=1/60/60,Outliers=0,obl/obu=0/0) (5.937 ms/1635289677.802274)
>> [  1] 2.00-3.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/4991 us  8
>> [  1] 2.00-3.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,48:1,49:2,50:2,51:1,60:1,64:1 (5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.307 ms/1635289678.794326)
>> [  1] 3.00-4.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/4610 us  9
>> [  1] 3.00-4.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,49:3,50:3,56:1,64:1 (5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.362 ms/1635289679.794335)
>> [  1] 4.00-5.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5028 us  8
>> [  1] 4.00-5.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,49:6,59:1,64:1 (5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.367 ms/1635289680.794399)
>> [  1] 5.00-6.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5113 us  8
>> [  1] 5.00-6.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,49:3,50:2,58:1,60:1,65:1 (5.00/95.00/99.7%=1/65/65,Outliers=0,obl/obu=0/0) (6.442 ms/1635289681.794392)
>> [  1] 6.00-7.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5054 us  8
>> [  1] 6.00-7.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,39:1,49:3,51:1,60:2,64:1 (5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.374 ms/1635289682.794335)
>> [  1] 7.00-8.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5138 us  8
>> [  1] 7.00-8.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,39:2,40:1,49:2,50:1,60:1,64:1 (5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.396 ms/1635289683.794338)
>> [  1] 8.00-9.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5329 us  8
>> [  1] 8.00-9.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,38:1,45:2,49:1,50:3,63:1 (5.00/95.00/99.7%=1/63/63,Outliers=0,obl/obu=0/0) (6.292 ms/1635289684.794262)
>> [  1] 9.00-10.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5329 us  8
>> [  1] 9.00-10.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,39:1,49:3,50:3,84:1 (5.00/95.00/99.7%=1/84/84,Outliers=0,obl/obu=0/0) (8.306 ms/1635289685.796315)
>> [  1] 0.00-10.01 sec   400 KBytes   327 Kbits/sec  102/0          0       14K/6331 us  6
>> [  1] 0.00-10.01 sec S8(f)-PDF: bin(w=100us):cnt(100)=1:19,38:2,39:5,40:2,44:1,45:3,47:1,48:1,49:26,50:17,51:4,52:1,56:1,58:1,59:1,60:7,63:1,64:5,65:1,84:1 (5.00/95.00/99.7%=1/64/84,Outliers=0,obl/obu=0/0) (8.306 ms/1635289685.796315)
>>
>> Bob
>>
>> On Tue, Oct 26, 2021 at 11:45 AM Christoph Paasch <cpaasch@apple.com <mailto:cpaasch@apple.com>> wrote:
>>
>>     Hello,
>>
>>     > On Oct 25, 2021, at 9:24 PM, Eric Dumazet <eric.dumazet@gmail.com <mailto:eric.dumazet@gmail.com>> wrote:
>>     >
>>     >
>>     >
>>     > On 10/25/21 8:11 PM, Stuart Cheshire via Bloat wrote:
>>     >> On 21 Oct 2021, at 17:51, Bob McMahon via Make-wifi-fast <make-wifi-fast@lists.bufferbloat.net <mailto:make-wifi-fast@lists.bufferbloat.net>> wrote:
>>     >>
>>     >>> Hi All,
>>     >>>
>>     >>> Sorry for the spam. I'm trying to support a meaningful TCP message latency w/iperf 2 from the sender side w/o requiring e2e clock synchronization. I thought I'd try to use the TCP_NOTSENT_LOWAT event to help with this. It seems that this event goes off when the bytes are in flight vs have reached the destination network stack. If that's the case, then iperf 2 client (sender) may be able to produce the message latency by adding the drain time (write start to TCP_NOTSENT_LOWAT) and the sampled RTT.
>>     >>>
>>     >>> Does this seem reasonable?
>>     >>
>>     >> I’m not 100% sure what you’re asking, but I will try to help.
>>     >>
>>     >> When you set TCP_NOTSENT_LOWAT, the TCP implementation won’t report your endpoint as writable (e.g., via kqueue or epoll) until less than that threshold of data remains unsent. It won’t stop you writing more bytes if you want to, up to the socket send buffer size, but it won’t *ask* you for more data until the TCP_NOTSENT_LOWAT threshold is reached.
>>     >
>>     >
>>     > When I implemented TCP_NOTSENT_LOWAT back in 2013 [1], I made sure that sendmsg() would actually
>>     > stop feeding more bytes in TCP transmit queue if the current amount of unsent bytes
>>     > was above the threshold.
>>     >
>>     > So it looks like Apple implementation is different, based on your description ?
>>
>>     Yes, TCP_NOTSENT_LOWAT only impacts the wakeup on iOS/macOS/...
>>
>>     An app can still fill the send-buffer if it does a sendmsg() with a large buffer or does repeated calls to sendmsg().
>>
>>     Fur Apple, the goal of TCP_NOTSENT_LOWAT was to allow an app to quickly change the data it "scheduled" to send. And thus allow the app to write the smallest "logical unit" it has. If that unit is 512KB large, the app is allowed to send that.
>>     For example, in case of video-streaming one may want to skip ahead in the video. In that case the app still needs to transmit the remaining parts of the previous frame anyways, before it can send the new video frame.
>>     That's the reason why the Apple implementation allows one to write more than just the lowat threshold.
>>
>>
>>     That being said, I do think that Linux's way allows for an easier API because the app does not need to be careful at how much data it sends after an epoll/kqueue wakeup. So, the latency-benefits will be easier to get.
>>
>>
>>     Christoph
>>
>>
>>
>>     > [1] https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=c9bee3b7fdecb0c1d070c7b54113b3bdfb9a3d36 <https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=c9bee3b7fdecb0c1d070c7b54113b3bdfb9a3d36>
>>     >
>>     > netperf does not use epoll(), but rather a loop over sendmsg().
>>     >
>>     > One of the point of TCP_NOTSENT_LOWAT for Google was to be able to considerably increase
>>     > max number of bytes in transmit queues (3rd column of /proc/sys/net/ipv4/tcp_wmem)
>>     > by 10x, allowing for autotune to increase BDP for big RTT flows, this without
>>     > increasing memory needs for flows with small RTT.
>>     >
>>     > In other words, the TCP implementation attempts to keep BDP bytes in flight + TCP_NOTSENT_LOWAT bytes buffered and ready to go. The BDP of bytes in flight is necessary to fill the network pipe and get good throughput. The TCP_NOTSENT_LOWAT of bytes buffered and ready to go is provided to give the source software some advance notice that the TCP implementation will soon be looking for more bytes to send, so that the buffer doesn’t run dry, thereby lowering throughput. (The old SO_SNDBUF option conflates both “bytes in flight” and “bytes buffered and ready to go” into the same number.)
>>     >>
>>     >> If you wait for the TCP_NOTSENT_LOWAT notification, write a chunk of n bytes of data, and then wait for the next TCP_NOTSENT_LOWAT notification, that will tell you roughly how long it took n bytes to depart the machine. You won’t know why, though. The bytes could depart the machine in response for acks indicating that the same number of bytes have been accepted at the receiver. But the bytes can also depart the machine because CWND is growing. Of course, both of those things are usually happening at the same time.
>>     >>
>>     >> How to use TCP_NOTSENT_LOWAT is explained in this video:
>>     >>
>>     >> <https://developer.apple.com/videos/play/wwdc2015/719/?time=2199 <https://developer.apple.com/videos/play/wwdc2015/719/?time=2199>>
>>     >>
>>     >> Later in the same video is a two-minute demo (time offset 42:00 to time offset 44:00) showing a “before and after” demo illustrating the dramatic difference this makes for screen sharing responsiveness.
>>     >>
>>     >> <https://developer.apple.com/videos/play/wwdc2015/719/?time=2520 <https://developer.apple.com/videos/play/wwdc2015/719/?time=2520>>
>>     >>
>>     >> Stuart Cheshire
>>     >> _______________________________________________
>>     >> Bloat mailing list
>>     >> Bloat@lists.bufferbloat.net <mailto:Bloat@lists.bufferbloat.net>
>>     >> https://lists.bufferbloat.net/listinfo/bloat <https://lists.bufferbloat.net/listinfo/bloat>
>>     >>
>>     > _______________________________________________
>>     > Bloat mailing list
>>     > Bloat@lists.bufferbloat.net <mailto:Bloat@lists.bufferbloat.net>
>>     > https://lists.bufferbloat.net/listinfo/bloat <https://lists.bufferbloat.net/listinfo/bloat>
>>
>>
>> This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.