[NNagain] Metrics for Network Managers (was FCC NOI due dec 1 on broadband speed standards)

[rjmcmahon]

Hi Jack,

We do have challenges with network metrics and network telemetry. While 
better than the days of SNMP, there is a long way to go. Any agency, 
e.g. the FCC, trying to make quality statements about broadband networks 
really need to be steeped in these topics. Particularly before govt 
allocates financial resources to help out.

The goal with iperf 2 is to behave as a application using TCP end to end 
as that's typically the performance folks primarily care about. 
Microsecond timestamps are supported at the app level. It does require 
clock sync for one way delays and the GPS atomic clock is pretty good. 
But Richard Roy will remind us that there really is no such thing as 
clock sync per general relativity. The kernel does provide TCP stack 
stats in realtime too.

I don't think we've instrumented most networks well enough even today. 
The focus has mostly been on speeds & feeds and then now it's becoming 
low latency (though latency became important for high frequency traders 
about a decode ago for their latency arbitrage in trading - so those 
networks are close to the limits of nature and spacetime) and less on 
test & measurement. The few of us in T&M do our best.

I added little's law to iperf 2's output probably 5 years ago or more. 
It's still not utilized well across the industry. One can tell because 
the displacement units of bloat are bytes or packets but instead we use 
time or round trips. The bandwidth delay of a link isn't hard to figure 
out so determining the amount of excess queue memory in proper units has 
been doable for awhile now.

We're way behind on multivariate analysis too. Even basic statistical 
process controls (SPC) aren't heavily used. Machines could be carrying 
more of the load than they do.

There is a lot to do and only so much time. So-called AI and ChatGPT has 
most everyone's attention it seems.

Bob
> During the 90s, I was part of a small team responsible for operating
> a corporate small-i internet.  We used cisco equipment, in offices in
> 100 or so countries, all interconnected with circuits from various
> "phone companies".  No fiber or wifi around yet.  Our internet was not
> connected to The Internet, but used all the same equipment and
> software.
> 
> To figure out how to respond to user complaints (e.g., "The network is
> really slow today!"), we discovered that we needed good
> instrumentation not only in the various routers but also in the
> computers in the user environments.   TCP, implemented in the user
> computers, did a good job at hiding problems.  For example, TCPs could
> get into a semi-stable state where every datagram was getting
> transmitted twice with the second copy being discarded at the
> destination.  That caused "network throughput" (datagrams/second) to
> go way up, but user performance to go way down.
> 
> To get the data, we used SNMP and simple scripts to probe the various
> devices from our NOC, stuff all the data into a database, and use the
> available data analysis tools to figure out what was happening.  At
> the time, SNMP "MIB"s were defined for both routers and computers.
> So we could collect data from the TCPs involved in a problem, as well
> as data from router operations.  When routers reported N
> datagrams/time, the associated TCPs might report N/2 retransmissions
> from the sender, and N/2 discards from the receiver.
> 
> With today's ability to synchronize clocks across the network, it
> should now also be possible to report data about latency.
> Interactive operations - (A/V conferencing, gaming, telepresence,
> etc.) probably have lots of metrics that could be useful much as we
> found TCP's to be.
> 
> I'm way out of touch with network operations now.  Do current network
> managers look at metrics from the enduser computers' perspective?
> Can latency be measured between two user devices involved in a problem
> report?
> 
> Jack
> 
> On 11/14/23 13:45, rjmcmahon via Nnagain wrote:
> 
>> I think the missing metrics & test vectors are around latency more
>> than bandwidth.
>> 
>> I've attached a WiFi low latency table. Feel free to comment.
>> 
>> Good metrics could allow for a comprehensive analysis, at least from
>> a WiFi perspective.
>> 
>> Latency under load is a good start, but likely not enough.
>> 
>> Bob
>> 
>> PS. Agreed, the digital transition with storage has many engineers
>> who have contributed over decades. I'm very grateful to all of them.
>> 
>> 
>> I am glad you are reaching out, but it may be difficult for us to do
>> a
>> joint filing.
>> 
>> In particular I question the seeming assumption that more wifi
>> devices
>> will drive demand for more bandwidth,
>> and extrapolating from 18 devices forward may also well be a trend
>> that will reverse completely in favor of more bluetooth and thread
>> implementations from phone to device.
>> 
>> Of those 20 wifi devices today are probably
>> 
>> 1 or more laptops
>> 1 or more tablets
>> 1 or more phones
>> 1 or more tvs
>> 
>> and of those usually only one will be active per person, while they
>> are in the home, and even then....as one semi-hard number, even at
>> peak hours (with the libreqos data I have), only 1/6th of households
>> 
>> are watching video, and very, very few, more than one stream at the
>> same time.
>> 
>> The steady upload bandwidth pumpers are primarily video surveillance
>> 
>> devices (which as a personal preference I would prefer remain in the
>> 
>> home unless otherwise activated). I do not presently know much about
>> 
>> the frame rates or real bandwidth requirements of popular devices
>> like
>> ring, etc.  Similarly I am biased towards "Babycams" sending video
>> from up to downstairs only and not into the cloud. I know I am
>> bucking
>> the trend on this, but it will make me skeptical of much "data" that
>> 
>> exists today on it.
>> 
>> Then you have loads of extremely low bandwidth devices - alexa and
>> other automation is measured in bits/ms, light switches, a couple
>> bits
>> a day, audio streaming 128kbit/s (when you use it). Automatic
>> updates
>> to phones and tablets, etc, take place entirely asynchronously
>> nowadays and do not need much bandwidth. A small business just needs
>> 
>> to
>> *reliably* clear credit card transactions every few minutes.
>> 
>> Perhaps the biggest steady-state bandwidth suck is home gaming
>> updates, but while a big market, if you haven't noticed birthrates
>> are
>> down, and immigration being canceled.
>> 
>> Thus I feel that the opposite number of 70-80% two people or less
>> per
>> household  that you are not optimizing for, dominates the curves.
>> 
>> Looking at the actual useage disparity (delta) between fiber'd
>> cities
>> and rural, uptake of passive video streaming services vs spotify,
>> would give me a more pessimistic projection than most. Regrettably I
>> 
>> lack the time and as few fund accurate scenarios, I would merely be
>> willing to write down my estimate and find some sort of online
>> "futures" market to place puts on.
>> 
>> Lastly, a goodly percentage of the people I know just need food,
>> shelter, a job and a phone, and with broadband costs skyrocketing,
>> aside from the gaming market and business, that is all they can
>> afford, even with ACP.  And all that they need. Nobody has a
>> landline
>> anymore, and if it weren't for "Tv", few would want broadband at
>> even
>> 25/10.
>> 
>> On Tue, Nov 14, 2023 at 2:40 PM rjmcmahon via Nnagain
>> <nnagain at lists.bufferbloat.net> wrote:
>> 
>> Thanks for sharing this. I agree this works for researchers.
>> 
>> I think we're at a different state and economic returns matter too.
>> 
>> I sent the following to our engineers in hopes we can all better
>> understand what we're all trying to accomplish.
>> 
>> Hi All,
>> 
>> The attached Notice of Inquiry by the FCC shows how much our work
>> matters to most everyone in our country (and, by inference,
>> worldwide.)
>> Broadband networks are no longer entertainment or social networks
>> but
>> they are critical to all regardless of gender, age, race, ethnic
>> group,
>> etc. People's health, learning, and ability to earn for their
>> families
>> all depend on us providing world class engineering to our customers
>> who
>> in turn provide these networks for each and all of us, our friends &
>> 
>> families, our neighbors, and most everyone else.
>> 
>> Early in my career, I worked at Cisco and had the privilege to work
>> on
>> some of the first BGP routers that enabled the commercial build out
>> of
>> the internet, and I'm very thankful we did that way ahead of the
>> 2019
>> pandemic. There was no "pandemic use case" that drove us - we just
>> wanted to build the best products that engineers could build. A
>> worldwide pandemic w/o the internet could have been disastrous - so
>> that
>> work by many in the mid 1990s seems to have paid off well.
>> 
>> I hope you each realize, today, what you've accomplished since then
>> and
>> continue to be a part of. It's truly significant. It's been a high
>> honor
>> to work with so many of you over the last 14+ years.
>> 
>> This is beautiful, btw. I feel much the same way about linux being
>> now
>> so used heavily in the space program,
>> and all our code, and hardware, that will propagate across the solar
>> 
>> system, and of the millions of people, that contributed to it.
>> 
>> To the FCC report:
>> 
>> We begin this annual inquiry in the wake of the COVID-19 pandemic
>> during
>> which time Americans increasingly turned to their broadband
>> connections
>> to conduct their lives online by using telemedicine to access
>> healthcare, working from home, attending classes remotely,
>> connecting by
>> video with out-of-town family and friends, and streaming
>> entertainment.
>> Our experiences with the pandemic made it clear that broadband is no
>> 
>> longer a luxury but a necessity that will only become more important
>> 
>> with time. Never before has the critical importance of ensuring that
>> all
>> Americans have access to high-speed, affordable broadband been more
>> evident.
>> 
>> Also note, we have more work to do. We need to increase resiliency
>> as an
>> example. Also, the thing I'm most passionate about is low latency.
>> The
>> FCC is now recognizing the importance of that. People are slowly
>> learning why latency is becoming equally important to capacity when
>> it
>> comes to quality of service.
>> 
>> Bob
>> 
>> PS. The rest is TLDR but I thought I post some snippets for those
>> interested
>> 
>> We believe that in examining household use cases, a simple summation
>> of
>> required speeds for individual activities may provide a misleading
>> picture of actual broadband needs for at least three reasons. First,
>> we
>> believe it is appropriate to take into account at least occasional
>> downloads of very large files which can be bandwidth-intensive.
>> Second,
>> it is important to account for larger households; in 2022,
>> approximately
>> 21% of all U.S. households had four or more people, and the number
>> of
>> families seeking out multigenerational homes to live with additional
>> 
>> relatives rose.57 Households of all sizes must have sufficient
>> bandwidth
>> to satisfy their needs. In addition, the number of connected devices
>> per
>> household continues to grow, from 18.6 in the average household in
>> 2021
>> to 20.2 in the first half of 2022.58 Taking these factors into
>> account
>> suggests that fixed broadband download/upload needs could easily
>> exceed
>> 100/20 Mbps.
>> 
>> ...
>> 
>> Service Quality. We recognize that other factors, besides the speed
>> of a
>> broadband connection, can affect consumers’ ability to use the
>> services
>> effectively. Chief among these factors is latency, which is the
>> measure
>> of the time it takes a packet of data to travel from one point in
>> the
>> network to another, and which is typically measured by round-trip
>> time
>> in milliseconds (ms). As a measurement of advanced
>> telecommunications
>> capability, latency can be critical because it affects a
>> consumer’s
>> ability to use real-time applications, including voice over Internet
>> 
>> Protocol, video calling, distance learningapplications, and online
>> gaming. Actual (as opposed to advertised) speed received,
>> consistency of
>> speed, and data allowances are also important. Such factors are not
>> simply a matter of service interruptions and consumer
>> satisfaction—they
>> have a real and significant effect on Americans’ ability to use
>> critical
>> web-based applications, including those that facilitate telehealth,
>> telework, and virtual learning.
>> 
>>> In the beginning days of the Arpanet, circa early 1970s, ARPA made
>> a
>>> policy decision about use of the Arpanet.  First, Arpa Program
>>> Managers, located on the East Coast of the US, were assigned
>> computer
>>> accounts on USC-ISIA, located on the West Coast in LA.  Thus to do
>> 
>>> their work, exchanging email, editting documents, and such, they
>> had
>>> to *use* the Arpanet to connect their terminals in Washington to
>> the
>>> PDP-10 in California - 3000 miles away.
>>> 
>>> Second, ARPA began requiring all of their contractors (researchers
>> at
>>> Universities etc.) to interact with Arpa using email and FTP.   If
>> 
>>> your site was "on the Arpanet", you had to use the Arpanet.  If
>> you
>>> wanted your proposal for next year's research to be funded, you
>> had to
>>> submit your proposal using the net.
>>> 
>>> This policy caused a profound attention, by everyone involved, to
>>> making the Arpanet work and be useful as a collaboration tool.
>>> 
>>> JCR Licklider (aka Lick) was my advisor at MIT, and then my boss
>> when
>>> I joined the Research Staff.   Lick had been at ARPA for a while,
>>> promoting his vision of a "Galactic Network" that resulted in the
>>> Arpanet as a first step.  At MIT, Lick still had need for lots of
>>> interactions with others.   My assignment was to build and operate
>> the
>>> email system for Lick's group at MIT on our own PDP-10.  Lick had
>> a
>>> terminal in his office and was online a lot.   If email didn't
>> work, I
>>> heard about it.   If the Arpanet didn't work, BBN heard about it.
>>> 
>>> This pressure was part of Arpa policy.   Sometimes it's referred
>> to as
>>> "eating your own dog food" -- i.e., making sure your "dog" will
>> get
>>> the same kind of nutrition you enjoy.   IMHO, that pressure policy
>> was
>>> important, perhaps crucial, to the success of the Arpanet.
>>> 
>>> In the 70s, meetings still occurred, but a lot of progress was
>> made
>>> through the use of the Arpanet.   You can only do so much with
>> email
>>> and file interactions.  Today, the possibilities for far richer
>>> interactions are much more prevalent.   But IMHO they are held
>> back,
>>> possibly because no one is feeling the pressure to "make it work".
>> 
>>> Gigabit throughputs are common, but why does my video and audio
>> still
>>> break up...?
>>> 
>>> It's important to have face-to-face meetings, but perhaps if the
>> IETF
>>> scheduled a future meeting to be online only, whatever needs to
>> happen
>>> to make it work would happen?  Perhaps...
>>> 
>>> Even a "game" might drive progress.  At Interop '92, we
>> resurrected
>>> the old "MazeWars" game using computers scattered across the show
>>> exhibit halls.  The engineers in the control room above the floor
>> felt
>>> the pressure to make sure the Game continued to run.  At the time,
>> the
>>> Internet itself was too slow for enjoyable gameplay at any
>> distance.
>>> Will the Internet 30 years later work?
>>> 
>>> Or perhaps the IETF, or ISOC, or someone could take on a highly
>>> visible demo involving non-techie end users.   An online meeting
>> of
>>> the UN General Assembly?   Or some government bodies - US
>> Congress,
>>> British Parliament, etc.
>>> 
>>> Such an event would surface the issues, both technical and policy,
>> to
>>> the engineers, corporations, policy-makers, and others who might
>> have
>>> the ability and interest to "make it work".
>>> 
>>> Jack
>>> 
>>> On 11/14/23 10:10, Sebastian Moeller wrote:
>>> 
>>>> Hi Jack,
>>>> 
>>>>> On Nov 14, 2023, at 13:02, Jack Haverty via Nnagain
>>>>> <nnagain at lists.bufferbloat.net> wrote:
>>>>> 
>>>>> If video conferencing worked well enough, they would not have to
>> 
>>>>> all get together in one place and would instead hold IETF
>> meetings
>>>>> online ...?
>>>> 
>>>> [SM] Turns out that humans are social creatures, and some things
>>>> work better face-to-face and in the hallway (and if that is only
>>>> building trust and sympathy) than over any remote technology.
>>>> 
>>>>> Did anyone measure latency?   Does anyone measure throughput of
>>>>> "useful" traffic - e.g., excluding video/audio data that didn't
>>>>> arrive in time to be actually used on the screen or speaker?
>>>> 
>>>> [SM] Utility is in the eye of the beholder, no?
>>>> 
>>>> Jack Haverty
>>>> 
>>>> On 11/14/23 09:25, Vint Cerf via Nnagain wrote:
>>>> 
>>>> if they had not been all together they would have been consuming
>>>> tons of video capacity doing video conference calls....
>>>> 
>>>> :-))
>>>> v
>>>> 
>>>> On Tue, Nov 14, 2023 at 10:46 AM Livingood, Jason via Nnagain
>>>> <nnagain at lists.bufferbloat.net> wrote:
>>>> On the subject of how much bandwidth does one household need,
>> here's
>>>> a fun stat for you.
>>>> 
>>>> At the IETF’s 118th meeting last week (Nov 4 – 10, 2023),
>> there
>>>> were over 1,000 engineers in attendance. At peak there were 870
>>>> devices connected to the WiFi network. Peak bandwidth usage:
>>>> 
>>>> • Downstream peak ~750 Mbps
>>>> • Upstream ~250 Mbps
>>>> 
>>>> From my pre-meeting Twitter poll
>>>> (https://twitter.com/jlivingood/status/1720060429311901873):
>>>> 
>>>> <image001.png>
>>>> 
>>>> _______________________________________________
>>>> Nnagain mailing list
>>>> Nnagain at lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/nnagain
>>>> 
>>>> --
>>>> Please send any postal/overnight deliveries to:
>>>> Vint Cerf
>>>> Google, LLC
>>>> 1900 Reston Metro Plaza, 16th Floor
>>>> Reston, VA 20190
>>>> +1 (571) 213 1346
>>>> 
>>>> until further notice
>>>> 
>>>> _______________________________________________
>>>> Nnagain mailing list
>>>> 
>>>> Nnagain at lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/nnagain
>>>> 
>>>> _______________________________________________
>>>> Nnagain mailing list
>>>> Nnagain at lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/nnagain
>>> _______________________________________________
>>> Nnagain mailing list
>>> Nnagain at lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/nnagain
>> _______________________________________________
>> Nnagain mailing list
>> Nnagain at lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/nnagain
> 
> _______________________________________________
> Nnagain mailing list
> Nnagain at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/nnagain
> _______________________________________________
> Nnagain mailing list
> Nnagain at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/nnagain