During the 90s, I was part of a small team responsible for operating a corporate small-i internet. We used cisco equipment, in offices in 100 or so countries, all interconnected with circuits from various "phone companies". No fiber or wifi around yet. Our internet was not connected to The Internet, but used all the same equipment and software.

To figure out how to respond to user complaints (e.g., "The network is really slow today!"), we discovered that we needed good instrumentation not only in the various routers but also in the computers in the user environments. TCP, implemented in the user computers, did a good job at hiding problems. For example, TCPs could get into a semi-stable state where every datagram was getting transmitted twice with the second copy being discarded at the destination. That caused "network throughput" (datagrams/second) to go way up, but user performance to go way down.

To get the data, we used SNMP and simple scripts to probe the various devices from our NOC, stuff all the data into a database, and use the available data analysis tools to figure out what was happening. At the time, SNMP "MIB"s were defined for both routers and computers. So we could collect data from the TCPs involved in a problem, as well as data from router operations. When routers reported N datagrams/time, the associated TCPs might report N/2 retransmissions from the sender, and N/2 discards from the receiver.

With today's ability to synchronize clocks across the network, it should now also be possible to report data about latency. Interactive operations - (A/V conferencing, gaming, telepresence, etc.) probably have lots of metrics that could be useful much as we found TCP's to be.

I'm way out of touch with network operations now. Do current network managers look at metrics from the enduser computers' perspective? Can latency be measured between two user devices involved in a problem report?

Jack

On 11/14/23 13:45, rjmcmahon via Nnagain wrote:

I think the missing metrics & test vectors are around latency more than bandwidth.

I've attached a WiFi low latency table. Feel free to comment.

Good metrics could allow for a comprehensive analysis, at least from a WiFi perspective.

Latency under load is a good start, but likely not enough.

Bob

PS. Agreed, the digital transition with storage has many engineers who have contributed over decades. I'm very grateful to all of them.

I am glad you are reaching out, but it may be difficult for us to do a
joint filing.

In particular I question the seeming assumption that more wifi devices
will drive demand for more bandwidth,
and extrapolating from 18 devices forward may also well be a trend
that will reverse completely in favor of more bluetooth and thread
implementations from phone to device.

Of those 20 wifi devices today are probably

1 or more laptops
1 or more tablets
1 or more phones
1 or more tvs

and of those usually only one will be active per person, while they
are in the home, and even then....as one semi-hard number, even at
peak hours (with the libreqos data I have), only 1/6th of households
are watching video, and very, very few, more than one stream at the
same time.

The steady upload bandwidth pumpers are primarily video surveillance
devices (which as a personal preference I would prefer remain in the
home unless otherwise activated). I do not presently know much about
the frame rates or real bandwidth requirements of popular devices like
ring, etc. Similarly I am biased towards "Babycams" sending video
from up to downstairs only and not into the cloud. I know I am bucking
the trend on this, but it will make me skeptical of much "data" that
exists today on it.

Then you have loads of extremely low bandwidth devices - alexa and
other automation is measured in bits/ms, light switches, a couple bits
a day, audio streaming 128kbit/s (when you use it). Automatic updates
to phones and tablets, etc, take place entirely asynchronously
nowadays and do not need much bandwidth. A small business just needs
to
*reliably* clear credit card transactions every few minutes.

Perhaps the biggest steady-state bandwidth suck is home gaming
updates, but while a big market, if you haven't noticed birthrates are
down, and immigration being canceled.

Thus I feel that the opposite number of 70-80% two people or less per
household that you are not optimizing for, dominates the curves.

Looking at the actual useage disparity (delta) between fiber'd cities
and rural, uptake of passive video streaming services vs spotify,
would give me a more pessimistic projection than most. Regrettably I
lack the time and as few fund accurate scenarios, I would merely be
willing to write down my estimate and find some sort of online
"futures" market to place puts on.

Lastly, a goodly percentage of the people I know just need food,
shelter, a job and a phone, and with broadband costs skyrocketing,
aside from the gaming market and business, that is all they can
afford, even with ACP. And all that they need. Nobody has a landline
anymore, and if it weren't for "Tv", few would want broadband at even
25/10.

On Tue, Nov 14, 2023 at 2:40 PM rjmcmahon via Nnagain
<nnagain@lists.bufferbloat.net> wrote:

Thanks for sharing this. I agree this works for researchers.

I think we're at a different state and economic returns matter too.

I sent the following to our engineers in hopes we can all better
understand what we're all trying to accomplish.

Hi All,

The attached Notice of Inquiry by the FCC shows how much our work
matters to most everyone in our country (and, by inference, worldwide.)
Broadband networks are no longer entertainment or social networks but
they are critical to all regardless of gender, age, race, ethnic group,
etc. People's health, learning, and ability to earn for their families
all depend on us providing world class engineering to our customers who
in turn provide these networks for each and all of us, our friends &
families, our neighbors, and most everyone else.

Early in my career, I worked at Cisco and had the privilege to work on
some of the first BGP routers that enabled the commercial build out of
the internet, and I'm very thankful we did that way ahead of the 2019
pandemic. There was no "pandemic use case" that drove us - we just
wanted to build the best products that engineers could build. A
worldwide pandemic w/o the internet could have been disastrous - so that
work by many in the mid 1990s seems to have paid off well.

I hope you each realize, today, what you've accomplished since then and
continue to be a part of. It's truly significant. It's been a high honor
to work with so many of you over the last 14+ years.

This is beautiful, btw. I feel much the same way about linux being now
so used heavily in the space program,
and all our code, and hardware, that will propagate across the solar
system, and of the millions of people, that contributed to it.

To the FCC report:

We begin this annual inquiry in the wake of the COVID-19 pandemic during
which time Americans increasingly turned to their broadband connections
to conduct their lives online by using telemedicine to access
healthcare, working from home, attending classes remotely, connecting by
video with out-of-town family and friends, and streaming entertainment.
Our experiences with the pandemic made it clear that broadband is no
longer a luxury but a necessity that will only become more important
with time. Never before has the critical importance of ensuring that all
Americans have access to high-speed, affordable broadband been more
evident.

Also note, we have more work to do. We need to increase resiliency as an
example. Also, the thing I'm most passionate about is low latency. The
FCC is now recognizing the importance of that. People are slowly
learning why latency is becoming equally important to capacity when it
comes to quality of service.

Bob

PS. The rest is TLDR but I thought I post some snippets for those
interested

We believe that in examining household use cases, a simple summation of
required speeds for individual activities may provide a misleading
picture of actual broadband needs for at least three reasons. First, we
believe it is appropriate to take into account at least occasional
downloads of very large files which can be bandwidth-intensive. Second,
it is important to account for larger households; in 2022, approximately
21% of all U.S. households had four or more people, and the number of
families seeking out multigenerational homes to live with additional
relatives rose.57 Households of all sizes must have sufficient bandwidth
to satisfy their needs. In addition, the number of connected devices per
household continues to grow, from 18.6 in the average household in 2021
to 20.2 in the first half of 2022.58 Taking these factors into account
suggests that fixed broadband download/upload needs could easily exceed
100/20 Mbps.

...

Service Quality. We recognize that other factors, besides the speed of a
broadband connection, can affect consumers’ ability to use the services
effectively. Chief among these factors is latency, which is the measure
of the time it takes a packet of data to travel from one point in the
network to another, and which is typically measured by round-trip time
in milliseconds (ms). As a measurement of advanced telecommunications
capability, latency can be critical because it affects a consumer’s
ability to use real-time applications, including voice over Internet
Protocol, video calling, distance learningapplications, and online
gaming. Actual (as opposed to advertised) speed received, consistency of
speed, and data allowances are also important. Such factors are not
simply a matter of service interruptions and consumer satisfaction—they
have a real and significant effect on Americans’ ability to use critical
web-based applications, including those that facilitate telehealth,
telework, and virtual learning.

> In the beginning days of the Arpanet, circa early 1970s, ARPA made a
> policy decision about use of the Arpanet. First, Arpa Program
> Managers, located on the East Coast of the US, were assigned computer
> accounts on USC-ISIA, located on the West Coast in LA. Thus to do
> their work, exchanging email, editting documents, and such, they had
> to *use* the Arpanet to connect their terminals in Washington to the
> PDP-10 in California - 3000 miles away.
>
> Second, ARPA began requiring all of their contractors (researchers at
> Universities etc.) to interact with Arpa using email and FTP.   If
> your site was "on the Arpanet", you had to use the Arpanet. If you
> wanted your proposal for next year's research to be funded, you had to
> submit your proposal using the net.
>
> This policy caused a profound attention, by everyone involved, to
> making the Arpanet work and be useful as a collaboration tool.
>
> JCR Licklider (aka Lick) was my advisor at MIT, and then my boss when
> I joined the Research Staff.   Lick had been at ARPA for a while,
> promoting his vision of a "Galactic Network" that resulted in the
> Arpanet as a first step. At MIT, Lick still had need for lots of
> interactions with others.   My assignment was to build and operate the
> email system for Lick's group at MIT on our own PDP-10. Lick had a
> terminal in his office and was online a lot.   If email didn't work, I
> heard about it.   If the Arpanet didn't work, BBN heard about it.
>
> This pressure was part of Arpa policy.   Sometimes it's referred to as
> "eating your own dog food" -- i.e., making sure your "dog" will get
> the same kind of nutrition you enjoy.   IMHO, that pressure policy was
> important, perhaps crucial, to the success of the Arpanet.
>
> In the 70s, meetings still occurred, but a lot of progress was made
> through the use of the Arpanet.   You can only do so much with email
> and file interactions. Today, the possibilities for far richer
> interactions are much more prevalent.   But IMHO they are held back,
> possibly because no one is feeling the pressure to "make it work".
> Gigabit throughputs are common, but why does my video and audio still
> break up...?
>
> It's important to have face-to-face meetings, but perhaps if the IETF
> scheduled a future meeting to be online only, whatever needs to happen
> to make it work would happen? Perhaps...
>
> Even a "game" might drive progress. At Interop '92, we resurrected
> the old "MazeWars" game using computers scattered across the show
> exhibit halls. The engineers in the control room above the floor felt
> the pressure to make sure the Game continued to run. At the time, the
> Internet itself was too slow for enjoyable gameplay at any distance.
> Will the Internet 30 years later work?
>
> Or perhaps the IETF, or ISOC, or someone could take on a highly
> visible demo involving non-techie end users.   An online meeting of
> the UN General Assembly?   Or some government bodies - US Congress,
> British Parliament, etc.
>
> Such an event would surface the issues, both technical and policy, to
> the engineers, corporations, policy-makers, and others who might have
> the ability and interest to "make it work".
>
> Jack
>
> On 11/14/23 10:10, Sebastian Moeller wrote:
>
>> Hi Jack,
>>
>>> On Nov 14, 2023, at 13:02, Jack Haverty via Nnagain
>>> <nnagain@lists.bufferbloat.net> wrote:
>>>
>>> If video conferencing worked well enough, they would not have to
>>> all get together in one place and would instead hold IETF meetings
>>> online ...?
>>
>> [SM] Turns out that humans are social creatures, and some things
>> work better face-to-face and in the hallway (and if that is only
>> building trust and sympathy) than over any remote technology.
>>
>>> Did anyone measure latency?   Does anyone measure throughput of
>>> "useful" traffic - e.g., excluding video/audio data that didn't
>>> arrive in time to be actually used on the screen or speaker?
>>
>> [SM] Utility is in the eye of the beholder, no?
>>
>> Jack Haverty
>>
>> On 11/14/23 09:25, Vint Cerf via Nnagain wrote:
>>
>> if they had not been all together they would have been consuming
>> tons of video capacity doing video conference calls....
>>
>> :-))
>> v
>>
>> On Tue, Nov 14, 2023 at 10:46 AM Livingood, Jason via Nnagain
>> <nnagain@lists.bufferbloat.net> wrote:
>> On the subject of how much bandwidth does one household need, here's
>> a fun stat for you.
>>
>> At the IETF’s 118th meeting last week (Nov 4 – 10, 2023), there
>> were over 1,000 engineers in attendance. At peak there were 870
>> devices connected to the WiFi network. Peak bandwidth usage:
>>
>> • Downstream peak ~750 Mbps
>> • Upstream ~250 Mbps
>>
>> From my pre-meeting Twitter poll
>> (https://twitter.com/jlivingood/status/1720060429311901873):
>>
>> <image001.png>
>>
>> _______________________________________________
>> Nnagain mailing list
>> Nnagain@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/nnagain
>>
>> --
>> Please send any postal/overnight deliveries to:
>> Vint Cerf
>> Google, LLC
>> 1900 Reston Metro Plaza, 16th Floor
>> Reston, VA 20190
>> +1 (571) 213 1346
>>
>> until further notice
>>
>> _______________________________________________
>> Nnagain mailing list
>>
>> Nnagain@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/nnagain
>>
>> _______________________________________________
>> Nnagain mailing list
>> Nnagain@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/nnagain
> _______________________________________________
> Nnagain mailing list
> Nnagain@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/nnagain
_______________________________________________
Nnagain mailing list
Nnagain@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/nnagain
_______________________________________________
Nnagain mailing list
Nnagain@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/nnagain