<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">That Flakeway tool makes me think of an early version of the Chaos Monkey. To that note, Apple maintains a developer tool called Network Link Conditioner that does a good job simulating reduced network performance.<br><div><br><blockquote type="cite"><div>On Oct 15, 2023, at 23:30, Jack Haverty via Nnagain <nnagain@lists.bufferbloat.net> wrote:</div><br class="Apple-interchange-newline"><div>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div>
Even back in 1978, I didn't think Source Quench would work. I
recall that I was trying to adapt my TCP2.5 Unix implementation to
become TCP4, and I asked what my TCP should do if it sent the first
IP datagram to open a TCP connection and received a Source Quench.
It wasn't clear at all how I should "slow down". Other TCP
implementors took the receipt of an SQ as an indication that a
datagram they had sent had been discarded, so the obvious reaction
for user satisfaction was to retransmit immediately. Slowing down
would simply degrade their user's experience.<br>
<br>
Glad to hear SQ is gone. I hope whatever replaced it works.<br>
<br>
There's some confusion about the Arpanet. The Arpanet was known as
a "packet switching network", but it had lots of internal mechanisms
that essentially created virtual circuits between attached
computers. Every packet sent in to the network by a user computer
came out at the destination intact, in order, and not duplicated or
lost. The Arpanet switches even had a hardware mechanism for flow
control; a switch could halt data transfer from a user computer when
necessary. During the 80s, the Arpanet evolved to have an X.25
interface, and operated as a true "virtual circuit" provider. Even
in the Defense Data Network (DDN), the network delivered a virtual
circuit service. The attached users' computers had TCP, but the TCP
didn't need to deal with most of the network behavior that TCP was
designed to handle. Congestion was similarly handled by internal
Arpanet mechanisms (there were several technical reports from BBN to
ARPA with details). I don't remember any time that "an <span style="white-space: pre-wrap">explicit ack for every packet was ripped out of the arpanet"</span>
None of those events happened when two TCP computers were connected
to the Arpanet.<br>
<br>
The Internet grew up around the Arpanet, which provided most of the
wide-area connectivity through the mid-80s. Since the Arpanet
provided the same "reliable byte stream" behavior as TCP provided,
and most user computers were physically attached to an Arpanet
switch, it wasn't obvious how to test a TCP implementation, to see
how well it dealt with reordering, duplication, dropping, or
corruption of IP datagrams. <br>
<br>
We (at BBN) actually had to implement a software package called a
"Flakeway", which ran on a SparcStation. Using a "feature" of
Ethernets and ARP (some would call it a vulnerability), the Flakeway
could insert itself invisibly in the stream of datagrams between any
two computers on that LAN (e.g., between a user computer and the
gateway/router providing a path to other sites). The Flakeway could
then simulate "real" Internet behavior by dropping, duplicating,
reordering, mangling, delaying, or otherwise interfering with the
flow. That was extremely useful in testing and diagnosing TCP
implementations.<br>
<br>
I understand that there has been a lot of technical work over the
years, and lots of new mechanisms defined for use in the Internet to
solve various problems. But one issue that has not been addressed
-- how do you know whether or not some such mechanism has actually
been implemented, and configured correctly, in the millions of
devices that are now using TCP (and UDP, IP, etc.)? AFAIK, there's
no way to tell unless you can examine the actual code.<br>
<br>
The Internet, and TCP, was an experiment. One aspect of that
experiment involved changing the traditional role of a network
"switch", and moving mechanisms for flow control, error control, and
other mechanisms used to create a "virtual circuit" behavior.
Instead of being implemented inside some switching equipment, TCP's
mechanisms are implemented inside users' computers. That was a
significant break from traditional network architecture.<br>
<br>
I didn't realize it at the time, but now, with users' devices being
uncountable handheld or desktop computers rather than huge racks in
relatively few data centers, moving all those mechanisms from
switches to users' computers significantly complicates the system
design and especially operation.<br>
<br>
That may be one of the more important results of the long-running
experiment.<br>
<br>
Jack Haverty<br>
<br>
<div class="moz-cite-prefix">On 10/15/23 18:39, Dave Taht wrote:<br>
</div>
<blockquote type="cite" cite="mid:CAA93jw5LBgm3JYdxyv8LQNXgFBX7aYr53gjH-XZACV__WkBzQw@mail.gmail.com">
<pre class="moz-quote-pre" wrap="">It is wonderful to have your original perspectives here, Jack.
But please, everyone, before a major subject change, change the subject?
Jack's email conflates a few things that probably deserve other
threads for them. One is VGV - great acronym! Another is about the
"Placeholders" of TTL, and TOS. The last is the history of congestion
control - and it's future! As being a part of the most recent episodes
here I have written extensively on the subject, but what I most like
to point people to is my fun talks trying to make it more accessible
like this one at apnic
<a class="moz-txt-link-freetext" href="https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-but-its-not-over-yet/">https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-but-its-not-over-yet/</a>
or my more recent one at tti/vanguard.
Most recently one of our LibreQos clients has been collecting 10ms
samples and movies of what real-world residential traffic actually
looks like:
<a class="moz-txt-link-freetext" href="https://www.youtube.com/@trendaltoews7143">https://www.youtube.com/@trendaltoews7143</a>
And it is my hope that that conveys intuition to others... as compared
to speedtest traffic, which prove nothing about the actual behaviors
of VGV traffic, which I ranted about here:
<a class="moz-txt-link-freetext" href="https://blog.cerowrt.org/post/speedtests/">https://blog.cerowrt.org/post/speedtests/</a> - I am glad that these
speedtests now have latency under load reports almost universally, but
see the rant for more detail.
Most people only have a picture of traffic in the large, over 5 minute
intervals, which behaves quite differently, or a pre-conception that
backpressure actually exists across the internet. It doesn't. An
explicit ack for every packet was ripped out of the arpanet as costing
too much time. Wifi, to some extent, recreates the arpanet problem by
having explicit acks on the local loop that are repeated until by god
the packet comes through, usually without exponential backoff.
We have some really amazing encoding schemes now - I do not understand
how starlink works without retries for example, an my grip on 5G's
encodings is non-existent, except knowing it is the most bufferbloated
of all our technologies.
...
Anyway, my hope for this list is that we come up with useful technical
feedback to the powers-that-be that want to regulate the internet
under some title ii provisions, and I certainly hope we can make
strides towards fixing bufferbloat along the way! There are many other
issues. Let's talk about those instead!
But...
...
In "brief" response to the notes below - source quench died due to
easy ddos, AQMs from RED (1992) until codel (2012) struggled with
measuring the wrong things ( Kathie's updated paper on red in a
different light: <a class="moz-txt-link-freetext" href="https://pollere.net/Codel.html">https://pollere.net/Codel.html</a> ), SFQ was adopted by
many devices, WRR used in others, ARED I think is common in juniper
boxes, fq_codel is pretty much the default now for most of linux, and
I helped write CAKE.
TCPs evolved from reno to vegas to cubic to bbr and the paper on BBR
is excellent: <a class="moz-txt-link-freetext" href="https://research.google/pubs/pub45646/">https://research.google/pubs/pub45646/</a> as is len
kleinrock's monograph on it. However problems with self congestion and
excessive packet loss were observed, and after entering the ietf
process, is now in it's 3rd revision, which looks pretty good.
Hardware pause frames in ethernet are often available, there are all
kinds of specialized new hardware flow control standards in 802.1, a
new more centralized controller in wifi7
To this day I have no idea how infiniband works. Or how ATM was
supposed to work. I have a good grip on wifi up to version 6, and the
work we did on wifi is in use now on a lot of wifi gear like openwrt,
eero and evenroute, and I am proudest of all my teams' work on
achieving airtime fairness, and better scheduling described in this
paper here: <a class="moz-txt-link-freetext" href="https://www.cs.kau.se/tohojo/airtime-fairness/">https://www.cs.kau.se/tohojo/airtime-fairness/</a> for wifi
and MOS to die for.
There is new work on this thing called L4S, which has a bunch of RFCs
for it, leverages multi-bit DCTCP style ECN and is under test by apple
and comcast, it is discussed on tsvwg list a lot. I encourage users to
jump in on the comcast/apple beta, and operators to at least read
this: <a class="moz-txt-link-freetext" href="https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4sops/">https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4sops/</a>
Knowing that there is a book or three left to write on this subject
that nobody will read is an issue, as is coming up with an
architecture to take packet handling as we know it, to the moon and
the rest of the solar system, seems kind of difficult.
Ideally I would love to be working on that earth-moon architecture
rather than trying to finish getting stuff we designed in 2012-2016
deployed.
I am going to pull out a few specific questions from the below and
answer separately.
On Sun, Oct 15, 2023 at 1:00 PM Jack Haverty via Nnagain
<a class="moz-txt-link-rfc2396E" href="mailto:nnagain@lists.bufferbloat.net"><nnagain@lists.bufferbloat.net></a> wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">The "VGV User" (Voice, Gaming, Videoconferencing) cares a lot about
latency. It's not just "rewarding" to have lower latencies; high
latencies may make VGV unusable. Average (or "typical") latency as the
FCC label proposes isn't a good metric to judge usability. A path which
has high variance in latency can be unusable even if the average is
quite low. Having your voice or video or gameplay "break up" every
minute or so when latency spikes to 500 msec makes the "user experience"
intolerable.
A few years ago, I ran some simple "ping" tests to help a friend who was
trying to use a gaming app. My data was only for one specific path so
it's anecdotal. What I saw was surprising - zero data loss, every
datagram was delivered, but occasionally a datagram would take up to 30
seconds to arrive. I didn't have the ability to poke around inside, but
I suspected it was an experience of "bufferbloat", enabled by the
dramatic drop in price of memory over the decades.
It's been a long time since I was involved in operating any part of the
Internet, so I don't know much about the inner workings today. Apologies
for my ignorance....
There was a scenario in the early days of the Internet for which we
struggled to find a technical solution. Imagine some node in the bowels
of the network, with 3 connected "circuits" to some other nodes. On two
of those inputs, traffic is arriving to be forwarded out the third
circuit. The incoming flows are significantly more than the outgoing
path can accept.
What happens? How is "backpressure" generated so that the incoming
flows are reduced to the point that the outgoing circuit can handle the
traffic?
About 45 years ago, while we were defining TCPV4, we struggled with this
issue, but didn't find any consensus solutions. So "placeholder"
mechanisms were defined in TCPV4, to be replaced as research continued
and found a good solution.
In that "placeholder" scheme, the "Source Quench" (SQ) IP message was
defined; it was to be sent by a switching node back toward the sender of
any datagram that had to be discarded because there wasn't any place to
put it.
In addition, the TOS (Type Of Service) and TTL (Time To Live) fields
were defined in IP.
TOS would allow the sender to distinguish datagrams based on their
needs. For example, we thought "Interactive" service might be needed
for VGV traffic, where timeliness of delivery was most important.
"Bulk" service might be useful for activities like file transfers,
backups, et al. "Normal" service might now mean activities like using
the Web.
The TTL field was an attempt to inform each switching node about the
"expiration date" for a datagram. If a node somehow knew that a
particular datagram was unlikely to reach its destination in time to be
useful (such as a video datagram for a frame that has already been
displayed), the node could, and should, discard that datagram to free up
resources for useful traffic. Sadly we had no mechanisms for measuring
delay, either in transit or in queuing, so TTL was defined in terms of
"hops", which is not an accurate proxy for time. But it's all we had.
Part of the complexity was that the "flow control" mechanism of the
Internet had put much of the mechanism in the users' computers' TCP
implementations, rather than the switches which handle only IP. Without
mechanisms in the users' computers, all a switch could do is order more
circuits, and add more memory to the switches for queuing. Perhaps that
led to "bufferbloat".
So TOS, SQ, and TTL were all placeholders, for some mechanism in a
future release that would introduce a "real" form of Backpressure and
the ability to handle different types of traffic. Meanwhile, these
rudimentary mechanisms would provide some flow control. Hopefully the
users' computers sending the flows would respond to the SQ backpressure,
and switches would prioritize traffic using the TTL and TOS information.
But, being way out of touch, I don't know what actually happens today.
Perhaps the current operators and current government watchers can answer?:
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">I would love moe feedback about RED''s deployment at scale in particular.
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">1/ How do current switches exert Backpressure to reduce competing
traffic flows? Do they still send SQs?
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">Some send various forms of hardware flow control, an ethernet pause
frame derivative
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">2/ How do the current and proposed government regulations treat the
different needs of different types of traffic, e.g., "Bulk" versus
"Interactive" versus "Normal"? Are Internet carriers permitted to treat
traffic types differently? Are they permitted to charge different
amounts for different types of service?
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">Jack Haverty
On 10/15/23 09:45, Dave Taht via Nnagain wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">For starters I would like to apologize for cc-ing both nanog and my
new nn list. (I will add sender filters)
A bit more below.
On Sun, Oct 15, 2023 at 9:32 AM Tom Beecher <a class="moz-txt-link-rfc2396E" href="mailto:beecher@beecher.cc"><beecher@beecher.cc></a> wrote:
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">So for now, we'll keep paying for transit to get to the others (since it’s about as much as transporting IXP from Dallas), and hoping someone at Google finally sees Houston as more than a third rate city hanging off of Dallas. Or… someone finally brings a worthwhile IX to Houston that gets us more than peering to Kansas City. Yeah, I think the former is more likely. 😊
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">There is often a chicken/egg scenario here with the economics. As an eyeball network, your costs to build out and connect to Dallas are greater than your transit cost, so you do that. Totally fair.
However think about it from the content side. Say I want to build into to Houston. I have to put routers in, and a bunch of cache servers, so I have capital outlay , plus opex for space, power, IX/backhaul/transit costs. That's not cheap, so there's a lot of calculations that go into it. Is there enough total eyeball traffic there to make it worth it? Is saving 8-10ms enough of a performance boost to justify the spend? What are the long term trends in that market? These answers are of course different for a company running their own CDN vs the commercial CDNs.
I don't work for Google and obviously don't speak for them, but I would suspect that they're happy to eat a 8-10ms performance hit to serve from Dallas , versus the amount of capital outlay to build out there right now.
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">The three forms of traffic I care most about are voip, gaming, and
videoconferencing, which are rewarding to have at lower latencies.
When I was a kid, we had switched phone networks, and while the sound
quality was poorer than today, the voice latency cross-town was just
like "being there". Nowadays we see 500+ms latencies for this kind of
traffic.
As to how to make calls across town work that well again, cost-wise, I
do not know, but the volume of traffic that would be better served by
these interconnects quite low, respective to the overall gains in
lower latency experiences for them.
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">On Sat, Oct 14, 2023 at 11:47 PM Tim Burke <a class="moz-txt-link-rfc2396E" href="mailto:tim@mid.net"><tim@mid.net></a> wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">I would say that a 1Gbit IP transit in a carrier neutral DC can be had for a good bit less than $900 on the wholesale market.
Sadly, IXP’s are seemingly turning into a pay to play game, with rates almost costing as much as transit in many cases after you factor in loop costs.
For example, in the Houston market (one of the largest and fastest growing regions in the US!), we do not have a major IX, so to get up to Dallas it’s several thousand for a 100g wave, plus several thousand for a 100g port on one of those major IXes. Or, a better option, we can get a 100g flat internet transit for just a little bit more.
Fortunately, for us as an eyeball network, there are a good number of major content networks that are allowing for private peering in markets like Houston for just the cost of a cross connect and a QSFP if you’re in the right DC, with Google and some others being the outliers.
So for now, we'll keep paying for transit to get to the others (since it’s about as much as transporting IXP from Dallas), and hoping someone at Google finally sees Houston as more than a third rate city hanging off of Dallas. Or… someone finally brings a worthwhile IX to Houston that gets us more than peering to Kansas City. Yeah, I think the former is more likely. 😊
See y’all in San Diego this week,
Tim
On Oct 14, 2023, at 18:04, Dave Taht <a class="moz-txt-link-rfc2396E" href="mailto:dave.taht@gmail.com"><dave.taht@gmail.com></a> wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">This set of trendlines was very interesting. Unfortunately the data
stops in 2015. Does anyone have more recent data?
<a class="moz-txt-link-freetext" href="https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php">https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php</a>
I believe a gbit circuit that an ISP can resell still runs at about
$900 - $1.4k (?) in the usa? How about elsewhere?
...
I am under the impression that many IXPs remain very successful,
states without them suffer, and I also find the concept of doing micro
IXPs at the city level, appealing, and now achievable with cheap gear.
Finer grained cross connects between telco and ISP and IXP would lower
latencies across town quite hugely...
PS I hear ARIN is planning on dropping the price for, and bundling 3
BGP AS numbers at a time, as of the end of this year, also.
--
Oct 30: <a class="moz-txt-link-freetext" href="https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html">https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html</a>
Dave Täht CSO, LibreQos
</pre>
</blockquote>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Nnagain mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Nnagain@lists.bufferbloat.net">Nnagain@lists.bufferbloat.net</a>
<a class="moz-txt-link-freetext" href="https://lists.bufferbloat.net/listinfo/nnagain">https://lists.bufferbloat.net/listinfo/nnagain</a>
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
</pre>
</blockquote>
<br>
</div>
_______________________________________________<br>Nnagain mailing list<br>Nnagain@lists.bufferbloat.net<br>https://lists.bufferbloat.net/listinfo/nnagain<br></div></blockquote></div><br></body></html>