<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Thinking of networks not being fast enough .. we wrote this years
upon years ago at the Interop show. We shouldn't have been
surprised, but we were - a lot of "the press" believed this:</p>
<p><a class="moz-txt-link-freetext" href="https://www.cavebear.com/cb_catalog/techno/gaganet/">https://www.cavebear.com/cb_catalog/techno/gaganet/</a></p>
<p>Here's the introduction snippet, the rest is via the link above:<br>
</p>
<p><span class="tt">May 5, 1998:<br>
Las Vegas, Networld+Interop</span></p>
<p><span class="tt">Today, the worlds greatest collection of
networking
professionals gathered and constructed the first
trans-relativistic
network.</span></p>
<p><span class="tt"></span><span class="tt">The NOC Team used
hyper-fiber to
create the first network not limited by the speed of light. ...<br>
</span></p>
<p></p>
<p>etc etc</p>
<p> --karl--<br>
</p>
<div class="moz-cite-prefix">On 10/15/23 1:39 PM, rjmcmahon via
Nnagain wrote:<br>
</div>
<blockquote type="cite"
cite="mid:4c44a9ef4c4b14a06403e553e633717d@rjmcmahon.com">Hi Jack,
<br>
<br>
Thanks again for sharing. It's very interesting to me.
<br>
<br>
Today, the networks are shifting from capacity constrained to
latency constrained, as can be seen in the IX discussions about
how the speed of light over fiber is too slow even between Houston
& Dallas.
<br>
<br>
The mitigations against standing queues (which cause bloat today)
are:
<br>
<br>
o) Shrink the e2e bottleneck queue so it will drop packets in a
flow and TCP will respond to that "signal"
<br>
o) Use some form of ECN marking where the network forwarding plane
ultimately informs the TCP source state machine so it can slow
down or pace effectively. This can be an earlier feedback signal
and, if done well, can inform the sources to avoid bottleneck
queuing. There are couple of approaches with ECN. Comcast is
trialing L4S now which seems interesting to me as a WiFi test
& measurement engineer. The jury is still out on this and
measurements are needed.
<br>
o) Mitigate source side bloat via TCP_NOTSENT_LOWAT
<br>
<br>
The QoS priority approach per congestion is orthogonal by my
judgment as it's typically not supported e2e, many networks will
bleach DSCP markings. And it's really too late by my judgment.
<br>
<br>
Also, on clock sync, yes your generation did us both a service and
disservice by getting rid of the PSTN TDM clock ;) So IP
networking devices kinda ignored clock sync, which makes e2e one
way delay (OWD) measurements impossible. Thankfully, the GPS
atomic clock is now available mostly everywhere and many devices
use TCXO oscillators so it's possible to get clock sync and use
oscillators that can minimize drift. I pay $14 for a Rpi4 GPS chip
with pulse per second as an example.
<br>
<br>
It seems silly to me that clocks aren't synced to the GPS atomic
clock even if by a proxy even if only for measurement and
monitoring.
<br>
<br>
Note: As Richard Roy will point out, there really is no such thing
as synchronized clocks across geographies per general relativity -
so those syncing clocks need to keep those effects in mind. I
limited the iperf 2 timestamps to microsecond precision in hopes
avoiding those issues.
<br>
<br>
Note: With WiFi, a packet drop can occur because an intermittent
RF channel condition. TCP can't tell the difference between an RF
drop vs a congested queue drop. That's another reason ECN markings
from network devices may be better than dropped packets.
<br>
<br>
Note: I've added some iperf 2 test support around pacing as that
seems to be the direction the industry is heading as networks are
less and less capacity strained and user quality of experience is
being driven by tail latencies. One can also test with the Prague
CCA for the L4S scenarios. (This is a fun project:
<a class="moz-txt-link-freetext" href="https://www.l4sgear.com/">https://www.l4sgear.com/</a> and fairly low cost)
<br>
<br>
--fq-rate n[kmgKMG]
<br>
Set a rate to be used with fair-queuing based socket-level pacing,
in bytes or bits per second. Only available on platforms
supporting the SO_MAX_PACING_RATE socket option. (Note: Here the
suffixes indicate bytes/sec or bits/sec per use of uppercase or
lowercase, respectively)
<br>
<br>
--fq-rate-step n[kmgKMG]
<br>
Set a step of rate to be used with fair-queuing based socket-level
pacing, in bytes or bits per second. Step occurs every
fq-rate-step-interval (defaults to one second)
<br>
<br>
--fq-rate-step-interval n
<br>
Time in seconds before stepping the fq-rate
<br>
<br>
Bob
<br>
<br>
PS. Iperf 2 man page
<a class="moz-txt-link-freetext" href="https://iperf2.sourceforge.io/iperf-manpage.html">https://iperf2.sourceforge.io/iperf-manpage.html</a>
<br>
<br>
<blockquote type="cite">The "VGV User" (Voice, Gaming,
Videoconferencing) cares a lot about
<br>
latency. It's not just "rewarding" to have lower latencies;
high
<br>
latencies may make VGV unusable. Average (or "typical")
latency as
<br>
the FCC label proposes isn't a good metric to judge usability.
A path
<br>
which has high variance in latency can be unusable even if the
average
<br>
is quite low. Having your voice or video or gameplay "break
up"
<br>
every minute or so when latency spikes to 500 msec makes the
"user
<br>
experience" intolerable.
<br>
<br>
A few years ago, I ran some simple "ping" tests to help a friend
who
<br>
was trying to use a gaming app. My data was only for one
specific
<br>
path so it's anecdotal. What I saw was surprising - zero data
loss,
<br>
every datagram was delivered, but occasionally a datagram would
take
<br>
up to 30 seconds to arrive. I didn't have the ability to poke
around
<br>
inside, but I suspected it was an experience of "bufferbloat",
enabled
<br>
by the dramatic drop in price of memory over the decades.
<br>
<br>
It's been a long time since I was involved in operating any part
of
<br>
the Internet, so I don't know much about the inner workings
today.
<br>
Apologies for my ignorance....
<br>
<br>
There was a scenario in the early days of the Internet for which
we
<br>
struggled to find a technical solution. Imagine some node in
the
<br>
bowels of the network, with 3 connected "circuits" to some other
<br>
nodes. On two of those inputs, traffic is arriving to be
forwarded
<br>
out the third circuit. The incoming flows are significantly
more than
<br>
the outgoing path can accept.
<br>
<br>
What happens? How is "backpressure" generated so that the
incoming
<br>
flows are reduced to the point that the outgoing circuit can
handle
<br>
the traffic?
<br>
<br>
About 45 years ago, while we were defining TCPV4, we struggled
with
<br>
this issue, but didn't find any consensus solutions. So
"placeholder"
<br>
mechanisms were defined in TCPV4, to be replaced as research
continued
<br>
and found a good solution.
<br>
<br>
In that "placeholder" scheme, the "Source Quench" (SQ) IP
message was
<br>
defined; it was to be sent by a switching node back toward the
sender
<br>
of any datagram that had to be discarded because there wasn't
any
<br>
place to put it.
<br>
<br>
In addition, the TOS (Type Of Service) and TTL (Time To Live)
fields
<br>
were defined in IP.
<br>
<br>
TOS would allow the sender to distinguish datagrams based on
their
<br>
needs. For example, we thought "Interactive" service might be
needed
<br>
for VGV traffic, where timeliness of delivery was most
important.
<br>
"Bulk" service might be useful for activities like file
transfers,
<br>
backups, et al. "Normal" service might now mean activities
like
<br>
using the Web.
<br>
<br>
The TTL field was an attempt to inform each switching node about
the
<br>
"expiration date" for a datagram. If a node somehow knew that
a
<br>
particular datagram was unlikely to reach its destination in
time to
<br>
be useful (such as a video datagram for a frame that has already
been
<br>
displayed), the node could, and should, discard that datagram to
free
<br>
up resources for useful traffic. Sadly we had no mechanisms for
<br>
measuring delay, either in transit or in queuing, so TTL was
defined
<br>
in terms of "hops", which is not an accurate proxy for time.
But
<br>
it's all we had.
<br>
<br>
Part of the complexity was that the "flow control" mechanism of
the
<br>
Internet had put much of the mechanism in the users' computers'
TCP
<br>
implementations, rather than the switches which handle only IP.
<br>
Without mechanisms in the users' computers, all a switch could
do is
<br>
order more circuits, and add more memory to the switches for
queuing.
<br>
Perhaps that led to "bufferbloat".
<br>
<br>
So TOS, SQ, and TTL were all placeholders, for some mechanism in
a
<br>
future release that would introduce a "real" form of
Backpressure and
<br>
the ability to handle different types of traffic. Meanwhile,
these
<br>
rudimentary mechanisms would provide some flow control.
Hopefully the
<br>
users' computers sending the flows would respond to the SQ
<br>
backpressure, and switches would prioritize traffic using the
TTL and
<br>
TOS information.
<br>
<br>
But, being way out of touch, I don't know what actually happens
<br>
today. Perhaps the current operators and current government
watchers
<br>
can answer?:git clone
<a class="moz-txt-link-freetext" href="https://rjmcmahon@git.code.sf.net/p/iperf2/code">https://rjmcmahon@git.code.sf.net/p/iperf2/code</a> iperf2-code
<br>
<br>
1/ How do current switches exert Backpressure to reduce
competing
<br>
traffic flows? Do they still send SQs?
<br>
<br>
2/ How do the current and proposed government regulations treat
the
<br>
different needs of different types of traffic, e.g., "Bulk"
versus
<br>
"Interactive" versus "Normal"? Are Internet carriers permitted
to
<br>
treat traffic types differently? Are they permitted to charge
<br>
different amounts for different types of service?
<br>
<br>
Jack Haverty
<br>
<br>
On 10/15/23 09:45, Dave Taht via Nnagain wrote:
<br>
<blockquote type="cite">For starters I would like to apologize
for cc-ing both nanog and my
<br>
new nn list. (I will add sender filters)
<br>
<br>
A bit more below.
<br>
<br>
On Sun, Oct 15, 2023 at 9:32 AM Tom Beecher
<a class="moz-txt-link-rfc2396E" href="mailto:beecher@beecher.cc"><beecher@beecher.cc></a> wrote:
<br>
<blockquote type="cite">
<blockquote type="cite">So for now, we'll keep paying for
transit to get to the others (since it’s about as much as
transporting IXP from Dallas), and hoping someone at
Google finally sees Houston as more than a third rate city
hanging off of Dallas. Or… someone finally brings a
worthwhile IX to Houston that gets us more than peering to
Kansas City. Yeah, I think the former is more likely. 😊
<br>
</blockquote>
<br>
There is often a chicken/egg scenario here with the
economics. As an eyeball network, your costs to build out
and connect to Dallas are greater than your transit cost, so
you do that. Totally fair.
<br>
<br>
However think about it from the content side. Say I want to
build into to Houston. I have to put routers in, and a bunch
of cache servers, so I have capital outlay , plus opex for
space, power, IX/backhaul/transit costs. That's not cheap,
so there's a lot of calculations that go into it. Is there
enough total eyeball traffic there to make it worth it? Is
saving 8-10ms enough of a performance boost to justify the
spend? What are the long term trends in that market? These
answers are of course different for a company running their
own CDN vs the commercial CDNs.
<br>
<br>
I don't work for Google and obviously don't speak for them,
but I would suspect that they're happy to eat a 8-10ms
performance hit to serve from Dallas , versus the amount of
capital outlay to build out there right now.
<br>
</blockquote>
The three forms of traffic I care most about are voip, gaming,
and
<br>
videoconferencing, which are rewarding to have at lower
latencies.
<br>
When I was a kid, we had switched phone networks, and while
the sound
<br>
quality was poorer than today, the voice latency cross-town
was just
<br>
like "being there". Nowadays we see 500+ms latencies for this
kind of
<br>
traffic.
<br>
<br>
As to how to make calls across town work that well again,
cost-wise, I
<br>
do not know, but the volume of traffic that would be better
served by
<br>
these interconnects quite low, respective to the overall gains
in
<br>
lower latency experiences for them.
<br>
<br>
<br>
<br>
<blockquote type="cite">On Sat, Oct 14, 2023 at 11:47 PM Tim
Burke <a class="moz-txt-link-rfc2396E" href="mailto:tim@mid.net"><tim@mid.net></a> wrote:
<br>
<blockquote type="cite">I would say that a 1Gbit IP transit
in a carrier neutral DC can be had for a good bit less
than $900 on the wholesale market.
<br>
<br>
Sadly, IXP’s are seemingly turning into a pay to play
game, with rates almost costing as much as transit in many
cases after you factor in loop costs.
<br>
<br>
For example, in the Houston market (one of the largest and
fastest growing regions in the US!), we do not have a
major IX, so to get up to Dallas it’s several thousand for
a 100g wave, plus several thousand for a 100g port on one
of those major IXes. Or, a better option, we can get a
100g flat internet transit for just a little bit more.
<br>
<br>
Fortunately, for us as an eyeball network, there are a
good number of major content networks that are allowing
for private peering in markets like Houston for just the
cost of a cross connect and a QSFP if you’re in the right
DC, with Google and some others being the outliers.
<br>
<br>
So for now, we'll keep paying for transit to get to the
others (since it’s about as much as transporting IXP from
Dallas), and hoping someone at Google finally sees Houston
as more than a third rate city hanging off of Dallas. Or…
someone finally brings a worthwhile IX to Houston that
gets us more than peering to Kansas City. Yeah, I think
the former is more likely. 😊
<br>
<br>
See y’all in San Diego this week,
<br>
Tim
<br>
<br>
On Oct 14, 2023, at 18:04, Dave Taht
<a class="moz-txt-link-rfc2396E" href="mailto:dave.taht@gmail.com"><dave.taht@gmail.com></a> wrote:
<br>
<blockquote type="cite">This set of trendlines was very
interesting. Unfortunately the data
<br>
stops in 2015. Does anyone have more recent data?
<br>
<br>
<a class="moz-txt-link-freetext" href="https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php">https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php</a>
<br>
<br>
I believe a gbit circuit that an ISP can resell still
runs at about
<br>
$900 - $1.4k (?) in the usa? How about elsewhere?
<br>
<br>
...
<br>
<br>
I am under the impression that many IXPs remain very
successful,
<br>
states without them suffer, and I also find the concept
of doing micro
<br>
IXPs at the city level, appealing, and now achievable
with cheap gear.
<br>
Finer grained cross connects between telco and ISP and
IXP would lower
<br>
latencies across town quite hugely...
<br>
<br>
PS I hear ARIN is planning on dropping the price for,
and bundling 3
<br>
BGP AS numbers at a time, as of the end of this year,
also.
<br>
<br>
<br>
<br>
--
<br>
Oct 30:
<a class="moz-txt-link-freetext" href="https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html">https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html</a>
<br>
Dave Täht CSO, LibreQos
<br>
</blockquote>
</blockquote>
</blockquote>
<br>
<br>
</blockquote>
<br>
_______________________________________________
<br>
Nnagain mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:Nnagain@lists.bufferbloat.net">Nnagain@lists.bufferbloat.net</a>
<br>
<a class="moz-txt-link-freetext" href="https://lists.bufferbloat.net/listinfo/nnagain">https://lists.bufferbloat.net/listinfo/nnagain</a>
<br>
</blockquote>
_______________________________________________
<br>
Nnagain mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:Nnagain@lists.bufferbloat.net">Nnagain@lists.bufferbloat.net</a>
<br>
<a class="moz-txt-link-freetext" href="https://lists.bufferbloat.net/listinfo/nnagain">https://lists.bufferbloat.net/listinfo/nnagain</a>
<br>
</blockquote>
</body>
</html>