<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">I know this is a small side note but I felt compelled to speak up in defense of online gaming. I’m not a gamer at all and up till a year or two ago, would have agreed with Dick’s take about benefit to “society as a whole.” However, lately I’ve started hearing some research on the benefits of groups of friends using online games to socialize together, effectively using the game primarily as a group call.<div><br></div><div>There’s also this project, where people have collected banned/censored books into a library in Minecraft. Specifically as a solution to contexts where regulators/censors ban and monitor content through other channels (websites etc) but don’t surveil Minecraft... Presumably because they share Dick’s opinion ;-) <a href="https://www.uncensoredlibrary.com/en">https://www.uncensoredlibrary.com/en</a><div><div><br><blockquote type="cite"><div>On Oct 17, 2023, at 03:26, Sebastian Moeller via Nnagain <nnagain@lists.bufferbloat.net> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">Hi Richard,</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">On Oct 16, 2023, at 20:04, Dick Roy <dickroy@alum.mit.edu> wrote:<br><br>Good points all, Sebastien. How to "trade-off" a fixed capacity amongst many users is ultimately a game theoretic problem when users are allowed to make choices, which is certainly the case here. Secondly, any network that can and does generate "more traffic" (aka overhead such as ACKs NACKs and retries) reduces the capacity of the network, and ultimately can lead to the "user" capacity going to zero! Such is life in the fast lane (aka the internet).<br><br>Lastly, on the issue of low-latency real-time experience, there are many applications that need/want such capabilities that actually have a net benefit to the individuals involved AND to society as a whole. IMO, interactive gaming is NOT one of those.<br></blockquote><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span class="Apple-tab-span" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: pre; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"> </span><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">[SM] Yes, gaming is one obvious example of a class of uses that work best with low latency and low jitter, not necessarily an example for a use-case worthy enough to justify the work required to increase the responsiveness of the internet. Other examples are video conferences, VoIP, in extension of both musical collaboration over the internet, and surprising to some even plain old web browsing (it often needs to first read a page before being able to follow links and load resources, and every read takes at best a single RTT). None of these are inherently beneficial or detrimental to individuals or society, but most can be used to improve the status quo... I would argue that in the last 4 years the relevance of interactive use-cases has been made quite clear to a lot of folks...</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">OK, so now you know I don't engage in these time sinks with no redeeming social value.:)<br></blockquote><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span class="Apple-tab-span" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: pre; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"> </span><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">[SM] Duly noted ;)</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">Since it is not hard to argue that just like power distribution, information exchange/dissemination is "in the public interest", the question becomes "Do we allow any and all forms of information exchange/dissemination over what is becoming something akin to a public utility?" FWIW, I don't know the answer to this question! :)<br></blockquote><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span class="Apple-tab-span" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: pre; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"> </span><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">[SM] This is an interesting question and one (only) tangentially related to network neutrality... it is more related to freedom of speech and limits thereof. Maybe a question for another mailing list? Certainly one meriting a topic change...</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">Regards</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span class="Apple-tab-span" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: pre; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"> </span><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">Sebastian</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br>Cheers,<br><br>RR<br><br>-----Original Message-----<br>From: Sebastian Moeller [mailto:moeller0@gmx.de]<span class="Apple-converted-space"> </span><br>Sent: Monday, October 16, 2023 10:36 AM<br>To: dickroy@alum.mit.edu; Network Neutrality is back! Let´s make the technical aspects heard this time!<br>Subject: Re: [NNagain] transit and peering costs projections<br><br>Hi Richard,<br><br><br><blockquote type="cite">On Oct 16, 2023, at 19:01, Dick Roy via Nnagain <nnagain@lists.bufferbloat.net> wrote:<br><br>Just an observation: ANY type of congestion control that changes application behavior in response to congestion, or predicted congestion (ENC), begs the question "How does throttling of application information exchange rate (aka behavior) affect the user experience and will the user tolerate it?"<span class="Apple-converted-space"> </span><br></blockquote><br><span class="Apple-tab-span" style="white-space: pre;"> </span>[SM] The trade-off here is, if the application does not respond (or rather if no application would respond) we would end up with congestion collapse where no application would gain much of anything as the network busies itself trying to re-transmit dropped packets without making much head way... Simplistic game theory application might imply that individual applications could try to game this, and generally that seems to be true, but we have remedies for that available..<br><br><br><blockquote type="cite"><br>Given any (complex and packet-switched) network topology of interconnected nodes and links, each with possible a different capacity and characteristics, such as the internet today, IMO the two fundamental questions are:<br><br>1) How can a given network be operated/configured so as to maximize aggregate throughput (i.e. achieve its theoretical capacity), and<br>2) What things in the network need to change to increase the throughput (aka parameters in the network with the largest Lagrange multipliers associated with them)?<br></blockquote><br><span class="Apple-tab-span" style="white-space: pre;"> </span>[SM] The thing is we generally know how to maximize (average) throughput, just add (over-)generous amounts of buffering, the problem is that this screws up the other important quality axis, latency... We ideally want low latency and even more low latency variance (aka jitter) AND high throughput... Turns out though that above a certain throughput threshold* many users do not seem to care all that much for more throughput as long as interactive use cases are sufficiently responsive... but high responsiveness requires low latency and low jitter... This is actually a good thing, as that means we do not necessarily aim for 100% utilization (almost requiring deep buffers and hence resulting in compromised latency) but can get away with say 80-90% where shallow buffers will do (or rather where buffer filling stays shallow, there is IMHO still value in having deep buffers for rare events that need it).<br><br><br><br>*) This is not a hard physical law so the exact threshold is not set in stone, but unless one has many parallel users, something in the 20-50 Mbps range is plenty and that is only needed in the "loaded" direction, that is for pure consumers the upload can be thinner, for pure producers the download can be thinner.<br><br><br><blockquote type="cite"><br>I am not an expert in this field,<br></blockquote><br> [SM] Nor am I, I come from the wet-ware side of things so not even soft- or hard-ware ;)<br><br><br><blockquote type="cite">however it seems to me that answers to these questions would be useful, assuming they are not yet available!<br><br>Cheers,<br><br>RR<br><br><br><br>-----Original Message-----<br>From: Nnagain [mailto:nnagain-bounces@lists.bufferbloat.net] On Behalf Of rjmcmahon via Nnagain<br>Sent: Sunday, October 15, 2023 1:39 PM<br>To: Network Neutrality is back! Let´s make the technical aspects heard this time!<br>Cc: rjmcmahon<br>Subject: Re: [NNagain] transit and peering costs projections<br><br>Hi Jack,<br><br>Thanks again for sharing. It's very interesting to me.<br><br>Today, the networks are shifting from capacity constrained to latency<span class="Apple-converted-space"> </span><br>constrained, as can be seen in the IX discussions about how the speed of<span class="Apple-converted-space"> </span><br>light over fiber is too slow even between Houston & Dallas.<br><br>The mitigations against standing queues (which cause bloat today) are:<br><br>o) Shrink the e2e bottleneck queue so it will drop packets in a flow and<span class="Apple-converted-space"> </span><br>TCP will respond to that "signal"<br>o) Use some form of ECN marking where the network forwarding plane<span class="Apple-converted-space"> </span><br>ultimately informs the TCP source state machine so it can slow down or<span class="Apple-converted-space"> </span><br>pace effectively. This can be an earlier feedback signal and, if done<span class="Apple-converted-space"> </span><br>well, can inform the sources to avoid bottleneck queuing. There are<span class="Apple-converted-space"> </span><br>couple of approaches with ECN. Comcast is trialing L4S now which seems<span class="Apple-converted-space"> </span><br>interesting to me as a WiFi test & measurement engineer. The jury is<span class="Apple-converted-space"> </span><br>still out on this and measurements are needed.<br>o) Mitigate source side bloat via TCP_NOTSENT_LOWAT<br><br>The QoS priority approach per congestion is orthogonal by my judgment as<span class="Apple-converted-space"> </span><br>it's typically not supported e2e, many networks will bleach DSCP<span class="Apple-converted-space"> </span><br>markings. And it's really too late by my judgment.<br><br>Also, on clock sync, yes your generation did us both a service and<span class="Apple-converted-space"> </span><br>disservice by getting rid of the PSTN TDM clock ;) So IP networking<span class="Apple-converted-space"> </span><br>devices kinda ignored clock sync, which makes e2e one way delay (OWD)<span class="Apple-converted-space"> </span><br>measurements impossible. Thankfully, the GPS atomic clock is now<span class="Apple-converted-space"> </span><br>available mostly everywhere and many devices use TCXO oscillators so<span class="Apple-converted-space"> </span><br>it's possible to get clock sync and use oscillators that can minimize<span class="Apple-converted-space"> </span><br>drift. I pay $14 for a Rpi4 GPS chip with pulse per second as an<span class="Apple-converted-space"> </span><br>example.<br><br>It seems silly to me that clocks aren't synced to the GPS atomic clock<span class="Apple-converted-space"> </span><br>even if by a proxy even if only for measurement and monitoring.<br><br>Note: As Richard Roy will point out, there really is no such thing as<span class="Apple-converted-space"> </span><br>synchronized clocks across geographies per general relativity - so those<span class="Apple-converted-space"> </span><br>syncing clocks need to keep those effects in mind. I limited the iperf 2<span class="Apple-converted-space"> </span><br>timestamps to microsecond precision in hopes avoiding those issues.<br><br>Note: With WiFi, a packet drop can occur because an intermittent RF<span class="Apple-converted-space"> </span><br>channel condition. TCP can't tell the difference between an RF drop vs a<span class="Apple-converted-space"> </span><br>congested queue drop. That's another reason ECN markings from network<span class="Apple-converted-space"> </span><br>devices may be better than dropped packets.<br><br>Note: I've added some iperf 2 test support around pacing as that seems<span class="Apple-converted-space"> </span><br>to be the direction the industry is heading as networks are less and<span class="Apple-converted-space"> </span><br>less capacity strained and user quality of experience is being driven by<span class="Apple-converted-space"> </span><br>tail latencies. One can also test with the Prague CCA for the L4S<span class="Apple-converted-space"> </span><br>scenarios. (This is a fun project: https://www.l4sgear.com/ and fairly<span class="Apple-converted-space"> </span><br>low cost)<br><br>--fq-rate n[kmgKMG]<br>Set a rate to be used with fair-queuing based socket-level pacing, in<span class="Apple-converted-space"> </span><br>bytes or bits per second. Only available on platforms supporting the<span class="Apple-converted-space"> </span><br>SO_MAX_PACING_RATE socket option. (Note: Here the suffixes indicate<span class="Apple-converted-space"> </span><br>bytes/sec or bits/sec per use of uppercase or lowercase, respectively)<br><br>--fq-rate-step n[kmgKMG]<br>Set a step of rate to be used with fair-queuing based socket-level<span class="Apple-converted-space"> </span><br>pacing, in bytes or bits per second. Step occurs every<span class="Apple-converted-space"> </span><br>fq-rate-step-interval (defaults to one second)<br><br>--fq-rate-step-interval n<br>Time in seconds before stepping the fq-rate<br><br>Bob<br><br>PS. Iperf 2 man page https://iperf2.sourceforge.io/iperf-manpage.html<br><br><blockquote type="cite">The "VGV User" (Voice, Gaming, Videoconferencing) cares a lot about<br>latency. It's not just "rewarding" to have lower latencies; high<br>latencies may make VGV unusable. Average (or "typical") latency as<br>the FCC label proposes isn't a good metric to judge usability. A path<br>which has high variance in latency can be unusable even if the average<br>is quite low. Having your voice or video or gameplay "break up"<br>every minute or so when latency spikes to 500 msec makes the "user<br>experience" intolerable.<br><br>A few years ago, I ran some simple "ping" tests to help a friend who<br>was trying to use a gaming app. My data was only for one specific<br>path so it's anecdotal. What I saw was surprising - zero data loss,<br>every datagram was delivered, but occasionally a datagram would take<br>up to 30 seconds to arrive. I didn't have the ability to poke around<br>inside, but I suspected it was an experience of "bufferbloat", enabled<br>by the dramatic drop in price of memory over the decades.<br><br>It's been a long time since I was involved in operating any part of<br>the Internet, so I don't know much about the inner workings today.<br>Apologies for my ignorance....<br><br>There was a scenario in the early days of the Internet for which we<br>struggled to find a technical solution. Imagine some node in the<br>bowels of the network, with 3 connected "circuits" to some other<br>nodes. On two of those inputs, traffic is arriving to be forwarded<br>out the third circuit. The incoming flows are significantly more than<br>the outgoing path can accept.<br><br>What happens? How is "backpressure" generated so that the incoming<br>flows are reduced to the point that the outgoing circuit can handle<br>the traffic?<br><br>About 45 years ago, while we were defining TCPV4, we struggled with<br>this issue, but didn't find any consensus solutions. So "placeholder"<br>mechanisms were defined in TCPV4, to be replaced as research continued<br>and found a good solution.<br><br>In that "placeholder" scheme, the "Source Quench" (SQ) IP message was<br>defined; it was to be sent by a switching node back toward the sender<br>of any datagram that had to be discarded because there wasn't any<br>place to put it.<br><br>In addition, the TOS (Type Of Service) and TTL (Time To Live) fields<br>were defined in IP.<br><br>TOS would allow the sender to distinguish datagrams based on their<br>needs. For example, we thought "Interactive" service might be needed<br>for VGV traffic, where timeliness of delivery was most important.<span class="Apple-converted-space"> </span><br>"Bulk" service might be useful for activities like file transfers,<br>backups, et al. "Normal" service might now mean activities like<br>using the Web.<br><br>The TTL field was an attempt to inform each switching node about the<br>"expiration date" for a datagram. If a node somehow knew that a<br>particular datagram was unlikely to reach its destination in time to<br>be useful (such as a video datagram for a frame that has already been<br>displayed), the node could, and should, discard that datagram to free<br>up resources for useful traffic. Sadly we had no mechanisms for<br>measuring delay, either in transit or in queuing, so TTL was defined<br>in terms of "hops", which is not an accurate proxy for time. But<br>it's all we had.<br><br>Part of the complexity was that the "flow control" mechanism of the<br>Internet had put much of the mechanism in the users' computers' TCP<br>implementations, rather than the switches which handle only IP.<br>Without mechanisms in the users' computers, all a switch could do is<br>order more circuits, and add more memory to the switches for queuing.<span class="Apple-converted-space"> </span><br>Perhaps that led to "bufferbloat".<br><br>So TOS, SQ, and TTL were all placeholders, for some mechanism in a<br>future release that would introduce a "real" form of Backpressure and<br>the ability to handle different types of traffic. Meanwhile, these<br>rudimentary mechanisms would provide some flow control. Hopefully the<br>users' computers sending the flows would respond to the SQ<br>backpressure, and switches would prioritize traffic using the TTL and<br>TOS information.<br><br>But, being way out of touch, I don't know what actually happens<br>today. Perhaps the current operators and current government watchers<br>can answer?:git clone https://rjmcmahon@git.code.sf.net/p/iperf2/code<span class="Apple-converted-space"> </span><br>iperf2-code<br><br>1/ How do current switches exert Backpressure to reduce competing<br>traffic flows? Do they still send SQs?<br><br>2/ How do the current and proposed government regulations treat the<br>different needs of different types of traffic, e.g., "Bulk" versus<br>"Interactive" versus "Normal"? Are Internet carriers permitted to<br>treat traffic types differently? Are they permitted to charge<br>different amounts for different types of service?<br><br>Jack Haverty<br><br>On 10/15/23 09:45, Dave Taht via Nnagain wrote:<br><blockquote type="cite">For starters I would like to apologize for cc-ing both nanog and my<br>new nn list. (I will add sender filters)<br><br>A bit more below.<br><br>On Sun, Oct 15, 2023 at 9:32 AM Tom Beecher <beecher@beecher.cc><span class="Apple-converted-space"> </span><br>wrote:<br><blockquote type="cite"><blockquote type="cite">So for now, we'll keep paying for transit to get to the others<span class="Apple-converted-space"> </span><br>(since it’s about as much as transporting IXP from Dallas), and<span class="Apple-converted-space"> </span><br>hoping someone at Google finally sees Houston as more than a third<span class="Apple-converted-space"> </span><br>rate city hanging off of Dallas. Or… someone finally brings a<span class="Apple-converted-space"> </span><br>worthwhile IX to Houston that gets us more than peering to Kansas<span class="Apple-converted-space"> </span><br>City. Yeah, I think the former is more likely. 😊<br></blockquote><br>There is often a chicken/egg scenario here with the economics. As an<span class="Apple-converted-space"> </span><br>eyeball network, your costs to build out and connect to Dallas are<span class="Apple-converted-space"> </span><br>greater than your transit cost, so you do that. Totally fair.<br><br>However think about it from the content side. Say I want to build<span class="Apple-converted-space"> </span><br>into to Houston. I have to put routers in, and a bunch of cache<span class="Apple-converted-space"> </span><br>servers, so I have capital outlay , plus opex for space, power,<span class="Apple-converted-space"> </span><br>IX/backhaul/transit costs. That's not cheap, so there's a lot of<span class="Apple-converted-space"> </span><br>calculations that go into it. Is there enough total eyeball traffic<span class="Apple-converted-space"> </span><br>there to make it worth it? Is saving 8-10ms enough of a performance<span class="Apple-converted-space"> </span><br>boost to justify the spend? What are the long term trends in that<span class="Apple-converted-space"> </span><br>market? These answers are of course different for a company running<span class="Apple-converted-space"> </span><br>their own CDN vs the commercial CDNs.<br><br>I don't work for Google and obviously don't speak for them, but I<span class="Apple-converted-space"> </span><br>would suspect that they're happy to eat a 8-10ms performance hit to<span class="Apple-converted-space"> </span><br>serve from Dallas , versus the amount of capital outlay to build out<span class="Apple-converted-space"> </span><br>there right now.<br></blockquote>The three forms of traffic I care most about are voip, gaming, and<br>videoconferencing, which are rewarding to have at lower latencies.<br>When I was a kid, we had switched phone networks, and while the sound<br>quality was poorer than today, the voice latency cross-town was just<br>like "being there". Nowadays we see 500+ms latencies for this kind of<br>traffic.<br><br>As to how to make calls across town work that well again, cost-wise, I<br>do not know, but the volume of traffic that would be better served by<br>these interconnects quite low, respective to the overall gains in<br>lower latency experiences for them.<br><br><br><br><blockquote type="cite">On Sat, Oct 14, 2023 at 11:47 PM Tim Burke <tim@mid.net> wrote:<br><blockquote type="cite">I would say that a 1Gbit IP transit in a carrier neutral DC can be<span class="Apple-converted-space"> </span><br>had for a good bit less than $900 on the wholesale market.<br><br>Sadly, IXP’s are seemingly turning into a pay to play game, with<span class="Apple-converted-space"> </span><br>rates almost costing as much as transit in many cases after you<span class="Apple-converted-space"> </span><br>factor in loop costs.<br><br>For example, in the Houston market (one of the largest and fastest<span class="Apple-converted-space"> </span><br>growing regions in the US!), we do not have a major IX, so to get up<span class="Apple-converted-space"> </span><br>to Dallas it’s several thousand for a 100g wave, plus several<span class="Apple-converted-space"> </span><br>thousand for a 100g port on one of those major IXes. Or, a better<span class="Apple-converted-space"> </span><br>option, we can get a 100g flat internet transit for just a little<span class="Apple-converted-space"> </span><br>bit more.<br><br>Fortunately, for us as an eyeball network, there are a good number<span class="Apple-converted-space"> </span><br>of major content networks that are allowing for private peering in<span class="Apple-converted-space"> </span><br>markets like Houston for just the cost of a cross connect and a QSFP<span class="Apple-converted-space"> </span><br>if you’re in the right DC, with Google and some others being the<span class="Apple-converted-space"> </span><br>outliers.<br><br>So for now, we'll keep paying for transit to get to the others<span class="Apple-converted-space"> </span><br>(since it’s about as much as transporting IXP from Dallas), and<span class="Apple-converted-space"> </span><br>hoping someone at Google finally sees Houston as more than a third<span class="Apple-converted-space"> </span><br>rate city hanging off of Dallas. Or… someone finally brings a<span class="Apple-converted-space"> </span><br>worthwhile IX to Houston that gets us more than peering to Kansas<span class="Apple-converted-space"> </span><br>City. Yeah, I think the former is more likely. 😊<br><br>See y’all in San Diego this week,<br>Tim<br><br>On Oct 14, 2023, at 18:04, Dave Taht <dave.taht@gmail.com> wrote:<br><blockquote type="cite">This set of trendlines was very interesting. Unfortunately the<span class="Apple-converted-space"> </span><br>data<br>stops in 2015. Does anyone have more recent data?<br><br>https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php<br><br>I believe a gbit circuit that an ISP can resell still runs at about<br>$900 - $1.4k (?) in the usa? How about elsewhere?<br><br>...<br><br>I am under the impression that many IXPs remain very successful,<br>states without them suffer, and I also find the concept of doing<span class="Apple-converted-space"> </span><br>micro<br>IXPs at the city level, appealing, and now achievable with cheap<span class="Apple-converted-space"> </span><br>gear.<br>Finer grained cross connects between telco and ISP and IXP would<span class="Apple-converted-space"> </span><br>lower<br>latencies across town quite hugely...<br><br>PS I hear ARIN is planning on dropping the price for, and bundling<span class="Apple-converted-space"> </span><br>3<br>BGP AS numbers at a time, as of the end of this year, also.<br><br><br><br>--<br>Oct 30:<span class="Apple-converted-space"> </span><br>https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html<br>Dave Täht CSO, LibreQos<br></blockquote></blockquote></blockquote><br><br></blockquote><br>_______________________________________________<br>Nnagain mailing list<br>Nnagain@lists.bufferbloat.net<br>https://lists.bufferbloat.net/listinfo/nnagain<br></blockquote>_______________________________________________<br>Nnagain mailing list<br>Nnagain@lists.bufferbloat.net<br>https://lists.bufferbloat.net/listinfo/nnagain<br><br>_______________________________________________<br>Nnagain mailing list<br>Nnagain@lists.bufferbloat.net<br>https://lists.bufferbloat.net/listinfo/nnagain<br></blockquote><br></blockquote><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">Nnagain mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">Nnagain@lists.bufferbloat.net</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">https://lists.bufferbloat.net/listinfo/nnagain</span></div></blockquote></div><br></div></div></body></html>