From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from bobcat.rjmcmahon.com (bobcat.rjmcmahon.com [45.33.58.123]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 965323B29D for ; Mon, 16 Oct 2023 13:37:31 -0400 (EDT) Received: from [192.168.1.96] (c-69-181-111-171.hsd1.ca.comcast.net [69.181.111.171]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bobcat.rjmcmahon.com (Postfix) with ESMTPSA id 9006E1B26F; Mon, 16 Oct 2023 10:37:30 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 bobcat.rjmcmahon.com 9006E1B26F DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rjmcmahon.com; s=bobcat; t=1697477850; bh=yW159kIJUHXnW28266nw0Hz1tBn3ZHhOaI56HSPXRXk=; h=In-Reply-To:References:Subject:From:Date:To:CC:From; b=nJWNWZqTscwNbDtEUN3K3CeIQoQRAhBdCCfmGexKtMJPGjjIFy38+4qLbc78wrX+l /P3PmvM+CWfHTk4aJT+obICuvVf/PEORV/Pr4tqGSKJxcZAYtH4mEeqI36MV3QAWcv fKdIQg/zJBPdBqkk387pvHnM/E8P0XdYkeeCFuK8= In-Reply-To: References: <75bb9781-c7e5-4b15-a5ea-f825eb8c96a4@3kitty.org> X-Referenced-Uid: 0001150d567702d5 Thread-Topic: Re: [NNagain] The history of congestion control on the internet User-Agent: Android X-Is-Generated-Message-Id: true MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----1CHJ6OVW802LRQAZX6XKM757DM5BXD" Content-Transfer-Encoding: 7bit From: Robert McMahon Date: Mon, 16 Oct 2023 10:37:29 -0700 To: Spencer Sevilla via Nnagain Message-ID: Subject: Re: [NNagain] The history of congestion control on the internet X-BeenThere: nnagain@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: =?utf-8?q?Network_Neutrality_is_back!_Let=C2=B4s_make_the_technical_aspects_heard_this_time!?= List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Oct 2023 17:37:31 -0000 ------1CHJ6OVW802LRQAZX6XKM757DM5BXD Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 We in semiconductors test TCP on hundreds of test rigs and multiple operati= ng systems, use statistical process controls before sending our chips, and = support sw to system integrators or device manufacturers=2E Then, those com= panies do their work and test more before shipping to their customers=2E Th= ere is a lot of testing baked in now=2E If not, billions of TCP state machi= nes wouldn't function nor interoperate=2E And people then wouldn't buy thes= e products as networks are essential infrastructure=2E Reading the code do= esn't really work for this class of problem=2E Code reviews are good in hum= an processes, but escapes are quite high=2E Computers have to engage too an= d are now doing the heavy lifting Bob On Oct 16, 2023, 10:21 AM, at 10:2= 1 AM, Spencer Sevilla via Nnagain wrote= : >That Flakeway tool makes me think of an early version of the Chaos >Monk= ey=2E To that note, Apple maintains a developer tool called Network >Link C= onditioner that does a good job simulating reduced network >performance=2E = > >> On Oct 15, 2023, at 23:30, Jack Haverty via Nnagain > wrote: >> >> Even back in 1978, I didn't think Source Q= uench would work=2E I >recall that I was trying to adapt my TCP2=2E5 Unix= implementation to >become TCP4, and I asked what my TCP should do if it se= nt the first IP >datagram to open a TCP connection and received a Source Qu= ench=2E It >wasn't clear at all how I should "slow down"=2E Other TCP im= plementors >took the receipt of an SQ as an indication that a datagram they= had >sent had been discarded, so the obvious reaction for user satisfactio= n >was to retransmit immediately=2E Slowing down would simply degrade >th= eir user's experience=2E >> >> Glad to hear SQ is gone=2E I hope whateve= r replaced it works=2E >> >> There's some confusion about the Arpanet=2E = The Arpanet was known as a >"packet switching network", but it had lots of = internal mechanisms that >essentially created virtual circuits between atta= ched computers=2E >Every packet sent in to the network by a user computer= came out at the >destination intact, in order, and not duplicated or lost= =2E The Arpanet >switches even had a hardware mechanism for flow control;= a switch could >halt data transfer from a user computer when necessary=2E = During the >80s, the Arpanet evolved to have an X=2E25 interface, and ope= rated as a >true "virtual circuit" provider=2E Even in the Defense Data N= etwork >(DDN), the network delivered a virtual circuit service=2E The atta= ched >users' computers had TCP, but the TCP didn't need to deal with most o= f >the network behavior that TCP was designed to handle=2E Congestion was = >similarly handled by internal Arpanet mechanisms (there were several >tech= nical reports from BBN to ARPA with details)=2E I don't remember >any ti= me that "an explicit ack for every packet was ripped out of the >arpanet" N= one of those events happened when two TCP computers were >connected to the = Arpanet=2E >> >> The Internet grew up around the Arpanet, which provided m= ost of the >wide-area connectivity through the mid-80s=2E Since the Arpan= et >provided the same "reliable byte stream" behavior as TCP provided, and = >most user computers were physically attached to an Arpanet switch, it >was= n't obvious how to test a TCP implementation, to see how well it >dealt wit= h reordering, duplication, dropping, or corruption of IP >datagrams=2E >= > >> We (at BBN) actually had to implement a software package called a >"F= lakeway", which ran on a SparcStation=2E Using a "feature" of >Ethernets = and ARP (some would call it a vulnerability), the Flakeway >could insert it= self invisibly in the stream of datagrams between any >two computers on tha= t LAN (e=2Eg=2E, between a user computer and the >gateway/router providing = a path to other sites)=2E The Flakeway could >then simulate "real" Interne= t behavior by dropping, duplicating, >reordering, mangling, delaying, or ot= herwise interfering with the flow=2E >That was extremely useful in testing = and diagnosing TCP >implementations=2E >> >> I understand that there has b= een a lot of technical work over the >years, and lots of new mechanisms def= ined for use in the Internet to >solve various problems=2E But one issue t= hat has not been addressed -- >how do you know whether or not some such mec= hanism has actually been >implemented, and configured correctly, in the mil= lions of devices that >are now using TCP (and UDP, IP, etc=2E)? AFAIK, the= re's no way to tell >unless you can examine the actual code=2E >> >> The I= nternet, and TCP, was an experiment=2E One aspect of that >experiment invo= lved changing the traditional role of a network >"switch", and moving mecha= nisms for flow control, error control, and >other mechanisms used to create= a "virtual circuit" behavior=2E Instead >of being implemented inside som= e switching equipment, TCP's mechanisms >are implemented inside users' comp= uters=2E That was a significant >break from traditional network architec= ture=2E >> >> I didn't realize it at the time, but now, with users' device= s being >uncountable handheld or desktop computers rather than huge racks i= n >relatively few data centers, moving all those mechanisms from switches >= to users' computers significantly complicates the system design and >especi= ally operation=2E >> >> That may be one of the more important results of t= he long-running >experiment=2E >> >> Jack Haverty >> >> On 10/15/23 18:39= , Dave Taht wrote: >>> It is wonderful to have your original perspectives h= ere, Jack=2E >>> >>> But please, everyone, before a major subject change, = change the >subject? >>> >>> Jack's email conflates a few things that prob= ably deserve other >>> threads for them=2E One is VGV - great acronym! Anot= her is about the >>> "Placeholders" of TTL, and TOS=2E The last is the hist= ory of >congestion >>> control - and it's future! As being a part of the mo= st recent >episodes >>> here I have written extensively on the subject, but= what I most like >>> to point people to is my fun talks trying to make it = more accessible >>> like this one at apnic >>> >https://blog=2Eapnic=2Enet/= 2020/01/22/bufferbloat-may-be-solved-but-its-not-over-yet/ >>> or my more r= ecent one at tti/vanguard=2E >>> >>> Most recently one of our LibreQos cli= ents has been collecting 10ms >>> samples and movies of what real-world res= idential traffic actually >>> looks like: >>> >>> https://www=2Eyoutube=2E= com/@trendaltoews7143 >>> >>> And it is my hope that that conveys intuitio= n to others=2E=2E=2E as >compared >>> to speedtest traffic, which prove not= hing about the actual behaviors >>> of VGV traffic, which I ranted about he= re: >>> https://blog=2Ecerowrt=2Eorg/post/speedtests/ - I am glad that thes= e >>> speedtests now have latency under load reports almost universally, >b= ut >>> see the rant for more detail=2E >>> >>> Most people only have a pic= ture of traffic in the large, over 5 >minute >>> intervals, which behaves q= uite differently, or a pre-conception that >>> backpressure actually exists= across the internet=2E It doesn't=2E An >>> explicit ack for every packet = was ripped out of the arpanet as >costing >>> too much time=2E Wifi, to som= e extent, recreates the arpanet problem >by >>> having explicit acks on the= local loop that are repeated until by >god >>> the packet comes through, u= sually without exponential backoff=2E >>> >>> We have some really amazing = encoding schemes now - I do not >understand >>> how starlink works without = retries for example, an my grip on 5G's >>> encodings is non-existent, exce= pt knowing it is the most >bufferbloated >>> of all our technologies=2E >>>= >>> =2E=2E=2E >>> >>> Anyway, my hope for this list is that we come up w= ith useful >technical >>> feedback to the powers-that-be that want to regul= ate the internet >>> under some title ii provisions, and I certainly hope w= e can make >>> strides towards fixing bufferbloat along the way! There are = many >other >>> issues=2E Let's talk about those instead! >>> >>> But=2E= =2E=2E >>> =2E=2E=2E >>> >>> In "brief" response to the notes below - sour= ce quench died due to >>> easy ddos, AQMs from RED (1992) until codel (2012= ) struggled with >>> measuring the wrong things ( Kathie's updated paper on= red in a >>> different light: https://pollere=2Enet/Codel=2Ehtml ), SFQ wa= s adopted >by >>> many devices, WRR used in others, ARED I think is common = in juniper >>> boxes, fq_codel is pretty much the default now for most of l= inux, >and >>> I helped write CAKE=2E >>> >>> TCPs evolved from reno to ve= gas to cubic to bbr and the paper on BBR >>> is excellent: https://research= =2Egoogle/pubs/pub45646/ as is len >>> kleinrock's monograph on it=2E Howev= er problems with self congestion >and >>> excessive packet loss were observ= ed, and after entering the ietf >>> process, is now in it's 3rd revision, w= hich looks pretty good=2E >>> >>> Hardware pause frames in ethernet are of= ten available, there are all >>> kinds of specialized new hardware flow con= trol standards in 802=2E1, a >>> new more centralized controller in wifi7 >= >> >>> To this day I have no idea how infiniband works=2E Or how ATM was >= >> supposed to work=2E I have a good grip on wifi up to version 6, and >the= >>> work we did on wifi is in use now on a lot of wifi gear like >openwrt,= >>> eero and evenroute, and I am proudest of all my teams' work on >>> ach= ieving airtime fairness, and better scheduling described in this >>> paper = here: https://www=2Ecs=2Ekau=2Ese/tohojo/airtime-fairness/ for wifi >>> and= MOS to die for=2E >>> >>> There is new work on this thing called L4S, whi= ch has a bunch of >RFCs >>> for it, leverages multi-bit DCTCP style ECN and= is under test by >apple >>> and comcast, it is discussed on tsvwg list a l= ot=2E I encourage users >to >>> jump in on the comcast/apple beta, and oper= ators to at least read >>> this: https://datatracker=2Eietf=2Eorg/doc/draft= -ietf-tsvwg-l4sops/ >>> >>> Knowing that there is a book or three left to = write on this subject >>> that nobody will read is an issue, as is coming = up with an >>> architecture to take packet handling as we know it, to the m= oon and >>> the rest of the solar system, seems kind of difficult=2E >>> >= >> Ideally I would love to be working on that earth-moon architecture >>> r= ather than trying to finish getting stuff we designed in 2012-2016 >>> depl= oyed=2E >>> >>> I am going to pull out a few specific questions from the b= elow and >>> answer separately=2E >>> >>> On Sun, Oct 15, 2023 at 1:00=E2= =80=AFPM Jack Haverty via Nnagain >>> >= wrote: >>>> The "VGV User" (Voic= e, Gaming, Videoconferencing) cares a lot about >>>> latency=2E It's not = just "rewarding" to have lower latencies; high >>>> latencies may make VGV = unusable=2E Average (or "typical") latency >as the >>>> FCC label propose= s isn't a good metric to judge usability=2E A path >which >>>> has high va= riance in latency can be unusable even if the average is >>>> quite low=2E = Having your voice or video or gameplay "break up" >every >>>> minute or s= o when latency spikes to 500 msec makes the "user >experience" >>>> intoler= able=2E >>>> >>>> A few years ago, I ran some simple "ping" tests to help = a friend >who was >>>> trying to use a gaming app=2E My data was only for = one specific path >so >>>> it's anecdotal=2E What I saw was surprising - z= ero data loss, every >>>> datagram was delivered, but occasionally a datagr= am would take up >to 30 >>>> seconds to arrive=2E I didn't have the abilit= y to poke around >inside, but >>>> I suspected it was an experience of "buf= ferbloat", enabled by the >>>> dramatic drop in price of memory over the de= cades=2E >>>> >>>> It's been a long time since I was involved in operating= any part of >the >>>> Internet, so I don't know much about the inner worki= ngs today=2E >Apologies >>>> for my ignorance=2E=2E=2E=2E >>>> >>>> There = was a scenario in the early days of the Internet for which we >>>> struggle= d to find a technical solution=2E Imagine some node in the >bowels >>>> of= the network, with 3 connected "circuits" to some other nodes=2E >On two >= >>> of those inputs, traffic is arriving to be forwarded out the third >>>>= circuit=2E The incoming flows are significantly more than the >outgoing >= >>> path can accept=2E >>>> >>>> What happens? How is "backpressure" gen= erated so that the >incoming >>>> flows are reduced to the point that the o= utgoing circuit can handle >the >>>> traffic? >>>> >>>> About 45 years ago= , while we were defining TCPV4, we struggled with >this >>>> issue, but did= n't find any consensus solutions=2E So "placeholder" >>>> mechanisms were = defined in TCPV4, to be replaced as research >continued >>>> and found a go= od solution=2E >>>> >>>> In that "placeholder" scheme, the "Source Quench"= (SQ) IP message >was >>>> defined; it was to be sent by a switching node b= ack toward the >sender of >>>> any datagram that had to be discarded becaus= e there wasn't any >place to >>>> put it=2E >>>> >>>> In addition, the TOS= (Type Of Service) and TTL (Time To Live) >fields >>>> were defined in IP= =2E >>>> >>>> TOS would allow the sender to distinguish datagrams based on= their >>>> needs=2E For example, we thought "Interactive" service might b= e >needed >>>> for VGV traffic, where timeliness of delivery was most impor= tant=2E >>>> "Bulk" service might be useful for activities like file transf= ers, >>>> backups, et al=2E "Normal" service might now mean activities li= ke >using >>>> the Web=2E >>>> >>>> The TTL field was an attempt to inform= each switching node about >the >>>> "expiration date" for a datagram=2E = If a node somehow knew that a >>>> particular datagram was unlikely to reac= h its destination in time >to be >>>> useful (such as a video datagram for = a frame that has already been >>>> displayed), the node could, and should, = discard that datagram to >free up >>>> resources for useful traffic=2E Sad= ly we had no mechanisms for >measuring >>>> delay, either in transit or in = queuing, so TTL was defined in terms >of >>>> "hops", which is not an accur= ate proxy for time=2E But it's all we >had=2E >>>> >>>> Part of the comp= lexity was that the "flow control" mechanism of the >>>> Internet had put m= uch of the mechanism in the users' computers' TCP >>>> implementations, rat= her than the switches which handle only IP=2E >Without >>>> mechanisms in t= he users' computers, all a switch could do is order >more >>>> circuits, an= d add more memory to the switches for queuing=2E Perhaps >that >>>> led to= "bufferbloat"=2E >>>> >>>> So TOS, SQ, and TTL were all placeholders, for= some mechanism in a >>>> future release that would introduce a "real" form= of Backpressure >and >>>> the ability to handle different types of traffic= =2E Meanwhile, >these >>>> rudimentary mechanisms would provide some flow= control=2E Hopefully >the >>>> users' computers sending the flows would re= spond to the SQ >backpressure, >>>> and switches would prioritize traffic u= sing the TTL and TOS >information=2E >>>> >>>> But, being way out of touch= , I don't know what actually happens >today=2E >>>> Perhaps the current ope= rators and current government watchers can >answer?: >>> I would love moe f= eedback about RED''s deployment at scale in >particular=2E >>> >>>> 1/ How= do current switches exert Backpressure to reduce competing >>>> traffic f= lows? Do they still send SQs? >>> Some send various forms of hardware flow= control, an ethernet pause >>> frame derivative >>> >>>> 2/ How do the cu= rrent and proposed government regulations treat the >>>> different needs of= different types of traffic, e=2Eg=2E, "Bulk" versus >>>> "Interactive" ver= sus "Normal"? Are Internet carriers permitted to >treat >>>> traffic types= differently? Are they permitted to charge different >>>> amounts for diff= erent types of service? >>> >>>> Jack Haverty >>>> >>>> On 10/15/23 09:45= , Dave Taht via Nnagain wrote: >>>>> For starters I would like to apologize= for cc-ing both nanog and >my >>>>> new nn list=2E (I will add sender filt= ers) >>>>> >>>>> A bit more below=2E >>>>> >>>>> On Sun, Oct 15, 2023 at = 9:32=E2=80=AFAM Tom Beecher > wrote: >>>>>>> So for now, we'll keep paying for transit to get to t= he others >(since it=E2=80=99s about as much as transporting IXP from Dalla= s), and hoping >someone at Google finally sees Houston as more than a third= rate city >hanging off of Dallas=2E Or=E2=80=A6 someone finally brings a w= orthwhile IX to >Houston that gets us more than peering to Kansas City=2E Y= eah, I think >the former is more likely=2E =F0=9F=98=8A >>>>>> There is oft= en a chicken/egg scenario here with the economics=2E As >an eyeball network= , your costs to build out and connect to Dallas are >greater than your tran= sit cost, so you do that=2E Totally fair=2E >>>>>> >>>>>> However think ab= out it from the content side=2E Say I want to build >into to Houston=2E I h= ave to put routers in, and a bunch of cache >servers, so I have capital out= lay , plus opex for space, power, >IX/backhaul/transit costs=2E That's not = cheap, so there's a lot of >calculations that go into it=2E Is there enough= total eyeball traffic >there to make it worth it? Is saving 8-10ms enough = of a performance >boost to justify the spend? What are the long term trends= in that >market? These answers are of course different for a company runni= ng >their own CDN vs the commercial CDNs=2E >>>>>> >>>>>> I don't work for= Google and obviously don't speak for them, but I >would suspect that they'= re happy to eat a 8-10ms performance hit to >serve from Dallas , versus the= amount of capital outlay to build out >there right now=2E >>>>> The three = forms of traffic I care most about are voip, gaming, and >>>>> videoconfere= ncing, which are rewarding to have at lower latencies=2E >>>>> When I was a= kid, we had switched phone networks, and while the >sound >>>>> quality wa= s poorer than today, the voice latency cross-town was >just >>>>> like "bei= ng there"=2E Nowadays we see 500+ms latencies for this kind >of >>>>> traff= ic=2E >>>>> >>>>> As to how to make calls across town work that well again= , >cost-wise, I >>>>> do not know, but the volume of traffic that would be = better served >by >>>>> these interconnects quite low, respective to the ov= erall gains in >>>>> lower latency experiences for them=2E >>>>> >>>>> >>= >>> >>>>>> On Sat, Oct 14, 2023 at 11:47=E2=80=AFPM Tim Burke > wrote: >>>>>>> I would say that a 1Gbit IP tran= sit in a carrier neutral DC can >be had for a good bit less than $900 on th= e wholesale market=2E >>>>>>> >>>>>>> Sadly, IXP=E2=80=99s are seemingly t= urning into a pay to play game, with >rates almost costing as much as trans= it in many cases after you factor >in loop costs=2E >>>>>>> >>>>>>> For ex= ample, in the Houston market (one of the largest and >fastest growing regio= ns in the US!), we do not have a major IX, so to >get up to Dallas it=E2=80= =99s several thousand for a 100g wave, plus several >thousand for a 100g po= rt on one of those major IXes=2E Or, a better >option, we can get a 100g fl= at internet transit for just a little bit >more=2E >>>>>>> >>>>>>> Fortuna= tely, for us as an eyeball network, there are a good >number of major conte= nt networks that are allowing for private peering >in markets like Houston = for just the cost of a cross connect and a QSFP >if you=E2=80=99re in the r= ight DC, with Google and some others being the >outliers=2E >>>>>>> >>>>>>= > So for now, we'll keep paying for transit to get to the others >(since it= =E2=80=99s about as much as transporting IXP from Dallas), and hoping >some= one at Google finally sees Houston as more than a third rate city >hanging = off of Dallas=2E Or=E2=80=A6 someone finally brings a worthwhile IX to >Hou= ston that gets us more than peering to Kansas City=2E Yeah, I think >the fo= rmer is more likely=2E =F0=9F=98=8A >>>>>>> >>>>>>> See y=E2=80=99all in S= an Diego this week, >>>>>>> Tim >>>>>>> >>>>>>> On Oct 14, 2023, at 18:04,= Dave Taht > wrot= e: >>>>>>>> =EF=BB=BFThis set of trendlines was very interesting=2E Unfortu= nately the >data >>>>>>>> stops in 2015=2E Does anyone have more recent dat= a? >>>>>>>> >>>>>>>> >https://drpeering=2Enet/white-papers/Internet-Transi= t-Pricing-Historical-And-Projected=2Ephp >>>>>>>> >>>>>>>> I believe a gbi= t circuit that an ISP can resell still runs at >about >>>>>>>> $900 - $1=2E= 4k (?) in the usa? How about elsewhere? >>>>>>>> >>>>>>>> =2E=2E=2E >>>>>>= >> >>>>>>>> I am under the impression that many IXPs remain very >successf= ul, >>>>>>>> states without them suffer, and I also find the concept of >do= ing micro >>>>>>>> IXPs at the city level, appealing, and now achievable wi= th >cheap gear=2E >>>>>>>> Finer grained cross connects between telco and I= SP and IXP >would lower >>>>>>>> latencies across town quite hugely=2E=2E= =2E >>>>>>>> >>>>>>>> PS I hear ARIN is planning on dropping the price for= , and >bundling 3 >>>>>>>> BGP AS numbers at a time, as of the end of this = year, also=2E >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Oct 30: >h= ttps://netdevconf=2Einfo/0x17/news/the-maestro-and-the-music-bof=2Ehtml >>>= >>>>> Dave T=C3=A4ht CSO, LibreQos >>>>> >>>> ____________________________= ___________________ >>>> Nnagain mailing list >>>> Nnagain@lists=2Ebufferbl= oat=2Enet > >>>> https://lists=2E= bufferbloat=2Enet/listinfo/nnagain >>> >>> >> >> _______________________= ________________________ >> Nnagain mailing list >> Nnagain@lists=2Ebufferb= loat=2Enet >> https://lists=2Ebufferbloat=2Enet/listinfo/nnagain ------1CHJ6OVW802LRQAZX6XKM757DM5BXD Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
We in semicond= uctors test TCP on hundreds of test rigs and multiple operating systems, us= e statistical process controls before sending our chips, and support sw to = system integrators or device manufacturers=2E Then, those companies do thei= r work and test more before shipping to their customers=2E There is a lot o= f testing baked in now=2E If not, billions of TCP state machines wouldn't f= unction nor interoperate=2E And people then wouldn't buy these products as = networks are essential infrastructure=2E

Re= ading the code doesn't really work for this class of problem=2E Code review= s are good in human processes, but escapes are quite high=2E Computers have= to engage too and are now doing the heavy lifting

Bob
On Oct 16, 2023, at 10:21 A= M, Spencer Sevilla via Nnagain <nnagain@lists=2Ebufferbloat=2Enet> wr= ote:
That Flake= way tool makes me think of an early version of the Chaos Monkey=2E To that = note, Apple maintains a developer tool called Network Link Conditioner that= does a good job simulating reduced network performance=2E

=
On Oct 15, 2023, at 23:30, Jack Hav= erty via Nnagain <nnagain@lists=2Ebufferbloat=2Enet> wrote:
=
Even ba= ck in 1978, I didn't think Source Quench would work=2E   I recall= that I was trying to adapt my TCP2=2E5 Unix implementation to become TCP4,= and I asked what my TCP should do if it sent the first IP datagram to open= a TCP connection and received a Source Quench=2E  It wasn't clear at = all how I should "slow down"=2E   Other TCP implementors took the= receipt of an SQ as an indication that a datagram they had sent had been d= iscarded, so the obvious reaction for user satisfaction was to retransmit i= mmediately=2E   Slowing down would simply degrade their user's ex= perience=2E

Glad to hear SQ is gone=2E   I= hope whatever replaced it works=2E

There's some con= fusion about the Arpanet=2E  The Arpanet was known as a "packet switch= ing network", but it had lots of internal mechanisms that essentially creat= ed virtual circuits between attached computers=2E   Every packet sent = in to the network by a user computer came out at the destination intact, in= order, and not duplicated or lost=2E   The Arpanet switches even had = a hardware mechanism for flow control; a switch could halt data transfer fr= om a user computer when necessary=2E   During the 80s, the Arpane= t evolved to have an X=2E25 interface, and operated as a true "virtual circ= uit" provider=2E   Even in the Defense Data Network (DDN), the ne= twork delivered a virtual circuit service=2E  The attached users' comp= uters had TCP, but the TCP didn't need to deal with most of the network beh= avior that TCP was designed to handle=2E  Congestion was similarly han= dled by internal Arpanet mechanisms (there were several technical reports f= rom BBN to ARPA with details)=2E    I don't remember any tim= e that "an explicit ack for every pac= ket was ripped out of the arpanet" None of those events happened whe= n two TCP computers were connected to the Arpanet=2E

= The Internet grew up around the Arpanet, which provided most of the wide-a= rea connectivity through the mid-80s=2E   Since the Arpanet provi= ded the same "reliable byte stream" behavior as TCP provided, and most user= computers were physically attached to an Arpanet switch, it wasn't obvious= how to test a TCP implementation, to see how well it dealt with reordering= , duplication, dropping, or corruption of IP datagrams=2E   =

We (at BBN) actually had to implement a software package= called a "Flakeway", which ran on a SparcStation=2E   Using a "f= eature" of Ethernets and ARP (some would call it a vulnerability), the Flak= eway could insert itself invisibly in the stream of datagrams between any t= wo computers on that LAN (e=2Eg=2E, between a user computer and the gateway= /router providing a path to other sites)=2E  The Flakeway could then s= imulate "real" Internet behavior by dropping, duplicating, reordering, mang= ling, delaying, or otherwise interfering with the flow=2E   That = was extremely useful in testing and diagnosing TCP implementations=2E <= br>
I understand that there has been a lot of technical work = over the years, and lots of new mechanisms defined for use in the Internet = to solve various problems=2E  But one issue that has not been addresse= d -- how do you know whether or not some such mechanism has actually been i= mplemented, and configured correctly, in the millions of devices that are n= ow using TCP (and UDP, IP, etc=2E)?  AFAIK, there's no way to tell unl= ess you can examine the actual code=2E

The Internet,= and TCP, was an experiment=2E  One aspect of that experiment involved= changing the traditional role of a network "switch", and moving mechanisms= for flow control, error control, and other mechanisms used to create a "vi= rtual circuit" behavior=2E   Instead of being implemented inside = some switching equipment, TCP's mechanisms are implemented inside users' co= mputers=2E    That was a significant break from traditional = network architecture=2E

I didn't realize it at the t= ime, but now, with users' devices being uncountable handheld or desktop com= puters rather than huge racks in relatively few data centers, moving all th= ose mechanisms from switches to users' computers significantly complicates = the system design and especially operation=2E

That m= ay be one of the more important results of the long-running experiment=2E =

Jack Haverty

On 10/15/23 18:39, Dave Taht wrote:
= 
It is wonderful to have your original perspectives here, Jack=2E
=

But please, everyone, before a major subject change, change the subject?

=
Jack's email conflates a few things that probably deserve other
threads for=
 them=2E One is VGV - great acronym! Another is about the
"Placeholders" of=
 TTL, and TOS=2E The last is the history of congestion
control - and it's f=
uture! As being a part of the most recent episodes
here I have written exte=
nsively on the subject, but what I most like
to point people to is my fun t=
alks trying to make it more accessible
like this one at apnic
https://blog=2Eapnic=2Enet/2020/0=
1/22/bufferbloat-may-be-solved-but-its-not-over-yet/
or my more recent =
one at tti/vanguard=2E

Most recently one of our LibreQos clients has been =
collecting 10ms
samples and movies of what real-world residential traffic a=
ctually
looks like:

https://www=2Eyoutube=2Ecom/@trendaltoe=
ws7143

And it is my hope that that conveys intuition to others=2E=2E=
=2E as compared
to speedtest traffic, which prove nothing about the actual =
behaviors
of VGV traffic, which I ranted about here:
https:/=
/blog=2Ecerowrt=2Eorg/post/speedtests/ - I am glad that these
speedtest=
s now have latency under load reports almost universally, but
see the rant =
for more detail=2E

Most people only have a picture of traffic in the large=
, over 5 minute
intervals, which behaves quite differently, or a pre-concep=
tion that
backpressure actually exists across the internet=2E It doesn't=2E=
 An
explicit ack for every packet was ripped out of the arpanet as costing
=
too much time=2E Wifi, to some extent, recreates the arpanet problem by
hav=
ing explicit acks on the local loop that are repeated until by god
the pack=
et comes through, usually without exponential backoff=2E

We have some real=
ly amazing encoding schemes now - I do not understand
how starlink works wi=
thout retries for example, an my grip on 5G's
encodings is non-existent, ex=
cept knowing it is the most bufferbloated
of all our technologies=2E

=2E=
=2E=2E

Anyway, my hope for this list is that we come up with useful techni=
cal
feedback to the powers-that-be that want to regulate the internet
under=
 some title ii provisions, and I certainly hope we can make
strides towards=
 fixing bufferbloat along the way! There are many other
issues=2E Let's tal=
k about those instead!

But=2E=2E=2E
=2E=2E=2E

In "brief" response to the =
notes below - source quench died due to
easy ddos, AQMs from RED (1992) unt=
il codel (2012) struggled with
measuring the wrong things ( Kathie's update=
d paper on red in a
different light: https://pollere=2Enet/Codel=2Ehtml=
 ), SFQ was adopted by
------1CHJ6OVW802LRQAZX6XKM757DM5BXD--