From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp74.iad3a.emailsrvr.com (smtp74.iad3a.emailsrvr.com [173.203.187.74]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 7C6A33B2A4 for ; Sun, 26 Sep 2021 14:24:03 -0400 (EDT) Received: from app64.wa-webapps.iad3a (relay-webapps.rsapps.net [172.27.255.140]) by smtp18.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id E8D5120D35; Sun, 26 Sep 2021 14:24:02 -0400 (EDT) Received: from deepplum.com (localhost.localdomain [127.0.0.1]) by app64.wa-webapps.iad3a (Postfix) with ESMTP id D5558618D5; Sun, 26 Sep 2021 14:24:02 -0400 (EDT) Received: by apps.rackspace.com (Authenticated sender: dpreed@deepplum.com, from: dpreed@deepplum.com) with HTTP; Sun, 26 Sep 2021 14:24:02 -0400 (EDT) X-Auth-ID: dpreed@deepplum.com Date: Sun, 26 Sep 2021 14:24:02 -0400 (EDT) From: "David P. Reed" To: "Bob McMahon" Cc: "Vint Cerf" , "Steve Crocker" , "=?utf-8?Q?Valdis_Kl=C4=93tnieks?=" , starlink@lists.bufferbloat.net, "Make-Wifi-fast" , "Cake List" , "codel" , "cerowrt-devel" , "bloat" MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_20210926142402000000_22541" Importance: Normal X-Priority: 3 (Normal) X-Type: html In-Reply-To: References: <1625188609.32718319@apps.rackspace.com> <989de0c1-e06c-cda9-ebe6-1f33df8a4c24@candelatech.com> <1625773080.94974089@apps.rackspace.com> <1625859083.09751240@apps.rackspace.com> <257851.1632110422@turing-police> X-Client-IP: 209.6.168.128 Message-ID: <1632680642.869711321@apps.rackspace.com> X-Mailer: webmail/19.0.13-RC X-Classification-ID: 3b0fcce3-004d-4787-8e2e-1093ec843d11-1-1 Subject: Re: [Cerowrt-devel] [Starlink] [Bloat] Little's Law mea culpa, but not invalidating my main point X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Sep 2021 18:24:03 -0000 ------=_20210926142402000000_22541 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =0APretty good list, thanks for putting this together.=0A =0AThe only thing= I'd add, and I'm not able to formulate it very elegantly, is this personal= insight: One that I would research, because it can be a LOT more useful in= the end-to-end control loop than stuff like ECN, L4S, RED, ...=0A =0AFact:= Detecting congestion by allowing a queue to build up is a very lagging ind= icator of incipient congestion in the forwarding system. The delay added to= all paths by that queue buildup slows down the control loop's ability to r= espond by slowing the sources. It's the control loop delay that creates bot= h instability and continued congestion growth.=0AObservation: current forwa= rders forget what they have forwarded as soon as it is transmitted. This lo= ses all the information about incipient congestion and "fairness" among mul= tiple sources. Yet, there is no need to forget recent history at all after = the packets have been transmitted.=0A =0AAn idea I keep proposing is the id= ea of remembering the last K seconds of packets, their flow ids (source and= destination), the arrival time and departure time, and their channel occup= ancy on the outbound shared link. Then using this information to reflect in= cipient congestion information to the flows that need controlling, to be us= ed in their control loops.=0A =0ASo far, no one has taken me up on doing th= e research to try this in the field. Note: the signalling can be simple (se= nding ECN flags on all flows that transit the queue, even though there is n= o backlog, yet, when the queue is empty but transient overload seems likely= ), but the key thing is that we already assume that recent history of pack= ets is predictive of future overflow.=0AThis can be implemented locally on = any routing path that tends to be a bottleneck link. Such as the uplink of = a home network. It should work with TCP as is if the signalling causes wind= ow reduction (at first, just signal by dropping packets prematurely, but if= TCP will handle ECN aggressively - a single ECN mark causing window reduct= ion, then it will help that, too).=0A =0AThe insight is that from an "infor= mation and control theory" perspective, the packets that have already been = forwarded are incredibly valuable for congestion prediction.=0A =0APlease, = if possible, if anyone actually works on this and publishes, give me credit= for suggesting this.=0AJust because I've been suggesting it for about 15 y= ears now, and being ignored. It would be a mitzvah.=0A =0A =0AOn Thursday, = September 23, 2021 1:46pm, "Bob McMahon" said:= =0A=0A=0A=0AHi All,=0AI do appreciate this thread as well. As a test & meas= urement guy here are my conclusions around network performance. Thanks in a= dvance for any comments.=0A=0ACongestion can be mitigated the following way= s=0Ao) Size queues properly to minimize/negate bloat (easier said than done= with tech like WiFi)=0Ao) Use faster links on the service side such that a= queues' service rates exceeds the arrival rate, no congestion even in burs= ts, if possible=0Ao) Drop entries during oversubscribed states (queue proce= ssing can't "speed up" like water flow through a constricted pipe, must dro= p)=0Ao) Identify aggressor flows per congestion if possible=0Ao) Forwarding= planes can signal back the the sources "earlier" to minimize queue build u= ps per a "control loop request" asking sources to pace their writes=0Ao) tr= ansport layers use techniques a la BBR=0Ao) Use "home gateways" that suppor= t tech like FQ_CODEL=0ALatency can be mitigated the following ways=0Ao) Mit= igate or eliminate congestion, particularly around queueing delays=0Ao) End= host apps can use TCP_NOTSENT_LOWAT along with write()/select() to reduce = host sends of "better never than late" messages =0Ao) Move servers closer t= o the clients per fundamental limit of the speed of light (i.e. propagation= delay of energy over the wave guides), a la CDNs=0A(Except if you're a HFT= , separate servers across geography and make sure to have exclusive user ri= ghts over the lowest latency links)=0A=0ATransport control loop(s)=0Ao) Tra= nsport layer control loops are non linear systems so network tooling will s= truggle to emulate "end user experience"=0Ao) 1/2 RTT does not equal OWD us= ed to compute the bandwidth delay product, imbalance and effects need to be= measured=0Ao) forwarding planes signaling congestion to sources wasn't des= igned in TCP originally but the industry trend seems to be to moving toward= s this per things like L4S=0APhotons, radio & antenna design=0Ao) Find expe= rts who have experience & knowledge, e.g. many do here=0Ao) Photons don't r= eally have mass nor size, at least per my limited understanding of particle= physics and QED though, I must admit, came from reading things on the inte= rnet =0A=0ABob=0A=0A=0AOn Mon, Sep 20, 2021 at 7:40 PM Vint Cerf <[ vint@go= ogle.com ]( mailto:vint@google.com )> wrote:=0Asee [ https://mediatrust.com= / ]( https://mediatrust.com/ )=0Av=0A=0A=0AOn Mon, Sep 20, 2021 at 10:28 AM= Steve Crocker <[ steve@shinkuro.com ]( mailto:steve@shinkuro.com )> wrote:= =0A=0ARelated but slightly different: Attached is a slide some of my collea= gues put together a decade ago showing the number of DNS lookups involved i= n displaying CNN's front page.=0ASteve=0A=0A=0AOn Mon, Sep 20, 2021 at 8:18= AM Valdis Kl=C4=93tnieks <[ valdis.kletnieks@vt.edu ]( mailto:valdis.kletn= ieks@vt.edu )> wrote:On Sun, 19 Sep 2021 18:21:56 -0700, Dave Taht said:=0A= > what actually happens during a web page load,=0A=0A I'm pretty sure that= nobody actually understands that anymore, in any=0A more than handwaving l= evels.=0A=0A I have a nice Chrome extension called IPvFoo that actually tra= cks the IP=0A addresses contacted during the load of the displayed page. I'= ll let you make=0A a guess as to how many unique IP addresses were contacte= d during a load=0A of [ https://www.cnn.com ]( https://www.cnn.com )=0A=0A = ...=0A=0A=0A ...=0A=0A=0A ...=0A=0A=0A 145, at least half of which appeared= to be analytics. And that's only the=0A hosts that were contacted by my l= aptop for HTTP, and doesn't count DNS, or=0A load-balancing front ends, or = all the back-end boxes. As I commented over on=0A NANOG, we've gotten to a= point similar to that of AT&T long distance, where 60%=0A of the effort of= connecting a long distance phone call was the cost of=0A accounting and bi= lling for the call.=0A=0A=0A=0A=0A=0A=0A=0A=0A ____________________________= ___________________=0A Starlink mailing list=0A[ Starlink@lists.bufferbloat= .net ]( mailto:Starlink@lists.bufferbloat.net )=0A[ https://lists.bufferblo= at.net/listinfo/starlink ]( https://lists.bufferbloat.net/listinfo/starlink= )_______________________________________________=0A Starlink mailing list= =0A[ Starlink@lists.bufferbloat.net ]( mailto:Starlink@lists.bufferbloat.ne= t )=0A[ https://lists.bufferbloat.net/listinfo/starlink ]( https://lists.bu= fferbloat.net/listinfo/starlink )=0A-- =0A=0A=0A=0APlease send any postal/o= vernight deliveries to:=0AVint Cerf=0A1435 Woodhurst Blvd =0AMcLean, VA 221= 02=0A703-448-0965=0Auntil further notice=0AThis electronic communication an= d the information and any files transmitted with it, or attached to it, are= confidential and are intended solely for the use of the individual or enti= ty to whom it is addressed and may contain information that is confidential= , legally privileged, protected by privacy laws, or otherwise restricted fr= om disclosure to anyone else. If you are not the intended recipient or the = person responsible for delivering the e-mail to the intended recipient, you= are hereby notified that any use, copying, distributing, dissemination, fo= rwarding, printing, or copying of this e-mail is strictly prohibited. If yo= u received this e-mail in error, please return the e-mail to the sender, de= lete it from your computer, and destroy any printed copy of it. ------=_20210926142402000000_22541 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Pretty good list, than= ks for putting this together.

=0A

 

=0A

The only thing I'd add, and I'm not able to formulate it ve= ry elegantly, is this personal insight: One that I would research, because = it can be a LOT more useful in the end-to-end control loop than stuff like = ECN, L4S, RED, ...

=0A

 

=0A

Fact: Detecting congestion by allowing a queue to build up is a very l= agging indicator of incipient congestion in the forwarding system. The dela= y added to all paths by that queue buildup slows down the control loop's ab= ility to respond by slowing the sources. It's the control loop delay that c= reates both instability and continued congestion growth.

=0A

Observation: current forwarders forget what they have forwarded as= soon as it is transmitted. This loses all the information about incipient = congestion and "fairness" among multiple sources. Yet, there is no need to = forget recent history at all after the packets have been transmitted.

= =0A

 

=0A

An idea I keep pr= oposing is the idea of remembering the last K seconds of packets, their flo= w ids (source and destination), the arrival time and departure time, and th= eir channel occupancy on the outbound shared link. Then using this informat= ion to reflect incipient congestion information to the flows that need cont= rolling, to be used in their control loops.

=0A

&nbs= p;

=0A

So far, no one has taken me up on doing the r= esearch to try this in the field. Note: the signalling can be simple (sendi= ng ECN flags on all flows that transit the queue, even though there is no b= acklog, yet, when the queue is empty but transient overload seems likely), = but the key thing is that we already assume that  recent history of pa= ckets is predictive of future overflow.

=0A

This can= be implemented locally on any routing path that tends to be a bottleneck l= ink. Such as the uplink of a home network. It should work with TCP as is if= the signalling causes window reduction (at first, just signal by dropping = packets prematurely, but if TCP will handle ECN aggressively - a single ECN= mark causing window reduction, then it will help that, too).

=0A

 

=0A

The insight is that from an= "information and control theory" perspective, the packets that have alread= y been forwarded are incredibly valuable for congestion prediction.

=0A<= p style=3D"margin:0;padding:0;font-family: arial; font-size: 10pt; overflow= -wrap: break-word;"> 

=0A

Please, if possible, = if anyone actually works on this and publishes, give me credit for suggesti= ng this.

=0A

Just because I've been suggesting it fo= r about 15 years now, and being ignored. It would be a mitzvah.

=0A

 

=0A

 

=0A

On Thursday, September 23, 2021 1:46pm, "Bob McMahon" <bob.mcma= hon@broadcom.com> said:

=0A
=0A
Hi All,=0A
I do appreciate this thread as well. = As a test & measurement guy here are my conclusions around network perf= ormance. Thanks in advance for any comments.

Congestion can be m= itigated the following ways
=0A
o) Size queues properly to mi= nimize/negate bloat (easier said than done with tech like WiFi)
=0Ao) Use faster links on the service side such that a queues' service rates= exceeds the arrival rate, no congestion even in bursts, if possi= ble
o) Drop entries during oversubscribed states (queue processing can= 't "speed up" like water flow through a constricted pipe, must drop)
= =0A
o) Identify aggressor flows per congestion if possible
= =0A
o) Forwarding planes can signal back the the sources "earlier" to m= inimize queue build ups per a "control loop request" asking sources to= pace their writes
=0A
o) transport layers use techniques a la BBR=
=0A
o) Use "home gateways" that support tech like FQ_CODEL
= =0A
Latency can be mitigated the following ways
=0A
o) Mi= tigate or eliminate congestion, particularly around queueing delays=0A
o) End host apps can use TCP_NOTSENT_LOWAT along with write()/se= lect() to reduce host sends of "better never than late" messages 
o) Move servers closer to the clients per fundamental limit of the speed o= f light (i.e. propagation delay of energy over the wave guides), a la = CDNs
=0A
(Except if you're a HFT, separate servers across geograph= y and make sure to have exclusive user rights over the lowest latency = links)

Transport control loop(s)
=0A
o) Transport layer= control loops are non linear systems so network tooling will struggle to e= mulate "end user experience"
=0A
o) 1/2 RTT does not equal OW= D used to compute the bandwidth delay product, imbalance and effects need t= o be measured
=0A
o) forwarding planes signaling congestion to sou= rces wasn't designed in TCP originally but the industry trend seems to be t= o moving towards this per things like L4S
=0A
Photons, radio &= antenna design
o) Find experts who have experience & knowledge, e= .g. many do here
o) Photons don't really have mass nor size, at least&= nbsp;per my limited understanding of particle physics and QED though, I mus= t admit, came from reading things on the internet 

Bob=0A
=0A
=0A
=0A
On Mon, Sep 20, 2021 at 7:40 PM Vint Cerf <vint@google.com> wrote:
= =0A
=0A
see <= a href=3D"https://mediatrust.com/" target=3D"_blank">https://mediatrust.com= /=0A
v
=0A
=0A
=0A
=0AOn Mon, Sep 20, 2021 at 10:28 AM Steve Cr= ocker <steve@shi= nkuro.com> wrote:
=0A
=0A
=0A
Related but slight= ly different: Attached is a slide some of my colleagues put together a deca= de ago showing the number of DNS lookups involved in displaying CNN's front= page.
=0A
Steve
=0A
=0A
= =0A
=0A
On M= on, Sep 20, 2021 at 8:18 AM Valdis Kl=C4=93tnieks <valdis.kletnieks@vt.edu> wro= te:
=0A
On Sun, 19 Sep 2= 021 18:21:56 -0700, Dave Taht said:
> what actually happens during= a web page load,

I'm pretty sure that nobody actually understa= nds that anymore, in any
more than handwaving levels.

I h= ave a nice Chrome extension called IPvFoo that actually tracks the IP
= addresses contacted during the load of the displayed page. I'll let you ma= ke
a guess as to how many unique IP addresses were contacted during a= load
of https://www.cnn.com

...


...


...


145, at least half of which appeared to= be analytics.  And that's only the
hosts that were contacted by= my laptop for HTTP, and doesn't count DNS, or
load-balancing front e= nds, or all the back-end boxes.  As I commented over on
NANOG, w= e've gotten to a point similar to that of AT&T long distance, where 60%=
of the effort of connecting a long distance phone call was the cost = of
accounting and billing for the call.




=



_______________________________________________ Starlink mailing list
Starlink@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink=0A
=0A_______________________________________________
Starlink= mailing list
Starlink@lists.bufferbloat.net
h= ttps://lists.bufferbloat.net/listinfo/starlink
=0A
=0A=
--
=0A
=0A
=0A
Please send = any postal/overnight deliveries to:
=0A
Vint Cerf
=0A
143= 5 Woodhurst Blvd 
=0A
McLean, VA 22102
=0A
703-448-0= 965
=0A
until further notice
=0A
=0A
=0A=0A
=0A
This electronic communication and the information an= d any files transmitted with it, or attached to it, are confidential and ar= e intended solely for the use of the individual or entity to whom it is add= ressed and may contain information that is confidential, legally privileged= , protected by privacy laws, or otherwise restricted from disclosure to any= one else. If you are not the intended recipient or the person responsible f= or delivering the e-mail to the intended recipient, you are hereby notified= that any use, copying, distributing, dissemination, forwarding, printing, = or copying of this e-mail is strictly prohibited. If you received this e-ma= il in error, please return the e-mail to the sender, delete it from your co= mputer, and destroy any printed copy of it.
 ------=_20210926142402000000_22541--