From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp98.iad3a.emailsrvr.com (smtp98.iad3a.emailsrvr.com [173.203.187.98]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 7A2113B2A4 for ; Sat, 2 May 2020 19:23:36 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=g001.emailsrvr.com; s=20190322-9u7zjiwi; t=1588461816; bh=z0yrFaL7eMbONrnWn77BijNgCpiQ1mbGP4LG2U8uBzE=; h=Date:Subject:From:To:From; b=nUFtJDXzjjarKXsvz4eqJqD4rFrsYVybT0N/F2W3ucOQ7jg1RidKLKy5JlF4103+W +nHKiofxegvELVnlqx74f/J5mN4PX8WEeNqS7qEPQm97fJ0s63CuIzVEnbgRznuxgP zYkMvEJqlD+3WM0jX/fgJOGlvvKmVeHREwvf67hk= Received: from app66.wa-webapps.iad3a (relay-webapps.rsapps.net [172.27.255.140]) by smtp37.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id ED28E3D89; Sat, 2 May 2020 19:23:35 -0400 (EDT) X-Sender-Id: dpreed@deepplum.com Received: from app66.wa-webapps.iad3a (relay-webapps.rsapps.net [172.27.255.140]) by 0.0.0.0:25 (trex/5.7.12); Sat, 02 May 2020 19:23:36 -0400 Received: from deepplum.com (localhost.localdomain [127.0.0.1]) by app66.wa-webapps.iad3a (Postfix) with ESMTP id 530B56010B; Sat, 2 May 2020 19:23:33 -0400 (EDT) Received: by apps.rackspace.com (Authenticated sender: dpreed@deepplum.com, from: dpreed@deepplum.com) with HTTP; Sat, 2 May 2020 19:23:33 -0400 (EDT) X-Auth-ID: dpreed@deepplum.com Date: Sat, 2 May 2020 19:23:33 -0400 (EDT) From: "David P. Reed" To: "Sergey Fedorov" Cc: "Dave Taht" , "Benjamin Cronce" , "Michael Richardson" , "Jannie Hanekom" , "bloat" , "Cake List" , "Make-Wifi-fast" MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_20200502192333000000_52261" Importance: Normal X-Priority: 3 (Normal) X-Type: html In-Reply-To: References: <05410663-5E50-4CF5-8ADE-3BBB985E32B1@gmx.de> <24457.1588370840@localhost> <013601d6201f$04c7db50$0e5791f0$@hanekom.net> <1588441128.39172345@apps.rackspace.com> Message-ID: <1588461813.33611935@apps.rackspace.com> X-Mailer: webmail/17.3.10-RC X-Classification-ID: 2888d7f0-a44a-469e-ae3b-c1140c5b52e3-1-1 Subject: Re: [Cake] [Make-wifi-fast] [Bloat] dslreports is no longer free X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 May 2020 23:23:36 -0000 ------=_20200502192333000000_52261 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =0ASergey - I wasn't assuming anything about fast.com. The document you sha= red wasn't clear about the methodology's details here. Others sadly, have a= ctually used ICMP pings in the way I described. I was making a generic comm= ent of concern.=0A =0AThat said, it sounds like what you are doing is reall= y helpful (esp. given that your measure is aimed at end user experiential q= ualities).=0A =0AGood luck!=0A =0A =0AOn Saturday, May 2, 2020 3:00pm, "Ser= gey Fedorov" said:=0A=0A=0A=0A=0A=0ADave, thanks for= sharing interesting thoughts and context. I am still a bit worried about p= roperly defining "latency under load" for a NAT routed situation. If the te= st is based on ICMP Ping packets *from the server*, it will NOT be measuri= ng the full path latency, and if the potential congestion is in the uplink = path from the access provider's residential box to the access provider's ro= uter/switch, it will NOT measure congestion caused by bufferbloat reliably = on either side, since the bufferbloat will be outside the ICMP Ping path.= =0A =0AI realize that a browser based speed test has to be basically run fr= om the "server" end, because browsers are not that good at time measurement= on a packet basis. However, there are ways to solve this and avoid the ICM= P Ping issue, with a cooperative server.=0AThis erroneously assumes that [ = fast.com ]( http://fast.com ) measures latency from the server side. It doe= s not. The measurements are done from the client, over http, with a paralle= l connection(s) to the same or similar set of servers, by sending empty req= uests over a previously established connection (you can see that in the bro= wser web inspector).=0AIt should be noted that the value is not precisely t= he "RTT on a TCP/UDP flow that is loaded with traffic", but "user delay giv= en the presence of heavy parallel flows". With that, some of the challenges= you mentioned do not apply.=0AIn line with another point I've shared earli= er - the goal is to measure and explain the user experience, not to be a di= agnostic tool showing internal transport metrics.=0A=0A=0A=0A=0A=0A=0ASERGE= Y FEDOROV=0ADirector of Engineering=0A[ sfedorov@netflix.com ]( mailto:sfed= orov@netflix.com )=0A121 Albright Way | Los Gatos, CA 95032=0A=0A=0AOn Sat,= May 2, 2020 at 10:38 AM David P. Reed <[ dpreed@deepplum.com ]( mailto:dpr= eed@deepplum.com )> wrote:=0AI am still a bit worried about properly defini= ng "latency under load" for a NAT routed situation. If the test is based on= ICMP Ping packets *from the server*, it will NOT be measuring the full pa= th latency, and if the potential congestion is in the uplink path from the = access provider's residential box to the access provider's router/switch, i= t will NOT measure congestion caused by bufferbloat reliably on either side= , since the bufferbloat will be outside the ICMP Ping path.=0A =0AI realize= that a browser based speed test has to be basically run from the "server" = end, because browsers are not that good at time measurement on a packet bas= is. However, there are ways to solve this and avoid the ICMP Ping issue, wi= th a cooperative server.=0A =0AI once built a test that fixed this issue re= asonably well. It carefully created a TCP based RTT measurement channel (ov= er HTTP) that made the echo have to traverse the whole end-to-end path, whi= ch is the best and only way to accurately define lag under load from the us= er's perspective. The client end of an unloaded TCP connection can depend o= n TCP (properly prepared by getting it past slowstart) to generate a single= packet response.=0A =0AThis "TCP ping" is thus compatible with getting the= end-to-end measurement on the server end of a true RTT.=0A =0AIt's like tc= p-traceroute tool, in that it tricks anyone in the middle boxes into thinki= ng this is a real, serious packet, not an optional low priority packet.=0A = =0AThe same issue comes up with non-browser-based techniques for measuring = true lag-under-load.=0A =0ANow as we move HTTP to QUIC, this actually gets = easier to do.=0A =0AOne other opportunity I haven't explored, but which is = pregnant with potential is the use of WebRTC, which runs over UDP internall= y. Since JavaScript has direct access to create WebRTC connections (multipl= e ones), this makes detailed testing in the browser quite reasonable.=0A = =0AAnd the time measurements can resolve well below 100 microseconds, if th= e JS is based on modern JIT compilation (Chrome, Firefox, Edge all compile = to machine code speed if the code is restricted and in a loop). Then again,= there is Web Assembly if you want to write C code that runs in the brower = fast. WebAssembly is a low level language that compiles to machine code in = the browser execution, and still has access to all the browser networking f= acilities.=0A =0AOn Saturday, May 2, 2020 12:52pm, "Dave Taht" <[ dave.taht= @gmail.com ]( mailto:dave.taht@gmail.com )> said:=0A=0A=0A=0A> On Sat, May = 2, 2020 at 9:37 AM Benjamin Cronce <[ bcronce@gmail.com ]( mailto:bcronce@g= mail.com )> wrote:=0A> >=0A> > > Fast.com reports my unloaded latency as 4m= s, my loaded latency as ~7ms=0A> =0A> I guess one of my questions is that w= ith a switch to BBR netflix is=0A> going to do pretty well. If [ fast.com ]= ( http://fast.com ) is using bbr, well... that=0A> excludes much of the cur= rent side of the internet.=0A> =0A> > For download, I show 6ms unloaded and= 6-7 loaded. But for upload the loaded=0A> shows as 7-8 and I see it blip u= pwards of 12ms. But I am no longer using any=0A> traffic shaping. Any anti-= bufferbloat is from my ISP. A graph of the bloat would=0A> be nice.=0A> =0A= > The tests do need to last a fairly long time.=0A> =0A> > On Sat, May 2, 2= 020 at 9:51 AM Jannie Hanekom <[ jannie@hanekom.net ]( mailto:jannie@haneko= m.net )>=0A> wrote:=0A> >>=0A> >> Michael Richardson <[ mcr@sandelman.ca ](= mailto:mcr@sandelman.ca )>:=0A> >> > Does it find/use my nearest Netflix c= ache?=0A> >>=0A> >> Thankfully, it appears so. The DSLReports bloat test wa= s interesting,=0A> but=0A> >> the jitter on the ~240ms base latency from So= uth Africa (and other parts=0A> of=0A> >> the world) was significant enough= that the figures returned were often=0A> >> unreliable and largely unusabl= e - at least in my experience.=0A> >>=0A> >> Fast.com reports my unloaded l= atency as 4ms, my loaded latency as ~7ms=0A> and=0A> >> mentions servers lo= cated in local cities. I finally have a test I can=0A> share=0A> >> with lo= cal non-technical people!=0A> >>=0A> >> (Agreed, upload test would be nice,= but this is a huge step forward from=0A> >> what I had access to before.)= =0A> >>=0A> >> Jannie Hanekom=0A> >>=0A> >> _______________________________= ________________=0A> >> Cake mailing list=0A> >> [ Cake@lists.bufferbloat.n= et ]( mailto:Cake@lists.bufferbloat.net )=0A> >> [ https://lists.bufferbloa= t.net/listinfo/cake ]( https://lists.bufferbloat.net/listinfo/cake )=0A> >= =0A> > _______________________________________________=0A> > Cake mailing l= ist=0A> > [ Cake@lists.bufferbloat.net ]( mailto:Cake@lists.bufferbloat.net= )=0A> > [ https://lists.bufferbloat.net/listinfo/cake ]( https://lists.buf= ferbloat.net/listinfo/cake )=0A> =0A> =0A> =0A> --=0A> Make Music, Not War= =0A> =0A> Dave T=C3=A4ht=0A> CTO, TekLibre, LLC=0A> [ http://www.teklibre.c= om ]( http://www.teklibre.com )=0A> Tel: 1-831-435-0729=0A> _______________= ________________________________=0A> Cake mailing list=0A> [ Cake@lists.buf= ferbloat.net ]( mailto:Cake@lists.bufferbloat.net )=0A> [ https://lists.buf= ferbloat.net/listinfo/cake ]( https://lists.bufferbloat.net/listinfo/cake )= =0A> ------=_20200502192333000000_52261 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Sergey - I wasn't assu= ming anything about fast.com. The document you shared wasn't clear about th= e methodology's details here. Others sadly, have actually used ICMP pings i= n the way I described. I was making a generic comment of concern.

=0A

 

=0A

That said, it sounds li= ke what you are doing is really helpful (esp. given that your measure is ai= med at end user experiential qualities).

=0A

 <= /p>=0A

Good luck!

=0A

 

= =0A

 

=0A

On Saturday, May = 2, 2020 3:00pm, "Sergey Fedorov" <sfedorov@netflix.com> said:

=0A
=0A
=0A
=0A=
Dave, thanks for sharing interesting thoughts and context. 
= =0A
I am still a bit worried = about properly defining "latency under load" for a NAT routed situation. If= the test is based on ICMP Ping packets *from the server*,  it will NO= T be measuring the full path latency, and if the potential congestion is in= the uplink path from the access provider's residential box to the access p= rovider's router/switch, it will NOT measure congestion caused by bufferblo= at reliably on either side, since the bufferbloat will be outside the ICMP = Ping path.
 
I realize that a browser based speed test has t= o be basically run from the "server" end, because browsers are not that goo= d at time measurement on a packet basis. However, there are ways to solve t= his and avoid the ICMP Ping issue, with a cooperative server.
= =0A
=0A
This erroneously assumes that = fast.com measures latency from the server side. It does not. The measur= ements are done from the client, over http, with a parallel connection(s) t= o the same or similar set of servers, by sending empty requests over a= previously established connection (you can see that in the browser we= b inspector).
=0A
It should be noted that the value is not precise= ly the "RTT on a TCP/UDP flow that is loaded with traffic", but "user = delay given the presence of heavy parallel flows". With that, some of = the challenges you mentioned do not apply.
=0A
In line with a= nother point I've shared earlier - the goal is to measure and explain = the user experience, not to be a diagnostic tool showing internal transport= metrics.
=0A
=0A
=0A
=0A
=0A
=0A
=0A

SERGEY FED= OROV

=0A

Direct= or of Engineering

=0A

sfedorov@netflix.com

=0A

121 Albright Way | L= os Gatos, CA 95032

=0A= 3D""
= =0A
=0A
=0A
=0A
=0A
=0A
=0A
=0A
On Sat, May 2, 2020 at 10= :38 AM David P. Reed <dpreed@deep= plum.com> wrote:
=0A
=0A

I am still a bit worried about properly definin= g "latency under load" for a NAT routed situation. If the test is based on = ICMP Ping packets *from the server*,  it will NOT be measuring the ful= l path latency, and if the potential congestion is in the uplink path from = the access provider's residential box to the access provider's router/switc= h, it will NOT measure congestion caused by bufferbloat reliably on either = side, since the bufferbloat will be outside the ICMP Ping path.

=0A

 

=0A

I realize that a browser = based speed test has to be basically run from the "server" end, because bro= wsers are not that good at time measurement on a packet basis. However, the= re are ways to solve this and avoid the ICMP Ping issue, with a cooperative= server.

=0A

 

=0A

I onc= e built a test that fixed this issue reasonably well. It carefully created = a TCP based RTT measurement channel (over HTTP) that made the echo have to = traverse the whole end-to-end path, which is the best and only way to accur= ately define lag under load from the user's perspective. The client end of = an unloaded TCP connection can depend on TCP (properly prepared by getting = it past slowstart) to generate a single packet response.

=0A

 

=0A

This "TCP ping" is thus compatib= le with getting the end-to-end measurement on the server end of a true RTT.=

=0A

 

=0A

It's like tcp= -traceroute tool, in that it tricks anyone in the middle boxes into thinkin= g this is a real, serious packet, not an optional low priority packet.

= =0A

 

=0A

The same issue co= mes up with non-browser-based techniques for measuring true lag-under-load.=

=0A

 

=0A

Now as we mov= e HTTP to QUIC, this actually gets easier to do.

=0A

 

=0A

One other opportunity I haven't explored= , but which is pregnant with potential is the use of WebRTC, which runs ove= r UDP internally. Since JavaScript has direct access to create WebRTC conne= ctions (multiple ones), this makes detailed testing in the browser quite re= asonable.

=0A

 

=0A

And = the time measurements can resolve well below 100 microseconds, if the JS is= based on modern JIT compilation (Chrome, Firefox, Edge all compile to mach= ine code speed if the code is restricted and in a loop). Then again, there = is Web Assembly if you want to write C code that runs in the brower fast. W= ebAssembly is a low level language that compiles to machine code in the bro= wser execution, and still has access to all the browser networking faciliti= es.

=0A

 

=0A

On Saturda= y, May 2, 2020 12:52pm, "Dave Taht" <dave.taht@gmail.com> said:

=0A<= div id=3D"gmail-m_-5707858911278770127SafeStyles1588440044">=0A

> On Sat, May 2, 2020 at 9:37 AM Benjamin Cronce <bcronce@gmail.com> wrot= e:
> >
> > > Fast.com reports my unloaded latency = as 4ms, my loaded latency as ~7ms
>
> I guess one of my qu= estions is that with a switch to BBR netflix is
> going to do prett= y well. If fast.com is us= ing bbr, well... that
> excludes much of the current side of the in= ternet.
>
> > For download, I show 6ms unloaded and 6-7= loaded. But for upload the loaded
> shows as 7-8 and I see it blip= upwards of 12ms. But I am no longer using any
> traffic shaping. A= ny anti-bufferbloat is from my ISP. A graph of the bloat would
> be= nice.
>
> The tests do need to last a fairly long time.>
> > On Sat, May 2, 2020 at 9:51 AM Jannie Hanekom <= jannie@hanekom.net<= /a>>
> wrote:
> >>
> >> Michael Rich= ardson <mcr@sandel= man.ca>:
> >> > Does it find/use my nearest Netflix= cache?
> >>
> >> Thankfully, it appears so. Th= e DSLReports bloat test was interesting,
> but
> >> t= he jitter on the ~240ms base latency from South Africa (and other parts
> of
> >> the world) was significant enough that the fig= ures returned were often
> >> unreliable and largely unusable= - at least in my experience.
> >>
> >> Fast.co= m reports my unloaded latency as 4ms, my loaded latency as ~7ms
> a= nd
> >> mentions servers located in local cities. I finally h= ave a test I can
> share
> >> with local non-technica= l people!
> >>
> >> (Agreed, upload test would = be nice, but this is a huge step forward from
> >> what I had= access to before.)
> >>
> >> Jannie Hanekom> >>
> >> _______________________________________= ________
> >> Cake mailing list
> >> Cake@lists.bufferbloa= t.net
> >> https://lists.bufferbloat.net/listinfo/cake=
> >
> > ____________________________________________= ___
> > Cake mailing list
> > Cake@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cake
>
>
>
> --
> Make Music, Not War
>
> Dave T=C3=A4ht
> CTO, TekLibre, LLC
> http://www.teklibre.com
&g= t; Tel: 1-831-435-0729
> __________________________________________= _____
> Cake mailing list
> Cake@lists.bufferbloat.net
> <= a href=3D"https://lists.bufferbloat.net/listinfo/cake" target=3D"_blank">ht= tps://lists.bufferbloat.net/listinfo/cake
>

=0A
=0A=0A
=0A ------=_20200502192333000000_52261--