From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io1-xd30.google.com (mail-io1-xd30.google.com [IPv6:2607:f8b0:4864:20::d30]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id E83703CB37 for ; Sat, 2 May 2020 15:01:22 -0400 (EDT) Received: by mail-io1-xd30.google.com with SMTP id i19so8073042ioh.12 for ; Sat, 02 May 2020 12:01:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netflix.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=KaZxBE/HrJEvLXPV6r92pF0fykwY/5qIG1znDxnHOww=; b=U1UdKkjP8+gQOL5NJmqWhiLgDTP63rp2RddNRUo/Ry0kDeAMNPNX/rkXN1qRDNlTuM 0p0sfiVgD3CmSvBmVyYPwb3PCN0niRDkNOLwCHJMcW3tSwvmZEwdSFWPPLJ3NyKlQuWz Y6vgjsm0Z/m6JeSJkbIxwaqys2Yict6DcZh7g= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=KaZxBE/HrJEvLXPV6r92pF0fykwY/5qIG1znDxnHOww=; b=MV6azHXYV2oisS46mlXpkFiI9j3y9TyK9szkbAONkYm+sHsrGDtGsg14AuaMb1sCKA v5V/UXym5dTTirLQrCEzOBVHZmBsunHmqqmlJq39A0TowPVkhDq9osVTPzufNqMy518J MfiQHyNrgsfSsK9EyfYOn7G8FLInLW2Ql3DwmprYaCN7ANNj8L30UF9zDF+hk6Z7CgMH OR3I/QU87iGw0yupO85DpxTRG5ukLknYO/13aB1fvrFsYGoTtcZep8z73bWgftLrip6i c1klzxwtumevGbR+G0diQKe8lqKPYBZQfoGtROs2ZINI9mpEYv5m6P9zedalnK28C0vk WieQ== X-Gm-Message-State: AGi0PuYQBajwOBscz+HpV8hF18veSKwnkAt8liq5TNDV1sAOY+FLVGlS 8KTme5Okhab2ajtmBkRUuTDeyjZ74po2BuYEhBZIpg== X-Google-Smtp-Source: APiQypLNiaGiouj0twrbERht93ADf1FdQvywpXKCuCf50C2Dra2Jv87ovdNJS3MFbUncB2Jny6ydYDAKu69pigcdH+s= X-Received: by 2002:a5d:9604:: with SMTP id w4mr9289313iol.105.1588446082229; Sat, 02 May 2020 12:01:22 -0700 (PDT) MIME-Version: 1.0 References: <05410663-5E50-4CF5-8ADE-3BBB985E32B1@gmx.de> <24457.1588370840@localhost> <013601d6201f$04c7db50$0e5791f0$@hanekom.net> <1588441128.39172345@apps.rackspace.com> In-Reply-To: <1588441128.39172345@apps.rackspace.com> From: Sergey Fedorov Date: Sat, 2 May 2020 12:00:45 -0700 Message-ID: Subject: Re: [Cake] [Make-wifi-fast] [Bloat] dslreports is no longer free To: "David P. Reed" Cc: Dave Taht , Benjamin Cronce , Michael Richardson , Jannie Hanekom , bloat , Cake List , Make-Wifi-fast Content-Type: multipart/alternative; boundary="0000000000007a295b05a4aeec89" X-List-Received-Date: Sat, 02 May 2020 19:01:23 -0000 --0000000000007a295b05a4aeec89 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Dave, thanks for sharing interesting thoughts and context. > I am still a bit worried about properly defining "latency under load" for > a NAT routed situation. If the test is based on ICMP Ping packets *from t= he > server*, it will NOT be measuring the full path latency, and if the > potential congestion is in the uplink path from the access provider's > residential box to the access provider's router/switch, it will NOT measu= re > congestion caused by bufferbloat reliably on either side, since the > bufferbloat will be outside the ICMP Ping path. > > I realize that a browser based speed test has to be basically run from th= e > "server" end, because browsers are not that good at time measurement on a > packet basis. However, there are ways to solve this and avoid the ICMP Pi= ng > issue, with a cooperative server. This erroneously assumes that fast.com measures latency from the server side. It does not. The measurements are done from the client, over http, with a parallel connection(s) to the same or similar set of servers, by sending empty requests over a previously established connection (you can see that in the browser web inspector). It should be noted that the value is not precisely the "RTT on a TCP/UDP flow that is loaded with traffic", but "user delay given the presence of heavy parallel flows". With that, some of the challenges you mentioned do not apply. In line with another point I've shared earlier - the goal is to measure and explain the user experience, not to be a diagnostic tool showing internal transport metrics. SERGEY FEDOROV Director of Engineering sfedorov@netflix.com 121 Albright Way | Los Gatos, CA 95032 On Sat, May 2, 2020 at 10:38 AM David P. Reed wrote: > I am still a bit worried about properly defining "latency under load" for > a NAT routed situation. If the test is based on ICMP Ping packets *from t= he > server*, it will NOT be measuring the full path latency, and if the > potential congestion is in the uplink path from the access provider's > residential box to the access provider's router/switch, it will NOT measu= re > congestion caused by bufferbloat reliably on either side, since the > bufferbloat will be outside the ICMP Ping path. > > > > I realize that a browser based speed test has to be basically run from th= e > "server" end, because browsers are not that good at time measurement on a > packet basis. However, there are ways to solve this and avoid the ICMP Pi= ng > issue, with a cooperative server. > > > > I once built a test that fixed this issue reasonably well. It carefully > created a TCP based RTT measurement channel (over HTTP) that made the ech= o > have to traverse the whole end-to-end path, which is the best and only wa= y > to accurately define lag under load from the user's perspective. The clie= nt > end of an unloaded TCP connection can depend on TCP (properly prepared by > getting it past slowstart) to generate a single packet response. > > > > This "TCP ping" is thus compatible with getting the end-to-end measuremen= t > on the server end of a true RTT. > > > > It's like tcp-traceroute tool, in that it tricks anyone in the middle > boxes into thinking this is a real, serious packet, not an optional low > priority packet. > > > > The same issue comes up with non-browser-based techniques for measuring > true lag-under-load. > > > > Now as we move HTTP to QUIC, this actually gets easier to do. > > > > One other opportunity I haven't explored, but which is pregnant with > potential is the use of WebRTC, which runs over UDP internally. Since > JavaScript has direct access to create WebRTC connections (multiple ones)= , > this makes detailed testing in the browser quite reasonable. > > > > And the time measurements can resolve well below 100 microseconds, if the > JS is based on modern JIT compilation (Chrome, Firefox, Edge all compile = to > machine code speed if the code is restricted and in a loop). Then again, > there is Web Assembly if you want to write C code that runs in the brower > fast. WebAssembly is a low level language that compiles to machine code i= n > the browser execution, and still has access to all the browser networking > facilities. > > > > On Saturday, May 2, 2020 12:52pm, "Dave Taht" said: > > > On Sat, May 2, 2020 at 9:37 AM Benjamin Cronce > wrote: > > > > > > > Fast.com reports my unloaded latency as 4ms, my loaded latency as > ~7ms > > > > I guess one of my questions is that with a switch to BBR netflix is > > going to do pretty well. If fast.com is using bbr, well... that > > excludes much of the current side of the internet. > > > > > For download, I show 6ms unloaded and 6-7 loaded. But for upload the > loaded > > shows as 7-8 and I see it blip upwards of 12ms. But I am no longer usin= g > any > > traffic shaping. Any anti-bufferbloat is from my ISP. A graph of the > bloat would > > be nice. > > > > The tests do need to last a fairly long time. > > > > > On Sat, May 2, 2020 at 9:51 AM Jannie Hanekom > > wrote: > > >> > > >> Michael Richardson : > > >> > Does it find/use my nearest Netflix cache? > > >> > > >> Thankfully, it appears so. The DSLReports bloat test was interesting= , > > but > > >> the jitter on the ~240ms base latency from South Africa (and other > parts > > of > > >> the world) was significant enough that the figures returned were oft= en > > >> unreliable and largely unusable - at least in my experience. > > >> > > >> Fast.com reports my unloaded latency as 4ms, my loaded latency as ~7= ms > > and > > >> mentions servers located in local cities. I finally have a test I ca= n > > share > > >> with local non-technical people! > > >> > > >> (Agreed, upload test would be nice, but this is a huge step forward > from > > >> what I had access to before.) > > >> > > >> Jannie Hanekom > > >> > > >> _______________________________________________ > > >> Cake mailing list > > >> Cake@lists.bufferbloat.net > > >> https://lists.bufferbloat.net/listinfo/cake > > > > > > _______________________________________________ > > > Cake mailing list > > > Cake@lists.bufferbloat.net > > > https://lists.bufferbloat.net/listinfo/cake > > > > > > > > -- > > Make Music, Not War > > > > Dave T=C3=A4ht > > CTO, TekLibre, LLC > > http://www.teklibre.com > > Tel: 1-831-435-0729 > > _______________________________________________ > > Cake mailing list > > Cake@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/cake > > > --0000000000007a295b05a4aeec89 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Dave, thanks for sharing interesting thoughts an= d context.=C2=A0
I a= m still a bit worried about properly defining "latency under load"= ; for a NAT routed situation. If the test is based on ICMP Ping packets *fr= om the server*,=C2=A0 it will NOT be measuring the full path latency, and i= f the potential congestion is in the uplink path from the access provider&#= 39;s residential box to the access provider's router/switch, it will NO= T measure congestion caused by bufferbloat reliably on either side, since t= he bufferbloat will be outside the ICMP Ping path.
=C2=A0
I realize t= hat a browser based speed test has to be basically run from the "serve= r" end, because browsers are not that good at time measurement on a pa= cket basis. However, there are ways to solve this and avoid the ICMP Ping i= ssue, with a cooperative server.
This err= oneously assumes that fast.com measures lat= ency from the server side. It does not. The measurements are done from the = client, over http, with a parallel connection(s) to the same or similar set= of servers,=C2=A0by sending empty requests over a previously established= =C2=A0connection (you can see that in the browser web inspector).
It should be noted that the value is not precisely the "RTT on a TCP/= UDP=C2=A0flow that is loaded with traffic", but "user delay=C2=A0= given the presence of heavy parallel flows". With that, some of the ch= allenges=C2=A0you mentioned do not apply.
In line with another po= int I've=C2=A0shared earlier - the goal is to measure and explain the u= ser experience, not to be a diagnostic tool showing internal transport metr= ics.

=

SERGEY FEDOROV

Director of Engineering

sfedorov@netflix.co= m

121 Albright Way | Los Gatos, CA 95032


=


On Sat, May 2, 2020 at 10:38 AM David P. Reed= <dpreed@deepplum.com> wro= te:

I am still a bit worried about properly defining "late= ncy under load" for a NAT routed situation. If the test is based on IC= MP Ping packets *from the server*,=C2=A0 it will NOT be measuring the full = path latency, and if the potential congestion is in the uplink path from th= e access provider's residential box to the access provider's router= /switch, it will NOT measure congestion caused by bufferbloat reliably on e= ither side, since the bufferbloat will be outside the ICMP Ping path.

=C2=A0=

I real= ize that a browser based speed test has to be basically run from the "= server" end, because browsers are not that good at time measurement on= a packet basis. However, there are ways to solve this and avoid the ICMP P= ing issue, with a cooperative server.

=C2=A0=

I once= built a test that fixed this issue reasonably well. It carefully created a= TCP based RTT measurement channel (over HTTP) that made the echo have to t= raverse the whole end-to-end path, which is the best and only way to accura= tely define lag under load from the user's perspective. The client end = of an unloaded TCP connection can depend on TCP (properly prepared by getti= ng it past slowstart) to generate a single packet response.

=C2=A0=

This &= quot;TCP ping" is thus compatible with getting the end-to-end measurem= ent on the server end of a true RTT.

=C2=A0=

It'= ;s like tcp-traceroute tool, in that it tricks anyone in the middle boxes i= nto thinking this is a real, serious packet, not an optional low priority p= acket.

=C2=A0=

The sa= me issue comes up with non-browser-based techniques for measuring true lag-= under-load.

=C2=A0=

Now as= we move HTTP to QUIC, this actually gets easier to do.

=C2=A0=

One ot= her opportunity I haven't explored, but which is pregnant with potentia= l is the use of WebRTC, which runs over UDP internally. Since JavaScript ha= s direct access to create WebRTC connections (multiple ones), this makes de= tailed testing in the browser quite reasonable.

=C2=A0=

And th= e time measurements can resolve well below 100 microseconds, if the JS is b= ased on modern JIT compilation (Chrome, Firefox, Edge all compile to machin= e code speed if the code is restricted and in a loop). Then again, there is= Web Assembly if you want to write C code that runs in the brower fast. Web= Assembly is a low level language that compiles to machine code in the brows= er execution, and still has access to all the browser networking facilities= .

=C2=A0=

On Sat= urday, May 2, 2020 12:52pm, "Dave Taht" <dave.taht@gmail.com> said:

> O= n Sat, May 2, 2020 at 9:37 AM Benjamin Cronce <bcronce@gmail.com> wrote:
> >=
> > > Fast.com reports my unloaded latency as 4ms, my loaded l= atency as ~7ms
>
> I guess one of my questions is that with a = switch to BBR netflix is
> going to do pretty well. If fast.com is using bbr, well... that
= > excludes much of the current side of the internet.
>
> &g= t; For download, I show 6ms unloaded and 6-7 loaded. But for upload the loa= ded
> shows as 7-8 and I see it blip upwards of 12ms. But I am no lon= ger using any
> traffic shaping. Any anti-bufferbloat is from my ISP.= A graph of the bloat would
> be nice.
>
> The tests do = need to last a fairly long time.
>
> > On Sat, May 2, 2020 = at 9:51 AM Jannie Hanekom <jannie@hanekom.net>
> wrote:
> >>
&= gt; >> Michael Richardson <mcr@sandelman.ca>:
> >> > Does it find= /use my nearest Netflix cache?
> >>
> >> Thankfully= , it appears so. The DSLReports bloat test was interesting,
> but
= > >> the jitter on the ~240ms base latency from South Africa (and = other parts
> of
> >> the world) was significant enough t= hat the figures returned were often
> >> unreliable and largely= unusable - at least in my experience.
> >>
> >> Fa= st.com reports my unloaded latency as 4ms, my loaded latency as ~7ms
>= ; and
> >> mentions servers located in local cities. I finally = have a test I can
> share
> >> with local non-technical p= eople!
> >>
> >> (Agreed, upload test would be nice= , but this is a huge step forward from
> >> what I had access t= o before.)
> >>
> >> Jannie Hanekom
> >>= ;
> >> _______________________________________________
> = >> Cake mailing list
> >> Cake@lists.bufferbloat.net
> >= ;> https://lists.bufferbloat.net/listinfo/cake
> >
> &= gt; _______________________________________________
> > Cake maili= ng list
> > Cake@lists.bufferbloat.net
> > https://lists.buffer= bloat.net/listinfo/cake
>
>
>
> --
> M= ake Music, Not War
>
> Dave T=C3=A4ht
> CTO, TekLibre, L= LC
> http://www= .teklibre.com
> Tel: 1-831-435-0729
> _____________________= __________________________
> Cake mailing list
> Cake@lists.bufferbloat.ne= t
> https://lists.bufferbloat.net/listinfo/cake
>

--0000000000007a295b05a4aeec89--