From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io1-xd2f.google.com (mail-io1-xd2f.google.com [IPv6:2607:f8b0:4864:20::d2f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 7DD493CB35; Mon, 4 May 2020 20:10:52 -0400 (EDT) Received: by mail-io1-xd2f.google.com with SMTP id w4so131680ioc.6; Mon, 04 May 2020 17:10:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=WcGbmYidIJgj0dc0hf7fyGdRePJR52JpDAYukmsqpNg=; b=UiCbuOioaDp4RZf/JGzjtgEsFdVQQBx4v6VWRYfaFYuWVcs/PJhl3Zn9B6wJla/A92 ZLe1acFq4T8mPf1DZ5pVmVacBGilvo9HeqR6byc7yrGec2txzEHBFxqc6nvHdXiqw7QI 4mCBYwRpPXvOiM8g4ZFgakz3iag0i70K5dgEPThCC+m5pKXE3YpsTBH1hZF0C7JcPA+S +M/Ff0tMFS9HH1KbCIgLgMqwS6Rc3p8tYq0yHGhUBSYTvG3+2JkXZ4fEST8Z5JBtAikr HnZNITdppRgz2IoIFv6If/zGIPHiJnpIzm9Rs8O3xSYTt9GOsy6Nq27U+Uy+j4VDCg/6 DimA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=WcGbmYidIJgj0dc0hf7fyGdRePJR52JpDAYukmsqpNg=; b=Ks8Udo8G1IgGWdiTU5Y49nckUiNK1TkDTCIl5rsBSNA+zHMNvsb31vLNcWFuZrTZbF NbDZkWraf+5C7Zpa5lOpgCyQSoXI+F5xBv+GkhRE8YAjHqBwlngPCi1wANe3WbrThO/3 77qkcy1Wl4eDu7pimljDEHaWcQ4aDYRrcPPPYK4oK+uwCI6wGhL7ZIs3PshnLKA2FHaa D6RwkWFcpVkAurAwBsR3+wOiw66eBJwvL24GaDBiFMCIrKmvrmP9yf07h3pn3NpnLZIf seG1RSkskrqj0IsHDG1C7iA19OtMamrBAQAiLGAtpLTQQuRl9JlorZy0WYLzmWAVFznw smmQ== X-Gm-Message-State: AGi0Pua1vrR7npbr2entXj/o6tq99sl0Mams5b85AgLiCejfM85GgBWY v52Cd75Ag4nbGVzotUD+c9IeYUDXUoYKehlrnrs= X-Google-Smtp-Source: APiQypKitRyGumAmlQ+BH7YYmaDYZzWV/kFtjKzGzHIrNDTOFkUcEUmPzptX2nYpg7GYoVnkRVmrGe4K8zzt4hGZz6E= X-Received: by 2002:a5d:8786:: with SMTP id f6mr928711ion.100.1588637451856; Mon, 04 May 2020 17:10:51 -0700 (PDT) MIME-Version: 1.0 References: <1588518416.66682155@apps.rackspace.com> In-Reply-To: From: Dave Taht Date: Mon, 4 May 2020 17:10:35 -0700 Message-ID: To: Bob McMahon Cc: Sergey Fedorov , Make-Wifi-fast , bloat , "David P. Reed" , Cake List , Jannie Hanekom Content-Type: multipart/alternative; boundary="000000000000feb38205a4db7ab1" Subject: Re: [Make-wifi-fast] [Bloat] [Cake] dslreports is no longer free X-BeenThere: make-wifi-fast@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 May 2020 00:10:52 -0000 --000000000000feb38205a4db7ab1 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, May 4, 2020 at 5:03 PM Bob McMahon via Bloat < bloat@lists.bufferbloat.net> wrote: > > > > ---------- Forwarded message ---------- > From: Bob McMahon > To: Sergey Fedorov > Cc: "David P. Reed" , Michael Richardson < > mcr@sandelman.ca>, Make-Wifi-fast , > bloat , Cake List , > Jannie Hanekom > Bcc: > Date: Mon, 4 May 2020 17:03:02 -0700 > Subject: Re: [Make-wifi-fast] [Cake] [Bloat] dslreports is no longer free > Sorry for being a bit off topic but we find average latency not all that > useful. A full CDF is. The next best is a box plot with outliers which > can be presented parametrically as a few numbers. Most customers want > visibility into the PDF tail. > yea! Try never to discard the outliers anywhere in the core tests. I always point to this as a place where, if you stop thinking the noise is noise, caused by bird droppings in your receiver, you find structure where you thought it never existed before. https://theconversation.com/the-cmb-how-an-accidental-discovery-became-the-= key-to-understanding-the-universe-45126 A lot of times, just plotting the patterns of the outliers can be interesting. It's often a lot of bird droppings to sort through! Also, we're moving to socket write() to read() latencies for our end/end > measurements (using the iperf 2.0.14 --trip-times option assumes > synchronized clocks.). We also now measure TCP connects (3WHS) as well. > One thing that may or may not help is the sock_sent_lowat option. I note that with SSL so common, it helps to be using that, rather than straight tcp, so it's closer to a 5WHS Yes, generating the crypto exchange costs time, but with that as a baseline with the extra round trips... > Finally, since we have trip times and the application write rates we can > compute the amount of "end/end bytes in queue" per Little's law. > I will reserve comment on littles law for a bit. For fault isolation, in-band network telemetry (or something similar) can > be useful. https://p4.org/assets/INT-current-spec.pdf > > Bob > > On Mon, May 4, 2020 at 10:05 AM Sergey Fedorov via Make-wifi-fast < > make-wifi-fast@lists.bufferbloat.net> wrote: > >> >> >> >> ---------- Forwarded message ---------- >> From: Sergey Fedorov >> To: "David P. Reed" >> Cc: Sebastian Moeller , "Dave T=C3=A4ht" , >> Michael Richardson , Make-Wifi-fast < >> make-wifi-fast@lists.bufferbloat.net>, Jannie Hanekom , >> Cake List , bloat < >> bloat@lists.bufferbloat.net> >> Bcc: >> Date: Mon, 4 May 2020 10:04:19 -0700 >> Subject: Re: [Cake] [Make-wifi-fast] [Bloat] dslreports is no longer fre= e >> >>> Sergey - I wasn't assuming anything about fast.com. The document you >>> shared wasn't clear about the methodology's details here. Others sadly, >>> have actually used ICMP pings in the way I described. I was making a >>> generic comment of concern. >>> >>> That said, it sounds like what you are doing is really helpful (esp. >>> given that your measure is aimed at end user experiential qualities). >> >> David - my apologies, I incorrectly interpreted your statement as being >> said in context of fast.com measurements. The blog post linked indeed >> doesn't provide the latency measurement details - was written before we >> added the extra metrics. We'll see if we can publish an update. >> >> 1) a clear definition of lag under load that is from end-to-end in >>> latency, and involves, ideally, independent traffic from multiple sourc= es >>> through the bottleneck. >> >> Curious if by multiple sources you mean multiple clients (devices) or >> multiple connections sending data? >> >> >> SERGEY FEDOROV >> >> Director of Engineering >> >> sfedorov@netflix.com >> >> 121 Albright Way | Los Gatos, CA 95032 >> >> >> >> >> On Sun, May 3, 2020 at 8:07 AM David P. Reed wrote= : >> >>> Thanks Sebastian. I do agree that in many cases, reflecting the ICMP of= f >>> the entry device that has the external IP address for the NAT gets most= of >>> the RTT measure, and if there's no queueing built up in the NAT device, >>> that's a reasonable measure. But... >>> >>> >>> >>> However, if the router has "taken up the queueing delay" by rate >>> limiting its uplink traffic to slightly less than the capacity (as with >>> Cake and other TC shaping that isn't as good as cake), then there is a >>> queue in the TC layer itself. This is what concerns me as a distortion = in >>> the measurement that can fool one into thinking the TC shaper is doing = a >>> good job, when in fact, lag under load may be quite high from inside th= e >>> routed domain (the home). >>> >>> >>> >>> As you point out this unmeasured queueing delay can also be a problem >>> with WiFi inside the home. But it isn't limited to that. >>> >>> >>> >>> A badly set up shaping/congestion management subsystem inside the NAT >>> can look "very good" in its echo of ICMP packets, but be terrible in >>> response time to trivial HTTP requests from inside, or equally terrible= in >>> twitch games and video conferencing. >>> >>> >>> >>> So, for example, for tuning settings with "Cake" it is useless. >>> >>> >>> >>> To be fair, usually the Access Provider has no control of what is done >>> after the cable is terminated at the home, so as a way to decide if the >>> provider is badly engineering its side, a ping from a server is a >>> reasonable quality measure of the provider. >>> >>> >>> >>> But not a good measure of the user experience, and if the provider >>> provides the NAT box, even if it has a good shaper in it, like Cake or >>> fq_codel, it will just confuse the user and create the opportunity for = a >>> "finger pointing" argument where neither side understands what is going= on. >>> >>> >>> >>> This is why we need >>> >>> >>> >>> 1) a clear definition of lag under load that is from end-to-end in >>> latency, and involves, ideally, independent traffic from multiple sourc= es >>> through the bottleneck. >>> >>> >>> >>> 2) ideally, a better way to localize where the queues are building up >>> and present that to users and access providers. The flent graphs are n= ot >>> interpretable by most non-experts. What we need is a simple visualizati= on >>> of a sketch-map of the path (like traceroute might provide) with queuei= ng >>> delay measures shown at key points that the user can understand. >>> >>> On Saturday, May 2, 2020 4:19pm, "Sebastian Moeller" >>> said: >>> >>> > Hi David, >>> > >>> > in principle I agree, a NATed IPv4 ICMP probe will be at best >>> reflected at the NAT >>> > router (CPE) (some commercial home gateways do not respond to ICMP >>> echo requests >>> > in the name of security theatre). So it is pretty hard to measure the >>> full end to >>> > end path in that configuration. I believe that IPv6 should make that >>> > easier/simpler in that NAT hopefully will be out of the path (but >>> let's see what >>> > ingenuity ISPs will come up with). >>> > Then again, traditionally the relevant bottlenecks often are a) the >>> internet >>> > access link itself and there the CPE is in a reasonable position as a >>> reflector on >>> > the other side of the bottleneck as seen from an internet server, b) >>> the home >>> > network between CPE and end-host, often with variable rate wifi, here >>> I agree >>> > reflecting echos at the CPE hides part of the issue. >>> > >>> > >>> > >>> > > On May 2, 2020, at 19:38, David P. Reed wrote= : >>> > > >>> > > I am still a bit worried about properly defining "latency under >>> load" for a >>> > NAT routed situation. If the test is based on ICMP Ping packets *from >>> the server*, >>> > it will NOT be measuring the full path latency, and if the potential >>> congestion >>> > is in the uplink path from the access provider's residential box to >>> the access >>> > provider's router/switch, it will NOT measure congestion caused by >>> bufferbloat >>> > reliably on either side, since the bufferbloat will be outside the >>> ICMP Ping >>> > path. >>> > >>> > Puzzled, as i believe it is going to be the residential box that will >>> respond >>> > here, or will it be the AFTRs for CG-NAT that reflect the ICMP echo >>> requests? >>> > >>> > > >>> > > I realize that a browser based speed test has to be basically run >>> from the >>> > "server" end, because browsers are not that good at time measurement >>> on a packet >>> > basis. However, there are ways to solve this and avoid the ICMP Ping >>> issue, with a >>> > cooperative server. >>> > > >>> > > I once built a test that fixed this issue reasonably well. It >>> carefully >>> > created a TCP based RTT measurement channel (over HTTP) that made the >>> echo have to >>> > traverse the whole end-to-end path, which is the best and only way to >>> accurately >>> > define lag under load from the user's perspective. The client end of >>> an unloaded >>> > TCP connection can depend on TCP (properly prepared by getting it pas= t >>> slowstart) >>> > to generate a single packet response. >>> > > >>> > > This "TCP ping" is thus compatible with getting the end-to-end >>> measurement on >>> > the server end of a true RTT. >>> > > >>> > > It's like tcp-traceroute tool, in that it tricks anyone in the >>> middle boxes >>> > into thinking this is a real, serious packet, not an optional low >>> priority >>> > packet. >>> > > >>> > > The same issue comes up with non-browser-based techniques for >>> measuring true >>> > lag-under-load. >>> > > >>> > > Now as we move HTTP to QUIC, this actually gets easier to do. >>> > > >>> > > One other opportunity I haven't explored, but which is pregnant wit= h >>> > potential is the use of WebRTC, which runs over UDP internally. Since >>> JavaScript >>> > has direct access to create WebRTC connections (multiple ones), this >>> makes >>> > detailed testing in the browser quite reasonable. >>> > > >>> > > And the time measurements can resolve well below 100 microseconds, >>> if the JS >>> > is based on modern JIT compilation (Chrome, Firefox, Edge all compile >>> to machine >>> > code speed if the code is restricted and in a loop). Then again, ther= e >>> is Web >>> > Assembly if you want to write C code that runs in the brower fast. >>> WebAssembly is >>> > a low level language that compiles to machine code in the browser >>> execution, and >>> > still has access to all the browser networking facilities. >>> > >>> > Mmmh, according to https://github.com/w3c/hr-time/issues/56 due to >>> spectre >>> > side-channel vulnerabilities many browsers seemed to have lowered the >>> timer >>> > resolution, but even the ~1ms resolution should be fine for typical >>> RTTs. >>> > >>> > Best Regards >>> > Sebastian >>> > >>> > P.S.: I assume that I simply do not see/understand the full scope of >>> the issue at >>> > hand yet. >>> > >>> > >>> > > >>> > > On Saturday, May 2, 2020 12:52pm, "Dave Taht" >>> > said: >>> > > >>> > > > On Sat, May 2, 2020 at 9:37 AM Benjamin Cronce >>> > wrote: >>> > > > > >>> > > > > > Fast.com reports my unloaded latency as 4ms, my loaded latenc= y >>> > as ~7ms >>> > > > >>> > > > I guess one of my questions is that with a switch to BBR netflix = is >>> > > > going to do pretty well. If fast.com is using bbr, well... that >>> > > > excludes much of the current side of the internet. >>> > > > >>> > > > > For download, I show 6ms unloaded and 6-7 loaded. But for uploa= d >>> > the loaded >>> > > > shows as 7-8 and I see it blip upwards of 12ms. But I am no longe= r >>> using >>> > any >>> > > > traffic shaping. Any anti-bufferbloat is from my ISP. A graph of >>> the >>> > bloat would >>> > > > be nice. >>> > > > >>> > > > The tests do need to last a fairly long time. >>> > > > >>> > > > > On Sat, May 2, 2020 at 9:51 AM Jannie Hanekom >>> > >>> > > > wrote: >>> > > > >> >>> > > > >> Michael Richardson : >>> > > > >> > Does it find/use my nearest Netflix cache? >>> > > > >> >>> > > > >> Thankfully, it appears so. The DSLReports bloat test was >>> > interesting, >>> > > > but >>> > > > >> the jitter on the ~240ms base latency from South Africa (and >>> > other parts >>> > > > of >>> > > > >> the world) was significant enough that the figures returned >>> > were often >>> > > > >> unreliable and largely unusable - at least in my experience. >>> > > > >> >>> > > > >> Fast.com reports my unloaded latency as 4ms, my loaded latency >>> > as ~7ms >>> > > > and >>> > > > >> mentions servers located in local cities. I finally have a tes= t >>> > I can >>> > > > share >>> > > > >> with local non-technical people! >>> > > > >> >>> > > > >> (Agreed, upload test would be nice, but this is a huge step >>> > forward from >>> > > > >> what I had access to before.) >>> > > > >> >>> > > > >> Jannie Hanekom >>> > > > >> >>> > > > >> _______________________________________________ >>> > > > >> Cake mailing list >>> > > > >> Cake@lists.bufferbloat.net >>> > > > >> https://lists.bufferbloat.net/listinfo/cake >>> > > > > >>> > > > > _______________________________________________ >>> > > > > Cake mailing list >>> > > > > Cake@lists.bufferbloat.net >>> > > > > https://lists.bufferbloat.net/listinfo/cake >>> > > > >>> > > > >>> > > > >>> > > > -- >>> > > > Make Music, Not War >>> > > > >>> > > > Dave T=C3=A4ht >>> > > > CTO, TekLibre, LLC >>> > > > http://www.teklibre.com >>> > > > Tel: 1-831-435-0729 >>> > > > _______________________________________________ >>> > > > Cake mailing list >>> > > > Cake@lists.bufferbloat.net >>> > > > https://lists.bufferbloat.net/listinfo/cake >>> > > > >>> > > _______________________________________________ >>> > > Cake mailing list >>> > > Cake@lists.bufferbloat.net >>> > > https://lists.bufferbloat.net/listinfo/cake >>> > >>> > >>> >>> >>> >> >> >> >> ---------- Forwarded message ---------- >> From: Sergey Fedorov via Make-wifi-fast < >> make-wifi-fast@lists.bufferbloat.net> >> To: "David P. Reed" >> Cc: Michael Richardson , Make-Wifi-fast < >> make-wifi-fast@lists.bufferbloat.net>, bloat , >> Cake List , Jannie Hanekom < >> jannie@hanekom.net> >> Bcc: >> Date: Mon, 04 May 2020 10:05:04 -0700 (PDT) >> Subject: Re: [Make-wifi-fast] [Cake] [Bloat] dslreports is no longer fr= ee >> _______________________________________________ >> Make-wifi-fast mailing list >> Make-wifi-fast@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/make-wifi-fast > > > > > ---------- Forwarded message ---------- > From: Bob McMahon via Bloat > To: Sergey Fedorov > Cc: Make-Wifi-fast , bloat < > bloat@lists.bufferbloat.net>, "David P. Reed" , Cake > List , Jannie Hanekom > Bcc: > Date: Mon, 04 May 2020 17:03:19 -0700 (PDT) > Subject: Re: [Bloat] [Make-wifi-fast] [Cake] dslreports is no longer fre= e > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat > --=20 Make Music, Not War Dave T=C3=A4ht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-435-0729 --000000000000feb38205a4db7ab1 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Mon, May 4, 2020 at 5:03 PM Bob Mc= Mahon via Bloat <bloat@li= sts.bufferbloat.net> wrote:



---------- Forwarded message ----------
F= rom:=C2=A0Bob McMahon <bob.mcmahon@broadcom.com>
To:=C2=A0Sergey Fedorov &l= t;sfedorov@netfli= x.com>
Cc:=C2=A0"David P. Reed" <dpreed@deepplum.com>, Michael = Richardson <mcr@sa= ndelman.ca>, Make-Wifi-fast <make-wifi-fast@lists.bufferbloat.net<= /a>>, bloat <bloat@lists.bufferbloat.net>, Cake List <cake@lists.bufferbloat.net= >, Jannie Hanekom <jannie@hanekom.net>
Bcc:=C2=A0
Date:=C2=A0Mon, 4 M= ay 2020 17:03:02 -0700
Subject:=C2=A0Re: [Make-wifi-fast] [Cake] [Bloat]= dslreports is no longer free
Sorry for being a bit off= topic but we find average latency not all that useful.=C2=A0 A full CDF is= .=C2=A0 The next best is a box plot with outliers which can be presented pa= rametrically as a few numbers. Most customers want visibility into the PDF = tail.
=C2=A0
yea!

<= /div>
Try never to discard the outliers anywhere in the core tests. I a= lways point to this as a place where, if you stop thinking
the no= ise is noise, caused by bird droppings in your receiver, you find structure= where you thought it never existed before.

https://theconversation.c= om/the-cmb-how-an-accidental-discovery-became-the-key-to-understanding-the-= universe-45126

A lot of times, just plotting t= he patterns of the outliers can be interesting. It's often a lot of bir= d droppings to sort through!


Also, we're mov= ing to socket write() to read() latencies for our end/end measurements (usi= ng the iperf 2.0.14 --trip-times option assumes synchronized clocks.). We a= lso now measure TCP connects=C2=A0(3WHS) as well.=C2=A0
=

One thing that may or may not help is the sock_sent_low= at option.

I note that with SSL so common, i= t helps to be using that, rather than straight tcp, so it's closer to a= 5WHS

Yes, generating the crypto exchange cost= s time, but with that as a baseline with the extra round trips...
=
=C2=A0
Finally, since we have trip times and the application write = rates we can compute the amount of "end/end bytes in queue" per L= ittle's law.

I will reserve c= omment on littles law for a bit.

For fault isolation, in-b= and network telemetry (or something=C2=A0similar) can be useful.=C2=A0https:/= /p4.org/assets/INT-current-spec.pdf

Bob

On Mon, May 4, 2020 at 10= :05 AM Sergey Fedorov via Make-wifi-fast <make-wifi-fast@lists.bufferbloa= t.net> wrote:



---------- Forwarded message ----------
From:=C2=A0Serg= ey Fedorov <sf= edorov@netflix.com>
To:=C2=A0"David P. Reed" <dpreed@deepplum.com&= gt;
Cc:=C2=A0Sebastian Moeller <moeller0@gmx.de>, "Dave T=C3=A4ht" <dave.taht@gmail.com>, Michael Richardson <mcr@sandelman.ca>, Make-Wifi-fast <make-wifi-fast@list= s.bufferbloat.net>, Jannie Hanekom <jannie@hanekom.net>, Cake List <cake@lists.buffer= bloat.net>, bloat <bloat@lists.bufferbloat.net>
Bcc:=C2=A0
Dat= e:=C2=A0Mon, 4 May 2020 10:04:19 -0700
Subject:=C2=A0Re: [Cake] [Make-wi= fi-fast] [Bloat] dslreports is no longer free
David - m= y apologies, I incorrectly interpreted your statement as being said in cont= ext of fast.com measureme= nts. The blog post linked indeed doesn't provide the latency measuremen= t details - was written before we added the extra metrics. We'll see if= we can publish an update.=C2=A0

1) a clear definition of lag under load that is from end-to-end in latenc= y, and involves, ideally, independent traffic from multiple sources through= the bottleneck.
=C2=A0Curious if by multiple sourc= es you mean multiple clients (devices) or multiple connections sending data= ?=C2=A0
=C2=A0

SERGEY FEDOROV=

Director of Engineering

sfedo= rov@netflix.com

121 Albright Way | Los Gatos= , CA 95032




On Sun, May 3, 2020 at 8:07 AM David P. Reed <dpreed@deepplum.com> wro= te:

Thanks Sebastian. I do agree that in many cases, reflecting= the ICMP off the entry device that has the external IP address for the NAT= gets most of the RTT measure, and if there's no queueing built up in t= he NAT device, that's a reasonable measure. But...

=C2=A0=

Howeve= r, if the router has "taken up the queueing delay" by rate limiti= ng its uplink traffic to slightly less than the capacity (as with Cake and = other TC shaping that isn't as good as cake), then there is a queue in = the TC layer itself. This is what concerns me as a distortion in the measur= ement that can fool one into thinking the TC shaper is doing a good job, wh= en in fact, lag under load may be quite high from inside the routed domain = (the home).

=C2=A0=

As you= point out this unmeasured queueing delay can also be a problem with WiFi i= nside the home. But it isn't limited to that.

=C2=A0=

A badl= y set up shaping/congestion management subsystem inside the NAT can look &q= uot;very good" in its echo of ICMP packets, but be terrible in respons= e time to trivial HTTP requests from inside, or equally terrible in twitch = games and video conferencing.

=C2=A0=

So, fo= r example, for tuning settings with "Cake" it is useless.

=C2=A0=

To be = fair, usually the Access Provider has no control of what is done after the = cable is terminated at the home, so as a way to decide if the provider is b= adly engineering its side, a ping from a server is a reasonable quality mea= sure of the provider.=C2=A0

=C2=A0=

But no= t a good measure of the user experience, and if the provider provides the N= AT box, even if it has a good shaper in it, like Cake or fq_codel, it will = just confuse the user and create the opportunity for a "finger pointin= g" argument where neither side understands what is going on.

=C2=A0=

This i= s why we need=C2=A0

=C2=A0=

1) a c= lear definition of lag under load that is from end-to-end in latency, and i= nvolves, ideally, independent traffic from multiple sources through the bot= tleneck.

=C2=A0=

2) ide= ally, a better way to localize where the queues are building up and present= that to users and access providers.=C2=A0 The flent graphs are not interpr= etable by most non-experts. What we need is a simple visualization of a ske= tch-map of the path (like traceroute might provide) with queueing delay mea= sures=C2=A0 shown at key points that the user can understand.

On Sat= urday, May 2, 2020 4:19pm, "Sebastian Moeller" <moeller0@gmx.de> said:

> H= i David,
>
> in principle I agree, a NATed IPv4 ICMP probe wil= l be at best reflected at the NAT
> router (CPE) (some commercial hom= e gateways do not respond to ICMP echo requests
> in the name of secu= rity theatre). So it is pretty hard to measure the full end to
> end = path in that configuration. I believe that IPv6 should make that
> ea= sier/simpler in that NAT hopefully will be out of the path (but let's s= ee what
> ingenuity ISPs will come up with).
> Then again, trad= itionally the relevant bottlenecks often are a) the internet
> access= link itself and there the CPE is in a reasonable position as a reflector o= n
> the other side of the bottleneck as seen from an internet server,= b) the home
> network between CPE and end-host, often with variable = rate wifi, here I agree
> reflecting echos at the CPE hides part of t= he issue.
>
>
>
> > On May 2, 2020, at 19:38,= David P. Reed <dpreed@deepplum.com> wrote:
> >
> > I am still a = bit worried about properly defining "latency under load" for a> NAT routed situation. If the test is based on ICMP Ping packets *from= the server*,
> it will NOT be measuring the full path latency, and i= f the potential congestion
> is in the uplink path from the access pr= ovider's residential box to the access
> provider's router/sw= itch, it will NOT measure congestion caused by bufferbloat
> reliably= on either side, since the bufferbloat will be outside the ICMP Ping
>= ; path.
>
> Puzzled, as i believe it is going to be the reside= ntial box that will respond
> here, or will it be the AFTRs for CG-NA= T that reflect the ICMP echo requests?
>
> >
> > I= realize that a browser based speed test has to be basically run from the> "server" end, because browsers are not that good at time m= easurement on a packet
> basis. However, there are ways to solve this= and avoid the ICMP Ping issue, with a
> cooperative server.
> = >
> > I once built a test that fixed this issue reasonably well= . It carefully
> created a TCP based RTT measurement channel (over HT= TP) that made the echo have to
> traverse the whole end-to-end path, = which is the best and only way to accurately
> define lag under load = from the user's perspective. The client end of an unloaded
> TCP = connection can depend on TCP (properly prepared by getting it past slowstar= t)
> to generate a single packet response.
> >
> > = This "TCP ping" is thus compatible with getting the end-to-end me= asurement on
> the server end of a true RTT.
> >
> >= ; It's like tcp-traceroute tool, in that it tricks anyone in the middle= boxes
> into thinking this is a real, serious packet, not an optiona= l low priority
> packet.
> >
> > The same issue com= es up with non-browser-based techniques for measuring true
> lag-unde= r-load.
> >
> > Now as we move HTTP to QUIC, this actuall= y gets easier to do.
> >
> > One other opportunity I have= n't explored, but which is pregnant with
> potential is the use o= f WebRTC, which runs over UDP internally. Since JavaScript
> has dire= ct access to create WebRTC connections (multiple ones), this makes
> = detailed testing in the browser quite reasonable.
> >
> >= And the time measurements can resolve well below 100 microseconds, if the = JS
> is based on modern JIT compilation (Chrome, Firefox, Edge all co= mpile to machine
> code speed if the code is restricted and in a loop= ). Then again, there is Web
> Assembly if you want to write C code th= at runs in the brower fast. WebAssembly is
> a low level language tha= t compiles to machine code in the browser execution, and
> still has = access to all the browser networking facilities.
>
> Mmmh, acc= ording to https://github.com/w3c/hr-time/issues/56 due to spectre
> s= ide-channel vulnerabilities many browsers seemed to have lowered the timer<= br>> resolution, but even the ~1ms resolution should be fine for typical= RTTs.
>
> Best Regards
> Sebastian
>
> P.S= .: I assume that I simply do not see/understand the full scope of the issue= at
> hand yet.
>
>
> >
> > On Saturd= ay, May 2, 2020 12:52pm, "Dave Taht" <dave.taht@gmail.com>
> said:<= br>> >
> > > On Sat, May 2, 2020 at 9:37 AM Benjamin Cron= ce <bcronce@gmail= .com>
> wrote:
> > > >
> > > > &= gt; Fast.com reports my unloaded latency as 4ms, my loaded latency
> = as ~7ms
> > >
> > > I guess one of my questions is = that with a switch to BBR netflix is
> > > going to do pretty w= ell. If fast.com is using= bbr, well... that
> > > excludes much of the current side of t= he internet.
> > >
> > > > For download, I show = 6ms unloaded and 6-7 loaded. But for upload
> the loaded
> >= > shows as 7-8 and I see it blip upwards of 12ms. But I am no longer us= ing
> any
> > > traffic shaping. Any anti-bufferbloat is = from my ISP. A graph of the
> bloat would
> > > be nice.<= br>> > >
> > > The tests do need to last a fairly long= time.
> > >
> > > > On Sat, May 2, 2020 at 9:51= AM Jannie Hanekom
> <jannie@hanekom.net>
> > > wrote:
> >= ; > >>
> > > >> Michael Richardson <mcr@sandelman.ca>:> > > >> > Does it find/use my nearest Netflix cache?> > > >>
> > > >> Thankfully, it appears = so. The DSLReports bloat test was
> interesting,
> > > bu= t
> > > >> the jitter on the ~240ms base latency from Sou= th Africa (and
> other parts
> > > of
> > > &= gt;> the world) was significant enough that the figures returned
>= were often
> > > >> unreliable and largely unusable - at= least in my experience.
> > > >>
> > > >&= gt; Fast.com reports my unloaded latency as 4ms, my loaded latency
> = as ~7ms
> > > and
> > > >> mentions servers l= ocated in local cities. I finally have a test
> I can
> > &g= t; share
> > > >> with local non-technical people!
>= ; > > >>
> > > >> (Agreed, upload test would = be nice, but this is a huge step
> forward from
> > > >= ;> what I had access to before.)
> > > >>
> >= > >> Jannie Hanekom
> > > >>
> > > = >> _______________________________________________
> > > = >> Cake mailing list
> > > >> Cake@lists.bufferbloat.net> > > >> https://lists.bufferbloat.net/listinfo/cake> > > >
> > > > _______________________________= ________________
> > > > Cake mailing list
> > >= > Cake@= lists.bufferbloat.net
> > > > https://lists.bufferbloat.n= et/listinfo/cake
> > >
> > >
> > ><= br>> > > --
> > > Make Music, Not War
> > >= ;
> > > Dave T=C3=A4ht
> > > CTO, TekLibre, LLC
= > > > http:/= /www.teklibre.com
> > > Tel: 1-831-435-0729
> > &g= t; _______________________________________________
> > > Cake m= ailing list
> > > Cake@lists.bufferbloat.net
> > > https://= lists.bufferbloat.net/listinfo/cake
> > >
> > ____= ___________________________________________
> > Cake mailing list<= br>> > Cake@lists.bufferbloat.net
> > https://lists.bufferbloat.net/= listinfo/cake
>
>

=C2=A0=




---------- Forwarded message ----------
From:=C2=A0Sergey Fe= dorov via Make-wifi-fast <make-wifi-fast@lists.bufferbloat.net>To:=C2=A0"David P. Reed" <dpreed@deepplum.com>
Cc:=C2=A0Michael Richa= rdson <mcr@sandelm= an.ca>, Make-Wifi-fast <make-wifi-fast@lists.bufferbloat.net&g= t;, bloat <bloat@lists.bufferbloat.net>, Cake List <cake@lists.bufferbloat.net&= gt;, Jannie Hanekom <jannie@hanekom.net>
Bcc:=C2=A0
Date:=C2=A0Mon, 04 May 202= 0 10:05:04 -0700 (PDT)
Subject:=C2=A0Re: [Make-wifi-fast] [Cake]=C2=A0 [= Bloat] dslreports is no longer free
____________________________________= ___________
Make-wifi-fast mailing list
M= ake-wifi-fast@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/make-wif= i-fast



---------- Forwarded message ----------
From:=C2=A0Bob McMah= on via Bloat <bloat@lists.bufferbloat.net>
To:=C2=A0Sergey Fedorov <<= a href=3D"mailto:sfedorov@netflix.com" target=3D"_blank">sfedorov@netflix.c= om>
Cc:=C2=A0Make-Wifi-fast <make-wifi-fast@lists.bufferbloat.n= et>, bloat <bloat@lists.bufferbloat.net>, "David P. Reed" = <dpreed@deepplu= m.com>, Cake List <cake@lists.bufferbloat.net>, Jannie Hanekom <jannie@hanekom.net>
Bcc:=C2=A0
Date:=C2=A0Mon, 04 May 2020 17:03:19 -0700 (PDT)
= Subject:=C2=A0Re: [Bloat] [Make-wifi-fast] [Cake]=C2=A0 dslreports is no lo= nger free
_______________________________________________
Bloat mailing list
Bloat@list= s.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat


--
Make Music, Not War

Dave T=C3=A4ht
CTO, TekLibre,= LLC
http://www.te= klibre.com
Tel: 1-831-435-0729
--000000000000feb38205a4db7ab1--