From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io1-xd2d.google.com (mail-io1-xd2d.google.com [IPv6:2607:f8b0:4864:20::d2d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id DC6CD3CB35 for ; Mon, 4 May 2020 13:04:55 -0400 (EDT) Received: by mail-io1-xd2d.google.com with SMTP id u11so13120390iow.4 for ; Mon, 04 May 2020 10:04:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netflix.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=M2C9jRpm53m7yuEaoe5mLEzUdT38C4LBOTQ7OrtQqR4=; b=pCx5eIr8JWgUJyN2uXz8tFNo/Mj3+NBFiyp5eDYqgfSDJHN0Z7xTsdFlQ0v1LZDlhN qX+LFgInYOsg/5yuVhrDBRMAFm17VSCTeQUU4N/AaL3Uma6ulVsEFRZrKLfKBDaZU1Rq CM+Fwp6gZuAKbWUDYSWV34aW4dS4vVrfIta3E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=M2C9jRpm53m7yuEaoe5mLEzUdT38C4LBOTQ7OrtQqR4=; b=F+FPHlZ26y0xBwSaTdAqO3Qv8czz0DEkfdzVfcWMgHETR545jbUTA+B+9muzm4xFex RkDErdabSbtiNhla1T/jGnYcelLr4U4sCMcVeZdza8FzKxzkLoH2EceO9Tf2cbr1atg9 SaAmbW9SoNV6+YI9rk23Blm9TV8yem+ZKg0lgAw04w6gTsRf3TBRatPeDtjcjaGAxwsn WkW70Bs2Bvh046M7qf7LTm1031BMDfIvekVvmiOzD3Orh1w1npwODbelkzx60jaa7s8S VLM87w3KvbvHj6PEoJW0u+Wec0TuJKvgjQkYWIQGcP4FRh5HSVYb/Fu0XsxfVcNsoLBU yrVw== X-Gm-Message-State: AGi0PubGHhQDamyVhMRhJfOWRw1cSPEQS1DmMkInOZDGz9T+GHGGz/8J 0Zh+mRX6GW+3p4agfpbyysRRrrPdrKBopKVwSac8VA== X-Google-Smtp-Source: APiQypJ5GyXSpTAU09a5UE6v031tSkyMRF4mY08DGFJfQjc64H1Vt728QAnBIDHaVpf7EqpocNLXPBjVz4kQL6aTp2s= X-Received: by 2002:a02:a40b:: with SMTP id c11mr15621923jal.62.1588611895169; Mon, 04 May 2020 10:04:55 -0700 (PDT) MIME-Version: 1.0 References: <1588518416.66682155@apps.rackspace.com> In-Reply-To: <1588518416.66682155@apps.rackspace.com> From: Sergey Fedorov Date: Mon, 4 May 2020 10:04:19 -0700 Message-ID: Subject: Re: [Cake] [Make-wifi-fast] [Bloat] dslreports is no longer free To: "David P. Reed" Cc: Sebastian Moeller , =?UTF-8?Q?Dave_T=C3=A4ht?= , Michael Richardson , Make-Wifi-fast , Jannie Hanekom , Cake List , bloat Content-Type: multipart/alternative; boundary="000000000000b2eaf105a4d587ec" X-List-Received-Date: Mon, 04 May 2020 17:04:56 -0000 --000000000000b2eaf105a4d587ec Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable > > Sergey - I wasn't assuming anything about fast.com. The document you > shared wasn't clear about the methodology's details here. Others sadly, > have actually used ICMP pings in the way I described. I was making a > generic comment of concern. > > That said, it sounds like what you are doing is really helpful (esp. give= n > that your measure is aimed at end user experiential qualities). David - my apologies, I incorrectly interpreted your statement as being said in context of fast.com measurements. The blog post linked indeed doesn't provide the latency measurement details - was written before we added the extra metrics. We'll see if we can publish an update. 1) a clear definition of lag under load that is from end-to-end in latency, > and involves, ideally, independent traffic from multiple sources through > the bottleneck. Curious if by multiple sources you mean multiple clients (devices) or multiple connections sending data? SERGEY FEDOROV Director of Engineering sfedorov@netflix.com 121 Albright Way | Los Gatos, CA 95032 On Sun, May 3, 2020 at 8:07 AM David P. Reed wrote: > Thanks Sebastian. I do agree that in many cases, reflecting the ICMP off > the entry device that has the external IP address for the NAT gets most o= f > the RTT measure, and if there's no queueing built up in the NAT device, > that's a reasonable measure. But... > > > > However, if the router has "taken up the queueing delay" by rate limiting > its uplink traffic to slightly less than the capacity (as with Cake and > other TC shaping that isn't as good as cake), then there is a queue in th= e > TC layer itself. This is what concerns me as a distortion in the > measurement that can fool one into thinking the TC shaper is doing a good > job, when in fact, lag under load may be quite high from inside the route= d > domain (the home). > > > > As you point out this unmeasured queueing delay can also be a problem wit= h > WiFi inside the home. But it isn't limited to that. > > > > A badly set up shaping/congestion management subsystem inside the NAT can > look "very good" in its echo of ICMP packets, but be terrible in response > time to trivial HTTP requests from inside, or equally terrible in twitch > games and video conferencing. > > > > So, for example, for tuning settings with "Cake" it is useless. > > > > To be fair, usually the Access Provider has no control of what is done > after the cable is terminated at the home, so as a way to decide if the > provider is badly engineering its side, a ping from a server is a > reasonable quality measure of the provider. > > > > But not a good measure of the user experience, and if the provider > provides the NAT box, even if it has a good shaper in it, like Cake or > fq_codel, it will just confuse the user and create the opportunity for a > "finger pointing" argument where neither side understands what is going o= n. > > > > This is why we need > > > > 1) a clear definition of lag under load that is from end-to-end in > latency, and involves, ideally, independent traffic from multiple sources > through the bottleneck. > > > > 2) ideally, a better way to localize where the queues are building up and > present that to users and access providers. The flent graphs are not > interpretable by most non-experts. What we need is a simple visualization > of a sketch-map of the path (like traceroute might provide) with queueing > delay measures shown at key points that the user can understand. > > On Saturday, May 2, 2020 4:19pm, "Sebastian Moeller" > said: > > > Hi David, > > > > in principle I agree, a NATed IPv4 ICMP probe will be at best reflected > at the NAT > > router (CPE) (some commercial home gateways do not respond to ICMP echo > requests > > in the name of security theatre). So it is pretty hard to measure the > full end to > > end path in that configuration. I believe that IPv6 should make that > > easier/simpler in that NAT hopefully will be out of the path (but let's > see what > > ingenuity ISPs will come up with). > > Then again, traditionally the relevant bottlenecks often are a) the > internet > > access link itself and there the CPE is in a reasonable position as a > reflector on > > the other side of the bottleneck as seen from an internet server, b) th= e > home > > network between CPE and end-host, often with variable rate wifi, here I > agree > > reflecting echos at the CPE hides part of the issue. > > > > > > > > > On May 2, 2020, at 19:38, David P. Reed wrote: > > > > > > I am still a bit worried about properly defining "latency under load" > for a > > NAT routed situation. If the test is based on ICMP Ping packets *from > the server*, > > it will NOT be measuring the full path latency, and if the potential > congestion > > is in the uplink path from the access provider's residential box to the > access > > provider's router/switch, it will NOT measure congestion caused by > bufferbloat > > reliably on either side, since the bufferbloat will be outside the ICMP > Ping > > path. > > > > Puzzled, as i believe it is going to be the residential box that will > respond > > here, or will it be the AFTRs for CG-NAT that reflect the ICMP echo > requests? > > > > > > > > I realize that a browser based speed test has to be basically run fro= m > the > > "server" end, because browsers are not that good at time measurement on > a packet > > basis. However, there are ways to solve this and avoid the ICMP Ping > issue, with a > > cooperative server. > > > > > > I once built a test that fixed this issue reasonably well. It careful= ly > > created a TCP based RTT measurement channel (over HTTP) that made the > echo have to > > traverse the whole end-to-end path, which is the best and only way to > accurately > > define lag under load from the user's perspective. The client end of an > unloaded > > TCP connection can depend on TCP (properly prepared by getting it past > slowstart) > > to generate a single packet response. > > > > > > This "TCP ping" is thus compatible with getting the end-to-end > measurement on > > the server end of a true RTT. > > > > > > It's like tcp-traceroute tool, in that it tricks anyone in the middle > boxes > > into thinking this is a real, serious packet, not an optional low > priority > > packet. > > > > > > The same issue comes up with non-browser-based techniques for > measuring true > > lag-under-load. > > > > > > Now as we move HTTP to QUIC, this actually gets easier to do. > > > > > > One other opportunity I haven't explored, but which is pregnant with > > potential is the use of WebRTC, which runs over UDP internally. Since > JavaScript > > has direct access to create WebRTC connections (multiple ones), this > makes > > detailed testing in the browser quite reasonable. > > > > > > And the time measurements can resolve well below 100 microseconds, if > the JS > > is based on modern JIT compilation (Chrome, Firefox, Edge all compile t= o > machine > > code speed if the code is restricted and in a loop). Then again, there > is Web > > Assembly if you want to write C code that runs in the brower fast. > WebAssembly is > > a low level language that compiles to machine code in the browser > execution, and > > still has access to all the browser networking facilities. > > > > Mmmh, according to https://github.com/w3c/hr-time/issues/56 due to > spectre > > side-channel vulnerabilities many browsers seemed to have lowered the > timer > > resolution, but even the ~1ms resolution should be fine for typical RTT= s. > > > > Best Regards > > Sebastian > > > > P.S.: I assume that I simply do not see/understand the full scope of th= e > issue at > > hand yet. > > > > > > > > > > On Saturday, May 2, 2020 12:52pm, "Dave Taht" > > said: > > > > > > > On Sat, May 2, 2020 at 9:37 AM Benjamin Cronce > > wrote: > > > > > > > > > > > Fast.com reports my unloaded latency as 4ms, my loaded latency > > as ~7ms > > > > > > > > I guess one of my questions is that with a switch to BBR netflix is > > > > going to do pretty well. If fast.com is using bbr, well... that > > > > excludes much of the current side of the internet. > > > > > > > > > For download, I show 6ms unloaded and 6-7 loaded. But for upload > > the loaded > > > > shows as 7-8 and I see it blip upwards of 12ms. But I am no longer > using > > any > > > > traffic shaping. Any anti-bufferbloat is from my ISP. A graph of th= e > > bloat would > > > > be nice. > > > > > > > > The tests do need to last a fairly long time. > > > > > > > > > On Sat, May 2, 2020 at 9:51 AM Jannie Hanekom > > > > > > wrote: > > > > >> > > > > >> Michael Richardson : > > > > >> > Does it find/use my nearest Netflix cache? > > > > >> > > > > >> Thankfully, it appears so. The DSLReports bloat test was > > interesting, > > > > but > > > > >> the jitter on the ~240ms base latency from South Africa (and > > other parts > > > > of > > > > >> the world) was significant enough that the figures returned > > were often > > > > >> unreliable and largely unusable - at least in my experience. > > > > >> > > > > >> Fast.com reports my unloaded latency as 4ms, my loaded latency > > as ~7ms > > > > and > > > > >> mentions servers located in local cities. I finally have a test > > I can > > > > share > > > > >> with local non-technical people! > > > > >> > > > > >> (Agreed, upload test would be nice, but this is a huge step > > forward from > > > > >> what I had access to before.) > > > > >> > > > > >> Jannie Hanekom > > > > >> > > > > >> _______________________________________________ > > > > >> Cake mailing list > > > > >> Cake@lists.bufferbloat.net > > > > >> https://lists.bufferbloat.net/listinfo/cake > > > > > > > > > > _______________________________________________ > > > > > Cake mailing list > > > > > Cake@lists.bufferbloat.net > > > > > https://lists.bufferbloat.net/listinfo/cake > > > > > > > > > > > > > > > > -- > > > > Make Music, Not War > > > > > > > > Dave T=C3=A4ht > > > > CTO, TekLibre, LLC > > > > http://www.teklibre.com > > > > Tel: 1-831-435-0729 > > > > _______________________________________________ > > > > Cake mailing list > > > > Cake@lists.bufferbloat.net > > > > https://lists.bufferbloat.net/listinfo/cake > > > > > > > _______________________________________________ > > > Cake mailing list > > > Cake@lists.bufferbloat.net > > > https://lists.bufferbloat.net/listinfo/cake > > > > > > > --000000000000b2eaf105a4d587ec Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Sergey -= I wasn't assuming anything about=C2=A0fast.com. The document you shared wasn't clear about= the methodology's details here. Others sadly, have actually used ICMP = pings in the way I described. I was making a generic comment of concern.=C2=A0
That said, it sounds like what you are doing is really helpful (= esp. given that your measure is aimed at end user experiential qualities).<= /blockquote>
David - my apologies, I incorrectly interpreted your state= ment as being said in context of fast.com m= easurements. The blog post linked indeed doesn't provide the latency me= asurement details - was written before we added the extra metrics. We'l= l see if we can publish an update.=C2=A0

1) a clear definition of lag under load that is from end-to-end i= n latency, and involves, ideally, independent traffic from multiple sources= through the bottleneck.
=C2=A0Curious if by multip= le sources you mean multiple clients (devices) or multiple connections send= ing data?=C2=A0
=C2=A0

SERGEY FEDOROV

Dir= ector of Engineering

sfedorov@netflix.com

121 Albright Way | Los Gatos, CA 95032




On Sun, May 3, 2020 at 8:07 AM David P. Reed = <dpreed@deepplum.com> wrot= e:

Thanks Sebastian. I do agree that in many cases, reflecting th= e ICMP off the entry device that has the external IP address for the NAT ge= ts most of the RTT measure, and if there's no queueing built up in the = NAT device, that's a reasonable measure. But...

=C2=A0=

Howeve= r, if the router has "taken up the queueing delay" by rate limiti= ng its uplink traffic to slightly less than the capacity (as with Cake and = other TC shaping that isn't as good as cake), then there is a queue in = the TC layer itself. This is what concerns me as a distortion in the measur= ement that can fool one into thinking the TC shaper is doing a good job, wh= en in fact, lag under load may be quite high from inside the routed domain = (the home).

=C2=A0=

As you= point out this unmeasured queueing delay can also be a problem with WiFi i= nside the home. But it isn't limited to that.

=C2=A0=

A badl= y set up shaping/congestion management subsystem inside the NAT can look &q= uot;very good" in its echo of ICMP packets, but be terrible in respons= e time to trivial HTTP requests from inside, or equally terrible in twitch = games and video conferencing.

=C2=A0=

So, fo= r example, for tuning settings with "Cake" it is useless.

=C2=A0=

To be = fair, usually the Access Provider has no control of what is done after the = cable is terminated at the home, so as a way to decide if the provider is b= adly engineering its side, a ping from a server is a reasonable quality mea= sure of the provider.=C2=A0

=C2=A0=

But no= t a good measure of the user experience, and if the provider provides the N= AT box, even if it has a good shaper in it, like Cake or fq_codel, it will = just confuse the user and create the opportunity for a "finger pointin= g" argument where neither side understands what is going on.

=C2=A0=

This i= s why we need=C2=A0

=C2=A0=

1) a c= lear definition of lag under load that is from end-to-end in latency, and i= nvolves, ideally, independent traffic from multiple sources through the bot= tleneck.

=C2=A0=

2) ide= ally, a better way to localize where the queues are building up and present= that to users and access providers.=C2=A0 The flent graphs are not interpr= etable by most non-experts. What we need is a simple visualization of a ske= tch-map of the path (like traceroute might provide) with queueing delay mea= sures=C2=A0 shown at key points that the user can understand.

On Sat= urday, May 2, 2020 4:19pm, "Sebastian Moeller" <moeller0@gmx.de> said:

> H= i David,
>
> in principle I agree, a NATed IPv4 ICMP probe wil= l be at best reflected at the NAT
> router (CPE) (some commercial hom= e gateways do not respond to ICMP echo requests
> in the name of secu= rity theatre). So it is pretty hard to measure the full end to
> end = path in that configuration. I believe that IPv6 should make that
> ea= sier/simpler in that NAT hopefully will be out of the path (but let's s= ee what
> ingenuity ISPs will come up with).
> Then again, trad= itionally the relevant bottlenecks often are a) the internet
> access= link itself and there the CPE is in a reasonable position as a reflector o= n
> the other side of the bottleneck as seen from an internet server,= b) the home
> network between CPE and end-host, often with variable = rate wifi, here I agree
> reflecting echos at the CPE hides part of t= he issue.
>
>
>
> > On May 2, 2020, at 19:38,= David P. Reed <dpreed@deepplum.com> wrote:
> >
> > I am still a = bit worried about properly defining "latency under load" for a> NAT routed situation. If the test is based on ICMP Ping packets *from= the server*,
> it will NOT be measuring the full path latency, and i= f the potential congestion
> is in the uplink path from the access pr= ovider's residential box to the access
> provider's router/sw= itch, it will NOT measure congestion caused by bufferbloat
> reliably= on either side, since the bufferbloat will be outside the ICMP Ping
>= ; path.
>
> Puzzled, as i believe it is going to be the reside= ntial box that will respond
> here, or will it be the AFTRs for CG-NA= T that reflect the ICMP echo requests?
>
> >
> > I= realize that a browser based speed test has to be basically run from the> "server" end, because browsers are not that good at time m= easurement on a packet
> basis. However, there are ways to solve this= and avoid the ICMP Ping issue, with a
> cooperative server.
> = >
> > I once built a test that fixed this issue reasonably well= . It carefully
> created a TCP based RTT measurement channel (over HT= TP) that made the echo have to
> traverse the whole end-to-end path, = which is the best and only way to accurately
> define lag under load = from the user's perspective. The client end of an unloaded
> TCP = connection can depend on TCP (properly prepared by getting it past slowstar= t)
> to generate a single packet response.
> >
> > = This "TCP ping" is thus compatible with getting the end-to-end me= asurement on
> the server end of a true RTT.
> >
> >= ; It's like tcp-traceroute tool, in that it tricks anyone in the middle= boxes
> into thinking this is a real, serious packet, not an optiona= l low priority
> packet.
> >
> > The same issue com= es up with non-browser-based techniques for measuring true
> lag-unde= r-load.
> >
> > Now as we move HTTP to QUIC, this actuall= y gets easier to do.
> >
> > One other opportunity I have= n't explored, but which is pregnant with
> potential is the use o= f WebRTC, which runs over UDP internally. Since JavaScript
> has dire= ct access to create WebRTC connections (multiple ones), this makes
> = detailed testing in the browser quite reasonable.
> >
> >= And the time measurements can resolve well below 100 microseconds, if the = JS
> is based on modern JIT compilation (Chrome, Firefox, Edge all co= mpile to machine
> code speed if the code is restricted and in a loop= ). Then again, there is Web
> Assembly if you want to write C code th= at runs in the brower fast. WebAssembly is
> a low level language tha= t compiles to machine code in the browser execution, and
> still has = access to all the browser networking facilities.
>
> Mmmh, acc= ording to https://github.com/w3c/hr-time/issues/56 due to spectre
> s= ide-channel vulnerabilities many browsers seemed to have lowered the timer<= br>> resolution, but even the ~1ms resolution should be fine for typical= RTTs.
>
> Best Regards
> Sebastian
>
> P.S= .: I assume that I simply do not see/understand the full scope of the issue= at
> hand yet.
>
>
> >
> > On Saturd= ay, May 2, 2020 12:52pm, "Dave Taht" <dave.taht@gmail.com>
> said:<= br>> >
> > > On Sat, May 2, 2020 at 9:37 AM Benjamin Cron= ce <bcronce@gmail= .com>
> wrote:
> > > >
> > > > &= gt; Fast.com reports my unloaded latency as 4ms, my loaded latency
> = as ~7ms
> > >
> > > I guess one of my questions is = that with a switch to BBR netflix is
> > > going to do pretty w= ell. If fast.com is using= bbr, well... that
> > > excludes much of the current side of t= he internet.
> > >
> > > > For download, I show = 6ms unloaded and 6-7 loaded. But for upload
> the loaded
> >= > shows as 7-8 and I see it blip upwards of 12ms. But I am no longer us= ing
> any
> > > traffic shaping. Any anti-bufferbloat is = from my ISP. A graph of the
> bloat would
> > > be nice.<= br>> > >
> > > The tests do need to last a fairly long= time.
> > >
> > > > On Sat, May 2, 2020 at 9:51= AM Jannie Hanekom
> <jannie@hanekom.net>
> > > wrote:
> >= ; > >>
> > > >> Michael Richardson <mcr@sandelman.ca>:> > > >> > Does it find/use my nearest Netflix cache?> > > >>
> > > >> Thankfully, it appears = so. The DSLReports bloat test was
> interesting,
> > > bu= t
> > > >> the jitter on the ~240ms base latency from Sou= th Africa (and
> other parts
> > > of
> > > &= gt;> the world) was significant enough that the figures returned
>= were often
> > > >> unreliable and largely unusable - at= least in my experience.
> > > >>
> > > >&= gt; Fast.com reports my unloaded latency as 4ms, my loaded latency
> = as ~7ms
> > > and
> > > >> mentions servers l= ocated in local cities. I finally have a test
> I can
> > &g= t; share
> > > >> with local non-technical people!
>= ; > > >>
> > > >> (Agreed, upload test would = be nice, but this is a huge step
> forward from
> > > >= ;> what I had access to before.)
> > > >>
> >= > >> Jannie Hanekom
> > > >>
> > > = >> _______________________________________________
> > > = >> Cake mailing list
> > > >> Cake@lists.bufferbloat.net> > > >> https://lists.bufferbloat.net/listinfo/cake> > > >
> > > > _______________________________= ________________
> > > > Cake mailing list
> > >= > Cake@= lists.bufferbloat.net
> > > > https://lists.bufferbloat.n= et/listinfo/cake
> > >
> > >
> > ><= br>> > > --
> > > Make Music, Not War
> > >= ;
> > > Dave T=C3=A4ht
> > > CTO, TekLibre, LLC
= > > > http:/= /www.teklibre.com
> > > Tel: 1-831-435-0729
> > &g= t; _______________________________________________
> > > Cake m= ailing list
> > > Cake@lists.bufferbloat.net
> > > https://= lists.bufferbloat.net/listinfo/cake
> > >
> > ____= ___________________________________________
> > Cake mailing list<= br>> > Cake@lists.bufferbloat.net
> > https://lists.bufferbloat.net/= listinfo/cake
>
>

=C2=A0=

--000000000000b2eaf105a4d587ec--