From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr1-x442.google.com (mail-wr1-x442.google.com [IPv6:2a00:1450:4864:20::442]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 5DEE03BA8E for ; Tue, 5 Feb 2019 04:37:46 -0500 (EST) Received: by mail-wr1-x442.google.com with SMTP id t27so2816082wra.6 for ; Tue, 05 Feb 2019 01:37:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heistp.net; s=google; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=zHv62ZayiQ4u6V2Y8UCU+uM7luxovZX061iDuAkIR1Q=; b=HAKkoUH6BqtuWekZ+4YUTCJzc4/bA0/sPJbiAnTdQM+a4eOXJHSGwRpccIXExLg7zm zNUrSlzrIcgmLKg1C1VcOQKRGg8btermgj3WMDezKhj60CTMCrWhx3WlUgVnPWqN1M58 jUz6pxzZ5+6IX6L4EGvefMGF1QPn9qPColbZNBBsOXEOTFoJlKMjnRBwhg9e74A/ifAL sOvRmDpTAr3fXuXluNUaoGz7xioG/7+iKtTiFBHHL0ksU6Lw3jjXtDdfZu0hdPEgNSUL vMBnHDSznuAPWGA/lDrWitbbsofUk9ERd9L2OonCQwbwFqyVVSx3WytwwWeEf1tleAd+ Bf5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=zHv62ZayiQ4u6V2Y8UCU+uM7luxovZX061iDuAkIR1Q=; b=qomoeFwP0RIy5/Wvx4xlpd8C16smc6M0IIfxvNb/SQD8qyvA63/AT2tbqHYeQ9zuCy 9w7FI/YjKPJceqqUIjaesvUfLlq1sqmm/2FNSGwLjx6su0LxkBvykrYATj9VOUQld6Z/ kh/LMPOLSJnJ30u7QWzYb1y2jy4shIRVfD2Xr6Ixxck12dm8WuENu4+ak79iA6suHUlY UW6jxMw0bR7cAwchm06xGjKxNJZVUG00eo7EsP1zXw7EX4oyDJ4nA+OuoefvTph5yv5b ZeN6ZoNiK7Y5HQJNxRw8vUHEcAWR0unbczI8BBwsB/Y+tFiWOKtAakIXbc/wJkFKfyRB 3iHQ== X-Gm-Message-State: AHQUAubVM+s7IrN99HHKBNu571ePhvxzn4PtUDieXJai5VFXrGREFlKp W5MOAsUKWp78vJoFRfFgvIocRg== X-Google-Smtp-Source: AHgI3IaJ1tRYDomG+6MQpLjZwBRkEwLjl4w8rZ6ctkAk2LR1AOvschXWBZeTCGhvzJKF2Qt6hiStdg== X-Received: by 2002:a5d:6907:: with SMTP id t7mr2790035wru.226.1549359465364; Tue, 05 Feb 2019 01:37:45 -0800 (PST) Received: from tron.luk.heistp.net (h-1169.lbcfree.net. [185.193.85.130]) by smtp.gmail.com with ESMTPSA id x186sm25742278wmg.41.2019.02.05.01.37.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 05 Feb 2019 01:37:44 -0800 (PST) From: Pete Heist Message-Id: <2A29A2E5-5845-4CC4-BB5A-C92464C57920@heistp.net> Content-Type: multipart/alternative; boundary="Apple-Mail=_28CF7CCC-A94A-42B4-976C-BCE3951736CF" Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Date: Tue, 5 Feb 2019 10:37:43 +0100 In-Reply-To: Cc: bloat To: Dave Taht References: X-Mailer: Apple Mail (2.3445.9.1) Subject: Re: [Bloat] $106 achieved and flent-farm status X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2019 09:37:46 -0000 --Apple-Mail=_28CF7CCC-A94A-42B4-976C-BCE3951736CF Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Feb 5, 2019, at 4:37 AM, Dave Taht wrote: >=20 > Thank you mikael and jake, matt and matthew and richard! (and jon, and > dev for trying) +1 > and here's a puzzler for you! Both boxes are running ntp yet one box > was still *30* seconds off. I haven=E2=80=99t explored this fully but I find it works best when ntp = is configured in an identical way across servers, either all running = systemd-timesyncd, or ntpd, or chronyd, and to the same server pool. = When that=E2=80=99s the case, I often see it works within a few = milliseconds. For me, it looks like clocks are currently about 10ms off to london and = 75ms off to singapore. I=E2=80=99m using systemd-timesyncd with the = default Debian servers- "0.debian.pool.ntp.org 1.debian.pool.ntp.org = 2.debian.pool.ntp.org 3.debian.pool.ntp.org=E2=80=9D. > irtt 1ms between california and germany has a mindboggling amount of > loss... in america... ahh... tools=E2=80=A6 Curiously, I see less loss at 1ms to singapore than london, london=E2=80=99= s loss comes from the downstream, and I wasn=E2=80=99t seeing it four = days ago. The upstream loss is my NLOS uplink as I get it straight to my = next hop, which also has an irtt server. Next hop router: packets sent/received: 4999/4864 (2.70% loss) server packets received: 4864/4999 (2.70%/0.00% loss up/down) London: packets sent/received: 5000/4124 (17.52% loss) server packets received: 4817/5000 (3.66%/14.39% loss up/down) Singapore: packets sent/received: 5000/4830 (3.40% loss) server packets received: 4830/5000 (3.40%/0.00% loss up/down) Also curiously, RTT to london has increased for some reason, mainly with = increased receive delay, but this could easily be from our peering = provider, which we hope to switch soon for other reasons. Feb. 1, 2019 ------------ $ irtt client -q -i 10ms -d 1s flent-london.bufferbloat.net = [Connecting] connecting to flent-london.bufferbloat.net = [176.58.107.8:2112] [Connected] connection established [176.58.107.8:2112] [WaitForPackets] waiting 135ms for final packets Min Mean Median Max Stddev --- ---- ------ --- ------ RTT 31.08ms 37.09ms 37.04ms 44.98ms 2.75ms send delay 16.02ms 21.55ms 21.39ms 29.51ms 2.67ms receive delay 14.3ms 15.54ms 15.61ms 17.2ms 470=C2=B5s =20 IPDV (jitter) 49.5=C2=B5s 3.27ms 2.45ms 10.85ms 2.88ms send IPDV 63.4=C2=B5s 3.16ms 2.22ms 10.79ms 2.84ms receive IPDV 903ns 400=C2=B5s 203=C2=B5s 2.82ms = 506=C2=B5s =20 send call time 21.3=C2=B5s 79.1=C2=B5s 146=C2=B5s = 28.9=C2=B5s timer error 1.84=C2=B5s 721=C2=B5s 2.22ms = 501=C2=B5s server proc. time 1.29=C2=B5s 31.4=C2=B5s 2.23ms = 222=C2=B5s duration: 1.13s (wait 135ms) packets sent/received: 100/100 (0.00% loss) server packets received: 100/100 (0.00%/0.00% loss up/down) bytes sent/received: 6000/6000 send/receive rate: 48.5 Kbps / 48.8 Kbps packet length: 60 bytes timer stats: 0/100 (0.00%) missed, 7.21% error Feb. 5, 2019 ------------ $ irtt client -q -i 10ms -d 1s flent-london.bufferbloat.net [Connecting] connecting to flent-london.bufferbloat.net [176.58.107.8:2112] [Connected] connection established [176.58.107.8:2112] [WaitForPackets] waiting 176.8ms for final packets Min Mean Median Max Stddev --- ---- ------ --- ------ RTT 53.38ms 54.59ms 54.38ms 58.93ms 1.05ms send delay 16.81ms 18.07ms 17.87ms 22.17ms 998=C2=B5s receive delay 35.72ms 36.52ms 36.5ms 38.07ms 333=C2=B5s =20 IPDV (jitter) 47.1=C2=B5s 960=C2=B5s 370=C2=B5s 4.69ms = 1.03ms send IPDV 13.4=C2=B5s 974=C2=B5s 646=C2=B5s 4.69ms = 1.04ms receive IPDV 5.05=C2=B5s 379=C2=B5s 287=C2=B5s 1.51ms = 322=C2=B5s =20 send call time 32.5=C2=B5s 35.5=C2=B5s 61=C2=B5s = 3.65=C2=B5s timer error 90ns 15.1=C2=B5s 190=C2=B5s = 25.5=C2=B5s server proc. time 7.47=C2=B5s 13.4=C2=B5s 191=C2=B5s = 20.6=C2=B5s duration: 1.17s (wait 176.8ms) packets sent/received: 99/77 (22.22% loss) server packets received: 99/99 (0.00%/22.22% loss up/down) bytes sent/received: 5940/4620 send/receive rate: 48.0 Kbps / 37.3 Kbps packet length: 60 bytes timer stats: 1/100 (1.00%) missed, 0.15% error --Apple-Mail=_28CF7CCC-A94A-42B4-976C-BCE3951736CF Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
On = Feb 5, 2019, at 4:37 AM, Dave Taht <dave.taht@gmail.com>= wrote:

Thank you mikael and jake, matt and matthew and richard! (and = jon, and
dev for trying)

+1

and here's a puzzler for you! = Both boxes are running ntp yet one box
was still *30* = seconds off.

I haven=E2=80=99t explored this fully but I find = it works best when ntp is configured in an identical way across servers, = either all running systemd-timesyncd, or ntpd, or chronyd, and to the = same server pool. When that=E2=80=99s the case, I often see it works = within a few milliseconds.

For me, = it looks like clocks are currently about 10ms off to london and 75ms off = to singapore. I=E2=80=99m using systemd-timesyncd with the default = Debian servers- "0.debian.pool.ntp.org 1.debian.pool.ntp.org= 2.debian.pool.ntp.org 3.debian.pool.ntp.org=E2=80=9D.

irtt 1ms between california and germany has a mindboggling = amount of
loss... in america... ahh... = tools=E2=80=A6

Curiously, I see less loss at 1ms to singapore = than london, london=E2=80=99s loss comes from the downstream, and I = wasn=E2=80=99t seeing it four days ago. The upstream loss is my NLOS = uplink as I get it straight to my next hop, which also has an irtt = server.

Next hop = router:
   packets = sent/received: 4999/4864 (2.70% loss)
 server = packets received: 4864/4999 (2.70%/0.00% loss = up/down)

London:
   packets = sent/received: 5000/4124 (17.52% loss)
 server packets = received: 4817/5000 (3.66%/14.39% loss up/down)

Singapore:
   packets sent/received: 5000/4830 (3.40% = loss)
 server packets received: 4830/5000 = (3.40%/0.00% loss up/down)

Also curiously, RTT to london has = increased for some reason, mainly with increased receive delay, but this = could easily be from our peering provider, which we hope to switch soon = for other reasons.

Feb. 1, = 2019
------------

$ irtt client -q -i 10ms -d 1s flent-london.bufferbloat.net
[Connecting] connecting = to flent-london.bufferbloat.net
[176.58.107.8:2112] = [Connected] connection established
[176.58.107.8:2112] [WaitForPackets] waiting = 135ms for final packets

            =              Min     Mean =   Median      Max  Stddev
        =                  ---   =   ----   ------      --- =  ------
                RTT =  31.08ms  37.09ms  37.04ms  44.98ms =  2.75ms
         send delay  16.02ms =  21.55ms  21.39ms  29.51ms  2.67ms
      receive = delay   14.3ms  15.54ms  15.61ms   17.2ms   = 470=C2=B5s
                =                     =                     =        
      IPDV (jitter)   = 49.5=C2=B5s   3.27ms   2.45ms  10.85ms =  2.88ms
          send IPDV   63.4=C2=B5s=   3.16ms   2.22ms  10.79ms  2.84ms
      =  receive IPDV    903ns    400=C2=B5s   =  203=C2=B5s   2.82ms   506=C2=B5s
        =                     =                     =                
     send = call time   21.3=C2=B5s   79.1=C2=B5s       =       146=C2=B5s  28.9=C2=B5s
        = timer error   1.84=C2=B5s    721=C2=B5s     =        2.22ms   501=C2=B5s
  server proc. time =   1.29=C2=B5s   31.4=C2=B5s           =  2.23ms   222=C2=B5s

        =         duration: 1.13s (wait = 135ms)
 =  packets sent/received: 100/100 (0.00% loss)
 server packets = received: 100/100 (0.00%/0.00% loss up/down)
     bytes = sent/received: 6000/6000
       send/receive rate: 48.5 Kbps / = 48.8 Kbps
           packet length: 60 = bytes
  =            timer stats: 0/100 (0.00%) = missed, 7.21% error

Feb. 5, = 2019
------------

$ irtt client -q -i 10ms -d 1s flent-london.bufferbloat.net
[Connecting] connecting to = flent-london.bufferbloat.net
[176.58.107.8:2112] = [Connected] connection established
[176.58.107.8:2112] [WaitForPackets] waiting = 176.8ms for final packets

        =                  Min   =   Mean   Median      Max =  Stddev
                =          ---     ----   ------ =      ---  ------
            =     RTT  53.38ms  54.59ms  54.38ms =  58.93ms  1.05ms
         send delay =  16.81ms  18.07ms  17.87ms  22.17ms   = 998=C2=B5s
      receive delay  35.72ms =  36.52ms   36.5ms  38.07ms   333=C2=B5s
      =                     =                     =                 =  
 =     IPDV (jitter)   47.1=C2=B5s    960=C2=B5s =    370=C2=B5s   4.69ms  1.03ms
        =   send IPDV   13.4=C2=B5s    974=C2=B5s   =  646=C2=B5s   4.69ms  1.04ms
      =  receive IPDV   5.05=C2=B5s    379=C2=B5s   =  287=C2=B5s   1.51ms   322=C2=B5s
        =                     =                     =                
     send = call time   32.5=C2=B5s   35.5=C2=B5s       =        61=C2=B5s  3.65=C2=B5s
        = timer error     90ns   15.1=C2=B5s       =       190=C2=B5s  25.5=C2=B5s
  server proc. time =   7.47=C2=B5s   13.4=C2=B5s           =   191=C2=B5s  20.6=C2=B5s

        =         duration: 1.17s (wait = 176.8ms)
   packets sent/received: 99/77 (22.22% = loss)
 server packets received: 99/99 (0.00%/22.22% loss = up/down)
     bytes sent/received: = 5940/4620
       send/receive rate: 48.0 Kbps / = 37.3 Kbps
           packet length: 60 = bytes
  =            timer stats: 1/100 (1.00%) = missed, 0.15% error

= --Apple-Mail=_28CF7CCC-A94A-42B4-976C-BCE3951736CF--