From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qt1-x834.google.com (mail-qt1-x834.google.com [IPv6:2607:f8b0:4864:20::834]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 944E03BA8E for ; Mon, 4 Feb 2019 22:37:29 -0500 (EST) Received: by mail-qt1-x834.google.com with SMTP id b8so2472604qtj.1 for ; Mon, 04 Feb 2019 19:37:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to :content-transfer-encoding; bh=ov8UDl7zKGHCejpWXK8L1Fe/2Yi34NyMvIym/kwfTwY=; b=ryyhCzy3/lo7UUfpXCBZksL34CIlNedXYxLi8xLkp+xoQePc1eHd1zMV4RAMo1gZ6h 2yd3SkCuVx6akekT0dkVROzfl6v2nIclpl2g3K85EIlJnjAtvzi8Dyym8djmHhUonTvv nUSZew8Se1eVTan6Ct0caemZl7cK9r8Wxgw2idZCToBEgZdhVort4lMDE8o0uImaCkS+ bGz4Qq2iujRy7LM4yRyRv22FCDn7wWkDu/AzKoy3zEi756VAP+IYKlnjlGupA4BXkr4T uVvpLUOs7z4iuMs4HWFPPmnfzutfh65h5JvhLue8rCi0AktDsSUfmIHxDLFDkK2Lu05/ vHww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to :content-transfer-encoding; bh=ov8UDl7zKGHCejpWXK8L1Fe/2Yi34NyMvIym/kwfTwY=; b=pNj8M6F5Teb/lAo3GOBs2fl72WJu7WgFKn0yXdQ2vevHQEYmirW7Owp0ERpnbuyzJd sw4HtfTO//GTQHKPnNa2WCEvrTrLFFUkQSZn1l61CP9QfMj48FIGMiwQmVfXqgzlfX+e gay8s7nvRfZrAQu534up94icClTtIiMBCC2TEbBUWaUvoKlu2U01vaBMheMvKE1YC11U b/N7RlXS3NGaAe75XHkwo83XG1P5i02aGjT3oDSioVAVQndqssKZdg5wgLbYxpjvBxnf Z+07ob9n9oQ4Q97DZbbRQMW3VXGly0rGWxbjAaKvFFxkYYfnls3VgBFZGD8vUCIhPmn0 vq+A== X-Gm-Message-State: AHQUAuZ7h2j+Lz+ik1nv7KPiIGGVWzXHDmKxNUdO+mCgaXRowlpVtYmc HWKIky4AzyL8HqSlFXWfvD0QEKMGXOeYZkfvUU+O6Pyh X-Google-Smtp-Source: AHgI3IYpjvUaJl7uRFH28S39Lf7/41wnDHwFslBpmMsliAy1B94z85T+bh8Hni4xcd6Pvsnk5vLb2bxWCNuKH+aFCSs= X-Received: by 2002:a0c:8a5a:: with SMTP id 26mr2056762qvu.94.1549337848062; Mon, 04 Feb 2019 19:37:28 -0800 (PST) MIME-Version: 1.0 From: Dave Taht Date: Mon, 4 Feb 2019 19:37:14 -0800 Message-ID: To: bloat Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: [Bloat] $106 achieved and flent-farm status X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2019 03:37:29 -0000 Thank you mikael and jake, matt and matthew and richard! (and jon, and dev for trying) the linode bill is now paid. For the record, peak "earnings" for the patreon contribution page was $212 in may 2016, and fell below $100 in May, 2018. I forget when I scaled the number of servers back in between those times. In cerowrt's glory days we were burning ~2k a month on the openwrt build farm, which I had to quit doing 6 months after the google grant did. Prior to that we had donated servers from ISC, which had to get out of the free ISP business. I kind of miss running on bare metal, with tons of disk space. Cloud services are pretty cheap, cloud storage isn't. Oy! I remember the headaches and hassles when the number of openwrt buildslaves dropped to 1 for 6 weeks during a critical phase.... Thankfully that farm looks healthy at the moment: http://phase1.builds.openwrt.org/buildslaves Costs on the "flent-farm" continue to drop. Our earliest linode servers cost $20/month and our two latest ones (nanoservers) cost $5/month. For "science!" I've been generally unwilling to update/change these much, the most critical keeping the same kernel versions they had for the last couple years. I note that linode at least a year+ ago, started defaulting to a kernel with fq_codel enabled *by default*, bql just works, irtt, flent, netperf all "just install" from apt, etc, etc. It's currently a matter of a few minutes to get a basic flent server running in the cloud, and sometimes I wish we had more worldwide coverage - australia - aws - france - china - It turns out arm based servers are quite cheap nowadays... yep, here I am 6 bucks to the good and trying to figure out another server on which to spend it. :) I have not been maintaining the flent network all that well, but your support inspired me to go fix two of them. flent-singapore.taht.net: the newest of these servers de.taht.net: the second newest now support BBR, cubic, and reno, and have irtt running with support for 1ms intervals. (thx pete for showing me how on flent-london.bufferbloat.net) Linux flent-singapore.taht.net 4.18.16-x86_64-linode118 #1 SMP PREEMPT Mon Oct 29 15:38:25 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Linux de.taht.net 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux In the teklibre sub-basement are a few dozen tiny openwrt/armbian boxes, and two honkin, but ancient 12 core xeon servers ("spaceheater" and "ceres") with 10TB of storage. I keep a variety of vms on those in complicated network emulation setups, an indexed copy of all the source code in the world, my openwrt build system, etc. and here's a puzzler for you! Both boxes are running ntp yet one box was still *30* seconds off. irtt 1ms between california and germany has a mindboggling amount of loss... in america... ahh... tools... 5. be-3651-cr02.sunnyvale.ca.ibone.comcast.net 71.9% 33 11.4 11.6 9.4 16.0 2.3 6. be-11083-pe02.529bryant.ca.ibone.comcast.net 56.2% 33 13.1 11.3 9.5 14.8 1.6 Min Mean Median Max Stddev --- ---- ------ --- ------ RTT 166.2ms 175.4ms 174.3ms 232.7ms 4.85ms send delay 27.9s 27.9s 27.9s 28s 4.07ms receive delay -27.7s -27.7s -27.7s -27.7s 1.98ms IPDV (jitter) 434ns 1.46ms 979=C2=B5s 37.64ms 1.78ms send IPDV 408ns 1.36ms 954=C2=B5s 34.92ms 1.51ms receive IPDV 2ns 378=C2=B5s 72.2=C2=B5s 32.98ms 1.08ms send call time 3.55=C2=B5s 14.9=C2=B5s 1.61ms 16.7=C2= =B5s timer error 0s 19.6=C2=B5s 5.52ms 68.3=C2=B5s server proc. time 710ns 7.67=C2=B5s 10.33ms 91.6=C2=B5s duration: 1m1s (wait 698.1ms) packets sent/received: 58852/44855 (23.78% loss) server packets received: 45579/58852 (22.55%/1.59% loss up/down) bytes sent/received: 3531120/2691300 send/receive rate: 470.8 Kbps / 358.8 Kbps packet length: 60 bytes timer stats: 1148/60000 (1.91%) missed, 1.96% error restarted ntp, and: Min Mean Median Max Stddev --- ---- ------ --- ------ RTT 166.3ms 175.9ms 175.1ms 233ms 4.78ms send delay 83.97ms 92.57ms 91.9ms 137.4ms 4.1ms receive delay 80.86ms 83.32ms 82.86ms 137.4ms 1.89ms IPDV (jitter) 316ns 1.44ms 977=C2=B5s 63.68ms 1.75ms send IPDV 41ns 1.35ms 933=C2=B5s 35.49ms 1.5ms receive IPDV 2ns 345=C2=B5s 83.8=C2=B5s 55.06ms 971=C2= =B5s send call time 3.55=C2=B5s 16.4=C2=B5s 1.07ms 11.8=C2= =B5s timer error 1ns 23.4=C2=B5s 5.02ms 76.8=C2=B5s server proc. time 716ns 8.22=C2=B5s 13.13ms 117=C2=B5s duration: 1m1s (wait 699.1ms) packets sent/received: 58779/45154 (23.18% loss) server packets received: 45912/58779 (21.89%/1.65% loss up/down) bytes sent/received: 3526740/2709240 send/receive rate: 470.2 Kbps / 361.2 Kbps packet length: 60 bytes timer stats: 1221/60000 (2.04%) missed, 2.34% error Verses singapore. Min Mean Median Max Stddev --- ---- ------ --- ------ RTT 173.6ms 179.2ms 178.7ms 194.7ms 2.5ms send delay 113ms 118.5ms 118ms 131.9ms 2.32ms receive delay 59.7ms 60.68ms 60.45ms 74.75ms 980=C2=B5s IPDV (jitter) 203ns 1.24ms 978=C2=B5s 18.8ms 1.01ms send IPDV 20ns 1.19ms 983=C2=B5s 14ms 881=C2=B5s receive IPDV 0s 195=C2=B5s 20.3=C2=B5s 14.11ms 539=C2= =B5s send call time 3.67=C2=B5s 14.7=C2=B5s 1.75ms 13.2=C2= =B5s timer error 1ns 21.4=C2=B5s 7.82ms 93.9=C2=B5s server proc. time 680ns 3.27=C2=B5s 435=C2=B5s 9.35=C2= =B5s duration: 1m1s (wait 584ms) packets sent/received: 58709/55848 (4.87% loss) server packets received: 55848/58709 (4.87%/0.00% loss up/down) bytes sent/received: 3522540/3350880 send/receive rate: 469.7 Kbps / 446.8 Kbps packet length: 60 bytes timer stats: 1290/59999 (2.15%) missed, 2.14% error --=20 Dave T=C3=A4ht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740