From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ob0-x22c.google.com (mail-ob0-x22c.google.com [IPv6:2607:f8b0:4003:c01::22c]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 0F46021F29C for ; Sat, 15 Nov 2014 14:47:07 -0800 (PST) Received: by mail-ob0-f172.google.com with SMTP id wn1so760019obc.31 for ; Sat, 15 Nov 2014 14:47:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=SEd3c1jXXCDDGVjs89pALZ8CN3f/2exBfwZfHz6kRMU=; b=iNV2xneXEDzA4Z7sFxRTPje7kJF/Rxv3zIfqmJqo4U9SCOo/h3rvrh75ktjdbHJLyV xwVBF5hjbWWu2IRy1lobTaw21HVS62QKWbnLToEg3ZWtBZtfqmlfWopkNXg7byGOx+A6 2CWk7TAYlzNWMg7e3aRl6SoJ1MSv+c/4KzuE76E54H9JKDlETVlXAuUF7JYKOwMYCZmb NB8KdL7BL7fshcV+ULkhDbrQsm99cSPaFccIKnhA20aQLZgRiZp1+gBsR7/X8aO50qpl RQDZ89N5HyWTIrC56n9AUKnuUEhVDhLbhrbVMz4KRRPMpDPeq3WQE1M26AHzcsXOcZRk mFSg== MIME-Version: 1.0 X-Received: by 10.182.79.41 with SMTP id g9mr15257197obx.14.1416091626971; Sat, 15 Nov 2014 14:47:06 -0800 (PST) Received: by 10.202.227.211 with HTTP; Sat, 15 Nov 2014 14:47:06 -0800 (PST) In-Reply-To: References: Date: Sat, 15 Nov 2014 14:47:06 -0800 Message-ID: From: Dave Taht To: Frank Horowitz Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "cerowrt-devel@lists.bufferbloat.net" Subject: Re: [Cerowrt-devel] High Performance (SSH) Data Transfers using fq_codel? X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Nov 2014 22:47:36 -0000 On Fri, Oct 17, 2014 at 4:21 PM, Frank Horowitz wrote: > G=E2=80=99Day folks, > > Long time lurker. I=E2=80=99ve been using Cero for my home router for qui= te a while now, with reasonable results (modulo bloody OSX wifi stuffola). > > I=E2=80=99m running into issues doing zfs send/receive over ssh across a = (mostly) internet2 backbone between Cornell (where I work) and West Virgini= a University (where we have a collaborator on a DOE sponsored project. Both= ends are linux machines running fq_codel configured like so: > tc qdisc > qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 = quantum 1514 target 5.0ms interval 100.0ms ecn So, in re-reading this now, I think I now grok that that A) Frank is using cerowrt at home, and B) he has it hooked up on two cool boxes connected on either side of Internet2, and B) is what the question was about. so my general assumption is that his boxes are x86, and hooked up at 1GigE or so to the Internet2. So answers A) If after a big transfer If a tc -s qdisc show dev eth0 # on both sides shows no drops or ecn marks, his linux servers are not the bottleneck link, and he should use mtr to find the real bottleneck link elsewhere, during that transfer. IF, after a big transfer drops are seen, you are (at least some of the time) a bottleneck link. Enable ecn between both tcps. And if you are willing to tolerate more latency on the link, feel free to increase the target and interval to values you are more comfortable with, but you won=C2=B4t increase actual bandwidth by all that much. Personally I suspect =C2=A8A=C2=A8 as the problem. And as per my original msg, it always helps to measure, and the rrul test between the two points is the best thing we got. > I stumbled across hpn-ssh and =E2= =80=94 of particular interest to this group =E2=80=94 their page on tuning= TCP parameters: > > > > N.B. their advice to increase buffer size=E2=80=A6 > > I=E2=80=99m curious, what part (if any) of that advice survives with fq_c= odel running on both ends? Most of that seems to apply to TCPs. I would suspect that enabling TCP pacing between the two points might be helpful, but without data on whatever problem(s) you are experiencing on your path, can=C2=B4t help. Amusingly, matt mathis is one the original authors of that page, and perhaps he has new advice. > Any advice from the experts here would be gratefully received! > > (And thanks for all of your collective and individual efforts!) > > Cheers, > Frank Horowitz > > > _______________________________________________ > Cerowrt-devel mailing list > Cerowrt-devel@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cerowrt-devel > --=20 Dave T=C3=A4ht thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks