From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ia0-x229.google.com (ia-in-x0229.1e100.net [IPv6:2607:f8b0:4001:c02::229]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id C606021F0AE; Fri, 22 Feb 2013 08:48:40 -0800 (PST) Received: by mail-ia0-f169.google.com with SMTP id j5so720714iaf.0 for ; Fri, 22 Feb 2013 08:48:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:date:message-id:subject:from:to :content-type; bh=ERY/waotBc899CEVQksj615mJ9VgGl1x1DTlhyet0ks=; b=zsH/nRpz/zXL0QDkSu+pY99dN/nu5jhK9uTuxd96993JtZ416xx22MLnjWD2uaQAoO apFxljOJUDzYtQlq+Fvd1qqSVDvAnVvf+/PtWlUsULQY6XqnuObKsADdOCWt6IA/vWJm QLhUeWjSLaF8a0hS4h7be8q09ZylIppLcvF1Rzucnrs2D7lQ8ziXwtUmXqvBUcJiUOU5 kscsXrcOt46MuvMnU/aY5YK+s4CRp8so6bvYifV3TEfqzGWYFpmzxgo4KwpqL6cmFktj yX9VyVDKpTKDQxhalqVbPUCnn5z8WyLT23wH40f0YA6GyxeO8XzxjvZg5CjRbb78vvr9 414g== MIME-Version: 1.0 X-Received: by 10.50.196.234 with SMTP id ip10mr1185405igc.27.1361551719571; Fri, 22 Feb 2013 08:48:39 -0800 (PST) Received: by 10.64.77.1 with HTTP; Fri, 22 Feb 2013 08:48:39 -0800 (PST) Date: Fri, 22 Feb 2013 11:48:39 -0500 Message-ID: From: Dave Taht To: cerowrt-devel@lists.bufferbloat.net, bloat Content-Type: multipart/alternative; boundary=14dae934121396aa3f04d652f65b Subject: [Cerowrt-devel] Fwd: High Speed WAN Rsync now possible via UDT X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Feb 2013 16:48:41 -0000 --14dae934121396aa3f04d652f65b Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Since we are essentially observing "wan" latencies anyway I am curious as to how the UDT protocol functions... has anyone tried it? If useful (or scary) I'll try to find the time to package it up for cero. I liked how mosh solved the terminal emulation problem over lousy links (in fact, reflecting on it, had it existed in 1998 when I was mucking with the strip protocol, I'd have not bothered with getting tcp to work at all. :) ) I also get a kick out of people using ssh to "authenticate" and then dropping to something else.... ---------- Forwarded message ---------- From: Erich Weiler Date: Fri, Feb 22, 2013 at 10:54 AM Subject: High Speed WAN Rsync now possible!!! To: rsync@lists.samba.org Folks- Just wanted to plug a totally awesome software package from a group I know: UDR (UDT Enabled Rsync). For those not familiar with UDT, it is a low level network protocol based on UDP that allows for high speed transfers over high latency WAN networks: http://udt.sourceforge.net/ For a while the UDT API was available, developed by the Laboratory for Advanced Computing at the University of Chicago, but there were little development around it for actually developing a software suite to allow for high speed WAN transfers, such as can be achieved by Aspera, GridFTP, FDT, and a couple others. The problem with those often is: Aspera: Great but *crazy* expensive GridFTP: Not bad but non-trivial to set up FDT: Java (Eh...) But the awesome thing here is that UDR is a lightweight wrapper for rsync that allows for rsync functionality, but uses UDT as the underlying protocol for transfer (no rsh or ssh). It authenticates over ssh and then transfers the data over UDT streams. And supports encryption. Right now it is kind of in beta but we've been using it for a while and it's very stable. It has been tested on Linux, BSD and OSX. It may compile on other platforms but not much testing has been done on those. Written in C++. https://github.com/LabAdvComp/**UDR You can clone the repo with 'git clone' and build the code. It compiles very easily and only requires the OpenSSL library as a dependency. As an example: We have been trying to transfer data from California to our servers in Germany for a while, and have only be getting 10Mb/s. With UDR we get 700Mb/s. Not bad. There are details on the GitHub page. Check it out!! -erich --=20 Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/** mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/**smart-questions.html --=20 Dave T=E4ht Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html --14dae934121396aa3f04d652f65b Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Since we are essentially observing "wan" latencies anyway I am cu= rious as to how the UDT protocol functions... has anyone tried it?

I= f useful (or scary) I'll try to find the time to package it up for cero= .

I liked how mosh solved the terminal emulation problem over lousy links= (in fact, reflecting on it, had it existed in 1998 when I was mucking with= the strip protocol, I'd have not bothered with getting tcp to work at = all. :) ) I also get a kick out of people using ssh to "authenticate&q= uot; and then dropping to something else....


---------- Forwarded message ----------<= br>From: Erich Weiler &= lt;weiler@soe.ucsc.edu>
Date: Fri, Feb 22, 2013 at 10:54 AM
Subject: High Speed WAN Rsync now po= ssible!!!
To: rsync@lists.samba= .org


Folks-

Just wanted to plug a totally awesome software package from a group I know:= UDR (UDT Enabled Rsync).

For those not familiar with UDT, it is a low level network protocol based o= n UDP that allows for high speed transfers over high latency WAN networks:<= br>
http://udt.source= forge.net/

For a while the UDT API was available, developed by the Laboratory for Adva= nced Computing at the University of Chicago, but there were little developm= ent around it for actually developing a software suite to allow for high sp= eed WAN transfers, such as can be achieved by Aspera, GridFTP, FDT, and a c= ouple others. =A0The problem with those often is:

Aspera: Great but *crazy* expensive
GridFTP: Not bad but non-trivial to set up
FDT: Java (Eh...)

But the awesome thing here is that UDR is a lightweight wrapper for rsync t= hat allows for rsync functionality, but uses UDT as the underlying protocol= for transfer (no rsh or ssh). =A0It authenticates over ssh and then transf= ers the data over UDT streams. =A0And supports encryption.

Right now it is kind of in beta but we've been using it for a while and= it's very stable. =A0It has been tested on Linux, BSD and OSX. =A0It m= ay compile on other platforms but not much testing has been done on those. = =A0Written in C++.

https://git= hub.com/LabAdvComp/UDR

You can clone the repo with 'git clone' and build the code. =A0It c= ompiles very easily and only requires the OpenSSL library as a dependency.<= br>
As an example: =A0We have been trying to transfer data from California to o= ur servers in Germany for a while, and have only be getting 10Mb/s. With UD= R we get 700Mb/s. =A0Not bad.

There are details on the GitHub page. =A0Check it out!!

-erich
--
Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/= listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-ques= tions.html



--
Dave T=E4ht

Fix= ing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.h= tml=20 --14dae934121396aa3f04d652f65b--