[Cerowrt-devel] High Performance (SSH) Data Transfers using fq_codel?

[Cerowrt-devel] High Performance (SSH) Data Transfers using fq_codel?

[Frank Horowitz]

[-- Attachment #1: Type: text/plain, Size: 1077 bytes --]

G’Day folks,

Long time lurker. I’ve been using Cero for my home router for quite a while now, with reasonable results (modulo bloody OSX wifi stuffola).

I’m running into issues doing zfs send/receive over ssh across a (mostly) internet2 backbone between Cornell (where I work) and West Virginia University (where we have a collaborator on a DOE sponsored project. Both ends are linux machines running fq_codel configured like so:
	tc qdisc
	qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn

I stumbled across hpn-ssh <https://www.psc.edu/index.php/hpn-ssh> and —  of particular interest to this group — their page on tuning TCP parameters:

<http://www.psc.edu/index.php/networking/641-tcp-tune>

N.B. their advice to increase buffer size…

I’m curious, what part (if any) of that advice survives with fq_codel running on both ends? 

Any advice from the experts here would be gratefully received!

(And thanks for all of your collective and individual efforts!)

Cheers,
	Frank Horowitz


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 841 bytes --]

Re: [Cerowrt-devel] High Performance (SSH) Data Transfers using fq_codel?

[Dave Taht]

One thing the HPC crowd has missed is that in their quest for big
buffers for contenental distances, they hurt themselves on shorter
ones...

... and also that big buffers with FQ on them works just fine in the
general case.

As always I recomend benchmarking - do a rrul test between the two
points, for example, with their recomendations.


On Fri, Oct 17, 2014 at 4:21 PM, Frank Horowitz <frank@horow.net> wrote:
> G’Day folks,
>
> Long time lurker. I’ve been using Cero for my home router for quite a while now, with reasonable results (modulo bloody OSX wifi stuffola).
>
> I’m running into issues doing zfs send/receive over ssh across a (mostly) internet2 backbone between Cornell (where I work) and West Virginia University (where we have a collaborator on a DOE sponsored project. Both ends are linux machines running fq_codel configured like so:
>         tc qdisc
>         qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
>
> I stumbled across hpn-ssh <https://www.psc.edu/index.php/hpn-ssh> and —  of particular interest to this group — their page on tuning TCP parameters:
>
> <http://www.psc.edu/index.php/networking/641-tcp-tune>
>
> N.B. their advice to increase buffer size…
>
> I’m curious, what part (if any) of that advice survives with fq_codel running on both ends?
>
> Any advice from the experts here would be gratefully received!
>
> (And thanks for all of your collective and individual efforts!)
>
> Cheers,
>         Frank Horowitz
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>



-- 
Dave Täht

thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

Re: [Cerowrt-devel] High Performance (SSH) Data Transfers using fq_codel?

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 2725 bytes --]

I have a couple friends in that crowd, and they _also_ aren't using shared
lines.  So they don't worry in the slightest about congestion when they're
trying to keep dedicated links fully saturated.  They're big issue with
dropped packets is that some of the TCP congestion-control algorithms kick
in on a single dropped packet:
http://fasterdata.es.net/network-tuning/tcp-issues-explained/packet-loss/

I'm thinking that some forward-error-correction would make their lives
much, much better.

-Aaron

On Thu, Nov 13, 2014 at 7:11 PM, Dave Taht <dave.taht@gmail.com> wrote:

> One thing the HPC crowd has missed is that in their quest for big
> buffers for contenental distances, they hurt themselves on shorter
> ones...
>
> ... and also that big buffers with FQ on them works just fine in the
> general case.
>
> As always I recomend benchmarking - do a rrul test between the two
> points, for example, with their recomendations.
>
>
> On Fri, Oct 17, 2014 at 4:21 PM, Frank Horowitz <frank@horow.net> wrote:
> > G’Day folks,
> >
> > Long time lurker. I’ve been using Cero for my home router for quite a
> while now, with reasonable results (modulo bloody OSX wifi stuffola).
> >
> > I’m running into issues doing zfs send/receive over ssh across a
> (mostly) internet2 backbone between Cornell (where I work) and West
> Virginia University (where we have a collaborator on a DOE sponsored
> project. Both ends are linux machines running fq_codel configured like so:
> >         tc qdisc
> >         qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024
> quantum 1514 target 5.0ms interval 100.0ms ecn
> >
> > I stumbled across hpn-ssh <https://www.psc.edu/index.php/hpn-ssh> and
> —  of particular interest to this group — their page on tuning TCP
> parameters:
> >
> > <http://www.psc.edu/index.php/networking/641-tcp-tune>
> >
> > N.B. their advice to increase buffer size…
> >
> > I’m curious, what part (if any) of that advice survives with fq_codel
> running on both ends?
> >
> > Any advice from the experts here would be gratefully received!
> >
> > (And thanks for all of your collective and individual efforts!)
> >
> > Cheers,
> >         Frank Horowitz
> >
> >
> > _______________________________________________
> > Cerowrt-devel mailing list
> > Cerowrt-devel@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cerowrt-devel
> >
>
>
>
> --
> Dave Täht
>
> thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>

[-- Attachment #2: Type: text/html, Size: 4021 bytes --]

Re: [Cerowrt-devel] High Performance (SSH) Data Transfers using fq_codel?

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 3703 bytes --]

Filling intermediate buffers doesn't make the tcp congestion algorithms work.  They just kick in when the buffers are full! And then you end up with a pile of packets that will be duplicated which amplifies the pressure on buffers!

If there could be no buffering,the big file transfers would home in on the available capacity more quickly and waste fewer retransmit - while being friendly to folks sharing the bottleneck!

The HPC guys really don't understand a thing about control theory...

On Nov 13, 2014, Aaron Wood <woody77@gmail.com> wrote:
>I have a couple friends in that crowd, and they _also_ aren't using
>shared
>lines.  So they don't worry in the slightest about congestion when
>they're
>trying to keep dedicated links fully saturated.  They're big issue with
>dropped packets is that some of the TCP congestion-control algorithms
>kick
>in on a single dropped packet:
>http://fasterdata.es.net/network-tuning/tcp-issues-explained/packet-loss/
>
>I'm thinking that some forward-error-correction would make their lives
>much, much better.
>
>-Aaron
>
>On Thu, Nov 13, 2014 at 7:11 PM, Dave Taht <dave.taht@gmail.com> wrote:
>
>> One thing the HPC crowd has missed is that in their quest for big
>> buffers for contenental distances, they hurt themselves on shorter
>> ones...
>>
>> ... and also that big buffers with FQ on them works just fine in the
>> general case.
>>
>> As always I recomend benchmarking - do a rrul test between the two
>> points, for example, with their recomendations.
>>
>>
>> On Fri, Oct 17, 2014 at 4:21 PM, Frank Horowitz <frank@horow.net>
>wrote:
>> > G’Day folks,
>> >
>> > Long time lurker. I’ve been using Cero for my home router for quite
>a
>> while now, with reasonable results (modulo bloody OSX wifi stuffola).
>> >
>> > I’m running into issues doing zfs send/receive over ssh across a
>> (mostly) internet2 backbone between Cornell (where I work) and West
>> Virginia University (where we have a collaborator on a DOE sponsored
>> project. Both ends are linux machines running fq_codel configured
>like so:
>> >         tc qdisc
>> >         qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows
>1024
>> quantum 1514 target 5.0ms interval 100.0ms ecn
>> >
>> > I stumbled across hpn-ssh <https://www.psc.edu/index.php/hpn-ssh>
>and
>> —  of particular interest to this group — their page on tuning TCP
>> parameters:
>> >
>> > <http://www.psc.edu/index.php/networking/641-tcp-tune>
>> >
>> > N.B. their advice to increase buffer size…
>> >
>> > I’m curious, what part (if any) of that advice survives with
>fq_codel
>> running on both ends?
>> >
>> > Any advice from the experts here would be gratefully received!
>> >
>> > (And thanks for all of your collective and individual efforts!)
>> >
>> > Cheers,
>> >         Frank Horowitz
>> >
>> >
>> > _______________________________________________
>> > Cerowrt-devel mailing list
>> > Cerowrt-devel@lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> >
>>
>>
>>
>> --
>> Dave Täht
>>
>> thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Cerowrt-devel mailing list
>Cerowrt-devel@lists.bufferbloat.net
>https://lists.bufferbloat.net/listinfo/cerowrt-devel

-- Sent from my Android device with K-@ Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 6434 bytes --]

Re: [Cerowrt-devel] High Performance (SSH) Data Transfers using fq_codel?

[Dave Taht]

On Fri, Oct 17, 2014 at 4:21 PM, Frank Horowitz <frank@horow.net> wrote:
> G’Day folks,
>
> Long time lurker. I’ve been using Cero for my home router for quite a while now, with reasonable results (modulo bloody OSX wifi stuffola).
>
> I’m running into issues doing zfs send/receive over ssh across a (mostly) internet2 backbone between Cornell (where I work) and West Virginia University (where we have a collaborator on a DOE sponsored project. Both ends are linux machines running fq_codel configured like so:
>         tc qdisc
>         qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn

So, in re-reading this now, I think I now grok that that A) Frank is
using cerowrt at home, and B) he has it hooked up on two cool boxes
connected on either side of Internet2, and B) is what the question was
about.

so my general assumption is that his boxes are x86, and hooked up at
1GigE or so to the Internet2.

So answers A) If after a big transfer If a

tc -s qdisc show dev eth0 # on both sides

shows no drops or ecn marks, his linux servers are not the bottleneck
link, and he should use mtr to find the real bottleneck link
elsewhere, during that transfer.

IF, after a big transfer drops are seen, you are (at least some of the
time) a bottleneck link. Enable ecn between both tcps. And if you are
willing to tolerate more latency on the link, feel free to increase
the target and interval to values you are more comfortable with, but
you won´t increase actual bandwidth by all that much.

Personally I suspect ¨A¨ as the problem.

And as per my original msg, it always helps to measure, and the rrul
test between the two points is the best thing we got.


> I stumbled across hpn-ssh <https://www.psc.edu/index.php/hpn-ssh> and —  of particular interest to this group — their page on tuning TCP parameters:
>
> <http://www.psc.edu/index.php/networking/641-tcp-tune>
>
> N.B. their advice to increase buffer size…
>
> I’m curious, what part (if any) of that advice survives with fq_codel running on both ends?

Most of that seems to apply to TCPs.

I would suspect that enabling TCP pacing between the two points might
be helpful, but without data on whatever problem(s) you are
experiencing on your path, can´t help. Amusingly, matt mathis is one
the original authors of that page, and perhaps he has new advice.

> Any advice from the experts here would be gratefully received!
>
> (And thanks for all of your collective and individual efforts!)
>
> Cheers,
>         Frank Horowitz
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>



-- 
Dave Täht

thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks