1

I'm copying large files over LAN, and I've noticed that with doing one ssh pipe, I can get around 10MB/s transfer speed. However, if I shoot off four of those at the same time, I get around 40MB/s transfer speeds in total.

So I'd like to split up a large file (eg. 1GB) into parts, and pipe each part on its own, then reassemble on other side. Need to tar/zip before transfer also, and untar/unzip on other side.

How can I do this?

Improve this question
6
  • is this an option? Commented Mar 27, 2015 at 17:35
  • Hum, perhaps. Not sure what is the bottleneck, actually. Any clue? Commented Mar 27, 2015 at 17:50
  • 2
    It's doable but it seems a rather horrible workaround. Wouldn't it be better to find out why you are not getting full speed under ssh with one pipe? Which os? Which ssh? Commented Mar 27, 2015 at 18:44
  • 1
    is there a reason why you don't have/want more recent versions? My ubuntu 14.04.2 has OpenSSH_6.6.1p1 Ubuntu-2ubuntu2, OpenSSL 1.0.1f 6 Jan 2014? The latest openssh is 6.8. Some (old) speed comparison info here - best results were tar over ssh using arcfour128 (a low security cipher, disabled by default in openssh 6.7+) or umac-64. Also consider trying HPN-SSH - patch files for openssh 6.6 and earlier are available on sourceforce. Commented Mar 28, 2015 at 6:41
  • 1
    A Ubuntu ppa for Openssh-hpn, for easy installation, is here. Commented Mar 28, 2015 at 6:48

2 Answers 2

Reset to default
2

Ended up with HPN-SSHand pigz.

tar -cf - -C [RELATIVEFOLDER] [FILENAME] | pigz | ssh px "pigz -d | tar xf - -C [REMOTEFOLDER]"

Improvement by power of ten.

For reference, installing HPN-SSH and pigz on Ubuntu 14.04 is easy as:

# hpn-patched ssh from ppa
sudo apt-get install python-software-properties
sudo add-apt-repository ppa:w-rouesnel/openssh-hpn
sudo apt-get update -y  
sudo apt-get install openssh-server
ssh -V # should have 'hpn' in it somewhere

# pigz
sudo apt-get install pigz
Improve this answer
2
  • I'd be interested if you know how much of this 10x speed improvement (to 100MB/sec i.e. network limited?) is because of using HPN-SSH (as I suggested) in place of standard openssh? Commented Mar 28, 2015 at 17:21
  • @gogoud basically all of it, as the bottleneck was SSH encrypting on one CPU only. Never going back! :) Commented Mar 29, 2015 at 14:37
0

Well there is zipsplit(1) and split(1), that you could used to split up the large files before using say scp(1) to copy them over. Or you could split the output into named pipes and then cat the named pipes to ssh, if you wish a pipelined effect.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.