3

I am running my below shell script from machineA which is copying the files machineB and machineC into machineA. If the files are not there in machineB, then it should be there in machineC.

The below shell script will copy the files into TEST1 and TEST2 directory in machineA..

#!/bin/bash
set -e

readonly TEST1=/data01/test1
readonly TEST2=/data02/test2
readonly SERVER_LOCATION=(machineB machineC)
readonly FILE_LOCATION=/data/snapshot

dir1=$(ssh -o "StrictHostKeyChecking no" david@${SERVER_LOCATION[0]} ls -dt1 "$FILE_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)
dir2=$(ssh -o "StrictHostKeyChecking no" david@${SERVER_LOCATION[1]} ls -dt1 "$FILE_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)

echo $dir1
echo $dir2

if [ "$dir1" = "$dir2" ]
then
    rm -rf $TEST1/*
    rm -rf $TEST2/*
    for el in $test1_partition
    do
        scp david@${SERVER_LOCATION[0]}:$dir1/pp_monthly_9800_"$el"_200003_5.data $TEST1/. || scp david@${SERVER_LOCATION[1]}:$dir2/pp_monthly_9800_"$el"_200003_5.data $TEST1/.
    done
    for sl in $test2_partition
    do    
        scp david@${SERVER_LOCATION[0]}:$dir1/pp_monthly_9800_"$sl"_200003_5.data $TEST2/. || scp david@${SERVER_LOCATION[1]}:$dir2/pp_monthly_9800_"$sl"_200003_5.data $TEST2/.
    done
fi

Is there a way to run process parallelly in the loop of a bash script

Currently it copies the file from machineB and machineC into machineA TEST1 directory first, and if it is done, then only it will go and copy the files from machineB and machineC into machineA TEST2 directory.. Is there any way I transfer the files both in TEST1 and TEST2 directory simultaneously?

I am running Ubuntu 12.04

Improve this question
0

3 Answers 3

Reset to default
2

In addition to sending them to the background, use the wait built in to wait for all background processes to finish before continuing.

for el in $test1_partition
do
    (scp david@${SERVER_LOCATION[0]}:$dir1/pp_monthly_9800_"$el"_200003_5.data $TEST1/. || scp david@${SERVER_LOCATION[1]}:$dir2/pp_monthly_9800_"$el"_200003_5.data $TEST1/.) &
    WAITPID="$WAITPID $!"
done
for sl in $test2_partition
do    
    (scp david@${SERVER_LOCATION[0]}:$dir1/pp_monthly_9800_"$sl"_200003_5.data $TEST2/. || scp david@${SERVER_LOCATION[1]}:$dir2/pp_monthly_9800_"$sl"_200003_5.data $TEST2/.) &
    WAITPID="$WAITPID $!"
done
wait $WAITPID
echo "All files done copying."
Improve this answer
5
  • Thanks for the suggestion. What will happen in case of any errors? Will I be able to identify those if I go with your suggestion? In general if there is any error, I would like to show them on the console and stopped executing the script so that's why I was using set -e Commented Feb 16, 2014 at 4:39
  • Handle the error in the subshell. I.e., whatever you put after the || is what handles the error. Commented Feb 16, 2014 at 5:31
  • Suppose for whatever reason if the files are not present in both the machine, then what will happen in that case, it will throw an error right? Commented Feb 16, 2014 at 5:34
  • Yes, but you'll still need to handle the error in the subshell. It won't be surfaced to the parent process. Commented Feb 16, 2014 at 5:42
  • hmmm.. Can you provide an example how would I do this in this example? I would like to clearly indicate to parent process about this error.. Commented Feb 16, 2014 at 5:48
1

You should take a look to pssh project, there's a pscp command that run parallel scp

Improve this answer
1

You can do:

scp david@${SERVER_LOCATION[0]}:$dir1/pp_monthly_9800_"$el"_200003_5.data $TEST1/ ||
scp david@${SERVER_LOCATION[1]}:$dir2/pp_monthly_9800_"$el"_200003_5.data $TEST1/ &

The & at the end will detach the command, returning control to the script.

If you put this in a loop, this will run all the bodies in parallel. To wait for all these background jobs to finish, use

wait
Improve this answer
2
  • Thanks. In this, after running the script, control will come out of the script and script will end successfully but the file transfer will happen in the background? Is there any way to wait for the script to finish but the file transfer should happened in the background? I also need to see whether there was any exception happened during the file transfer process or not.. If the control comes out of the script, then I am assuming shell script will always show success status. Commented Feb 15, 2014 at 15:13
  • This isn't going to work: different files are being copied to the two servers ($test1_partition vs $test2_partition). Also you're going to have a hard time detecting errors. Commented Feb 15, 2014 at 23:21

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.