Variable value is not recognized after using gnu parallel?

Question

I have a below shell script from which I am trying to copy 5 files in parallel. I am running my below shell script on machineA which tries to copy the file from machineB and machineC.

If the file is not there in machineB, then it should be there in machineC for sure.

I am using GNU Parallel here to download five files in parallel.

#!/bin/bash

readonly PRIMARY=/tech01/primary
readonly FILERS_LOCATION=(machineB machineC)
readonly MEMORY_MAPPED_LOCATION=/techbat/data/be_t1_snapshot
PRIMARY_PARTITION=(550 274 2 546 278 6 558 282 10 554 286 14) # this will have more file numbers

dir1=/techbat/data/be_t1_snapshot/20140501

find "$PRIMARY" -mindepth 1 -delete

do_copy() {
  el=$1
  scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/.
}
export -f do_copy
parallel -j 5 do_copy ::: "${PRIMARY_PARTITION[@]}"

Problem Statement:-

The problem I am facing with the above script is - It is not able to recognize ${FILERS_LOCATION[0]}, ${FILERS_LOCATION[1]}, $dir1 and $PRIMARY inside do_copy method? And I am not sure why?

If I try to print out like this inside do_copy method nothing is printed out?

  echo ${FILERS_LOCATION[0]}    
  echo ${FILERS_LOCATION[1]}

But if I print out same thing just above do_copy method, then it works fine?

Anything I am missing here?

Update:-

Below is the code I am using -

#!/bin/bash

export PRIMARY=/tech01/primary
export FILERS_LOCATION=(machineB machineC)
export MEMORY_MAPPED_LOCATION=/techbat/data/be_t1_snapshot
PRIMARY_PARTITION=(0 548 272 4 544 276 8 556 280)

export dir1=/techbat/data/be_t1_snapshot/20140501

find "$PRIMARY" -mindepth 1 -delete

do_copy() {
  el=$1
  scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/.
}
export -f do_copy
parallel -j 8 do_copy ::: "${PRIMARY_PARTITION[@]}"

Another Update:-

This is what I got after running the below script -

#!/bin/bash

export PRIMARY=/tech01/primary
export FILERS_LOCATION=(slc4b03c-407d.stratus.slc.ebay.com chd1b02c-0db8.stratus.phx.ebay.com)
export MEMORY_MAPPED_LOCATION=/techbat/data/be_t1_snapshot
PRIMARY_PARTITION=(0 548 272 4 544)

export dir1=/techbat/data/be_t1_snapshot/20140501

find "$PRIMARY" -mindepth 1 -delete

 echo ${FILERS_LOCATION[0]}    
 echo ${FILERS_LOCATION[1]}

do_copy() {
  el=$1
  echo "scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 bullseye@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 bullseye@${FILERS_LOCATION[1]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/."
}
export -f do_copy
parallel -j 3 do_copy ::: "${PRIMARY_PARTITION[@]}"

Output I got -

david@tvxdbx1143:/home/david$ ./scp_files5.sh
machineB
machineC
When using programs that use GNU Parallel to process data for publication please cite:

  O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
  ;login: The USENIX Magazine, February 2011:42-47.

This helps funding further development; and it won't cost you a cent.

To silence this citation notice run 'parallel --bibtex' once or use '--no-notice'.

scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@:/techbat/data/be_t1_snapshot/20140501/t1_weekly_1680_0_200003_5.data /tech01/primary/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@:/techbat/data/be_t1_snapshot/20140501/t1_weekly_1680_0_200003_5.data /tech01/primary/.
scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@:/techbat/data/be_t1_snapshot/20140501/t1_weekly_1680_548_200003_5.data /tech01/primary/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@:/techbat/data/be_t1_snapshot/20140501/t1_weekly_1680_548_200003_5.data /tech01/primary/.
scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@:/techbat/data/be_t1_snapshot/20140501/t1_weekly_1680_272_200003_5.data /tech01/primary/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@:/techbat/data/be_t1_snapshot/20140501/t1_weekly_1680_272_200003_5.data /tech01/primary/.
scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@:/techbat/data/be_t1_snapshot/20140501/t1_weekly_1680_4_200003_5.data /tech01/primary/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@:/techbat/data/be_t1_snapshot/20140501/t1_weekly_1680_4_200003_5.data /tech01/primary/.
scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@:/techbat/data/be_t1_snapshot/20140501/t1_weekly_1680_544_200003_5.data /tech01/primary/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@:/techbat/data/be_t1_snapshot/20140501/t1_weekly_1680_544_200003_5.data /tech01/primary/.

Prepend scp with 'echo' and see if you get the correct output. — Ole Tange
– Ole Tange, Commented May 5, 2014 at 19:07
@OleTange: I have updated the question with the scp output. It is not even making correct scp output. — arsenal
– arsenal, Commented May 5, 2014 at 19:33

Ole Tange · Accepted Answer · 2014-05-06 06:34:57Z

2

Try exporting them and removing the array as bash can not export arrays:

export PRIMARY=/data01/primary
export FILERS_LOCATION_1=machineB
export FILERS_LOCATION_2=machineC
export MEMORY_MAPPED_LOCATION=/bexbat/data/be_t1_snapshot

export dir1=/bexbat/data/be_t1_snapshot/20140501

Or simply put all the constant variables into the function:

#!/bin/bash

PRIMARY_PARTITION=(0 548 272 4 544 276 8 556 280)

PRIMARY=/data01/primary
find "$PRIMARY" -mindepth 1 -delete

do_copy() {
  el=$1

  PRIMARY=/data01/primary
  FILERS_LOCATION=(machineB machineC)
  MEMORY_MAPPED_LOCATION=/bexbat/data/be_t1_snapshot

  dir1=/bexbat/data/be_t1_snapshot/20140501

  scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/.
}
export -f do_copy
parallel -j 8 do_copy ::: "${PRIMARY_PARTITION[@]}"

Depending on what kinds of files you are copying you should look into rsync -z instead of scp. And consider running parallel --bibtex once (as suggested by parallel).

edited May 6, 2014 at 6:34

answered May 5, 2014 at 7:31

Ole Tange

37.6k34 gold badges119 silver badges228 bronze badges

I tried this and it doesn't work somehow? I have updated the question with the shell script details. May be I am doing something wrong?

arsenal
– arsenal

2014-05-05 18:20:00 +00:00
Commented May 5, 2014 at 18:20
2

Apparently bash arrays can not be exported. Use FILERS_LOCATION_1 and FILERS_LOCATION_2 instead.

Matt
– Matt

2014-05-05 22:28:33 +00:00
Commented May 5, 2014 at 22:28
Actually Ole said that code worked for him. What versions of bash do you both have? bash --version

Matt
– Matt

2014-05-05 22:31:10 +00:00
Commented May 5, 2014 at 22:31
@mtm bash version I have is GNU bash, version 4.2.24(1)-release (x86_64-pc-linux-gnu)

arsenal
– arsenal

2014-05-05 23:37:36 +00:00
Commented May 5, 2014 at 23:37
It seems you need bash 4.3 to export arrays. stackoverflow.com/questions/5564418/… Follow mtm's suggestion.

Ole Tange
– Ole Tange

2014-05-06 06:25:02 +00:00
Commented May 6, 2014 at 6:25

Add a comment |

Matt · Accepted Answer · 2014-05-05 21:15:28Z

1

You have exported a function but not the variables you are trying to use directly in the function.

parallel will launch a new shell for each run of do_copy and in that shell the variables are interpreted and do not exist.

If the -s SERVER option is used the --env VAR option will copy VAR from the initial parallel shell to the remote environment the command is run in:

parallel -j 5 -S localhost --env do_copy --env PRIMARY --env FILERS_LOCATION do_copy ::: "${PRIMARY_PARTITION[@]}"

You may be able to get away with the above localhost hack as I can't see a simple way to implement your multiple server logic into the parallel server -S options (unless you are guaranteed one server won't have the file?)

The nicer way would be to export the variables as Ole suggested, or passing all the require values as parameters to the do_copy function

edited May 5, 2014 at 21:15

answered May 5, 2014 at 7:31

Matt

9,3912 gold badges29 silver badges33 bronze badges

Thanks for suggestion. I tried running like this and still it doesn't work? parallel -j 5 --env PRIMARY --env FILERS_LOCATION --env dir1 do_copy ::: "${PRIMARY_PARTITION[@]}" Anything wrong?

arsenal
– arsenal

2014-05-05 18:28:48 +00:00
Commented May 5, 2014 at 18:28
--env is only for remote runnning.

Ole Tange
– Ole Tange

2014-05-05 18:29:53 +00:00
Commented May 5, 2014 at 18:29
@OleTange: I see, and I tried your suggestion as well. Am I doing something wrong as well?

arsenal
– arsenal

2014-05-05 18:33:42 +00:00
Commented May 5, 2014 at 18:33
aahh forgot about that, noted. cheers

Matt
– Matt

2014-05-05 21:18:21 +00:00
Commented May 5, 2014 at 21:18
@mtm: I also tried Ole suggestion but that is also not working for me. I have updated the question as well. Can you take a look?

arsenal
– arsenal

2014-05-05 21:59:36 +00:00
Commented May 5, 2014 at 21:59

Add a comment |

Stack Exchange Network

Variable value is not recognized after using gnu parallel?

2 Answers 2

You must log in to answer this question.

Hot Network Questions

Variable value is not recognized after using gnu parallel?

2 Answers 2

You must log in to answer this question.

Related

Hot Network Questions