I currently have the current script.
#!/bin/bash
# script.sh
for i in {0..99}; do
script-to-run.sh input/ output/ $i
done
I wish to run it in parallel using xargs. I have tried
script.sh | xargs -P8
But doing the above only executed once at the time. No luck with -n8 as well. Adding & at the end of the line to be executed in the script for loop would try to run the script 99 times at once. How do I execute the loop only 8 at the time, up to 100 total.
From the xargs man page:
This manual page documents the GNU version of xargs. xargs reads items from the standard input, delimited by blanks (which can be protected with double or single quotes or a backslash) or newlines, and executes the command (default is /bin/echo) one or more times with any initial- arguments followed by items read from standard input. Blank lines on the standard input are ignored.
Which means that for your example xargs is waiting and collecting all of the output from your script and then running echo <that output>. Not exactly all that useful nor what you wanted.
The -n argument is how many items from the input to use with each command that gets run (nothing, by itself, about parallelism here).
To do what you want with xargs you would need to do something more like this (untested):
printf %s\\n {0..99} | xargs -n 1 -P 8 script-to-run.sh input/ output/
Which breaks down like this.
printf %s\\n {0..99} - Print one number per-line from 0 to 99.xargs
echo {0..99} | would work just as well. <<<{0..99} doesn't seem to work; although <<<word is documented as brace-expanding word, it doesn't do so with any version of bash I have handy.
<< but does for <<< so *shrug*). The expansions that do and don't happen in here docs and here strings are a bit odd to my mind.
time head -12 <(yes "1") | xargs -n1 -P4 sleep will run 12 sleep 1 commands, 4 parallel. The command will take 3 seconds.
-P 0 will use the number of cpus on the systemHere's an example running commands in parallel in conjuction with find:
find -name "*.wav" -print0 | xargs -0 -t -I % -P $(nproc) flac %
-print0 terminates filenames with a null byte rather than a newline so we can use -0 in xargs to prevent filenames with spaces being treated as two separate arguments.
-t means verbose, makes xargs print every command it's executing, can be useful, remove if not needed.
-I % means replace occurrences of % in the command with arguments read from standard input.
-P $(nproc) means run a maximum of nproc instances of our command in parallel (nproc prints the number of available processing units).
flac % is our command, the -I % from earlier means this will become flac foo.wav
See also: Manual for xargs(1)
bash -c. So for example maybe you wanted to rm the original file, you could do bash -c "flac \"%\" && rm \"%\""
--verbose instead of -t and --max-procs= instead of -P.You can use this simple 1 line command
seq 1 500 | xargs -n 1 -P 8 script-to-run.sh input/ output/