I'm trying to make a slurm file that runs a python script on tens/hundreds of input files. Each process is single-threaded and totally independent from each other and I want to speed things up by running them simultaneously.
Currently I'm doing
#SBATCH --nodes=1 --ntasks-per-node=1 --time=0:20:00
python3 xxx.py -i /path/input1
and I want to change it to something like
python3 xxx.py -i /path/*input
How should I change it so that all process could run in parallel?
Try the following
#SBATCH --nodes=1 --time=0:20:00
for i in {1..100}
do
srun -n 1 python3 xxx.py -i /path/input$i &
done
wait
This will call your python script for the files ending with the name (1-100) i.e. /path/input1 till /path/input100. Also, specify the --ntasks-per-node as the maximum allowed (eg: number of cpus).
The idea here is you can do scripting inside your sbatch job script.
The for loop will call the srun multiple times with different arguments (& ensures that srun will run in background without blocking). wait at the end will make the script wait till all the background jobs are finished. You can use variety of techniques to achieve the same objective.
Also you can use job array to do the same more elegantly.
srun wait for the task to finish, or it will just launch 100 sruns without waiting for the individual jobs to finish?
