Nesting mpi calls with mpi4py

Question

I am trying to use mpi4py to call a second instance of an mpi executable.

I am getting the error:

Open MPI does not support recursive calls of mpirun

But I was under the impression that is exactly what Spawn is supposed to be able to handle - i.e. setting up a new communicator within which another mpi command could be launched.

The test code:

parent.py:

#!/usr/bin/env python
from mpi4py import MPI
import numpy
import sys


rank = MPI.COMM_WORLD.Get_rank()
new_comm = MPI.COMM_WORLD.Split(color=rank, key=rank)
print(new_comm.Get_rank())
new_comm.Spawn(sys.executable,
                           args=['test.py'],
                           maxprocs=4)

which calls test.py:

#!/usr/bin/env python
from mpi4py import MPI
import numpy
import os
import sys

comm = MPI.Comm.Get_parent()
rank = comm.Get_rank()

cwd=os.getcwd()
directory=os.path.join(cwd,str(rank))
os.chdir(directory)


os.system('{}'.format('mpirun -np 4 SOME_MPI_EXECUTABLE_HERE'))



print("Finished in "+directory)
os.chdir(cwd)

comm.Disconnect()

I'm running with:

mpirun --oversubscribe -np 1 parent.py

Using openmpi 2.0.0 with gcc, and python/3.4.2

Anyone have any bright ideas as to why this is happening.....

Thanks!

What are you trying to achieve with the test.py wrapper? I believe you are supposed to spawn(SOME_MPI_EXECUTABLE_HERE), .... — Zulan
– Zulan, Commented Sep 21, 2016 at 13:46
The wrapper is needed as the MPI_EXECUTABLE_HERE that I call is not written in python, and so will not run with the sys.executable interpreter. The piece of mpi code that i am calling within the python script s written in fortran. — abinitio
– abinitio, Commented Sep 21, 2016 at 14:16
I agree with @Zulan, you should start the executable directly via the MPI_Spawn call. — Georg Bisseling
– Georg Bisseling, Commented Sep 21, 2016 at 14:23
Please post the code that resolved your issue as answer to your own question and feel free to accept it so that it is immediately clear that you have found a solution. — Zulan
– Zulan, Commented Sep 22, 2016 at 10:33

abinitio · Accepted Answer · 2016-09-22 11:17:36Z

3

The following code seems to perform the way I wanted.

#!/usr/bin/env python
from mpi4py import MPI
import numpy
import sys
import os

rank = MPI.COMM_WORLD.Get_rank()
new_comm = MPI.COMM_WORLD.Split(color=rank, key=rank)
print(new_comm.Get_rank())

cwd=os.getcwd()
os.mkdir(str(rank))
directory=os.path.join(cwd,str(rank))
print(rank,directory)
os.chdir(directory)


new_comm.Spawn("SOME_MPI_EXECUTABLE_HERE",
                  args=[""],
                           maxprocs=4)

run with:

mpirun --oversubscribe -np 4 parent.py

Seems to start 4 instances of SOME_MPI_EXECUTABLE each running on 4 cores.

(Thanks to Zulan)

answered Sep 22, 2016 at 11:17

abinitio

8491 gold badge10 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Bananach Over a year ago

The comment of @Hristo under the original question still applies. Instead of creating new_comm you can simply use MPI.COMM_SELF.Spawn. This will give the same results as the code you posted here and would make this answer even more helpful (I myself started off from this answer, thanks btw., and it took me a while to understand the meaning of Split, color, key before I understood that this isn't even necessary)

Collectives™ on Stack Overflow

Nesting mpi calls with mpi4py

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related