I want to implement a system, where there are one receiver and multiple senders. Each sender keeps sending data to the receiver. The receiver waits to receive data and process it. Here is the toy example.
#include <iostream>
#include <cstdlib>
#include <mpi.h>
using namespace std;
int main(int argc, char *argv[]) {
int _mpi_numWorkers, _mpi_rank;
// Initialize openMPI
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &_mpi_numWorkers);
MPI_Comm_rank(MPI_COMM_WORLD, &_mpi_rank);
MPI_Barrier(MPI_COMM_WORLD);
float *send_data = (float *)malloc(360 * 5000 * sizeof(float));
MPI_Status receive_status;
if (_mpi_rank == 0) {
while (1) {
MPI_Recv(send_data, 360 * 5000, MPI_FLOAT, MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, &receive_status);
cout << "Receive from " << receive_status.MPI_SOURCE << endl;
}
} else {
while (1){
MPI_Send(send_data, 360 * 5000, MPI_FLOAT, 0, 0, MPI_COMM_WORLD);
//sleep(1);
}
}
// Terminate
MPI_Finalize();
return 0;
}
The issue is that MPI_recv can only receive messages from up to two processes no matter how many processes I set it to run (when no sleep). I have tested this code on one single machine, and on multiple machines cases:
Single Machine Case
I run this code through the following command:
mpiexec -n 5 ./test_mpi
Then, the receiver only receive from senders with rank 1 and 2.
Multiple Machine Case
I run 4 senders and 1 receiver on 5 homogeneous physical machines. All of them connect to a 100Mbps switch. In this case, the receiver also only receives data from a subset of senders. I use the tcpdump to check the packet, and observe that some senders do not even send the message. (Those senders are blocked at MPI_send, but no tcp sequence increases and no re-transmission.)
For those two cases, if I make each sender sleep some time (decreasing the sending rate), the receiver can receive data from more senders.
Can somebody help me to understand why this happens?
Environment
Debian testing with openmpi-1.6
Edit 2/4/16
I include <cstdlib> in the code to prevent any compilation issues.
<cstdlib>helps to definemallocand running it asmpirun -np 5 test_mpiseems to work fine for me. (Compiling asmpic++ test_mpi.cpp -o test_mpi)