How to optimize NFS concurrent access performance

Test program:

#include <fcntl.h>
#include <unistd.h>
#include <string>
#include <stdexcept>
#include <cstring>
#include <iostream>

int main(int argc, char* argv[]) {
    if (argc != 2) {
        std::cerr << "Usage: " << argv[0] << " <filename>\n";
        return 1;
    }

    const std::string filename = argv[1];
    const int numIterations = 10000;

    for (int i = 0; i < numIterations; ++i) {
        int fd = open(filename.c_str(), O_RDONLY);

        if (fd == -1) {
            throw std::runtime_error("Failed to open " + filename + " (Iteration " + std::to_string(i + 1) + "): " + strerror(errno));
        }

        if (close(fd) == -1) {
            throw std::runtime_error("Failed to close " + filename + " (Iteration " + std::to_string(i + 1) + "): " + strerror(errno));
        }
    }

    return 0;
}

g++ -O2 test.cpp -o test_open_close

Run a single program:

$ time /tmp/test_open_close /mnt/nfs_mnt/xxxxxx

real    0m0.115s
user    0m0.003s
sys 0m0.066s

Use a wrapper script to run this program in parallel:

#!/bin/bash

# The command to be executed
command="/tmp/test_open_close /mnt/nfs_mnt/xxxxxx"

# Number of concurrent executions
num_executions=360

# Function to execute the command and capture its exit status
run_command() {
    $command 2>/dev/null
}

# Start time
start_time=$(date +%s.%N)

# Run the commands concurrently
for i in $(seq 1 $num_executions); do
    run_command &
done

# Wait for all background processes to finish
wait

# End time
end_time=$(date +%s.%N)

# Calculate duration
duration=$(echo "$end_time - $start_time" | bc)

echo "Executed $num_executions processes in $duration seconds"

$ bash /tmp/test.sh
Executed 360 processes in 35.127017601 seconds

The total used time is almost the same as run them in serial.

There's another strange thing. During the test.sh script is running, I run time /tmp/test_open_close /mnt/nfs_mnt/xxxxxx in another shell, and the used time is different when using different users.

1. Use the same user as test.sh script.

$ time /tmp/test_open_close /mnt/nfs_mnt/xxxxxx

real    0m22.753s
user    0m0.005s
sys 0m0.121s

2. Use a different user.

$ time /tmp/test_open_close /mnt/nfs_mnt/xxxxxx

real    0m0.197s
user    0m0.000s
sys 0m0.158s

3. Use the same user but on different host.

$ time /tmp/test_open_close /mnt/nfs_mnt/xxxxxx

real    0m3.055s
user    0m0.003s
sys 0m0.227s

System enviroment:

Rocky 8.8, 4.18.0.

The test host has 378 cores.

My problem is:

1. What the bottleneck is in this case? The lock in NFS client?

2. Why different user has different behavior?

3. How to optimization the performance in this case?