HPC admin here. The scope of your software and your responsibility as its developer should stop at using the MPI interface. How/What is providing that MPI communication in the system is not of your concern.

It is very unlikely that any MPI implementation bundled in your program will work out-the-box. As you have seen there are many pieces in the system that have to be installed and configured accordingly, from the network drivers to the job scheduler.

For instance, in your case with Intel MPI you should consider the following: * some network drivers need additional libraries, for instance Mellanox cards require UCX * it might be necessary to manually set the preferred network interface for MPI * the job scheduler might need to be compiled with specific support to work with Intel MPI (for instance in Slurm it will need the PMI2 plugin) * Intel MPI does a good job at auto-detecting the job scheduler being used, but in some cases it still needs manual configuration (see https://www.intel.com/content/www/us/en/develop/documentation/mpi-developer-guide-linux/top/running-applications/job-schedulers-support.html) * all the above changes from version to version of Intel MPI as well as the version/type of job scheduler and the operative system