Explaining How to Perform Node-to-Node Parallel (Distributed Memory Parallel) Processing with OpenMPI on SQUID
Compilation Method
$ module load BaseGCC
$ mpif90 [options] source_file (For Fortran,)
$ mpicc [options] source_file (For C,)
$ mpic++ [options] source_file (For C++,)
If you are using GCC 11.4 + OpenMPI 4.1.5, specify the following instead of BaseGCC:
$ module load /system/apps/env/gcc/Compiler/gcc/11.4.0
$ module load /system/apps/env/gcc/mpi/gcc11.4.0/openmpi/4.1.5cu11.8
Execution Script
The command to execute MPI is as follows:
mpirun ${NQSV_MPIOPTS} -np (total parallel number) executable_name
An example script for executing an MPI batch request with a duration of 1 hour, running in parallel across 4 nodes, with 76 parallel executions per node (totaling 304 parallel executions), is as follows:
1 2 3 4 5 6 7 8 9 10 |
#!/bin/bash #PBS -q SQUID #PBS --group= [Group Name] #PBS -l elapstim_req=1:00:00 #PBS -T openmpi #PBS -v NQSV_MPI_MODULE=BaseGCC #PBS -b 4 cd $PBS_O_WORKDIR module load BaseCPU mpirun ${NQSV_MPIOPTS} -np 304 -npernode 76 ./a.out |
Points to Note for Multi-Node Execution
-
If you specify options or environment variables in the job script with setenv (option name), they will be set only on the master node and not on the slave nodes.
If you want to reflect the settings on all nodes, please specify #PBS -v (option name).
For more details, please refer to the following:
How to Write a Job Script - Specifying Environment Variables