Indiana University

On Quarry, how do I run MPI jobs using a TORQUE script?

To run parallel MPI jobs on the Quarry cluster at Indiana University, you must first add the necessary modules to your user environment. Once your user environment is configured properly, you can compile your code and submit your jobs to the TORQUE resource management system.

On this page:


Setting up your user environment using Modules

Open MPI wrapper compilers and libraries for the Intel Compiler Suite and GNU Compiler Collection (GCC) are available on Quarry. MPICH wrapper compilers and libraries for the Intel compilers are also available. The wrapper compilers add the necessary flags for compiling and linking MPI programs, and then invoke the respective back-end compilers to perform the actual commands.

You must add the appropriate compiler collection to your user environment on Quarry before adding the corresponding Open MPI or MPICH library:

  • For the Intel Open MPI or MPICH library:

    1. Add the Intel Compiler Collection to your user environment; on the command line, enter: module load intel
    2. Add the Intel Open MPI or MPICH library to your user environment. To load the desired module, on the command line, enter one of the following:

      • Open MPI (version 1.6.3): module load openmpi/intel
      • MPICH (version 3.0.1): module load mpich
  • For the GCC Open MPI library:

    1. Add the GNU Compiler Collection (GCC) to your user environment; on the command line, enter: module load gcc
    2. Add the GCC Open MPI library to your user environment; on the command line, enter: module load openmpi/gnu

To make permanent changes to your environment, edit your ~/.modules file. For more, see In Modules, how do I save my environment with a .modules file?

For more about Modules, see On Big Red II, Mason, Quarry, and Rockhopper at IU, how do I use Modules to manage my software environment?

Back to top

Compiling your code and submitting jobs to TORQUE

After adding the necessary modules to your user environment, you can compile your code and submit parallel MPI jobs to the TORQUE resource management system on Quarry:

  1. Compile your MPI code using the appropriate MPI wrapper compiler.

    MPI wrapper compilers
    Language Wrapper compiler
    C mpicc
    C++ mpic++ or
    mpicxx
    Fortran 77 mpif77
    Fortran 90 mpif90

    Note: Manual pages are available for the Open MPI wrapper compilers; for example, to view the manual page for the Open MPI Fortran 90 wrapper compiler, on the command line, enter:

    man mpif90

    For example:

    • To compile and link an MPI program written in C, on the command line, enter: mpicc -o myprog myprog.c
    • To compile and link an MPI program written in C++, on the command line, enter: mpicxx -o myprog myprog.cxx
    • To compile and link an MPI program written in Fortran 90, on the command line, enter: mpif90 -o myprog myprog.f
  2. Create a TORQUE script (e.g., submit_parallel.sh) that includes the correct number of nodes and tasks, and the appropriate output/error files; for example: #PBS -l nodes=2:ppn=8,walltime=5:00 #PBS -m ae #PBS -N job_myprog mpirun -machine $PBS_NODEFILE -np 16 ${HOME}/simple_quarry_jobs/myprog
  3. Submit the program to TORQUE. Use the qsub command from the command line; for example: [jjbinks@q0141 ~]$ qsub submit_parallel.sh

For more on TORQUE, see What is TORQUE, and how do I use it to submit and manage jobs on high-performance computing systems?

Back to top

Getting help

If you have questions about compilers, libraries, or programming on Quarry, email the Scientific Applications and Performance Tuning group for help.

Back to top