Indiana University
  •  
  •  
  •  

Quarry

On this page:


Introduction


Quarry (quarry.uits.indiana.edu) is Indiana University's primary Linux cluster computing environment for research and research instruction use. It also serves as a "condominium cluster" environment for researchers, research labs, departments, and schools that want to have computational nodes housed within the IU Bloomington Data Center and managed by UITS Research Technologies staff. Additionally, Quarry provides a virtual machine hosting environment for the Extreme Science and Engineering Discovery Environment (XSEDE), the National Science Foundation's largest advanced cyberinfrastructure facility.

Quarry nodes run Red Hat Enterprise Linux (RHEL 6.5). Job management is provided by the TORQUE resource manager and the Moab job scheduler. The Modules system is used to simplify application access and environment configuration.

Accounts on Quarry are available to all IU students, faculty, staff, and sponsored affiliated researchers. For more, see Requesting an account below.

Back to top

Condominium computing

If you or your IU sub-unit have money available to use for a computing cluster, you always have the option of buying and operating your own cluster, as well as managing it, backing it up, and securing it against hackers. Alternatively, you can purchase nodes that are compatible with IU's Quarry cluster, have them installed in the very secure IUB Data Center, have them available when you want to use them, and have them managed, backed up, and secured by UITS Research Technologies staff. With this option, you get access to your nodes within seconds of requesting their use. Additionally, when they are not in use, they become available to others in the IU community, thereby expanding the computing capability available to the IU community while conserving natural resources and energy. Since any piece of computing equipment has a relatively short useful life (about four years), and it takes considerable energy and a variety of metals to make a computer node, the least environmental impact is achieved by using computing equipment to its absolute maximum capability. Because of the benefits to the IU community, UITS generally hosts condominium computing nodes without charging maintenance or operations fees.

If you are interested in hosting computational nodes within Quarry in a "condominium computing" approach, contact Research Technologies.

Back to top

System information

Note: The scheduled monthly maintenance window for Quarry is the first Tuesday of each month, starting at 7am and ending the next morning (Wednesday) at 7am.

System summary
Machine type Cluster computing, high-throughput computing, and condominium computing for research and research instruction
Operating system RHEL 6.5
Memory model Distributed and shared
Local storage 42.06 TB aggregate
Rmax 26.1 TFLOPS

Back to top

File storage options

File storage options for Quarry users include:

Home directories

Your Quarry home directory disk space is allocated on a network-attached storage (NAS) device. You have a 10 GB disk quota, which is shared (if applicable) with your accounts on Big Red II, Mason, and the Research Database Complex.

The path to your home directory is (replace username with your IU Network ID username):

/N/u/username/Quarry

Research Storage systems

IU graduate students, faculty, and staff who need more than 10 GB of permanent storage can apply for accounts on the distributed storage systems administered by UITS Research Storage:

To request an account on the Scholarly Data Archive (SDA) or Research File System (RFS), use the Account Management Service (AMS):

  • For instructions on requesting an individual SDA or RFS account, see Instructions for getting additional computing accounts at IU

  • For instructions on requesting an SDA or RFS account for an IU group or department, see Requesting a departmental or group account.

    Note: In accordance with standards for access control mandated by the HIPAA Security Rule, you are not permitted to access ePHI data using a group (or departmental) account. To ensure accountability and enable only authorized users to access ePHI data, IU researchers must use their personal Network ID credentials for all work involving ePHI data.

After submitting your account request, UITS will notify you via email when your account is ready for use.

For eligibility requirements, see the "Research system accounts (all campuses)" section in Account availability and eligibility

For more information about distributed storage services at Indiana University, visit the Pervasive Technology Institute's Research Storage page.

See Applying for your SDA or RFS account

Local scratch space

Local serial-attached SCSI (SAS) scratch space is available on every node at /tmp.

Note: Files in /tmp are automatically deleted once they are 14 days old.

Shared scratch space

Shared scratch space is hosted on the Data Capacitor II (DC2) file system. The DC2 scratch directory is a temporary workspace. Scratch space is not allocated, and its total capacity fluctuates based on project space requirements. The DC2 file system is mounted on IU research systems as /N/dc2/scratch and behaves like any other disk device. If you have an account on an IU research system, you can access /N/dc2/scratch/username (replace username with your IU Network ID username). Access to /N/dc2/projects requires an allocation. For details, see Data Capacitor II and DCWAN. Files in shared scratch space more than 60 days old are periodically purged, following user notification.

Note: The Data Capacitor II (DC2) high-speed, high-capacity, storage facility for very large data sets replaces the former Data Capacitor file system, which was decommissioned January 7, 2014. The DC2 scratch file system (/N/dc2/scratch) is mounted on Big Red II, Quarry, and Mason. Project directories on the former Data Capacitor were migrated to DC2 by UITS before the system was decommissioned. All data on the Data Capacitor scratch file system (/N/dc/scratch) were deleted when the system was decommissioned. If you have questions about the Data Capacitor's retirement, email the UITS High Performance File Systems group.

Back to top

Working with ePHI research data

The Health Insurance Portability and Accountability Act of 1996 (HIPAA) established rules protecting the privacy and security of personal health data. The HIPAA Security Rule set national standards specifically for the security of protected health information (PHI) that is created, stored, transmitted, or received electronically (i.e., electronic protected health information, or ePHI). To ensure the confidentiality, integrity, and availability of ePHI data, the HIPAA Security Rule requires organizations and individuals to implement a series of administrative, physical, and technical safeguards when working with ePHI data.

Although you can use this system for processing or storing electronic protected health information (ePHI) related to official IU research:

  • You and/or the project's principal investigator (PI) are responsible for ensuring the privacy and security of that data, and complying with applicable federal and state laws/regulations and institutional policies. IU's policies regarding HIPAA compliance require the appropriate Institutional Review Board (IRB) approvals and a data management plan.

  • You and/or the project's PI are responsible for implementing HIPAA-required administrative, physical, and technical safeguards to any person, process, application, or service used to collect, process, manage, analyze, or store ePHI data.

The UITS Advanced Biomedical IT Core (ABITC) provides consulting and online help for Indiana University researchers who need help securely processing, storing, and sharing ePHI research data. If you need help or have questions about managing HIPAA-regulated data at IU, contact the ABITC. For additional details about HIPAA compliance at IU, see HIPAA & ABITC and the Office of Vice President and General Counsel (OVPGC) HIPAA Privacy & Security page.

Important: Although UITS HIPAA-aligned resources are managed using standards surpassing official standards for managing institutional data at IU and are appropriate for storing HIPAA-regulated ePHI research data, they are not recognized by the IU Committee of Data Stewards as appropriate for storing institutional data elements classified as Critical that are not ePHI data. For help determining which institutional data elements classified as Critical are considered ePHI, see Which data elements in the classifications of institutional data are considered protected health information (PHI)?

The IU Committee of Data Stewards and the University Information Policy Office (UIPO) set official classification levels and data management standards for institutional data in accordance with the university's Management of Institutional Data policy. If you have questions about the classifications of institutional data, contact the appropriate Data Steward. To determine the most sensitive classification of institutional data you can store on any given UITS service, see the "Choosing an appropriate storage solution" section of At IU, which dedicated file storage services and IT services with storage components are appropriate for sensitive institutional data, including ePHI research data?

Note: In accordance with standards for access control mandated by the HIPAA Security Rule, you are not permitted to access ePHI data using a group (or departmental) account. To ensure accountability and enable only authorized users to access ePHI data, IU researchers must use their personal Network ID credentials for all work involving ePHI data.

Back to top

Available software

Software installed on Quarry is made available to users via Modules, an environment management system that lets you easily and dynamically add software packages to your user environment. For a list of software modules available on Quarry, see Quarry Modules in the IU Cyberinfrastructure Gateway.

For more on Modules, see the Using Modules to set up your software environment below. For more on the IU Cyberinfrastructure Gateway, see The IU Cyberinfrastructure Gateway

Quarry users can request software using the Research Systems Software Request form.

Back to top

Requesting an account

To request an account on Quarry, IU students, faculty, staff, and sponsored affiliated researchers can use the IU Account Management Service (AMS):

Note: Accounts remain valid only while the account holder is a registered IU student, or an IU faculty or staff member. On Big Red II, Quarry, Mason, and the Research Database Complex, accounts are disabled during the semester following the account holder's departure from IU, and then are purged within six months. To exempt a research systems account from disabling, email a request to the Accounts Administration team. If the request is approved, the account will remain activated for one calendar year beyond the user's departure from IU. Then, at the end of the year, the account will be purged. Extensions beyond one year for research accounts are granted only for accounts involved in funded research and having an IU faculty sponsor, or with approval of the Dean or Director of Research and Academic Computing.

Back to top

Accessing Quarry via SSH

Access Quarry using your favorite SSH client; see SSH/SFTP clients supported at IU

ssh quarry.uits.indiana.edu

Log in with your IU Network ID username and passphrase.

Make sure to read the message of the day (MOTD); it contains news and information regarding the status of the cluster.

Note the following:

  • Windows users: If you use an SSH client in Windows, you cannot open tools that need a graphical user interface (GUI), like the TotalView debugger. You'll need X Window emulation software, such as Cygwin. UITS recommends using XLiveCD created by the Research Technologies division of UITS.

  • X applications: To use the TotalView Debugger/Intel Trace Analyzer, or any other graphical application, from Quarry compute nodes, you must disable X forwarding by specifying the -x flag: ssh -x your_username@quarry.uits.indiana.edu

    To use common graphical applications, such as Emacs, from the head node, do not use the -x option; the default X forwarding must remain enabled.

  • Intra-cluster logins: When you log into your Quarry account for the first time, passphrase-less SSH keys will be automatically created in your home directory. Those keys should enable you to log into compute nodes that you have gained access to through TORQUE without entering a passphrase. Thus, parallel jobs should run seamlessly on multiple compute nodes without any manual intervention.

    However, you may see the following error message when you try to access assigned compute nodes:

    Permission denied (publickey,password,keyboard-interactive)

    This indicates that the intra-cluster RSA key pair in your home directory is either not present or corrupted. If this happens, enter gensshkeys (in /opt/xcat/sbin) from any login node. That will generate a passphrase-less key pair, and allow you seamless intra-cluster logins between any nodes in the cluster assigned for your use by TORQUE.

Back to top

Shell information

On Quarry, the default shell is bash. When you log in for the first time, you will be prompted to select your preferred login shell by the changeshell program:

jdoe@Quarry:~> changeshell This program will assist you in changing your login shell on all nodes of the Quarry cluster. . . . 1) bash 2) tcsh . . . 5) quit Select 1-5: 2 Changing login shell for jdoe Password: Shell changed. Your shell has been changed to the Cornell tcsh shell This will take effect on all nodes within 15 minutes
  • If you choose the Bash, Bourne, or Korn shell, the system will automatically read and execute commands from the /etc/profile and ~/.profile files (plus, for Bash, ~/.bash_profile).

  • If you choose the csh or tcsh shell, the system will automatically read and execute commands from the .login and .cshrc (or .tcshrc) files.

For details, see In Unix, what is the shell? or the Unix man pages.

Back to top

Login, interactive, and compute nodes

Nodes on Quarry are labeled using qXXXX designations (e.g., q0141):

  • Login nodes: When you log into Quarry, you will be placed on one of the system's login (or head) nodes; the node you are assigned is determined on a round-robin basis.

    User processes on login nodes are limited to 20 minutes of CPU time. Processes exceeding this limit are automatically terminated without warning.

  • Interactive nodes: Quarry's interactive nodes are for running interactive jobs that require more than 20 minutes (but less than 24 hours) of processor time. To use an interactive node, log into Quarry, and then SSH from the login node to q0145, q0146, q0147, or q0148; for example: [pamadala@q0143 ~]$ ssh q0145

    If your job requires more than 24 hours of CPU time, prepare a job script, and submit it to run as a batch job on the compute nodes; see Compiling programs and submitting batch jobs to TORQUE below.

  • Compute nodes: Quarry's compute nodes are for running batch jobs. They are accessible by the batch queuing system (and from each other) only; you cannot SSH directly to a compute node.

Back to top

Queue information

Quarry's queues have properties defined to maximize job throughput and minimize time spent waiting in queue. Quarry uses a TORQUE routing queue to funnel your job from the default batch queue into the SERIAL, NORMAL, or LONG queue, depending on the resource requirements specified in your job script. Your job will run in the SERIAL, NORMAL, or LONG queue unless you specifically submit it to the DEBUG queue.

Note: To best meet the needs of all research projects affiliated with Indiana University, the High Performance Systems (HPS) team administers the batch job queues on UITS Research Technologies supercomputers using resource management and job scheduling policies that optimize the overall efficiency and performance of workloads on those systems. If the structure or configuration of the batch queues on any of IU's supercomputing systems does not meet the needs of your research project, fill out and submit the Research Technologies Ask RT for Help form (for "Help Needed", select High Performance Systems job or queue help).

Quarry's batch queues have the following properties:

SERIAL:

  • Maximum walltime: 12 hours
  • Maximum nodes per job: 1 node
  • Maximum cores per job: 8 cores
  • Maximum number of jobs per queue: 2,000
  • Maximum number of jobs per user: 500
  • Direct submission: No

NORMAL:

  • Maximum walltime: 7 days
  • Maximum nodes per job: 6 nodes
  • Maximum cores per job: 48 cores
  • Maximum number of jobs per queue: 1,500
  • Maximum number of jobs per user: 500
  • Direct submission: No

LONG:

  • Maximum walltime: 14 days
  • Maximum nodes per job: 117 nodes
  • Maximum cores per job: 936 cores
  • Maximum number of jobs per queue: 550
  • Maximum number of jobs per user: 75
  • Direct submission: No

DEBUG:

  • Maximum walltime: 30 minutes
  • Maximum nodes per job: 2 nodes
  • Maximum cores per job: 16 cores
  • Maximum number of jobs per queue: None
  • Maximum number of jobs per user: 2
  • Direct submission: Yes (qsub -q debug)

Note: The DEBUG queue is intended for short, quick-turnaround test jobs requiring less than 30 minutes of wall clock time. Once code has been debugged, you may submit it to the other work queues via the normal qsub mechanism. The batch queue will dispatch your jobs to the appropriate work queue based on the resources you have requested.

CONDO queue: A separate CONDO queue is available for users who purchase condominium computing nodes on the Quarry cluster. Condominium computing users may submit jobs to either the regular batch queues or the CONDO queue. CONDO queue fairshare calculations will be separate from non-condominium usage. Jobs that run in the CONDO queue will not affect fairshare calculations for jobs submitted to the regular batch queue, and vice versa. If you are interested in condominium computing on Quarry, contact UITS Research Technologies.

Back to top

Requesting single user time

Although UITS Research Technologies cannot provide dedicated access to an entire compute system during the course of normal operations, "single user time" is made available by request one day a month during each system's regularly scheduled maintenance window to accommodate IU researchers with tasks requiring dedicated access to an entire compute system. To request single user time on one of IU's research computing systems, fill out and submit the Research Technologies Ask RT for Help form (for "Help Needed", select Request to run jobs in single user time on HPS systems). If you have questions about single user time on IU research computing systems, email the HPS team.

Back to top

Managing local and remote files

Like other Unix systems, Linux uses a hierarchical, tree-structured directory system to organize files. Data on local disks reside in journaled file systems that contain a root directory from which associated files and subdirectories branch. Directories are catalogs of files that associate names with files, and are used to segregate files into related groups. To display information about currently mounted file systems, at the shell prompt, enter:

df

Each file is described by an inode. An inode contains critical information about the file, such as file type (directory, ordinary, character special, block special, or pipe), ownership, access permissions, group ID, file size, file creation, modification, and access times, and a pointer to its data blocks. To list information about the files within your current working directory, at the shell prompt, enter ls -al. Following is a sample of ls output:

$ ls -al drwxr-xr-x 5 jdoe chem 1024 Feb 21 16:06 . drwxr-xr-x 12 bin bin 512 Feb 05 10:56 .. -rw-r--r-- 1 jdoe chem 13250 Feb 14 11:29 .Xdefaults -rw-r--r-- 1 jdoe chem 120 Dec 19 14:41 .aliases.ksh -rwxr----- 1 jdoe chem 24 Jan 13 11:50 .exrc -rw-r--r-- 1 jdoe chem 471 Jan 29 16:37 .profile -rw-r--r-- 1 jdoe chem 255 Jan 29 16:18 .rhosts drwxr-xr-x 3 jdoe chem 512 Dec 19 14:48 bin -rw------- 1 jdoe chem 2054 Feb 21 16:07 .sh_history -rw-r--r-- 1 jdoe chem 37479 Feb 21 14:51 whitepaper.pdf -rwxr-x--- 1 jdoe chem 45 Jan 16 15:22 submit.sh -rwxr-x--- 1 jdoe chem 79 Feb 19 13:02 myjob.c -rwxr-x--- 1 jdoe chem 53 Feb 19 16:45 myjob.c.old -rwxr-xr-x 1 jdoe chem 93 Jan 29 16:10 myjob.o drwxr-xr-x 2 jdoe chem 512 Dec 20 08:41 project1 drwxr-xr-x 2 jdoe chem 512 Dec 20 08:41 project2

The first character in the first field indicates the file type (e.g., - for ordinary file, d for directory). The next nine characters in the first field indicate the file access permissions: r for read, w for write, and x for execute; a hyphen (-) indicates that the corresponding read, write, or execute permission is denied. Characters 2, 3, and 4 indicate the file owner's permissions, characters 5, 6, and 7 indicate the group's permissions, and the last three characters indicate all others' permissions. Execute permission for a directory allows users to search through and list its contents.

Field 2 indicates the number of links to the file. The next two fields show the owner and the group associated with the file. Field 5 gives the size of the file in bytes. Field 6 is the time the file was created or last modified. The last field is the name of the file. The files named . and .. indicate the current and the parent directory, respectively. For more about ls and available options, enter man ls.

To change directories, use the cd command. You can use an absolute or relative path name with the cd command. For example, assume you are currently in the /usr/local directory. To get to the /usr/local/bin directory, you could enter either cd /usr/local/bin (using the absolute path name) or cd bin (using the path name relative to /usr/local). To get back to /usr/local, you could enter cd /usr/local or just cd .. (to go up to the parent directory). Entering cd without an argument will put you into your home directory. To determine what your current working directory is, enter pwd.

Following are some common commands for working with files:

  • To view a file on your screen, enter cat. To view one screen at a time, enter cat | more.

  • To copy a file to the same or a different directory, enter cp. This command overwrites any existing file of the same name. For more information, enter man cp.

  • To rename a file or to move it to a new directory, enter mv. For more information, enter man mv.

  • To delete a file, enter rm. For more about deleting single or multiple files, enter man rm.

  • To create a directory, use the mkdir command. To delete an empty directory, use the rmdir command. For more information, enter man mkdir or man rmdir.

You can also use the commands for managing local files to manage remote NFS files, provided that the access permissions allow it. For example, you cannot use rm on a file in a remote file system that's mounted as read-only.

For access to remote files other than those already accessible to Quarry, you may access a remote host using the Secure Shell scp or sftp commands to copy files to or from the remote system.

To see how many 1 KB blocks of disk space you are using, enter du -k. To see the number of 1 KB blocks in your disk quota, enter quota.

You may exceed your soft quota up to the limit of your hard quota. If you exceed your soft quota, you will receive a message to that effect. You will then have seven days to reduce your use below your soft quota. If you do not reduce your disk usage below your soft quota, you will receive a grace period of seven days, after which you will not be able to write to your disk until you reduce your disk usage below your soft quota.

Back to top

Editing files

Using vi

Before entering vi, make sure your environment variable TERM is set correctly. When vi starts, it looks at TERM to see what kind of terminal you are using. It then loads the proper terminal control information from /etc/termcap (the terminal capabilities file). Enter the appropriate commands below depending on your shell:

  • If you use the Bourne (sh) or Korn (ksh) shell: $ TERM=vt100 $ export TERM $ vi filename
  • If you use the C shell (csh): % setenv TERM vt100 % vi filename

For information about common vi commands, see How do I use the vi text editor?, A quick reference list of vi editor commands, or In vi, how can I access deleted text?

You can find more information in Learning the vi Editor by Linda Lamb (O'Reilly and Associates, Inc., 1992).

Using Emacs

GNU Emacs is a version of Emacs written by the author of the original (PDP-10) Emacs, Richard Stallman. GNU Emacs retains all the functionality of other Emacs editors, but it is also customizable and extensible by modifying its Lisp editing commands. GNU Emacs is designed to work well with X Window systems, but it also functions with ASCII terminals.

The account creation process on Quarry copies a .emacs file into the newly created home directory:

[jdoe@Quarry] cat .emacs ;; .emacs ;;; uncomment this line to disable loading of "default.el" at startup ;; (setq inhibit-default-init t) ;; turn on font-lock mode (when (fboundp 'global-font-lock-mode) (global-font-lock-mode t)) ;; enable visual feedback on selections ;(setq transient-mark-mode t) ;; default to better frame titles (setq frame-title-format (concat "%b - emacs@" system-name))

Because GNU Emacs uses the Ctrl-q and Ctrl-s key sequences as commands, disable the use of Ctrl-q and Ctrl-s as terminal output start/stop signals before entering GNU Emacs. To do this, enter stty -ixon. After completing your GNU Emacs session, you may re-enable terminal start/stop signals by entering stty ixon.

GNU Emacs has an extensive interactive help facility, but to use it you must know how to manipulate Emacs windows and buffers. (For help with the Emacs notation that follows, see In Emacs, how are keystrokes denoted?) To enter Emacs help, with an Emacs window open, press C-h. To access an interactive tutorial that teaches the fundamentals of Emacs, press C-h t. To find a command given its functionality, press C-h a. The Help Character command (C-h c) describes a given character's effect, and the Help Function command (C-h f) describes a given Lisp function specified by name.

For information about common Emacs commands, see GNU Emacs Quick Reference Guide.

Back to top

Command files (shell scripts)

Shell scripts are command files interpreted by a shell. These scripts may contain any statements valid in the shell in which they are interpreted. You can use them to perform repetitive tasks, such as setting up your environment, compiling and executing programs, submitting batch jobs, and performing system management tasks. Locally written shell script utilities are contained in /usr/local/bin.

Shell programming constructs include variable assignment, argument handling, evaluation of integer expressions, conditional execution, flow control, file status checking, string and integer comparisons, reading of input, interrupt signal traps, and menu screen generation.

The following example of a C shell script illustrates some basic shell programming constructs.

C shell script

# Scriptname: showuser # Purpose: display information about a user set USAGE = "Usage: showuser userid" # Test for argument if ($#argv > 1) then echo $USAGE exit 1 endif if ($#argv == 1) then set USERNAME = $1 else if ($#argv == 0) then echo "Enter the user's login name: " set USERNAME = $< endif set IFS = ":" # internal field separator # Use sed to edit /etc/passwd, putting blank in empty field sed "s/$IFS/ /g" < /etc/passwd > $HOME/passwd echo "_________________________________________________________" echo "Userid UID GID Full Name Home Directory Login Shell" echo " " # Use awk to find the correct line and print it out awk "/$USERNAME/" $HOME/passwd echo "_________________________________________________________" rm $HOME/passwd exit

The following reference guides may be of interest:

Back to top

Using Modules to set up your software environment

Research computing resources at Indiana University (Big Red II, Quarry, Mason, and Rockhopper) use the Modules environment management package, which allows you to dynamically customize your software environment on each system.

Some common Modules commands include:

Command Action
module avail List all software packages available on the system.
module avail package List all versions of package available on the system, for example:

module avail openmpi
module list List all packages currently loaded in your environment.
module load package/version Add the specified version of the package to your environment, for example:

module load intel/11.1

To load the default version of the package, use:

module load package
module unload package Remove the specified package from your environment.
module swap package_A package_B Swap the loaded package (package_A) with another package (package_B).

This is synonymous with:

module switch package_A package_B
module show package Shows what changes will be made to your environment (e.g., paths to libraries and executables) by loading the specified package.

This is synonymous with:

module display package

To make permanent changes to your environment, edit your ~/.modules file. For more, see In Modules, how do I save my environment with a .modules file?

For more about the Modules package, see the module manual page and the modulefile manual page. Additionally, see On Big Red II, Mason, Quarry, and Rockhopper at IU, how do I use Modules to manage my software environment?

Back to top

Parallel computing and MPI

The Quarry cluster is equipped with a proprietary low-latency interconnect. Parallel jobs can be run over gigabit Ethernet; the cluster is connected to a Force10 E1200 switch.

To see which versions of Open MPI and MPICH are currently available, use the appropriate module avail command:

module avail mpich module avail openmpi

Note: Open MPI and MPICH compiled with the 64-bit Intel v10 compilers are available. The MPICH package is currently available compiled only with the Intel compiler suite.

To add either package to your environment:

  1. Add the necessary compiler suite to your environment:

    • GNU Compiler Collection (GCC): module load gcc
    • Intel Compiler Suite: module load intel
  2. Load the module for the appropriate package, compiler suite (suite), and version (<version>):

    • MPICH: module load mpich/<suite>/<version>
    • Open MPI: module load openmpi/<suite>/<version>

Briefly, use mpicc or mpif77 to compile and link MPI-enabled software. Then, use mpirun to execute the job on multiple nodes of the cluster. In practice, you must submit your job through the TORQUE resource manager to ensure fair and efficient use of the cluster.

Note: On Quarry, Moab serves as the job scheduler for the TORQUE resource manager. For information on Moab, see:

Back to top

Compiling programs and submitting batch jobs to TORQUE

Two simple programs are listed below for your convenience:

C program

/* * Copyright 2005, The Trustees of Indiana University. * Original author: Arvind Gopu (UITS-RAC-HPC group)... * . . . [ snip ] . . . */ #include<stdio.h> #include<math.h> int main () { double PI2=3.141592654/2.0, theta, sintheta; int i, N=4; for (i=0; i<=N; i++) { theta = i * (PI2/N); sintheta = sin (theta); printf (" Sin (%8.6lf) = %8.6lf \n", theta, sintheta); } return 0; }

Fortran program

C C Copyright 2005, The Trustees of Indiana University. C Original author: Don Berry (UITS-RAC-HPC group); C . . . [ snip ] . . . C program sine real :: PI2=3.141592654/2.0 integer :: N=4 real x,s integer i do i=0,N x=i*(PI2/N) s=sin(x) write(6,"(f11.6 f11.6)") x,s end do end

These commands use the GNU compilers (gcc and g77 for C and Fortran programs, respectively) to compile the two example programs:

jdoe@Quarry: ~> gcc -o sine_c sine.c -lm jdoe@Quarry: ~> g77 -o sine_f sine.f

Submit job(s) to TORQUE

It's best to write your job in its own script file, and tell TORQUE to execute that. Following is a script that runs the example sine_c program:

[jdoe@Quarry]$ cat submit_sine_c.sh #PBS -l nodes=1:ppn=1,walltime=5:00 #PBS -m ae #PBS -N job_sine_c ${HOME}/simple_quarry_jobs/sine_c

Use the script to submit your job to TORQUE:

[jdoe@Quarry]$ qsub submit_sine_c.sh 95.m2.quarry.teragrid.iu.edu

Check the job status with the qstat command:

[jdoe@Quarry]$ qstat | grep jdoe [jdoe@Quarry]$ qstat -u jdoe Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time --------------- -------- -------- ---------- ------ --- --- ------ ----- - ----- 95.m2.quarry.teragri jdoe batch job_sine_c -- 1 1 -- 00:05 R -- 96.m2.quarry.teragri jdoe batch job_sine_f -- 1 1 -- 00:05 Q --

Note: Quarry will email job-related messages to the address specified in the ~/.forward file in your home directory. (Note the period [.] preceding the filename.) By default, this is the email address you provided when you requested your account. To change this email address, on the command line, enter (replace username@host.com with your email address):

echo "username@host.com" > ~/.forward

For more about submitting and tracking jobs, see What is TORQUE, and how do I use it to submit and manage jobs on high-performance computing systems?

Output and error files

Assuming your job runs to completion, you can find messages it tried to print on the console in an output file. You can specify the directories where the output file and the error file go using the -o and -e flags, respectively. The default file name of these files is of the format job_name.osequence_number and job_name.esequence_number, where job_name is the name of the job (check out the -N option) and sequence_number is the job number assigned when the job is submitted.

For example, if TORQUE assigned job id 95.m2.quarry.teragrid.iu.edu to your job, with jobname job_sine_c, the output file would be named job_sine_c.o95 and the error file would be named job_sine_c.e95. (Also see the use of the -j TORQUE directive as explained in the section above.)

[jdoe@Quarry]$ ls job_sine_* job_sine_c.e348921 job_sine_c.o348921 job_sine_f.e348922 job_sine_f.o348922

Check the output files:

[jdoe@Quarry]$ cat job_sine_c.o348921 Sin (0.000000) = 0.000000 Sin (0.392699) = 0.382683 Sin (0.785398) = 0.707107 Sin (1.178097) = 0.923880 Sin (1.570796) = 1.000000 [jdoe@Quarry]$ cat job_sine_f.o348922 0.000000 0.000000 0.392699 0.382683 0.785398 0.707107 1.178097 0.923880 1.570796 1.000000

Back to top

Compiling and submitting parallel jobs

Here is a simple C program you can submit to use multiple processors:

#include <stdio.h> #include <mpi.h> /* #include "VT.h" */ /* only needed for Vampir Trace */ int main( int argc, char *argv[] ) { /* Remember: We are programming according to the SPMD model! */ /* This means that each and every processor executes this program. */ /* It is convenient to reference the typical processor in the */ /* first person, using "I" and "me" and "my." In other words, */ /* read through this program as if you were one of the processors. */ /* My ID number or my "rank": */ int myrank, length; char myname[BUFSIZ]; /* I initialize for myself the MPI API: */ MPI_Init(&argc, &argv); /* Who am I? I request my ID number: */ MPI_Comm_rank(MPI_COMM_WORLD, &myrank); MPI_Get_processor_name (myname, &length); //VT_symdef( 200, "for-loop", "Calculation" ); /* I print the standard greeting along with my ID: */ printf( "Hello, parallel worlds! This is processor %s and my rank is %d!\n", myname, myrank ); /* Finally I close the MPI API: */ MPI_Finalize(); return 0; }

Here is the script file to submit the job:

#! /bin/bash #PBS -l nodes=1:ppn=4,walltime=1:00 #PBS -m ea #PBS -N My_mpi_job # Get the number of processors NP=`wc -l $PBS_NODEFILE | awk '{print $1}'` # print out the nodes where the job runs echo "Execute node list:" sort -u $PBS_NODEFILE # change to the working directory cd simple_quarry_jobs # run an mpi job: mpirun -np $NP -machinefile $PBS_NODEFILE helloWorlds

Compile and submit:

[jdoe@Quarry]$ mpicc -o helloWorlds helloWorlds.c [jdoe@Quarry]$ qsub submit_parallel.sh 97.m2.quarry.teragrid.iu.edu

Look at the output file:

[jdoe@Quarry]$ cat job_helloWorlds.o348923 LAM 7.1.1/MPI 2 C++/ROMIO - Indiana University Hello, parallel worlds! This is processor bc27 and my rank is 1! Hello, parallel worlds! This is processor bc25 and my rank is 2! Hello, parallel worlds! This is processor bc27 and my rank is 0! Hello, parallel worlds! This is processor bc25 and my rank is 3! LAM 7.1.1/MPI 2 C++/ROMIO - Indiana University

Back to top

Usage policies

For information about your responsibilities as a computer user at IU, see:

Back to top

Acknowledging grant support

The Indiana University cyberinfrastructure managed by the Research Technologies division of UITS is supported by funding from several grants, each of which requires you to acknowledge its support in all presentations and published works stemming from research it has helped to fund. Conscientious acknowledgment of support from past grants also enhances the chances of IU's research community securing funding from grants in the future. For the acknowledgment statement(s) required for scholarly printed works, web pages, talks, online publications, and other presentations that make use of this and/or other grant-funded systems at IU, see Grants to cite in published papers and presentations

Back to top

Support

Support for research computing systems at Indiana University is provided by various units within the Systems area of the Research Technologies division of UITS:

To ask any other question about Research Technologies systems and services, use the Request help or information form.

Back to top