Bridges-2 EM

Jobs run in the EM partition and are charged in core-hours, where 1 core-hour equals 1 Service Unit (SU). For example, using one full node (96 cores) for one hour results in 96 SUs. Specific details are documented on the Accounting for Usage page on the Bridges-2 guide.

Jobs can use at most one full EM node and must specify the number of cores requested. Core counts must be requested in multiples of 24 (24, 48, 72, or 96 cores). Memory is allocated proportionally based on the number of cores requested, at approximately 1 TB per 24 cores.

Users must also specify a walltime limit when submitting jobs, or system defaults will be applied.

For more detail, please visit the Bridges-2 Extreme Memory page or the Summary table for the EM partition on the Bridges-2 guide.

Note that the EM partition does not support interactive jobs or access through OnDemand. EM jobs must be submitted in batch mode. Because of this, EM jobs must be submitted from the command line on a Bridges-2 login node. Connect via SSH first (see the Login to Bridges-2 section below), then save your batch script to your file space and submit it with the sbatch command.

When the job is submitted, sbatch echoes back a job id. You can check your job's progress with:

squeue -u PSC-username

A sample script job on the EM partition would look like this:

#!/bin/bash
#SBATCH -N 1
#SBATCH -p EM
#SBATCH -t 5:00:00
#SBATCH -n 96

# type 'man sbatch' for more information and options
# this job will ask for 1 full EM node (96 cores) and 5 hours of runtime
# this job would potentially charge 480 EM SUs

# echo commands to stdout
set -x

# move to working directory
# this job assumes:
# - all input data is stored in this directory
# - all output should be stored in this directory
# - please note that groupname should be replaced by your groupname
# - PSC-username should be replaced by your PSC username
# - path-to-directory should be replaced by the path to your directory where the executable is

cd /ocean/projects/groupname/PSC-username/path-to-directory

#run pre-compiled program which is already in your project space

./a.out

After that, you need a sbatch command to submit a job to the EM partition. An example requesting an entire node for 5 hours is:

sbatch -p EM -t 5:00:00 --ntasks-per-node=96 myscript.job

where:

-p indicates the intended partition
-t is the walltime requested in the format HH:MM:SS
--ntasks-per-node is the number of cores requested per node
myscript.job is the name of your batch script.

Detailed information about batch jobs can be found in the Batch Jobs page of the guide.