Bridges-2 GPU

Jobs on Bridges-2's GPU nodes are submitted through the Slurm scheduler, using either the GPU or GPU-shared partition. Jobs in the GPU partition take one or more whole nodes, receiving all 8 GPUs on each (16 on the DGX-2). Jobs in the GPU-shared partition take up to half of a single node, between 1 and 4 GPUs, and share that node with other jobs.

Use sbatch to submit batch jobs and interact for interactive sessions. Either way, you specify the GPU type and count with --gpus=type:n or --gres=gpu:type:n, where the type is one of h100-80, l40s-48, v100-32, or v100-16. Note that n means the total GPUs for the job in batch mode (a multiple of 8), but GPUs per node in interactive mode (8, or 16 for the DGX-2).

Both partitions default to a 1-hour walltime, up to a 48-hour maximum. GPU-partition jobs are charged for the full node regardless of how many GPUs they use; GPU-shared jobs are charged only for the GPUs they request. Specific details are documented on the Accounting for Usage page on the Bridges-2 guide.

For more detail, please visit the Bridges-2 GPU and GPU-Shared partitions page or the Summary table for the GPU partition on the Bridges-2 guide.

A sample job script for the GPU partition would look like this:

#!/bin/bash
#SBATCH -N 1
#SBATCH -p GPU
#SBATCH -t 5:00:00
#SBATCH --gpus=v100-32:8

#type 'man sbatch' for more information and options
#this job will ask for 1 full v100-32 GPU node(8 V100 GPUs) for 5 hours
#this job would potentially charge 40 GPU SUs

#echo commands to stdout
set -x

# move to working directory
# this job assumes:
# - all input data is stored in this directory
# - all output should be stored in this directory
# - please note that groupname should be replaced by your groupname
# - PSC-username should be replaced by your PSC username
# - path-to-directory should be replaced by the path to your directory where the executable is

cd /ocean/projects/groupname/PSC-username/path-to-directory

#run pre-compiled program which is already in your project space

./gpua.out

After that, you need a sbatch command to submit a job to he GPU partition. A sample sbatch command to submit a job to the GPU partition to use 2 full GPU v100-16 nodes  and all 8 GPUs on each node for 5 hours is:

sbatch -p GPU -N 2 --gpus=v100-16:16 -t 5:00:00 jobname

where:

-p indicates the intended partition
-N 2 requests two v100-16 GPU nodes
--gpus=v100-16:16  requests the use of all 8 GPUs on both v100-16 nodes, for a total of 16 for the job
-t is the walltime requested in the format HH:MM:SS
jobname is the name of your batch script

Detailed information about batch jobs can be found in the Batch Jobs page of the guide.