SDSC Expanse GPU

GPU nodes are allocated as a separate resource. The GPU nodes can be accessed via either the "gpu" or the "gpu-shared" partitions.

#SBATCH -p gpu

or

#SBATCH -p gpu-shared

When users request 1 GPU, in gpu-shared partition, by default they will also receive, 1 CPU, and 1G memory.  Here is an example AMBER script using the gpu-shared queue.

GPU JOB

#!/bin/bash
#SBATCH --job-name="ambergpu"
#SBATCH --output="ambergpu.%j.%N.out"
#SBATCH --partition=gpu
#SBATCH --nodes=1
#SBATCH --gpus=4
#SBATCH --mem=377300M
#SBATCH --account=<<project*>>
#SBATCH --no-requeue
#SBATCH -t 01:00:00

module purge
module load gpu
module load slurm
module load openmpi			
module load amber
pmemd.cuda -O -i mdin.GPU -o mdout.GPU.$SLURM_JOBID -x mdcrd.$SLURM_JOBID \
    -nf mdinfo.$SLURM_JOBID -1 mdlog.$SLURM_JOBID -p prmtop -c inpcrd

* Expanse  requires users to enter a valid project name; users can list valid project by running the expanse-client script.

GPU-SHARED JOB

#!/bin/bash
#SBATCH --job-name="ambergpushared"
#SBATCH --output="ambergpu.%j.%N.out"
#SBATCH --partition=gpu-shared
#SBATCH --nodes=1
#SBATCH --gpus=2
#SBATCH --cpus-per-task=1
#SBATCH --mem=93G
#SBATCH --account=<<project*>>
#SBATCH --no-requeue
#SBATCH -t 01:00:00

module purge
module load gpu
module load slurm
module load openmpi			
module load amber
pmemd.cuda -O -i mdin.GPU -o mdout-OneGPU.$SLURM_JOBID -p prmtop -c inpcrd

* Expanse  requires users to enter a valid project name; users can list valid project by running the expanse-client script.

Users can find application specific example job script on the system in directory /cm/shared/examples/sdsc/.

GPU modes can be controlled for jobs in the "gpu" partition.  By default, the GPUs are in non-exclusive mode and the persistence mode is 'on'.  If a particular "gpu"  partition job needs exclusive access the following options should be set in your batch script:

#SBATCH --constraint=exclusive

To turn persistence off add the following line to your batch script:

#SBATCH --constraint=persistenceoff

The charging equation will be:

GPU SUs = (Number of GPUs) x (wall clock time)

Regular SUs = (Number of Cores) x (wall clock time)