SDSC Expanse CPU

The job charge for a compute node is 128 SU per hour run ( 128 cores in one node x 1 hour = 128 SU ). 

Each standard compute node has ~256 GB of memory and 128 cores

• Each standard node core will be allocated 1 GB of memory, users should explicitly include the --mem directive to request additional memory
• Max. available memory per compute node --mem = 249208M

Requesting interactive resources using srun

You can request an interactive session using the srun command. The following example will request one regular compute node, 4 cores,  in the debug partition for 30 minutes.

srun --partition=debug  --pty --account=<<project>> --nodes=1 --ntasks-per-node=4 \
    --mem=8G -t 00:30:00 --wait=0 --export=ALL /bin/bash

 Expanse uses the Simple Linux Utility for Resource Management (SLURM) batch environment. When you run in the batch mode, you submit jobs to be run on the compute nodes using the sbatch command as described below. Remember that computationally intensive jobs should be run only on the compute nodes and not the login nodes.

Expanse places limits on the number of jobs queued and running on a per group (allocation) and partition basis. Please note that submitting a large number of jobs (especially very short ones) can impact the overall  scheduler response for all users. If you are anticipating submitting a lot of jobs, please contact the SDSC consulting staff before you submit them. We can work to check if there are bundling options that make your workflow more efficient and reduce the impact on the scheduler.

The limits for each partition are noted in the table below. Partition limits are subject to change based on Early User Period evaluation.

Requesting interactive resources using srun

You can request an interactive session using the srun command. The following example will request one regular compute node, 4 cores,  in the debug partition for 30 minutes.

srun --partition=debug  --pty --account=<<project>> --nodes=1 --ntasks-per-node=4 \
    --mem=8G -t 00:30:00 --wait=0 --export=ALL /bin/bash

The following example will request a GPU node, 10 cores, 1 GPU and 96G  in the debug partition for 30 minutes.  To ensure the GPU environment is properly loaded, please be sure run both the module purge and module restore commands.

login01$ srun --partition=gpu-debug --pty --account=<<project>> --ntasks-per-node=10 \
    --nodes=1 --mem=96G --gpus=1 -t 00:30:00 --wait=0 --export=ALL /bin/bash

srun: job 1336890 queued and waiting for resources
srun: job 1336890 has been allocated resources
exp-7-59$ module purge
exp-7-59$ module restore

Resetting modules to system default. Resetting $MODULEPATH back to system default.
    All extra directories will be removed from $MODULEPATH.

Submitting Jobs Using sbatch

Jobs can be submitted to the sbatch partitions using the sbatch command as follows:

 sbatch jobscriptfile

where jobscriptfile is the name of a UNIX format file containing special statements (corresponding to sbatch options), resource specifications and shell commands. Several example SLURM scripts are given below:

BASIC MPI JOB

#!/bin/bash
#SBATCH --job-name="hellompi"
#SBATCH --output="hellompi.%j.%N.out"
#SBATCH --partition=compute
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=128
#SBATCH --mem=0 
#SBATCH --account=<<project*>>
#SBATCH --export=ALL
#SBATCH -t 01:30:00

#This job runs with 2 nodes, 128 cores per node for a total of 256 tasks.

module purge
module load cpu
#Load module file(s) into the shell environment
module load gcc
module load mvapich2
module load slurm

srun --mpi=pmi2 -n 256 ../hello_mpi