Expanse CPU

Expanse CPU is a compute cluster consisting of 728 nodes. It is particularly well suited for parallel workloads that scale across many nodes, and is often used for large MPI simulations and other tightly-coupled parallel work.

Get Support

GET AN ACCOUNT ON EXPANSE CPU

Request an account

Submitting Jobs Documentation

Expanse uses the Simple Linux Utility for Resource Management (SLURM) batch environment. When you run in the batch mode, you submit jobs to be run on the compute nodes using the sbatch command as described in Expanse Running Jobs.

Remember that computationally intensive jobs should be run only on the compute nodes and not the login nodes. 

Queue specifications

Metrics updated 2026-09-29

Queue CPU cores / node Num nodes Node RAM Max wallclock Wait time
30-day trend
 Wall time
30-day trend
 Number of jobs run
30 days
compute
Compute Node Usage
 AMD EPYC 7742 (128 cores) 734 256 GB 48h
compute wait time: average 5.5 hours, range 0 to 20.6 hours over 30 days
compute wall time: average 7.5 hours, range 0.8 to 10.2 hours over 30 days, wall-time limit 48h
17,240
debug
Priority access to shared nodes set aside for testing of jobs with short wall time and limited resources
 AMD EPYC 7742 (128 cores) 2 256 GB 30m
debug wait time: average 0.0 hours, range 0 to 0.3 hours over 30 days
debug wall time: average 0.0 hours, range 0 to 0.2 hours over 30 days, wall-time limit 30m
12,283
preempt
Non-refundable discounted jobs to run on free nodes that can be pre-empted by jobs submitted to any other queue
 AMD EPYC 7742 (128 cores) 726 256 GB 168h
preempt wait time: average 0.2 hours, range 0 to 2.8 hours over 30 days
preempt wall time: average 0.2 hours, range 0.1 to 18.5 hours over 30 days, wall-time limit 168h
18,871
large-shared
Single-node jobs using large memory up to 2 TB (minimum memory required 256G)
 AMD EPYC 7742 (128 cores) 4 2 TB 48h
large-shared wait time: average 0.4 hours, range 0 to 4.1 hours over 30 days
large-shared wall time: average 8.2 hours, range 0 to 16 hours over 30 days, wall-time limit 48h
166
shared
Jobs using part of a single 256 GB node, sharing the node with other jobs.
 AMD EPYC 7742 (128 cores) 734 256 GB 48h
shared wait time: average 4.1 hours, range 1.3 to 12.8 hours over 30 days
shared wall time: average 2.4 hours, range 0.6 to 3.8 hours over 30 days, wall-time limit 48h
260,077

Software Documentation

No software usage data is currently reported for Expanse CPU in XDMoD.

SEE ALL SOFTWARE AVAILABLE ON EXPANSE CPU

Storage Documentation

Expanse has a small backed-up Home for source code and configuration, a 12 PB Lustre parallel file system for active scratch and project data, and fast node-local SSD that exists only for the duration of a job. There is no archival tier and the Lustre file system is not backed up, so anything you cannot lose has to be copied off Expanse yourself.

SDSC limits each user to 2 million files under /expanse/lustre/scratch. If your workflow needs a very large number of small files, contact support first - it loads the metadata server and can affect the whole system. Node-local scratch is a different size on each node type; the Notes column gives the figures.

Jobs that need Lustre must ask for it explicitly with #SBATCH --constraint="lustre", which can be combined with other constraints. A job that needs Lustre and omits the constraint is scheduled on a node without it and will fail. See [Expanse Storage] for the full description.

File System

Directory Path Quota Purge Backup Notes
Scratch Lustre /expanse/lustre/scratch 10 TB 90 days after allocation expiration. Not backed up​ This is not an archival file system, it is not backed up, and will be purged according to purge policy.
Scratch Compute Node /scratch/$USER/job_$SLURM_JOB_ID 1 TB Users only have access to these SSDs during job execution at the local file system path to the compute node.
Home /home 100 GB N/A 8 week rolling backup The home directory is limited in space and should be used only for source code storage. Jobs should never be run from the home file system, as it is not set up for high performance throughput.

External Storage Documentation

Expanse has no archival tier of its own. Project space is purged 90 days after the allocation expires and Lustre is never backed up, so move anything you need to keep to an external archive before your allocation ends.

To see which projects you can charge to and how much of your allocation is left, run expanse-client on a login node. See [Expanse Account Management] for its syntax.

File Transfer Documentation

Use Globus for anything large or made of many files - it retries and resumes on its own. Expanse has no separate data transfer node, so command-line transfers with scp, sftp or rsync go to login.expanse.sdsc.edu, and SDSC asks you not to use the login nodes for large or numerous transfers. See [Expanse Data Movement] for the Globus collections and their mount points.

Supported Methods Data Transfer Node / Globus Collection Notes
GLOBUS | RECOMMENDED SDSC HPC - Expanse Lustre Globus

Datasets

Name Description
OceanTopography

OpenTopography provides efficient, user-friendly access to high-resolution topography data, processing tools, and resources to advance understanding of the Earth's surface, vegetation, and built environment.
OpenAltimetry

OpenAltimetry is a web based data visualization and discovery tool for exploring surface elevation profiles over time using satellite altimetry data from NASA's ICESat and ICESat-2 missions.
OpenForest4D

OpenForest4D is a web-based platform that leverages multi-source remote sensing data and artificial intelligence to generate on-demand, research-grade estimates of forest structure and above-ground biomass in four dimensions for global forest monitoring.