Expanse Storage

Expanse Storage provides a 12 PB Lustre parallel filesystem for fast scratch and project I/O and a 7 PB Ceph object store for object-storage workflows. Both are accessible from every Expanse node.

Submitting Jobs Documentation

In addition to the local scratch storage, users will have access to global parallel filesystems on Expanse. Every Expanse node has access to a 12 PB Lustre parallel file system (provided by Aeon Computing) and a 7 PB Ceph Object Store system, 140 GB/second performance storage. SDSC limits the number of files that can be stored in the /lustre/scratch filesystem to 2 million files per user. Users should contact support for assistance at the ACCESS Help Desk if their workflow requires extensive small I/O, to avoid causing system issues assosiated with load on the metadata server.

The two Lustre filesystems available on Expanse are:

Lustre Expanse scratch filesystem: /expanse/lustre/scratch/$USER/temp_project
Lustre NSF projects filesystem: /expanse/lustre/projects/

SUBMITTING JOBS USING LUSTRE

Jobs that need to use the Lustre filesystem should explicitly reqeust the feature by including the following line to their script:

#SBATCH --constraint="lustre"

This constraint can be used in combination with any other constraints you are already using. For example:

#SBATCH --constraint="lustre&persistenceoff&exclusive"

Jobs submitted without --constraint="lustre" that need the Lustre filesystem will be scheduled on nodes without Lustre and will FAIL.

Software

No software usage data is currently reported for Expanse Storage in XDMoD.

SEE ALL SOFTWARE AVAILABLE ON EXPANSE STORAGE

Expanse Storage is not currently reporting available software to the Software Documentation Service.
See Provider Software Documentation

Storage Documentation

Expanse has a small backed-up Home for source code and configuration, a 12 PB Lustre parallel file system for active scratch and project data, and fast node-local SSD that exists only for the duration of a job. There is no archival tier and the Lustre file system is not backed up, so anything you cannot lose has to be copied off Expanse yourself.

SDSC limits each user to 2 million files under /expanse/lustre/scratch. If your workflow needs a very large number of small files, contact support first - it loads the metadata server and can affect the whole system. Node-local scratch is a different size on each node type; the Notes column gives the figures.

Jobs that need Lustre must ask for it explicitly with #SBATCH --constraint="lustre", which can be combined with other constraints. A job that needs Lustre and omits the constraint is scheduled on a node without it and will fail. See [Expanse Storage] for the full description.

File System

Directory	Path	Quota	Purge	Backup	Notes
Scratch Lustre	/expanse/lustre/scratch	See notes	90 days after allocation expiration.	Not backed up	This is not an archival file system, it is not backed up, and will be purged according to policy.
Scratch Large Memory Node	/scratch/$USER/job_$SLURM_JOB_ID	See notes			Users only have access to these SSDs during job execution at the local file system path to the compute node.
Home	/home	See notes		8 week rolling backup	The home directory is limited in space and should be used only for source code storage. Jobs should never be run from the home file system, as it is not set up for high performance throughput.

External Storage Documentation

Expanse has no archival tier of its own. Project space is purged 90 days after the allocation expires and Lustre is never backed up, so move anything you need to keep to an external archive before your allocation ends.

To see which projects you can charge to and how much of your allocation is left, run expanse-client on a login node. See [Expanse Account Management] for its syntax.

File Transfer Documentation

Use Globus for anything large or made of many files - it retries and resumes on its own. Expanse has no separate data transfer node, so command-line transfers with scp, sftp or rsync go to login.expanse.sdsc.edu, and SDSC asks you not to use the login nodes for large or numerous transfers. See [Expanse Data Movement] for the Globus collections and their mount points.

Supported Methods	Data Transfer Node / Globus Collection	Notes
GLOBUS \| RECOMMENDED	SDSC HPC - Expanse Lustre	Globus

Login to Expanse Storage Documentation

2FA/MFA required for login

Sign in by following the [Expanse System Access] guide, or open the [Expanse User Portal] web interface. Before your first sign-in:

You need an active ACCESS allocation on Expanse.
Two-factor authentication is required. Enroll your authenticator app before you connect: enrollment link.
SSH keys are supported, but only together with a TOTP code. A key on its own will not get you in.

There is no separate SDSC account to create - an ACCESS allocation request or a Trial Account is what gives you access.

SSH Login

$ ssh <your_username>@login.expanse.sdsc.edu

ACCESS OnDemand Login

How do I use ACCESS OnDemand?

Enroll in SDSC two-factor authentication Login Instructions for Expanse

Datasets

Name	Description
OceanTopography	OpenTopography provides efficient, user-friendly access to high-resolution topography data, processing tools, and resources to advance understanding of the Earth's surface, vegetation, and built environment.
OpenAltimetry	OpenAltimetry is a web based data visualization and discovery tool for exploring surface elevation profiles over time using satellite altimetry data from NASA's ICESat and ICESat-2 missions.
OpenForest4D	OpenForest4D is a web-based platform that leverages multi-source remote sensing data and artificial intelligence to generate on-demand, research-grade estimates of forest structure and above-ground biomass in four dimensions for global forest monitoring.