Derecho-GPU

RP account needed

Derecho-GPU is a GPU-accelerated compute cluster, the GPU portion of an HPE Cray EX supercomputer, consisting of 82 nodes with 64 AMD EPYC 7763 "Milan" cores, 512 GB of memory, and four 40 GB NVIDIA A100 GPUs per node. It is particularly well suited for AI and machine learning training and inference and for multi-GPU scientific simulation, and is often used for model training, large-scale inference, and GPU-accelerated modelling. It includes a great deal of GPU-enabled AI/ML, scientific computing, and parallel programming software.

Get Support

OFFICE HOURS

Mon-Fri 8 AM - 5 PM MT

GET AN ACCOUNT ON DERECHO-GPU

NCAR Account Portal

Submitting Jobs Documentation

You can run jobs at different sizes and durations on Derecho-GPU. The following lists the different queues that you can submit to, describing how many nodes you get, how long you can run, the type of resources you get, and the average wait time.

Derecho-GPU uses the PBS Professional scheduler. Use qsub to submit batch scripts, qcmd to schedule a single command, and qinteractive to start interactive work. Use qstat to monitor active jobs, qhist to inspect completed jobs, and qdel to cancel jobs. For instructions, see [Run jobs on Derecho with PBS]. GPU job scripts are available in [Derecho Job Script Examples].

Use main for production GPU jobs. Use preempt for interruptible jobs that can tolerate preemption and loss of unsaved progress. Use develop for short interactive, development, and debugging work on shared GPU nodes. Jobs submitted to main and preempt receive exclusive nodes and are charged for all four GPUs on each allocated node; jobs submitted to develop share nodes and are charged only for the requested GPUs.

Queue limits and how each queue is charged are listed in [Derecho-GPU Job submission queues and charges].

Queue specifications Documentation

Queue CPU cores / node GPUs / node Num nodes Node RAM Max wallclock
develop
Short interactive or batch development and debugging on shared GPU nodes; routes to the gpu execution queue and allows up to 8 concurrent GPUs per user
 1x AMD EPYC 7763 "Milan" (64 cores) 4 NVIDIA A100 (40 GB vRAM) N/A 487 GB 6h
main
Production GPU jobs requiring one or more exclusive nodes; routes to the gpu execution queue
 1x AMD EPYC 7763 "Milan" (64 cores) 4 NVIDIA A100 (40 GB vRAM) 82 512 GB 12h
preempt
Interruptible GPU jobs that can tolerate preemption and loss of unsaved progress; runs on otherwise idle nodes and routes to the pgpu execution queue
 1x AMD EPYC 7763 "Milan" (64 cores) 4 NVIDIA A100 (40 GB vRAM) 82 512 GB 100h

Software Documentation

No software usage data is currently reported for Derecho-GPU in XDMoD.

SEE ALL SOFTWARE AVAILABLE ON DERECHO-GPU

Storage Documentation

Derecho primarily uses NCAR's shared GLADE file systems. Use Home for small files that require backup, Scratch for temporary job data and large-scale workflows, Work for active data needed longer than the Scratch retention period, and Campaign Storage for allocation-based project storage. Check your usage with gladequota. For details, see [Derecho GLADE file spaces].

File System Documentation

Directory Path Quota Purge Backup Notes
Home /glade/u/home/<ncar-username> 50 GB Purged 30 days after account deactivation Backed up several times per week; backups retained several weeks Small scripts, source code, configuration files, and small datasets. gladequota may display 100 GB because NCAR stores dual copies, but the usable quota is 50 GB. NCAR publishes no file-count limit for this space.
Scratch /glade/derecho/scratch/<ncar-username> 30 TB, 10,000,000 files Purged 180 days after last access Not backed up Temporary output and large-scale computing or analysis workflows. Purging is based on file-access time, and deleted files cannot be recovered. Temporary quota increases may be requested.
Work /glade/work/<ncar-username> 2 TB Purged 30 days after account deactivation Not backed up Active datasets, compiled software, and conda environments needed longer than the Scratch retention period. NCAR publishes no file-count limit for this space.
Campaign Storage /glade/campaign/<project_path> See notes Purged 90 days after allocation expires Not backed up Shared project storage available through a Campaign Storage allocation. Quota depends on the approved project request; this is not permanent archival storage.

External Storage

Quasar is NCAR's cold, tape-based archive for curated data collections with an indefinite lifetime. It is not available for university project data, so an ACCESS allocation on Derecho does not include archival storage - copy anything you need to keep beyond the Campaign Storage retention period to storage you control. For details, see [Quasar archive for data collections].

File Transfer Documentation

Globus is recommended for large or many-file transfers between NCAR storage and other sites; NCAR suggests reaching for it beyond roughly 1,000 files or 200 MB. Derecho has no separate data transfer node, so scp, rsync, sftp and rclone transfers run through the login node at derecho.hpc.ucar.edu. For instructions, see [Derecho Data transfers and sharing].

Supported Methods Data Transfer Node / Globus Collection Notes
GLOBUS | RECOMMENDED NCAR GLADE Globus file transfers
SCP derecho.hpc.ucar.edu SSH-based data transfer tools
RSYNC derecho.hpc.ucar.edu SSH-based data transfer tools
SFTP derecho.hpc.ucar.edu SSH-based data transfer tools
RCLONE derecho.hpc.ucar.edu Rclone

Datasets

Name Description
CMIP Analysis Platform

Climate data from the Coupled Model Intercomparison Project, mirrored on NCAR's GLADE storage under /glade/campaign/collections/cmip.mirror/. Request datasets that are not already on GLADE with esgfsearch and esgfdownload. For current holdings, see [Derecho CMIP Analysis Platform].