Derecho

RP account needed

Derecho is a CPU compute cluster, the CPU-focused portion of an HPE Cray EX supercomputer, consisting of 2,488 dual-socket nodes with 128 AMD EPYC 7763 "Milan" cores and 256 GB of memory per node. It is particularly well suited for large-scale distributed-memory and hybrid MPI/OpenMP workloads, and is often used for weather and climate modeling, scientific simulations, and data processing. It includes a great deal of compiler, MPI, numerical-library, and scientific-modeling software for high-performance computing.

Get Support

OFFICE HOURS

Mon-Fri 8 AM - 5 PM MT

GET AN ACCOUNT ON DERECHO

NCAR Account Portal

Submitting Jobs Documentation

You can run jobs at different sizes and durations on Derecho. The following lists the different queues that you can submit to, describing how many nodes you get, how long you can run, the type of resources you get, and the average wait time.

Derecho uses the PBS Professional scheduler. Use qsub to submit batch scripts, qcmd to schedule a single command, and qinteractive to start interactive work. Use qstat to monitor active jobs, qhist to inspect completed jobs, and qdel to cancel jobs. For instructions, see [Run jobs on Derecho with PBS]. Worked scripts are in [Derecho Job Script Examples].

Use main for production CPU jobs. Use preempt for interruptible jobs that can tolerate preemption and loss of unsaved progress. Use develop for short interactive, development, and debugging work on shared CPU nodes. Jobs submitted to main and preempt receive exclusive nodes and are charged for all 128 CPU cores on each allocated node; jobs submitted to develop share nodes and are charged only for the requested cores.

If you do not request resources explicitly, qcmd runs on one node with 32 cores and qinteractive gives you 32 cores and 55 GB of memory, both on the develop queue. Queue limits and how each queue is charged are listed in [Derecho Job submission queues and charges].

Queue specifications Documentation

Queue CPU cores / node GPUs / node Num nodes Node RAM Max wallclock
develop
Short interactive or batch development and debugging on shared CPU nodes; routes to the cpudev execution queue and allows up to 256 concurrent CPU cores per user
 2x AMD EPYC 7763 "Milan" (128 cores) None N/A 235 GB 6h
main
Production CPU jobs requiring one or more exclusive nodes; routes to the cpu execution queue
 2x AMD EPYC 7763 "Milan" (128 cores) None 2,488 256 GB 12h
preempt
Interruptible CPU jobs that can tolerate preemption and loss of unsaved progress; runs on otherwise idle nodes and routes to the pcpu execution queue
 2x AMD EPYC 7763 "Milan" (128 cores) None 2,488 256 GB 100h

Software Documentation

No software usage data is currently reported for Derecho in XDMoD.

SEE ALL SOFTWARE AVAILABLE ON DERECHO

Storage Documentation

Derecho primarily uses NCAR's shared GLADE file systems. Use Home for small files that require backup, Scratch for temporary job data and large-scale workflows, Work for active data needed longer than the Scratch retention period, and Campaign Storage for allocation-based project storage. Check your usage with gladequota. For details, see [Derecho GLADE file spaces].

File System Documentation

Directory Path Quota Purge Backup Notes
Home /glade/u/home/<ncar-username> 50 GB Purged 30 days after account deactivation Backed up several times per week; backups retained several weeks Small scripts, source code, configuration files, and small datasets. gladequota may display 100 GB because NCAR stores dual copies, but the usable quota is 50 GB. NCAR publishes no file-count limit for this space.
Scratch /glade/derecho/scratch/<ncar-username> 30 TB, 10,000,000 files Purged 180 days after last access Not backed up Temporary output and large-scale computing or analysis workflows. Purging is based on file-access time, and deleted files cannot be recovered. Temporary quota increases may be requested.
Work /glade/work/<ncar-username> 2 TB Purged 30 days after account deactivation Not backed up Active datasets, compiled software, and conda environments needed longer than the Scratch retention period. NCAR publishes no file-count limit for this space.
Campaign Storage /glade/campaign/<project_path> See notes Purged 90 days after allocation expires Not backed up Shared project storage available through a Campaign Storage allocation. Quota depends on the approved project request; this is not permanent archival storage.

External Storage

Quasar is NCAR's cold, tape-based archive for curated data collections with an indefinite lifetime. It is not available for university project data, so an ACCESS allocation on Derecho does not include archival storage - copy anything you need to keep beyond the Campaign Storage retention period to storage you control. For details, see [Quasar archive for data collections].

File Transfer Documentation

Globus is recommended for large or many-file transfers between NCAR storage and other sites; NCAR suggests reaching for it beyond roughly 1,000 files or 200 MB. Derecho has no separate data transfer node, so scp, rsync, sftp and rclone transfers run through the login node at derecho.hpc.ucar.edu. For instructions, see [Derecho Data transfers and sharing].

Supported Methods Data Transfer Node / Globus Collection Notes
GLOBUS | RECOMMENDED NCAR GLADE Globus file transfers
SCP derecho.hpc.ucar.edu SSH-based data transfer tools
RSYNC derecho.hpc.ucar.edu SSH-based data transfer tools
SFTP derecho.hpc.ucar.edu SSH-based data transfer tools
RCLONE derecho.hpc.ucar.edu Rclone

Datasets

Name Description
CMIP Analysis Platform

Climate data from the Coupled Model Intercomparison Project, mirrored on NCAR's GLADE storage under /glade/campaign/collections/cmip.mirror/. Request datasets that are not already on GLADE with esgfsearch and esgfdownload. For current holdings, see [Derecho CMIP Analysis Platform].