Derecho is NSF NCAR’s HPE Cray EX supercomputer. This ACCESS Derecho resource represents the CPU-focused portion of the system: 2,488 CPU-only compute nodes, each with 128 AMD EPYC Milan cores and 256 GB RAM. Derecho CPU resources are suited for large-scale distributed-memory simulation, data processing, and MPI or hybrid MPI/OpenMP workloads that scale across many nodes.
Jobs
Jobs on Derecho CPU resources are submitted with PBS Professional and run on CPU-only compute nodes. Users can submit batch jobs with qsub and monitor jobs with commands such as qstat and qhist.
Derecho CPU jobs commonly use MPI or hybrid MPI/OpenMP parallelism. Production CPU jobs are typically submitted to the main queue and routed to the cpu execution queue. Development and debugging work can use the develop queue, while interruptible jobs that can tolerate preemption can use the preempt queue.
Example CPU resource request:
#PBS -l select=10:ncpus=128:mpiprocs=128:ompthreads=1For more information about submitting jobs on Derecho, see [Running Jobs on Derecho]. For sample PBS scripts, see [Derecho Batch Job Script Examples].
Queue specifications
| Name | Purpose | CPUs | GPUs | RAM | Jobs
30 days
|
Wait Time
30-day trend
|
Wall Time
30-day trend
|
|---|---|---|---|---|---|---|---|
| main | Primary queue for production CPU jobs; routes to the cpu execution queue. Nodes are allocated for exclusive use. | 128 cores per node | - | 256 GB per node | — | — | — |
| develop | Shared development/debugging queue for short interactive or batch CPU work; routes to the cpudev execution queue. | 256 cores | - | 235 GB per node | — | — | — |
| preempt | Preemptible queue for jobs that can run on otherwise idle resources and tolerate interruption; routes CPU jobs to the pcpu execution queue. | 128 cores per node | - | 256 GB per node | — | — | — |
Datasets
| Name | Description |
|---|---|
| CMIP Analysis Platform | Climate data from the Coupled Model Intercomparison Project available on CISL’s GLADE disk storage resource under |
Storage
File System
| Directory | Path | Quota | Purge | Backup | Notes |
|---|---|---|---|---|---|
| Home | /glade/u/home/<username> | 50 GB | Not purged | Yes (Weekly) | User home directory. Ideal for small scripts, source code, and configuration files that benefit from backup. |
| Scratch | /glade/derecho/scratch/<username> | 30 TB / 10M files | 180 Days | No | Temporary space. Derecho's scratch file system also includes a limit of 10 Million on a users' total number of files |
| Work | /glade/work/<username> | 2 TB | Not purged | No | User work space. Ideal for compiled code, conda environments, and similar large holdings that do not require backup. |
| Campaign Storage | /glade/campaign | N/A | Not purged | No | Project space allocations (via allocation request) |
File Transfer
Derecho has access to NCAR’s GLADE storage resources, and users can transfer data to and from Derecho through GLADE. For large transfers between NCAR-managed systems, university storage resources, desktops/laptops, Campaign Storage, or other external systems, NCAR recommends using Globus. SSH-based tools such as SCP, SFTP, rsync, and rclone are also available for smaller transfers or workflows where Globus is not suitable.
For further information on transferring data to and from Derecho, see [NCAR Data Transfer].
| Supported Methods | Data Transfer Node | URL |
|---|---|---|
| SCP | derecho.hpc.ucar.edu | |
| RSYNC | derecho.hpc.ucar.edu | |
| SFTP | derecho.hpc.ucar.edu | |
| RCLONE | derecho.hpc.ucar.edu | https://rclone.org/downloads/ |
| GLOBUS | RECOMMENDED | https://www.globus.org/ |
Login to Derecho
Direct SSH access to Derecho is available at derecho.hpc.ucar.edu. Users must have an NCAR account, a Derecho allocation, and Duo two-factor authentication.
For further information on accessing Derecho, see [Logging in to Derecho].