Derecho GPU is a GPU-accelerated computing resource within the Derecho system designed for machine learning, AI, and GPU-enabled scientific applications. It consists of dedicated GPU nodes equipped with NVIDIA A100 Tensor Core GPUs and high-bandwidth memory.
Each GPU node includes 4 NVIDIA A100 GPUs, 512 GB of system memory, and high-speed NVLink interconnects between GPUs, enabling efficient multi-GPU computation.
This resource is optimized for workloads requiring GPU acceleration, including deep learning, simulation, and GPU-enabled HPC applications.
File Transfer
Derecho uses NCAR’s GLADE storage system with multiple file spaces.
| Supported Methods | Data Transfer Node | URL |
|---|---|---|
| SCP | derecho.hpc.ucar.edu | |
| RSYNC | derecho.hpc.ucar.edu | |
| SFTP | derecho.hpc.ucar.edu | |
| RCLONE | RECOMMENDED | derecho.hpc.ucar.edu | https://rclone.org/downloads/ |
| GLOBUS | RECOMMENDED | https://www.globus.org/ |
Storage
File System
| Directory | Path | Quota | Purge | Backup | Notes |
|---|---|---|---|---|---|
| Home | /glade/u/home/<username> | 50 GB | Not purged | Yes | User home directory. Ideal for small scripts, source code, and configuration files that benefit from backup. |
| Scratch | /glade/derecho/scratch/<username> | 30 TB / 10M | 180 Days | No | Temporary space. Derecho's scratch file system also includes a limit of 10 Million on a users' total number of files |
| Work | /glade/work/<username> | 2 TB | Not purged | No | User work space. Ideal for compiled code, conda environments, and similar large holdings that do not require backup. |
| Campaign Storage | /glade/campaign | N/A | Not purged | No | Project space allocations (via allocation request) |
Jobs
Jobs on Derecho GPU resources are submitted through PBS Professional and run on GPU-enabled nodes. Each GPU node includes 4 A100 GPUs and associated CPU resources.
Users request GPU resources in their job scripts by specifying the number of GPUs per node. Example:
#PBS -l select=N:ncpus=64:mpiprocs=4:ngpus=4GPU workloads include machine learning training, AI inference, and GPU-accelerated simulations. Jobs are scheduled using the same queues as CPU jobs but require explicit GPU resource requests.
Queue specifications
| Name | Purpose | CPUs | GPUs | RAM | Jobs
30 days
|
Wait Time
30-day trend
|
Wall Time
30-day trend
|
|---|---|---|---|---|---|---|---|
| Derecho--GPU | AMD EPYC 7742 (Milan) 2,25 Ghz | NVIDIA A100 (40 GB) | 8 GB per CPU Core | — | — | — |