PNRP

PNRP is a Kubernetes compute cluster consisting of 8 NVIDIA HGX A100 servers (8 A100 80 GB GPUs each) and 32 composable Xilinx Alveo U55C FPGAs at SDSC, plus 36 nodes with 8 NVIDIA A10 GPUs each at UNL and MGHPCC, all part of the national Nautilus cluster. It is particularly well suited for containerized GPU and FPGA workloads and for work with data spread across sites, and is often used for AI model training and inference, classroom teaching and workshops, and networking research. It includes a great deal of managed services, including JupyterHub, Coder, and NRP-hosted large language models.

Get Support

GET AN ACCOUNT ON PNRP

Create your PNRP account

Submitting Jobs Documentation

You can run jobs at different sizes and durations on PNRP. The following lists the different queues that you can submit to, describing how many nodes you get, how long you can run, the type of resources you get, and the average wait time.

PNRP does not use Slurm or batch queues. Work runs as Kubernetes workloads inside a namespace, and the limits depend on the type of workload:

  • Jobs (Kubernetes Job) are the recommended way to run computations. NRP does not publish a maximum run time for Jobs.
  • Pods started without a controller are treated as interactive. They are deleted after 6 hours and are limited to 2 GPUs, 32 GB RAM and 16 CPU cores.
  • Deployments (long-running services) are deleted after 2 weeks and cannot request GPUs.
  • A100 GPUs need an approved request: every namespace starts with an A100 quota of zero. Pods that set priorityClassName: opportunistic skip the quota but can be preempted at any time.

Set resource limits within 20% of your requests. A user cannot run more than 4 pods that use less than 40% of their requested GPUs, fall outside 20–200% of requested CPU, or fall outside 20–150% of requested memory. Jobs that run sleep infinity, or scripts that end in sleep, get the user banned. See PNRP Running Batch Jobs and PNRP Cluster Policies for the full rules.

Software Documentation

No software usage data is currently reported for PNRP in XDMoD.

SEE ALL SOFTWARE AVAILABLE ON PNRP

Storage Documentation

PNRP storage is requested as Kubernetes volumes (PersistentVolumeClaims) in your namespace, plus node-local scratch and S3 object storage. Nothing persists inside a container: data is lost on restart unless it is on a volume. Do not install conda or pip packages on CephFS.

File System Documentation

Directory Path Quota Purge Backup Notes
Shared (CephFS) rook-cephfs See notes Not backed up Size is set when you create the volume; no namespace limit is published. Mountable by many pods at once. 16 TB per-file limit.
Block (RBD) rook-ceph-block See notes Not backed up Size is set when you create the volume. One pod at a time. Use for conda/pip installs, code builds and databases.
Node-local emptyDir See notes Purged at job end Not backed up Set ephemeral-storage requests to the size you need; pods using more than 50 GiB without a request can be evicted.
Object (S3) Ceph S3 endpoint See notes Not backed up Account must be requested. 5 TiB per-file limit. Fastest for large transfers.

File Transfer Documentation

Use S3 object storage for large or many-file transfers. Pods have no outside address and PNRP has no separate data transfer node, so other data is pulled from inside a pod (rclone, scp, wget) or synced through Nextcloud.

Supported Methods Data Transfer Node / Globus Collection Notes
S3 | RECOMMENDED PNRP Ceph S3
RCLONE OR SCP FROM A POD PNRP Pulling data from inside the pod
GLOBUS PNRP Globus Connect
NEXTCLOUD PNRP Nextcloud