Voyager

RP account needed

Voyager is an AI testbed system consisting of 42 Habana Gaudi training nodes with eight training processors each, two first-generation Habana inference nodes, 36 x86 compute nodes and a 400 gigabit-per-second Arista interconnect. It is particularly well suited for deep learning at scale, and is often used for training and inference in science and engineering research that depends on artificial intelligence. It includes a great deal of container-based machine learning software, with Habana-optimized TensorFlow and PyTorch images.

Get Support

GET AN ACCOUNT ON VOYAGER

Request Voyager access from SDSC HPC Consulting

Submitting Jobs Documentation

You can run jobs at different sizes and durations on Voyager. The following lists the different queues that you can submit to, describing how many nodes you get, how long you can run, the type of resources you get, and the average wait time.

Voyager does not use Slurm. Work is scheduled by Kubernetes, so there is no sbatch and no squeue. Load the scheduler with module load kubernetes/voyager, describe the work in a YAML file, and submit it with kubectl apply -f. YAML is whitespace sensitive, so an example copied out of a web page often fails to parse.

The smallest unit you can ask for is a Pod, and accelerators are requested individually, so a job can take part of a node: ask for habana.ai/gaudi on the training nodes and habana.ai/goya on the inference nodes. Node type is chosen with the node selector brightcomputing.com/node-category, which takes goya or compute. A job that spans more than one node has to be an MPIJob, not a plain Job.

Work runs in containers, and the image has to carry the Habana drivers - a stock x86 container will not use the accelerators. SDSC publishes no charging rates for Voyager during its testbed phase.

There is no default node type; the pod spec decides. During the testbed phase every node category has a 48 hour maximum wall time, and a single job can use at most 42 Gaudi nodes, 2 inference nodes or 36 compute nodes. For the pod, job and MPIJob templates and the full kubectl reference see [Voyager Running Jobs].

Queue specifications Documentation

Queue CPU cores / node GPUs / node Num nodes Node RAM Max wallclock
gaudi
Deep learning training on the Gaudi accelerators. Choose this for model training, from a single accelerator up to 42 nodes.
 2x Intel Xeon Gold 6336 8 Habana Gaudi (32 GB vRAM) 42 512 GB 48h
goya
Inference work on the first-generation Habana inference processors. Choose this for serving or evaluating a trained model rather than training one.
 2x Intel Xeon Gold 6240 (40 cores) 8 Habana Goya 2 384 GB 48h
compute
Data preparation and post-processing that does not need an accelerator. Choose this for staging data, converting datasets and analysing results.
 2x Intel x86 (2 cores) None 36 384 GB 48h

Software Documentation

No software usage data is currently reported for Voyager in XDMoD.

SEE ALL SOFTWARE AVAILABLE ON VOYAGER

Storage Documentation

Voyager has a small 200 GB Home for source code and configuration, a Ceph parallel file system at /voyager/ceph/users/ for active job data, and NFS project space at /voyager/projects/. Each node also has local NVMe scratch, which you mount as an emptyDir volume and which exists only while your pod is running on that node.

None of these file systems is backed up and there is no archival tier on Voyager, so anything you cannot lose has to be copied off the system yourself. SDSC limits each user to 2 million files on the Ceph file system. See [Voyager Storage] for the full description.

File System Documentation

Directory Path Quota Purge Backup Notes
Home /home/<sdsc-username> 200 GB Never purged Not backed up Source code and configuration only. Home is small and is not a place to run jobs from - use the Ceph file system or your project space for job data. No per-user file-count limit is published for Home; the 2 million file limit applies to Ceph.
Projects /voyager/projects/<project>/<sdsc-username> 153 TB Never purged Not backed up Project space shared by everyone on the project. SDSC publishes 153 TB as the total project space available on Voyager, not a per-project quota, so no figure is shown here - ask SDSC Consulting for your project's share.
Ceph /voyager/ceph/users/<sdsc-username> 2,000,000 files Not backed up Main working space for active job data: 3 PB in total at about 140 GB/s. It is not an archive, and SDSC publishes no per-user capacity quota - only the file count shown here. Copy anything you cannot lose off Voyager yourself.
Node-local See notes Purged at job end Not backed up Mount it as an emptyDir volume in your pod spec; it exists only while the pod is on that node. Capacity is 6.4 TB on the Gaudi nodes, 3.2 TB on the compute nodes and 1.6 TB on the inference nodes.

File Transfer Documentation

Voyager has no data transfer node. SDSC lists Globus endpoints for /voyager/projects as coming soon and has not published an endpoint or collection name yet, so there is no Globus route today.

Until then, command-line transfers go through the login node login.voyager.sdsc.edu, which SDSC asks you not to use for large or numerous transfers. For a bulk move, contact SDSC HPC Consulting at consult@sdsc.edu first. See [Voyager Data Movement] for what SDSC has published so far.

Supported Methods Data Transfer Node / Globus Collection Notes
GLOBUS (COMING SOON) Voyager Data Movement