TACC Stampede3 is a general-purpose supercomputer designed for large-scale simulation, high-throughput computing, artificial intelligence, GPU-accelerated workloads, and large-memory applications. Stampede3 provides multiple node types to support a wide range of research needs, including Intel Sapphire Rapids high-bandwidth-memory nodes, Ice Lake CPU nodes, large-memory Ice Lake NVDIMM nodes, NVIDIA H100 GPU nodes, Intel Data Center GPU Max nodes, and Skylake CPU nodes.
Jobs
Stampede3 uses Slurm for Job Scheduling. Create a job script and use Slurm's sbatch command to submit a job.
Stampede3's accounting system is based on node-hours: one unadjusted Service Unit (SU) represents a single compute node used for one hour (a node-hour). The queue charge rates are determined by the supply and demand for that particular queue or type of node used and are subject to change.
Stampede3 SUs billed = (# nodes) x max(job duration in wall clock hours,.25) x (charge rate per node-hour)
The Slurm scheduler tracks and charges for usage to a granularity of a few seconds of wall clock time. The system charges only for the resources you actually use, not those you request; however, there is a 15 minute minimum charge of queue time.
To display a summary of your TACC project balances and disk quotas at any time, execute:login1$ /usr/local/etc/taccinfo
For more information on jobs, visit these links on:
Job Scripts, Running Jobs, Submitting Batch Jobs, Job Management, and Launching Applications.
Queue specifications
| Name | Purpose | Nodes | CPU cores / node | GPUs / node | Node RAM | Jobs
30 days
|
Wait Time
30-day trend
|
Wall Time
30-day trend
|
|---|---|---|---|---|---|---|---|---|
| nvdimm | Ultra-high memory jobs. | 3 | Intel Xeon Platinum 8380 ("Ice Lake") (80 cores) | — | 4 TB | — | — | — |
| icx | General CPU jobs in ICX nodes. | 224 | Intel Xeon Platinum 8380 ("Ice Lake") (80 cores) | — | 256 GB | — | — | — |
| spr | High-throughput, memory bandwidth intensive jobs. | 560 | Intel Xeon CPU MAX 9480 ("Sapphire Rapids HBM") (112 cores) | — | 128 GB | — | — | — |
| skx | General CPU jobs on SKX nodes. | 1,060 | Intel Xeon Platinum 8160 ("Skylake") (48 cores) | — | 192 GB | — | — | — |
| skx-dev | Development and testing on SKX. | 1,060 | Intel Xeon Platinum 8160 ("Skylake") (48 cores) | — | 192 GB | — | — | — |
| h100 | NVIDIA GPU workloads. | 24 | Intel Xeon Platinum 8468 ("Sapphire Rapids") (96 cores) | 4 NVDIA H100 SXM5 (96 GB vRAM) | 4 TB | — | — | — |
| pvc | Intel GPU workloads. | 20 | Intel Xeon Platinum 8480 ("Sapphire Rapids") (96 cores) | 4 Intel Data Center GPU Max 1550s ("Ponte Vecchio") (124 GB vRAM) | 1 TB | — | — | — |
Storage
File System
| Directory | Path | Quota | Purge | Backup | Notes |
|---|---|---|---|---|---|
| $HOME | $HOME | 15 GB, 300,000 files | After account deactivation | Yes - backed up regularly | VAST filesystem. Personal scripts, config files; not for high-intensity |
| $WORK | $WORK | 1 TB, 3,000,000 files | Not on a schedule | Not backed up | Lustre filesystem. Quota is across all TACC systems. Not intended for parallel or high−intensity file operations. |
| $SCRATCH | $SCRATCH | Subject to purge if access time is more than 10 days old. | Not backed up | VAST filesystem. See Scratch File System Purge Policy. https://docs.tacc.utexas.edu/hpc/stampede3/#scratchpolicy |
External Storage
RANCH (Long-term Archival Tape Storage)
RANCH (Long-term Archival Tape Storage) is an allocated resource that is only available to users who have an allocation on a TACC system such as Stampede3. The default allocation for RANCH is 2 TB, and it is provided at no additional cost, meaning it does not consume ACCESS credits. It is intended for final data that is unlikely to change and does not require frequent access. RANCH can be accessed through the Globus endpoint Ranch3, and files can also be transferred using SCP with a command such as scp myfile ${ARCHIVER}:${ARCHIVE}/myfilepath.
Corral (Online Project Storage)
Corral (Online Project Storage) is TACC’s online project storage system. It is designed for users and projects that need more storage than $HOME and $WORK provide. Corral can be accessed from Stampede3 through supported TACC data transfer methods, including Globus.
For more information on managing your files, see this page.
File Transfer
Stampede3 supports two primary data transfer technologies: SSH-based tools (scp, sftp, rsync) and Globus.
For smaller transfers less than ~200GB, ssh tools are typically easier to work with. See https://docs.tacc.utexas.edu/datatransfer/ssh/ for more information.
For transfer larger than 200GB, Globus is the recommended transfer method. For more information on Globus on Stampede3, see https://docs.tacc.utexas.edu/datatransfer/globus/.
For more information about Data Transfer at TACC, see this page.
| Supported Methods | Data Transfer Node | URL |
|---|---|---|
| SCP | stampede3.tacc.utexas.edu | Using scp |
| GLOBUS | stampede3.tacc.utexas.edu | Globus Data Transfer Guide at TACC |
| SFTP | stampede3.tacc.utexas.edu | Using sftp |
| RSYNC | stampede3.tacc.utexas.edu | Using rsync |
Login to TACC Stampede3
2FA/MFA required for login
To log into Stampede3, users will use standard SSH, using their TACC account credentials. Users will receive their project allocation through ACCESS, but Stampede3 login access will go through TACC's systems. See TACC Portal.
In terminal, run: $ssh <username>@stampede3.tacc.utexas.edu then enter your TACC password and MFA respectively when prompted.
If you receive error message "Corrupted MAC on input" run:
$ssh -m hmac-sha2-512 <username>@stampede3.tacc.utexas.edu
For more information, Accessing the System, MFA at TACC, and TACC Accounts Portal.
Additional Note: Do NOT run ssh-keygen while on Stampede3. See Secure Shell (SSH)
SSH Login
$ ssh <your_username>@stampede3.tacc.utexas.edu