The OSG provides technologies and services for supporting pools of distributed high-throughput computing (dHTC) and data capacity that serve a number of research communities, worldwide. Among these services, the OSG operates the Open Science Pool (OSPool), which includes capacity contributed from clusters at campuses and national labs across and beyond the US, organized as a single virtual cluster.
Jobs
Jobs are submitted using the HTCondor software. This process involves preparing an executable script and a submit file to submit the job. From there, the status of jobs in the queue can be reviewed. Upon conclusion, the results can be examined. An example of this process can be found here. Additionally, a video can be found here that goes into more detail.
It is important to determine the amount of resources that need to be requested by starting with test job submissions before scaling up. A job duration category must also be specified upon job submission. Tips on submitting test jobs and determining resources can be found on this page. Additional information can be found in this video on high throughput computing with HTCondor.
HTCondor can submit multiple jobs using queue. Information on this process can be found on this page.
Information on converting workflow from slurm to HTCondor can be found here.
Storage
File System
| Directory | Path | Quota | Purge | Backup | Notes |
|---|---|---|---|---|---|
| Home | /home | 50GB | Use the /home directory to stage input and output files where input files total less than 1GB and output files per job are less than 1GB per file. | ||
| Data | /ospool/*/data | 100GB | An OSDF directory. Quota increase requests can be sent here: http://osg-htc.org/ospool-quota-increase.html | ||
| Projects | /ospool/*/projects | varies | An OSDF directory. Quota increase requests can be sent here: http://osg-htc.org/ospool-quota-increase.html |
File Transfer
- Smaller job files should be stored in the /home directory and can be transferred using HTCondor.
- Larger job files should be stored in and transferred using OSDF.
- If some of the data or software the jobs depend on is available via the web, files can be transferred using the appropriate HTTP address.
- When using a command line application for performing file transfers, utilities such as scp can be used.
- Other GUI tools can be used for transferring files and folders.
| Supported Methods | Data Transfer Node | URL |
|---|---|---|
| HTCONDOR | ap4#.uw.osg-htc.org | Transferring smaller job files to and from /home using HTCondor |
| OSDF | ap4#.uw.osg-htc.org | Using OSDF to transfer larger files and containers. |
| SCP | ap4#.uw.osg-htc.org | Use scp To Transfer Files To and From Access Point |
| WINSCP | ap4#.uw.osg-htc.org | WinSCP |
| FILEZILLA | ap4#.uw.osg-htc.org | FileZilla |
| CYBERDUCK | ap4#.uw.osg-htc.org | Cyberduck |
Login to OSPool
OSPool can be accessed by launching an OSPool Notebooks Instance or through an access node.
OSPool Notebooks:
- Start an instance at https://notebook.ospool.osg-htc.org/
Access Point:
The access point can be logged into via SSH Key Pair Authentication or Browser-based web authentication.
Login via SSH Key Pair:
- Generate an SSH Key Pair and upload it to your COmanage account.
- Open a terminal and run ssh username@ap4#.uw.osg-htc.org below. Replace username with your username and specify the assigned ap4# (located in your Welcome Email).
- After hitting enter to run the command, enter your corresponding passphrase (if prompted to) to complete the login process.
Login via Web Authentication:
- Open a terminal and run ssh username@ap4#.uw.osg-htc.org below. Replace username with your username and specify the assigned ap4# (located in your Welcome Email).
- Upon hitting enter, some text containing a unique URL will appear. Copy the link and paste it into your web browser to open it.
- At the link, login using your institutional credentials. A new page should appear with a message of success if login was correct.
- Return to your terminal and press the 'Enter' key to complete the login process.
Consult this page if there's issues logging into the access point.