Managing Python Packages on an HPC Cluster
1
This workshop will go into the different ways python packages can be managed in a cluster environment using conda and python virtual environments both in batch mode from the command line and with Jupyter Notebooks and Jupyter Lab on the cluster. The examples will be run on the GMU HOPPER Cluster.
ACCESS Pegasus Documentation
1
The documentation provides an overview of using Pegasus, a workflow management system, on ACCESS resources for high throughput computing (HTC) workloads, covering logging in, workflow creation, resource configuration, and monitoring options.
Gentle Introduction to Programming With Python
1
This course from MIT OpenCourseWare (OCW) covers very basic information on how to get started with programming using Python. Lectures are available, along with practice assignments, to users at no cost. Python has many applications in tech today, from web frameworks to machine learning. This course will also instruct users on how to get set up with an IDE, which will allow for way more efficient debugging.
Higher Ed Controlled Unclassified Information Slack (HigherEdCUI)
0
Slack channel for the Higher Ed CUI community
Bioinformatics Workflow Management with Nextflow
0
Nextflow is an open-source, domain-specific language and workflow manager designed for the execution and coordination of scientific and data-intensive computational workflows. It was specifically created to address the challenges faced by researchers and scientists when dealing with complex and scalable computational pipelines, particularly in fields such as bioinformatics, genomics, and data analysis.
Here provided some links to start with.
How to Build a Great Relationship with a Mentor
0
Emphasizes benefits of being mentored. Describes how to identify and choose a mentor. Suggests a path forward. Not mentor or two-way focused.
Setting up PyFR flow solver on clusters
0
These instructions were executed on the FASTER and Grace cluster computing facilities at Texas A&M University. However, the process can be applied to other clusters with similar environments. For local installation, please refer to the PyFR documentation.
Please note that these instructions were valid at the time of writing. Depending on the time you're executing these, the versions of the modules may need to be updated.
1. Loading Modules
The first step involves loading pre-installed software libraries required for PyFR. Execute the following commands in your terminal to load these modules:
module load foss/2022b
module load libffi/3.4.4
module load OpenSSL/1.1.1k
module load METIS/5.1.0
module load HDF5/1.13.1
2. Python Installation from Source
Choose a location for Python 3.11.1 installation, preferably in a .local directory. Navigate to the directory containing the Python 3.11.1 source code. Then configure and install Python:
cd $INSTALL/Python-3.11.1/
./configure --prefix=$LOCAL --enable-shared --with-system-ffi --with-openssl=/sw/eb/sw/OpenSSL/1.1.1k-GCCcore-11.2.0/ PKG_CONFIG_PATH=$LOCAL/pkgconfig LDFLAGS=/usr/lib64/libffi.so.6.0.2
make clean; make -j20; make install;
3. Virtual Environment Setup
A virtual environment allows you to isolate Python packages for this project from others on your system. Create and activate a virtual environment using:
pip3.11 install virtualenv
python3.11 -m venv pyfr-venv
. pyfr-venv/bin/activate
4. Install PyFR Dependencies
Several Python packages are required for PyFR. Install these packages using the following commands:
pip3 install --upgrade pip
pip3 install --no-cache-dir wheel
pip3 install --no-cache-dir botorch pandas matplotlib pyfr
pip3 uninstall -y pyfr
5. Install PyFR from Source
Finally, navigate to the directory containing the PyFR source code, and then install PyFR:
cd /scratch/user/sambit98/github/PyFR/
python3 setup.py develop
Congratulations! You've successfully set up PyFR on the FASTER and Grace cluster computing facilities. You should now be able to use PyFR for your computational fluid dynamics simulations.
ACCESS Guide (originally given at Duke OIT)
0
A guide for Duke OIT on how to advise users on using ACCESS and allocation credits to jetstream 2 for Duke University members. This can be used for non Duke members. Assumes the reader has basic knowledge of ACCESS.
Geocomputation with R (Free Reference Book)
0
Below is a link for a book that focuses on how to use "sf" and "terra" packages for GIS computations. As of 5/1/2023, this book is up to date and examples are error free. The book has a lot of information but provides a good overview and example workflows on how to use these tools.
OnShape FeatureScripts: Custom features for everyone
0
OnShape FeatureScripts allow users to create their own features via OnShape's programming language. The user can make these as simple or complex as they need, and they can save tons of time for heavy OnShape users or complex projects!
CI Computing Module For All
0
Computing Module: Introduces fundamental concepts and skills of Cyberinfrastructure (CI) and High-Performance Computing (HPC) to lower the barrier to becoming CI users in disaster management research. The module will cover the critical topics of CI and HPC with hands-on sessions.
Disaster Data Module: Introduces concepts of geospatial big data in disaster management. Students will learn how to access and process disaster data.
Geospatial Analytic Module: Introduces geospatial analytics skills to address real-world challenges in disaster management. The module will use the data introduced in the Disaster Data Module and cover various geospatial analytics topics such as geosimulation, spatial optimization, network analysis, terrain analysis, Geospatial Artificial Intelligence (GeoAI), social sensing, and CyberGIS.
ACCESS Campus Champion Example Allocation
0
ACCESS requests proposals to be written following NSF proposal guidelines. The link provides an example of an ACCESS proposal using an NSF LaTeX template. The request is at the DISCOVER level appropriate for Campus Champions. The file is 2 pages: the first page details the motivation, approach, and resources requested; and the second page is a 1-page bio.
Feed Forward NNs and Gradient Descent
0
Feed-forward neural networks are a simple type of network that simply rely on data to be "fed-forward" through a series of layers that makes decisions on how to categorize datum. Gradient descent is a type of optimization tool that is often used to train machines. These two areas in ML are good starting points and are the easiest types of neural network/optimization to understand.
Tutorial for OpenMP Building up and Utilization
0
The following link elaborates the usage of OpenMP API and its related syntax. There are also several exercises available for learners to help them get familiar with this widely-used tool for multi-threaded realization.
Cyber Security
0
learning cybersecurity is crucial for personal protection, safeguarding digital assets, financial security, and national security. It is important when it comes to consumer data protection for business, creating long lasting relationships with customers.
NCSA HPC-Moodle
0
Self-paced tutorials on high-end computing topics such as parallel computing, multi-core performance, and performance tools. Some of the tutorials also offer digital badges.
Factor Graphs and the Sum-Product Algorithm
0
A tutorial paper that presents a generic message-passing algorithm, the sum-product algorithm, that operates in a factor graph. Following a single, simple computational rule, the sum-product algorithm computes either exactly or approximately various marginal functions derived from the global function. A wide variety of algorithms developed in artificial intelligence, signal processing, and digital communications can be derived as specific instances of the sum-product algorithm, including the forward/backward algorithm, the Viterbi algorithm, the iterative "turbo" decoding algorithm, Pearl's (1988) belief propagation algorithm for Bayesian networks, the Kalman filter, and certain fast Fourier transform (FFT) algorithms
AI for improved HPC research - Cursor and Termius - Powerpoint
0
These slides provide an introduction on how Termius and Cursor, two new and freemium apps that use AI to perform more efficient work, can be used for faster HPC research.
FSL Lectures
0
This is the official University of Oxford FSL group lecture page. This includes information on upcoming and past courses (online and in-person), as well as lecture materials. Available lecture materials includes slides and recordings on using FSL, MR physics, and applications of imaging data.
OpenStack Tutorial For Beginners
0
OpenStack Tutorial For Beginners
Open OnDemand Documentation Repository
0
This is the main documentation repo for the Open OnDemand Portal which enables researchers to access HPC resources from a familiar web interface.
Advanced Compilers: The Self-Guided Online Course
0
This is a self guided online course on compilers. The topics covered throughout the course include universal compilers topics like intermediate representations, data flow, and “classic” optimizations as well as more research focusedtopics such as parallelization, just-in-time compilation, and garbage collection.
Using Dask on HPC Systems
0
A tutorial on the effective use of Dask on HPC resources. The four-hour tutorial will be split into two sections, with early topics focused on novice Dask users and later topics focused on intermediate usage on HPC and associated best practices. The knowledge areas covered include (but are not limited to):
Beginner section
High-level collections including dask.array and dask.dataframe
Distributed Dask clusters using HPC job schedulers
Earth Science data analysis using Dask with Xarray
Using the Dask dashboard to understand your computation
Intermediate section
Optimizing the number of workers and memory allocation
Choosing appropriate chunk shapes and sizes for Dask collections
Querying resource usage and debugging errors
Machine Learning with sci-kit learn
0
In the realm of Python-based machine learning, Scikit-Learn stands out as one of the most powerful and versatile tools available. This introductory post serves as a gateway to understanding Scikit-Learn through explanations of introductory ML concepts along with implementations examples in Python.
Cybersecurity Guide
0
Cybersecurity Guide is a comprehensive resource for students and early career professionals that provides users with a wide range of resources and up-to-date information on cybersecurity, including cybersecurity degree programs and bootcamps, career guides, as well as online courses and training opportunities. Additionally, it covers trends, best practices, and much more.