Tools in Practice: Researcher Stories

The CryoEM facility (LBSD-CryoEM) at Texas A&M University supports a wide range of researchers in structural biology, providing critical resources for studies that rely on determining protein structures at near-atomic resolution. This work enables scientists to understand how proteins and protein complexes interact with binding partners or drug-like molecules, and to explore their potential as therapeutic targets.

Data collection happens on a Titan Krios G4 equipped with a Gatan K3 camera and BioContinuum Imaging Filter, generating terabytes of data per day. Managing that volume was one of the central challenges. Their local storage infrastructure couldn't accommodate the sustained data rates, and initial attempts at manual transfers created significant bottlenecks that delayed processing by days. The team transferred data to ACES using Globus, which allowed them to move large datasets reliably without tying up local resources and enabled automated transfers that could run overnight. Microscope data is written to a local Globus endpoint and transferred in near real time using parallel transfers and checkpointing for speed and reliability. A consistent directory structure supports downstream tasks, and once the data arrive, CryoSPARC Live is started for automatic data processing, minimizing delay between collection and analysis.

Processing is handled through CryoSPARC and RELION, accessed via the Open OnDemand portal on ACES. They structured jobs to take advantage of ACES' GPU nodes, running motion correction and CTF estimation interactively during data collection. The iterative nature of particle picking and classification meant they could rapidly test parameters through Open OnDemand's interactive sessions, then submit optimized workflows as batch jobs. The team uses CryoSPARC Live for automatic job scheduling and sbatch for RELION job scheduling. This hybrid approach reduced their time-to-first-results from days to hours, critical when working with time-sensitive samples. The ability to run computationally intensive jobs at scale was critical given the dataset sizes involved in resolving these channel structures.

For cryo-ET, they analyze these complexes in their native cellular context using IMOD for tomogram reconstruction and PEET/emClarity for subtomogram averaging. The ACES cluster's large-memory nodes proved essential for handling the memory-intensive 3D reconstruction algorithms required for tomographic data. The team developed custom scripts to integrate single-particle and tomographic workflows, providing structural insight that complements the single-particle work and informs their understanding of how these channels behave in situ. A typical cryoEM project using CryoSPARC Live requires 5–10 TB of storage, 128–256 GB of system memory, and three GPUs with 24–48 GB of VRAM each.

The combination of Globus data management, Open OnDemand accessibility, and ACES' specialized hardware enabled the research group to process datasets that would have been computationally prohibitive on traditional university clusters, ultimately allowing them to resolve ion channel structures to resolutions sufficient for drug design — work that directly informs their collaborative efforts with pharmaceutical partners developing new therapeutics for neurological disorders.

Training AI models to reason through math and code problems is deceptively hard. Standard reinforcement learning methods teach a model by rewarding correct answers, but they only optimize for "what's the chance I get this right on the first try?" That turns out to be a first-order approximation of a deeper, more principled objective: maximum likelihood.

Fahim Tajwar, Andrea Zanette, and collaborators at Carnegie Mellon University identified a surprisingly simple fix. Their framework, Maximum Likelihood Reinforcement Learning (MaxRL), shows that normalizing the learning signal by the number of successful attempts rather than the total number of attempts causes the training objective to converge to exact maximum likelihood as more compute is applied. The practical change is a single line in the advantage calculation.

The mathematical basis comes from a Maclaurin expansion of the log-likelihood objective. Standard RL (and GRPO) optimizes only pass@1. MaxRL optimizes a harmonic mixture of successively higher pass@k gradients, meaning harder problems where correct solutions are rare receive stronger learning signals rather than being effectively ignored.

The experiments ran across two ACCESS GPU resources: NCSA Delta and NCSA DeltaAI. The paper is structured around four compute tiers, designed to both validate MaxRL progressively and make the results accessible to research groups with different amounts of compute available:

• ImageNet (image classification): Runnable on a single GPU. A single experiment takes 10–15 hours depending on GPU type (A100 vs. H200/GH200).
• Maze navigation: Requires autoregressive generation from a transformer, making it significantly more compute-intensive. One full node of 4×GH200 for 48–72 hours.
• GSM8K math reasoning (SmolLM2-360M): Requires 2 nodes of 4×GH200, or 1 node of 8×H200 on Delta, for up to 96 hours (two 48-hour jobs run in succession).
• Large-scale LLM training (Qwen3-1.7B and 4B on POLARIS-53K): Requires 2–4 nodes of 4×GH200 for approximately 1–1.5 weeks per full training run.

Scaling across two clusters brought practical challenges, and the Delta and DeltaAI support staff were immensely helpful throughout. The team used Open OnDemand for job submission, monitoring, and interactive debugging sessions on both systems, particularly useful when iterating on environment setup. The most significant technical issue was package compatibility: while building Flash-Attention and vLLM is straightforward on Delta, DeltaAI's aarch64 (ARM) CPUs often lack pre-built package wheels. The team worked around this by first compiling from source, then later using pre-compiled aarch64 wheels available as open-source modules (flash-attention-prebuild-wheels). Multi-node communication and checkpoint management were also non-trivial, since MaxRL's longer training runs often exceed a single job allocation. Globus handled data and checkpoint transfers between Delta and DeltaAI reliably across the two systems. The team's solutions, including proper use of the ray Python package for resource management and automatic checkpoint resumption, are open-sourced in the MaxRL code repository.

The team used XDMoD throughout the project to track compute usage across both clusters, monitoring CPU efficiency and idle cores in real time, complementing the job-level visibility they got through Open OnDemand. This proved directly useful: XDMoD revealed that vLLM, which is optimized for large models (1B+ parameters), introduced significant inefficiencies when applied to the small 3M-parameter Maze models. That insight led the team to build a custom inference system for the Maze experiments, recovering wasted compute hours that would otherwise have gone undetected.

MaxRL consistently matched or exceeded GRPO on pass@1 accuracy while dramatically improving pass@k, meaning the trained models find a correct answer far more reliably when given multiple attempts. Achieving the same coverage as MaxRL required GRPO to use 7.9× to 19.2× more samples at test time. MaxRL also showed less overfitting to the training distribution and maintained more solvable problems throughout training, enabling continued learning in later epochs.

For researchers looking to adopt MaxRL, an Explore or Discover tier allocation is a reasonable starting point. The team's recommended approach: find a smaller, representative setting that runs on a single node (8×H200 or 4×GH200 at most), validate the approach there, then scale up incrementally, the same ladder the paper itself follows.

Wong's team used large-scale ab initio molecular dynamics simulations to observe PFAS degradation in real time at the atomic scale. The research required tracking molecular behavior under applied electrical fields—a computationally intensive process that demanded quantum-mechanical calculations for thousands of electrons simultaneously.

The project utilized a strategic multi-cluster approach, beginning on SDSC's Expanse for initial development and on PSC's Bridges-2 for production simulations. The team used Open OnDemand to manage and monitor these complex, long-running VASP simulations across both systems. Open OnDemand was used to coordinate the workflow between Expanse and Bridges-2, enabling Wong’s team to explore different initial conditions for the calculations. Globus data management was used to ensure consistent data/settings among the quantum calculations.

The computational pipeline primarily utilized VASP for ab initio molecular dynamics calculations, supplemented by proprietary quantum chemistry software. The team found that simpler computational approaches that did not account for charged surfaces failed to show any bond breaking, validating the necessity of their full-scale electrochemical simulations. Specifically, a constant-electrode potential approach was used to introduce varying numbers of electrons into the entire chemical system to simulate the applied potential.

The simulations revealed that PFAS degradation occurs only when metal surfaces reach sufficiently negative electrical potentials, with electrons flowing into PFAS molecules, weakening carbon-fluorine bonds until they break. This provided direct evidence for stepwise degradation mechanisms that laboratory experiments had suggested but never directly observed.
 

The Agricultural Challenge

Modern agriculture increasingly depends on precise, data-driven decisions across large and variable farmlands. Rather than relying only on whole-field treatments, the ICICLE project demonstrates how AI-enabled analysis of drone and sensor data can support more targeted agricultural management.

Modern agriculture faces the challenge of making precise, data-driven decisions across vast farmlands. Traditional approaches often apply uniform treatments—pesticides, irrigation, fertilizers—across entire fields, wasting resources when only specific areas need attention. The ICICLE project demonstrates how AI-driven analysis of drone imagery can enable precision agriculture through accessible computational workflows.

Data Collection and Field Operations

The ICICLE team's agricultural drones capture thousands of high-resolution images during single flights, generating multi-gigabyte datasets per mission. These images, combined with sensor data measuring temperature differences that indicate crop stress and disease, provide comprehensive field monitoring capabilities. 

Data Transfer and Management

Raw imagery and sensor data are transferred from field collection systems to Ohio Supercomputer Center (OSC) resources using Globus. This automated data transfer system handles the large datasets reliably without requiring manual intervention, enabling continuous data flow from multiple drone flights across different agricultural sites.

Computational Access and Workflow Management

The computational analysis happens through ACCESS OnDemand, OSC's web-based portal that provides intuitive access to high-performance computing resources without requiring command-line expertise. Agricultural researchers can launch interactive sessions to examine drone imagery, submit batch jobs for large-scale AI model training, and monitor processing progress through a standard web browser. This accessibility proves crucial for interdisciplinary teams that include agricultural specialists without traditional HPC backgrounds.

AI-Driven Analysis Pipeline

The processing leverages OSC's computational resources to run AI algorithms that create infrared heat maps from drone imagery, enabling identification of crop stress, disease detection, and growth stage classification. The team uses performance benchmarking approaches to optimize data processing pipelines for the high-volume image analysis required for real-time farm management decisions. 

Real-Time Decision Support

The combination of Globus for seamless data movement and ACCESS OnDemand for accessible computing enables farmers and crop consultants to receive timely insights about soil moisture, crop disease, and growth patterns. This infrastructure transforms what would traditionally require weeks of manual analysis into rapid, data-driven decision making that can adapt to field conditions in real time.

Scalable Applications

The workflow extends beyond individual farm applications—the same cyberinfrastructure approach supports crop consultants serving multiple farms and can be adapted for environmental monitoring and urban planning applications. By making AI accessible through user-friendly interfaces and robust data management, ICICLE demonstrates how modern computing infrastructure can bridge the gap between advanced computational methods and practical agricultural needs. 

Traditional antiviral discovery requires physically synthesizing and testing thousands of compounds — a costly, time-intensive process. With monkeypox outbreaks highlighting the ongoing threat of orthopoxviruses, and Tecovirimat as the only FDA-approved treatment, this project used a two-stage computational screening approach to rapidly identify promising natural compound candidates from the COCONUT database, which contained approximately 600,000 molecules.

Stage 1: Virtual Screening Each molecule was docked to the VP37 protein using AutoDock Vina five times to calculate average binding affinity. The entire workflow — from file conversions through docking — was managed using Pegasus Workflow Management System on the OSG Open Science Pool. Setting up the pipeline took about two months, including writing the Pegasus scripts from scratch. The Pegasus support team was a significant help throughout the process, offering to write scripts to assist with data processing. Trang chose to study their initial work and build the rest independently — a decision that deepened his understanding of the workflow. Others running a similar pipeline could likely move faster now that the approach is established.

Stage 2: Molecular Dynamics Simulations The top 50 candidates advanced to 100 nanosecond molecular dynamics simulations using GROMACS, run on NCSA Delta GPU resources. These simulations assessed binding stability under physiological conditions — providing a more detailed picture of complex stability than docking scores alone. MD simulations are still ongoing.

Of the 600,000 screened compounds, Tecovirimat ranked only in the 95th percentile, meaning thousands of natural compounds showed stronger predicted binding. Top candidate CNP0000011.0 demonstrated a binding affinity of −10.2 kcal/mol compared to Tecovirimat's −8.9 kcal/mol. The next phase involves laboratory testing of the most promising computational candidates.

In this project, the team developed a novel computational fluid-structure interaction (FSI) framework to simulate the dynamics of two immiscible fluids moving through a soft capillary tube. The core challenge was coupling complex multi-physics phenomena: two-component fluid flow with topological changes, nonlinear solid deformation, surface tension effects, and dynamic wetting — a combination that had not previously been modeled together.

The team used a phase-field model for the fluids and a nonlinear large-deformation model for the solid tube, with both spatially discretized using Isogeometric Analysis (IGA). For the FSI simulations, the team used customized codes built using the open-source HPC libraries PETSc and PetIGA. The team installed their customized codes with support from the PSC team. Most importantly, because the relevant PETSc/PetIGA modules were not available on Bridges-2, the team had to ensure compatibility between the GCC, OpenMPI compilers and PETSc/PetIGA codes during compilation and installation.
The computations were too expensive to be run on the team’s local resources because solving the FSI problem required high memory, the need for parallel computing resources, and the need for high spatial and temporal accuracies. So, the team carried out simulations on Bridges-2 at the Pittsburgh Supercomputing Center via ACCESS. Each simulation used 128 CPU cores and typically ran for 12-15 hours. In total, the team performed about 200 simulation runs for code validation against experiments, mesh refinement studies, parametric studies and code optimizations.The team also used OnDemand to run and monitor the jobs, and to visualize the simulation results.

A key finding from this project was that tube's deformability delays or suppresses bubble pinch-off compared to rigid tubes — with different mechanisms at play in wetting versus non-wetting tubes. At the start of this project, the team hypothesized that non-wetting soft tubes would accelerate bubble pinch-off, but their simulations revealed two competing mechanisms in non-wetting tubes that eventually caused a delay (or suppression) in bubble pinch-off. The team's findings are important in many applications, including soft robotics, enhanced oil recovery, inkjet printing, bio-microfluidics, and manufacturing, where the deformability and wettability of the confining system can be tuned to control the size, shape, and timing of the pinched-off bubble.

When running challenging multiphysics simulations, the team advises future researchers to first perform preliminary scaling tests to identify the optimal number of CPU cores. Small test cases can be run on the RM-shared partition (maximum of 64 cores per node), while larger test cases can be run on RM nodes with 128 cores per node. For memory-intensive simulations, the team recommends the researchers to use the RM-512 partition with 512 GB RAM per node or request Extreme Memory nodes with up to 4 TB RAM per node. Linear solvers, nonlinear solvers and solver preconditioners should be carefully selected, as they can affect runtime. For restarting the simulations at a later time or simulations that exceed wall-time limits, researchers should use checkpoint/restart files. For deadline-driven jobs, the team advises researchers to reserve nodes ahead of time.