image: The 2023 Frontera User Meeting gave researchers an opportunity to share how they are maximizing system performance to support cutting-edge science. Shown is Paul Woodward, University of Minnesota.
Credit: TACC
What will the future bring for U.S. scientists using supercomputers to scale up their computations to the highest level? And what technologies should cyberinfrastructure providers deploy to match their ambitions?
These questions and more were explored at the 3rd annual Frontera User Meeting August 3-4, 2023, held at the Texas Advanced Computing Center (TACC).
"It's a great opportunity to hear about how Frontera is performing and for users to hear from each other about how they’re maximizing the system,” said Dan Stanzione, executive director of TACC and the principal investigator of the National Science Foundation (NSF)-funded Frontera supercomputer.
Frontera is the most powerful supercomputer ever deployed by the NSF, and it’s the fastest U.S. academic system according to the latest (June 2023) Top500 rankings. Frontera serves as the leading capability system in the national cyberinfrastructure intended for large applications that require thousands of compute nodes.
Over the past 12 months, Frontera has provided rock steady service with 99 percent uptime and an average continuous utilization of 95 percent of its cycles. It delivered more than 72 million node hours and completed over one million jobs, with a cumulative completion of more than 5.8 million jobs over its four years of life.
As of September 2023, Frontera has progressed through more than 80 percent of its projected lifespan with new technology coming that will extend its operation through late 2025.
Approximately 30 scientists participated in the 2023 Frontera User Meeting. The event featured 13 invited speakers who shared their recent experiences and findings while utilizing Frontera.
The presentations included projects taking advantage of the many ways that users get allocations on the system. Some focused on smaller "startup" activities for groups beginning the transition to very large-scale computing. Others, such as the Large-Scale Community Partnership allocation, are long-term collaborations with major experimental facilities and require over a million node-hours of computing resources.
Other presentations focused on more extensive initiatives, such as the Leadership Resource Allocations, which received up to five million node-hours of computational support. Additionally, certain awardees, known as Texascale Days recipients, were granted access to Frontera's full capacity, including its impressive 8,000+ nodes.
The presentations encompassed many domains of science ranging from cosmology to hurricanes, earthquakes to the memory center of the human brain, and more. All credited the unprecedented access to computing resources at the scale provided by Frontera as a cornerstone in allowing new understanding and discoveries in cutting-edge research.
Hurricane Storm Surge
Eirik Valseth, a research associate in the Computational Hydraulics Group, Oden Institute of UT Austin, described new work on Frontera to develop compound storm surge models that add river flooding effects with extreme resolution for the Texas coast. His group is also using Frontera to generate five-day hindcast and seven-day forecasts for global ocean storm surge in collaboration with The University of Notre Dame and the U.S. National Oceanic and Atmospheric Administration, in efforts to allow better planning for hurricanes.
Big One Along the San Andreas Fault
Yifeng Cui, the director of the High Performance GeoComputing Laboratory at the San Diego Supercomputer Center (SDSC), described nonlinear earthquake simulations performed by his team on Frontera during TACC’s Texascale Days. The simulations scaled up to 7,680 nodes and ran for 22.5 hours to simulate 83 seconds of shaking during a magnitude 7.8 quake on the southern San Andreas fault. More accurate simulations allow communities to plan better to withstand these large earthquakes, thus saving lives and property.
Child Brain Development
Jessica Church-Lang, an associate professor in the Department of Psychology at UT Austin, is using Frontera to analyze anonymized fMRI image data of brain activity in children to find connections between its various systems including control, visual, motor, auditory, and more. Frontera has helped her to construct 3D brain models from the fMRI images. “It takes about five hours, per child, on Frontera to run the analysis. It used to take three days on older computers. And this is just one step of our processing pipeline."
Brain Bubbles
Frontera is helping scientists probe the mysteries of how the brain forms thoughts in research led by Jose Rizo-Rey, a professor of biophysics at UT Southwestern Medical Center. His research, using all-atom molecular dynamics simulations on Frontera, investigates tiny bubbles called "vesicles" that shuttle neurotransmitters across the gap between neurons, carrying the signal the brain uses to communicate with itself and other parts of the body. “The process of fusion can happen in just a few micro seconds,” Rizo-Rey said. “That's why we hope that we can simulate this with Frontera.”
Memories, Models, and Optimizations
Research Engineer Ivan Raikov, Department of Neurosurgery at Stanford University, presented his progress on developing a large-scale model of the rodent hippocampus, a region of the brain associated with short-term memory and spatial navigation. The project is creating the first-of-its-kind biophysically detailed, full scale model of the hippocampal formation with as close as possible to a 1-to-1 scale representation of every neuron. “We start with a full-scale hippocampal model with one million neurons,” Raikov said. “It takes about six hours to simulate 10 seconds of hippocampal activity on 1,024 nodes of Frontera.”
Turbulent Times
P.K. Yeung, professor of aerospace engineering at Georgia Tech, presented his work using Frontera to study turbulent dispersion, an example of which is the spread of a candle’s smoke or how far disease agents travel through the atmosphere. Yeung’s simulations on Frontera track the motion of systems of more than a billion particles, calculating the trajectory and acceleration of each fluid element passing through a turbulent, high-rotation zone in what is known as Lagrangian intermittency in turbulence.
Star Turnover
Paul Woodward, the director of the Laboratory for Computational Science & Engineering and a professor in the School of Physics and Astronomy, University of Minnesota, performed 3D hydrodynamical simulations on runs of up to 3,510 compute nodes on Frontera of rotating, massive, main sequence stars to study convection in the interior of the star. “Frontera is powerful enough to permit us to run our non-rotating simulation forward in time for about three years, which is an amazing thing to have done,” Woodward said.
Black Hole Cosmology
Simeon Bird, an assistant professor in the Department of Physics & Astronomy, UC Riverside, presented a new suite of cosmological simulations called PRIYA (Sanskrit for ‘beloved’). The PRIYA simulations performed on Frontera are some of the largest cosmological simulations performed, needing over 100,000 core-hours to simulate a system of 3072^3 (about 29 billion) particles in a ‘box’ 120 megaparsecs on edge, or about 3.91 million light years across. “We run multiple models, interpolate them together and compare them to observational data of the real universe such as from the Sloan Digital Sky Survey and the Dark Energy Spectroscopic Instrument,” Bird said.
Space Plasma
Half of all the universe’s matter — composed of protons and neutrons — resides in space as plasma. The solar wind from stars such as our sun shapes clouds of space plasma. And on a much larger scale, cosmic magnetic fields knead space plasma across galaxies. “Some of our recently published work has made use of Frontera to study the turbulent dynamos in conducting plasma, which amplify cosmic magnetic fields and could help answer the question of the origin of magnetic fields in the universe,” said graduate student Michael Zhang, Princeton Program in Plasma Physics, Princeton University.
Tight Junctions
Tight junctions are multiprotein complexes in cells that control the permeability of ions and small molecules between cells, as well as supporting transport of nutrients, ions, and water. Sarah McGuinness, a PhD candidate in biomedical engineering at the University of Illinois, Chicago, presented progress using molecular dynamics simulations on Frontera to research Claudin-15, a protein which polymerizes into strands to form the backbone of tight junctions. "Computational simulations allow investigators to observe protein dynamics and atomic resolution with resources like Frontera,” McGuinness said.
Protein Sequencing
Behzad Mehrafrooz, a PhD student at the Center for Biophysics and Quantitative Biology, University of Illinois at Urbana-Champaign, outlined his group’s latest work extending the reach of nanopores to sequence entire proteins, which are much larger and more complex than DNA. “Thanks to Frontera, it was one of the longest, if not the longest molecular dynamics simulations for nanopore sequencing yet made," Mehrafrooz said. "And it confirmed the rapid, unidirectional translocation induced by guanidinium chloride and helped unravel the molecular mechanism behind it.”
Viral Packaging
Kush Coshic, a PhD student in the Aksimentiev Lab at the University of Illinois at Urbana-Champaign, described simulations that took more than four months to perform using Frontera’s GPU nodes to simulate the genomic packaging of a model herpes-like virus, applicable to developing new therapeutics. “Frontera enables us to perform unprecedented high throughput analysis of a 27 million atom system," Coshic said.
Spectral Function
“We've developed a new algorithm for calculating spectral functions with continuous momentum resolution that complements existing many-body techniques,” said Edwin Huang, an assistant professor in the Department of Physics & Astronomy at Notre Dame University. His team’s determinantal quantum Monte Carlo solver for computing the spectral function of fermionic models with local interactions required sampling over a billion state configurations on Frontera.
Path to Horizon
Planning is underway for a massive new system as part of the NSF-funded Leadership Class Computing Facility (LCCF), with a projected 10X the capabilities of Frontera. Early users of the new system, called Horizon, can expect it to start in the second half of 2025 and enter full production in 2026.
“There are still opportunities to talk about what goes into Horizon,” Stanzione said. “One of the points of this meeting is to continue requirement gathering.”
To unlock the potential of Horizon, the future system will need to provide robust support for both CPU- and GPU-based codes. Software performance directions being explored are mixed precision matrix operations in GPUs, which can offer a 30X advantage in performance over single precision vector units.
“Software enables science and it will drive our decisions about future systems. The most important thing for TACC is that we get to hear from users about what is working with Frontera, what can be improved, and what needs to change to meet their needs in future systems,” Stanzione concluded.