image: Simulations performed on Titan enable detailed tracking of two fluid phase systems during flow through porous media. In this case, the behavior of oil ganglia (yellow) can be observed as they are mobilized in an experimentally-imaged sandstone (the solid part of the sandstone is shown in blue). view more
Credit: James McClure, Virginia Tech
Ask a geoscientist how he or she feels about the phrase, "out of sight, out of mind," and you likely will meet some serious resistance. The scientist likely would explain that learning how different materials interact with one another below the Earth's surface is of the utmost importance.
Indeed, a robust research community of theoreticians, experimentalists, and computational scientists has been collaborating to gain greater insight into how different states, or phases, of matter interact with one another when they are below ground.
"These fluid flows affect society because of our desire to mitigate climate change using carbon sequestration, to extract energy resources from the subsurface, and to remediate the effects of environmental contamination," said James McClure, a computational scientist at Virginia Tech University.
Oil production, subsurface contamination, and carbon sequestration, among other research fields, share a common thread--they deal with multiphase flows, or situations where materials are flowing close together in different phases (solids, liquids, or gases). They also pertain to when the flow is composed of materials that have a common phase with a different chemical makeup that prevents mixing (such as oil and water).
McClure and his research team are using resources at the Oak Ridge Leadership Computing Facility (OLCF), a US Department of Energy (DOE) Office of Science User Facility located at DOE's Oak Ridge National Laboratory, to improve understanding of transport phenomena in multiphase systems.
"To better understand the processes influencing multiphase flow, we rely on the synergistic application of theory, experiment, and computation," McClure said. "We're interested in the fundamental mechanisms that influence multiphase systems and how we can best represent these mechanisms, both mathematically and computationally, within a multiscale framework."
Through the Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program, the team has created detailed models for simulating two-phase flow systems and has produced a roadmap to efficiently simulate increasingly complex multiphase flows.
Between increased computing power and advances in experimental instruments, the team is gaining unprecedented insight into how materials interact in porous media such as soil.
"It is an exciting time to be studying multiphase problems," McClure said. "New data acquisition technologies are transforming our understanding of geological materials. Recent theoretical breakthroughs can resolve problems that have existed for decades, and a massively parallel supercomputer like Titan allows us to simulate highly resolved systems at a scale that would have been far out of reach only a few years ago."
McClure noted that experimental advances such as microcomputed tomography (micro-CT) have allowed researchers to observe the structure of geologic materials with micron-scale resolution. Simulations are essential to connect the experimentally generated data sets to models that describe flow processes at length scales of kilometers, as they are needed to make predictions for a wide range of systems that impact society.
"The most exciting problems across the sciences involve systems in which a process with a distinct length scale affects the behavior of the system when viewed on a much different scale," McClure said. "Such so-called multiscale problems are extraordinarily difficult to solve because one must represent the effects of all smaller scale processes on the largest length scale of concern. Our research team has developed theory to help us better understand this multiscale problem for multiphase systems."
The team's current focus is to use experimental observations to evaluate, validate, and guide the refinement of models, such that the underlying processes are adequately resolved from the micrometer to the kilometer scale.
"That's what we are really interested in doing," McClure said. "We are not interested in one specific geologic system, but rather the basic theoretical approach that can be applied to a wide variety of applications. This sort of model development is essential to answer questions that arise when evaluating the efficacy of carbon sequestration at a given location or assessing the environmental risk resulting from hydraulic fracturing of porous medium systems."
Experimental analyses on the fly
Because the team members tackle a variety of multiphase transport problems, they require a versatile computational framework that can both simulate complex physics accurately and extract information that quantifies what really is happening.
The team uses its LBPM-WIA code, which stands for Lattice Boltzmann Methods for Porous Media with Internal Averaging. Lattice Boltzmann methods solve fluid transport equations, modeling the mass and momentum transport for multiphase flows that take place in complex geometries imaged using micro-CT.
The simulations are essentially four-dimensional, in that they map a geologic region in three spatial dimensions, while adding the temporal dimension of time. The team takes many snapshots during the simulations--roughly every 1,000 time steps. These snapshots capture the simulation's phase at a particular point in time. The team typically requires millions of time steps per simulation to characterize the multiphase behavior within a particular media.
The Lattice Boltzmann method allows the team to maximize its code's performance on the OLCF's Cray XK7 Titan supercomputer. Much of Titan's speed comes from NVIDIA Kepler GPUs, and Lattice Boltzmann calculations can take full advantage of those GPUs.
In fact, the ability to use GPUs has transformed the team's simulations. McClure said that the team does its calculations entirely on Titan's GPUs and uses the CPUs to do in situ (or real-time) data analyses on the machine.
OLCF computational scientist Mark Berrill helped optimize codes so that when running its largest simulations, the team can use Titan's CPUs to analyze the data, then send the relevant data sets to the Rhea data-analysis cluster to visualize the results from simulations as they are generated.
This novel approach has allowed the team to churn through hundreds of terabytes of data without any need for post-processing, providing continuous information about the simulation state.
"Based on our theoretical work, we know what information we need from the simulation," McClure said. "We perform the reduction of fine-scale detail to the desired measures of the system state as an integral part of the simulation and ultimately end up with reduced-sized output files that describe the system state that can be easily moved and archived."
The approach is already yielding results. The team's simulations performed on Titan have been instrumental in clarifying the role played by interfaces--the surface areas where two disparate substances meet in a two-phase system, such as where oil and water meet while flowing through a porous, spongy rock--in determining the behavior of these complex multi-scale systems. Analyzing the simulation state in situ enables unprecedented insight into how phases interact that could not otherwise be measured or observed.
Micro-CT imagery captures a snapshot of a geologic system, leaving experimentalists with a static image of a very dynamic process. Working with the OLCF, McClure has developed simulation tools that allow researchers to take the information from a three-dimensional micro-CT image and put it into motion.
With this approach, the team can study interfacial dynamics and other aspects of the system behavior quantitatively within a multiscale framework. The team can predict rocks' properties--such as relative permeability--from first principles to measure the resistance to flow that controls the movement of fluids at the reservoir scale.
These results are vital for situations where the mobility of trapped, non-wetting phases--in this case, oil or liquid carbon dioxide--is of essential importance, such as carbon sequestration, oil recovery, and contaminant transport. There is much societal benefit to understanding the conditions under which these non-wetting phases can be removed from the subsurface rock or placed to reside permanently in rock.
The team places emphasis on simulating real systems observed with the most advanced experimental methods in existence and on comparing simulations to aspects of the theory that can be resolved and tested independently.
Each year, synchrotron light sources such as DOE's National Synchrotron Light Source at Brookhaven National Laboratory are capable of generating larger and more well-resolved, three-dimensional image data of subsurface flows. Massively parallel computers such as Titan are indispensable to keep pace with these data sources--large computations are essential to fill in details that can't be measured from experiments. Such information includes the fluid pressure and velocity fields, which are essential to advance theory and understanding.
"Using high-performance computation to model state-of-the-art experiments when viewed with the lens of recent theoretical advancements across a range of length scales and ultimately to represent systems with the heterogeneity of real geologic systems are the short- and long-term goals of this project," McClure said.
As supercomputers get faster--such as the OLCF's next-generation system, Summit, set for production runs in 2018--McClure thinks his team's computational approach will readily scale with the computing power. "I'm really excited for Summit," McClure said. "We can already use a significant fraction of Titan for production runs, and we don't have a data analysis bottleneck in our code, which is a key attribute for scalable simulations."
"As computational hardware, and methods and algorithms, improve, we will be able to address increasingly complicated problems at increasingly larger scales with the desired resolution. The end result will be a more mature understanding and higher fidelity representations of flow in real subsurface systems," McClure said.
###