Now in 3D: Deep learning techniques help visualize X-ray data in three dimensions

Computers have been able to quickly process 2D images for some time. Your cell phone can snap digital photographs and manipulate them in a number of ways. Much more difficult, however, is processing an image in three dimensions, and doing it in a timely manner. The mathematics are more complex, and crunching those numbers, even on a supercomputer, takes time.

That’s the challenge a group of scientists from the U.S. Department of Energy’s (DOE) Argonne National Laboratory is working to overcome. Artificial intelligence has emerged as a versatile solution to the issues posed by big data processing. For scientists who use the Advanced Photon Source (APS), a DOE Office of Science User Facility at Argonne, to process 3D images, it may be the key to turning X-ray data into visible, understandable shapes at a much faster rate. A breakthrough in this area could have implications for astronomy, electron microscopy and other areas of science dependent on large amounts of 3D data.

“In order to make full use of what the upgraded APS will be capable of, we have to reinvent data analytics. Our current methods are not enough to keep up. Machine learning can make full use and go beyond what is currently possible.” — Mathew Cherukara, Argonne National Laboratory

The research team, which includes scientists from three Argonne divisions, has developed a new computational framework called 3D-CDI-NN, and has shown that it can create 3D visualizations from data collected at the APS hundreds of times faster than traditional methods can. The team’s research was published in Applied Physics Reviews, a publication of the American Institute of Physics.

CDI stands for coherent diffraction imaging, an X-ray technique that involves bouncing ultra-bright X-ray beams off of samples. Those beams of light will then be collected by detectors as data, and it takes some computational effort to turn that data into images. Part of the challenge, explains Mathew Cherukara, leader of the Computational X-ray Science group in Argonne’s X-ray Science Division (XSD), is that the detectors only capture some of the information from the beams.

But there is important information contained in the missing data, and scientists rely on computers to fill in that information. As Cherukara notes, while this takes some time to do in 2D, it takes even longer to do with 3D images. The solution, then, is to train an artificial intelligence to recognize objects and the microscopic changes they undergo directly from the raw data, without having to fill in the missing info.

To do this, the team started with simulated X-ray data to train the neural network. The NN in the framework’s title, a neural network is a series of algorithms that can teach a computer to predict outcomes based on data it receives. Henry Chan, the lead author on the paper and a postdoctoral researcher in the Center for Nanoscale Materials (CNM), a DOE Office of Science User Facility at Argonne, led this part of the work.

“We used computer simulations to create crystals of different shapes and sizes, and we converted them into images and diffraction patterns for the neural network to learn,” Chan said. “The ease of quickly generating many realistic crystals for training is the benefit of simulations.”

This work was done using the graphics processing unit resources at Argonne’s Joint Laboratory for System Evaluation, which deploys leading-edge testbeds to enable research on emerging high-performance computing platforms and capabilities.

Once the network is trained, says Stephan Hruszkewycz, physicist and group leader with Argonne’s Materials Science Division, it can come pretty close to the right answer, pretty quickly. However, there is still room for refinement, so the 3D-CDI-NN framework includes a process to get the network the rest of the way there. Hruszkewycz, along with Northwestern University graduate student Saugat Kandel, worked on this aspect of the project, which reduces the need for time-consuming iterative steps.

“The Materials Science Division cares about coherent diffraction because you can see materials at few-nanometer length scales — about 100,000 times smaller than the width of a human hair — with X-rays that penetrate into environments,” Hruszkewycz said. “This paper is a demonstration of these advanced methods, and it greatly facilitates the imaging process. We want to know what a material is, and how it changes over time, and this will help us make better pictures of it as we make measurements.”

As a final step, 3D-CDI-NN’s ability to fill in missing information and come up with a 3D visualization was tested on real X-ray data of tiny particles of gold, collected at beamline 34-ID-C at the APS. The result is a computational method that is hundreds of times faster on simulated data, and nearly that fast on real APS data. The tests also showed that the network can reconstruct images with less data than is usually required to compensate for the information not captured by the detectors.

The next step for this research, according to Chan, is to integrate the network into the APS’s workflow, so that it learns from data as it is taken. If the network learns from data at the beamline, he said, it will continuously improve.

For this team, there’s a time element to this research as well. As Cherukara points out, a massive upgrade of the APS is in the works, and the amount of data generated now will increase exponentially once the project is complete. The upgraded APS will generate X-ray beams that are up to 500 times brighter, and the coherence of the beam — the characteristic of light that allows it to diffract in a way that encodes more information about the sample — will be greatly increased.

That means that while it takes two to three minutes now to gather coherent diffraction imaging data from a sample and get an image, the data collection part of that process will soon be up to 500 times faster. The process of converting that data to a usable image also needs to be hundreds of times faster than it is now to keep up.

“In order to make full use of what the upgraded APS will be capable of, we have to reinvent data analytics,” Cherukara said. “Our current methods are not enough to keep up. Machine learning can make full use and go beyond what is currently possible.”

In addition to Chan, Cherukara and Hruszkewycz, authors on the paper include Subramanian Sankaranarayanan and Ross Harder, both of Argonne; Youssef Nashed of SLAC National Accelerator Laboratory; and Saugat Kandel of Northwestern University.

About Argonne’s Center for Nanoscale Materials
The Center for Nanoscale Materials is one of the five DOE Nanoscale Science Research Centers, premier national user facilities for interdisciplinary research at the nanoscale supported by the DOE Office of Science. Together the NSRCs comprise a suite of complementary facilities that provide researchers with state-of-the-art capabilities to fabricate, process, characterize and model nanoscale materials, and constitute the largest infrastructure investment of the National Nanotechnology Initiative. The NSRCs are located at DOE’s Argonne, Brookhaven, Lawrence Berkeley, Oak Ridge, Sandia and Los Alamos National Laboratories. For more information about the DOE NSRCs, please visit https://science.osti.gov/User-Facilities/User-Facilities-at-a-Glance.

About the Advanced Photon Source

The U. S. Department of Energy Office of Science’s Advanced Photon Source (APS) at Argonne National Laboratory is one of the world’s most productive X-ray light source facilities. The APS provides high-brightness X-ray beams to a diverse community of researchers in materials science, chemistry, condensed matter physics, the life and environmental sciences, and applied research. These X-rays are ideally suited for explorations of materials and biological structures; elemental distribution; chemical, magnetic, electronic states; and a wide range of technologically important engineering systems from batteries to fuel injector sprays, all of which are the foundations of our nation’s economic, technological, and physical well-being. Each year, more than 5,000 researchers use the APS to produce over 2,000 publications detailing impactful discoveries, and solve more vital biological protein structures than users of any other X-ray light source research facility. APS scientists and engineers innovate technology that is at the heart of advancing accelerator and light-source operations. This includes the insertion devices that produce extreme-brightness X-rays prized by researchers, lenses that focus the X-rays down to a few nanometers, instrumentation that maximizes the way the X-rays interact with samples being studied, and software that gathers and manages the massive quantity of data resulting from discovery research at the APS.

This research used resources of the Advanced Photon Source, a U.S. DOE Office of Science User Facility operated for the DOE Office of Science by Argonne National Laboratory under Contract No. DE-AC02-06CH11357.

Argonne National Laboratory seeks solutions to pressing national problems in science and technology. The nation’s first national laboratory, Argonne conducts leading-edge basic and applied scientific research in virtually every scientific discipline. Argonne researchers work closely with researchers from hundreds of companies, universities, and federal, state and municipal agencies to help them solve their specific problems, advance America’s scientific leadership and prepare the nation for a better future. With employees from more than 60 nations, Argonne is managed by UChicago Argonne, LLC for the U.S. Department of Energy’s Office of Science.

The U.S. Department of Energy’s Office of Science is the single largest supporter of basic research in the physical sciences in the United States and is working to address some of the most pressing challenges of our time. For more information, visit https://energy.gov/science.