image: Objects -- including books, a stuffed animal and a disco ball -- in and around a bookshelf tested the system's versatility in capturing light from different surfaces in a large-scale scene. view more
Credit: David Lindell
David Lindell, a graduate student in electrical engineering at Stanford University, donned a high visibility tracksuit and got to work, stretching, pacing and hopping across an empty room. Through a camera aimed away from Lindell - at what appeared to be a blank wall - his colleagues could watch his every move.
That's because, hidden to the naked eye, he was being scanned by a high powered laser and the single particles of light he reflected onto the walls around him were captured and reconstructed by the camera's advanced sensors and processing algorithm.
"People talk about building a camera that can see as well as humans for applications such as autonomous cars and robots, but we want to build systems that go well beyond that," said Gordon Wetzstein, an assistant professor of electrical engineering at Stanford. "We want to see things in 3D, around corners and beyond the visible light spectrum."
The camera system Lindell tested, which the researchers are presenting at the SIGGRAPH 2019 conference Aug. 1, builds upon previous around-the-corner cameras this team developed. It's able to capture more light from a greater variety of surfaces, see wider and farther away and is fast enough to monitor out-of-sight movement - such as Lindell's calisthenics - for the first time. Someday, the researchers hope superhuman vision systems could help autonomous cars and robots operate even more safely than they would with human guidance.
Practicality and seismology
Keeping their system practical is a high priority for these researchers. The hardware they chose, the scanning and image processing speeds, and the style of imaging are already common in autonomous car vision systems. Previous systems for viewing scenes outside a camera's line of sight relied on objects that either reflect light evenly or strongly. But real-world objects, including shiny cars, fall outside these categories, so this system can handle light bouncing off a range of surfaces, including disco balls, books and intricately textured statues.
Central to their advance was a laser 10,000 times more powerful than what they were using a year ago. The laser scans a wall opposite the scene of interest and that light bounces off the wall, hits the objects in the scene, bounces back to the wall and to the camera sensors. By the time the laser light reaches the camera only specks remain, but the sensor captures every one, sending it along to a highly efficient algorithm, also developed by this team, that untangles these echoes of light to decipher the hidden tableau.
"When you're watching the laser scanning it out, you don't see anything," described Lindell. "With this hardware, we can basically slow down time and reveal these tracks of light. It almost looks like magic."
The system can scan at four frames per second. It can reconstruct a scene at speeds of 60 frames per second on a computer with a graphics processing unit, which enhances graphics processing capabilities.
To advance their algorithm, the team looked to other fields for inspiration. The researchers were particularly drawn to seismic imaging systems - which bounce sound waves off underground layers of Earth to learn what's beneath the surface - and reconfigured their algorithm to likewise interpret bouncing light as waves emanating from the hidden objects. The result was the same high-speed and low memory usage with improvements in their abilities to see large scenes containing various materials.
"There are many ideas being used in other spaces - seismology, imaging with satellites, synthetic aperture radar - that are applicable to looking around corners," said Matthew O'Toole, an assistant professor at Carnegie Mellon University who was previously a postdoctoral fellow in Wetzstein's lab. "We're trying to take a little bit from these fields and we'll hopefully be able to give something back to them at some point."
Humble steps
Being able to see real-time movement from otherwise invisible light bounced around a corner was a thrilling moment for this team but a practical system for autonomous cars or robots will require further enhancements.
"It's very humble steps. The movement still looks low-resolution and it's not super-fast but compared to the state-of-the-art last year it is a significant improvement," said Wetzstein. "We were blown away the first time we saw these results because we've captured data that nobody's seen before."
The team hopes to move toward testing their system on autonomous research cars, while looking into other possible applications, such as medical imaging that can see through tissues. Among other improvements to speed and resolution, they'll also work at making their system even more versatile to address challenging visual conditions that drivers encounter, such as fog, rain, sandstorms and snow.
###
To read all stories about Stanford science, subscribe to the biweekly Stanford Science Digest.
Wetzstein is also a member of Stanford Bio-X and the Wu Tsai Neurosciences Institute.
This work was funded by a Stanford Graduate Fellowship, the National Science Foundation, a Terman Faculty Fellowship, a Sloan Fellowship, the King Abdullah University of Science and Technology, the Center for Automotive Research at Stanford, the Defense Advanced Research Projects Agency, the U.S. Army Research Laboratory.