"X-ray vision," once the province of comic book superheroes, is coming to a mere mortal near you: A budding technology called augmented reality uses computers and cameras to mesh views from different perspectives, giving the wearer of augmented reality goggles something like superhuman vision. Now, a group of computer scientists at the University of Rochester -- one of only a handful of groups working on augmented reality -- has proposed a new way for cameras and computers to cooperate to produce an image that very convincingly merges computer-generated images and real-world videos.
This ability to seamlessly combine a visible object, like a wall, with a whole database of images you can't see, such as an infrared view or a blueprint of the wall's interior, has many applications in such fields as medicine, entertainment, maintenance, and defense. For example, instead of having to glance back and forth between a patient and an MRI view of his brain tumor, surgeons at Brigham and Women's Hospital in Boston can see on a monitor the MRI view superimposed over a video image of the patient. The next step may be augmented reality goggles -- eyewear that superimposes the MRI view directly over the naked eye's view of the patient so that surgeons can even more precisely eradicate only the tumor.
"It sounds futuristic, but this technology's a lot closer than you think," says James Vallino, a Rochester graduate student who has been working on augmented reality with Assistant Professor Kiriakos Kutulakos. "It's been evolving slowly but steadily, led by military, design, and medical projects, some of which already use augmented reality."
A version of augmented reality already makes it possible for weather forecasters to appear on the air in front of computerized maps. There are many other high-tech applications being investigated by the Rochester researchers and others at institutions such as MIT, Carnegie Mellon University, and UNC- Chapel Hill:
- A soldier on the ground looking through augmented reality goggles could see infrared images of the landscape gleaned from a reconnaissance aircraft far above. A group of enemy troops just beyond the top of a ridge can't be seen by the soldier but can be detected by the aircraft -- which can then alert the soldier by superimposing its find onto the soldier's more limited view.
- Doctors, who must often glance back and forth from patient to medical imaging devices while performing surgery, are benefiting from being able to see everything they need to know at once. For example, augmented reality goggles superimpose an ultrasound image of a developing fetus on an external view of its mother's body, giving doctors an unprecedented view of babies as they develop in the womb. Physicians may even don augmented reality goggles to mix and match medical images from endoscopy, ultrasound, MRI scans, CT scans, and x-rays -- giving them the best views for carrying out treatment.
- Repairs to buildings or complex machinery can be made much simpler by augmented reality: If a wall's innards are stored as a computerized blueprint, that blueprint can be superimposed on a repairman's view of the wall to pinpoint where hidden ducts, beams, and wiring lie. Augmented reality could help homeowners envision home improvements before they're even started. A remodeling consultant discussing a new deck or addition with a homeowner would aim a video camera at the house and have the augmented reality system show what the home would look like when the work is completed.
The biggest problem faced by the few labs now studying augmented reality is how to flawlessly coordinate the video and computer graphic images in an augmented reality sequence. "Most people want to align the video image and the computer-generated image by building a super-accurate computer model of the three- dimensional setting in which augmentation takes place," says Vallino. "If you're working in a small room or an uncomplicated space, this is easy to do. But what if you're designing goggles for a soldier on a battlefield? Things are always changing, and it's impossible to model that setting."
"In previous approaches the environment had to be tightly controlled for augmented reality to work," Kutulakos adds. "We're now trying to move outside of the lab into less structured environments."
The Rochester group takes a unique approach to this problem of camera-computer coordination. Its approach uses what's called an affine set of axes to place computer graphics into videos using visible natural landmarks as reference points. "You could get the computer to precisely place a three-dimensional computer- generated character on top of a train," Vallino says. "When the train moves, the resulting motion of landmarks around the train is detected to keep the character correctly positioned in the image despite the motion. Augmented reality through such tracking is much easier to manage than the laborious frame-by-frame manipulation now used in Hollywood to make scenes like computerized Martians wandering through the White House."
Vallino and Kutulakos are joined in the augmented reality research by Chris Brown, a professor of computer science. The research is funded by the U.S. Department of Defense's Defense Advanced Research Projects Agency (DARPA) and the National Science Foundation.
EDITORS: For more information on augmented reality, including images, see Vallino's Web page at http://www.cs.rochester.edu /u/vallino/.