The facial appearance of actors can be transformed during live stage performances using a new advanced system developed by a team at Disney Research that can track an actor's movements and changing expressions so that the face can be painted with light, rather than physical makeup.
The projection system can track the actor's facial movements without use of facial markers. By adjusting illumination, the system can display any color or texture that an artist wants to achieve. That could include effects that make the actor appear older, or appear to grow facial hair, or have clownish face paint -- all in real-time during live performances.
"We've seen astounding advances in recent years in capturing facial performances of actors and transferring those expressions to virtual characters," said Markus Gross, vice president at Disney Research. "Leveraging these technologies to augment the appearance of live actors is the next step and could result in amazing transformations before our eyes of stage actors in theaters or other venues."
In contrast to previous work, the Disney team showed it can track facial movements and dynamically adjust the projections to a wide variety of arbitrary movements, rather than project augmentations onto static objects or rigidly constrain facial movements, said Anselm Grundhöfer, principal research engineer at Disney Research
The researchers will present their software and hardware system, called Makeup Lamps, April 24 at the European Association for Computer Graphics conference, Eurographics 2017, in Lyon, France.
"The key challenge of live augmentation is latency -- the time between generating an image that matches the actor's pose and when the image is displayed," Grundhöfer said. "The larger the latency, the more the actor's pose will have changed and the greater the potential misalignment between the augmentation and the face."
In designing Makeup Lamps, the research team aimed to reduce latency at each step of the process -- from capturing the facial pose, to processing to projection, he said. To do so, the team limited the complexity of its algorithms. They also employed a coaxial camera-projection setup, where the camera that detects facial movements shares the same optical axis as the projector that illuminates the face. This enabled the team to process images in two dimensions, rather than three, and still provide consistent augmentation of the face.
Some latency is unavoidable, of course, so the team used a method called Kalman filtering that uses measurements over time to make predictions, enabling slight adjustments that better align the augmentations with the facial performance.
Makeup Lamps is able to simulate different lighting conditions and facial effects, such as wrinkles, that are expression specific while allowing the performance to change over time," Grundhöfer said.
"We believe that projection-based non-rigid expressive augmentation could give rise to a wide variety of creative applications in the near future," he added. For instance, such as system might be used to simulate cosmetic plans on a face before actually applying makeup.
###
In addition to Grundhöfer, the research team included Amit Bermano, a post-doctoral researcher at Disney Research and Princeton University; Markus Billeter of Chalmers University and Daisuke Iwai of Osaka University.
Combining creativity and innovation, this research continues Disney's rich legacy of leveraging technology to enhance the tools and systems of tomorrow.
For more information, including a video, please visit the project web site at https://www.disneyresearch.com/publication/makeup-lamps-live-augmentation-of-human-faces-via-projection/.
About Disney Research
Disney Research is a network of research laboratories supporting The Walt Disney Company. Its purpose is to pursue scientific and technological innovation to advance the company's broad media and entertainment efforts. Vice President Markus Gross manages Disney Research facilities in Los Angeles, Pittsburgh and Zürich, and works closely with the Pixar and ILM research groups in the San Francisco Bay Area. Research topics include computer graphics, animation, video processing, computer vision, robotics, wireless & mobile computing, human computer interaction, displays, behavioral economics, and machine learning.
Website: http://www.disneyresearch.com
Twitter: @DisneyResearch
Facebook: http://www.facebook.com/DisneyResearch