Revolutionizing dynamic facial projection mapping: A leap forward in augmented reality

Augmented reality (AR) has become a hot topic in the entertainment, fashion, and makeup industries. Though a few different technologies exist in these fields, dynamic facial projection mapping (DFPM) is among the most sophisticated and visually stunning ones. Briefly put, DFPM consists of projecting dynamic visuals onto a person’s face in real-time, using advanced facial tracking to ensure projections adapt seamlessly to movements and expressions.

While imagination should ideally be the only thing limiting what’s possible with DFPM in AR, this approach is held back by technical challenges. Projecting visuals onto a moving face implies that the DFPM system can detect the user’s facial features, such as the eyes, nose, and mouth, within less than a millisecond. Even slight delays in processing or minuscule misalignments between the camera’s and projector’s image coordinates can result in projection errors—or “misalignment artifacts”—that viewers can notice, ruining the immersion.

Against this backdrop, a research team from Institute of Science Tokyo, Japan set out to find solutions to existing challenges in DFPM. Led by Associate Professor Yoshihiro Watanabe, also including graduate student Mr. Hao-Lun Peng, the team introduced a series of innovative strategies and techniques and combined them into a state-of-the-art high-speed DFPM system. Their findings were published in IEEE Transactions on Visualization and Computer Graphics on January 17, 2025.

First, the researchers developed a hybrid technique called the “high-speed face tracking method” that combines two different approaches in parallel to detect facial landmarks in real-time. On the one hand, they employed a method called Ensemble of Regression Trees (ERT) to realize fast detection. They also implemented a way to efficiently crop incoming pictures down to the user’s face to detect landmarks faster; they achieved this by leveraging temporal information from previous frames to limit the “search area.” To help ERT-based detection recover from errors or challenging situations, they combined it with a slower auxiliary method, which provides high accuracy at a lower speed.

Using this ingenious strategy, the researchers achieved unprecedented speed in DFPM. “By integrating the results of high-precision but slow detection and low-precision but fast detection techniques in parallel and compensating for temporal discrepancies, we reached a high-speed execution at just 0.107 milliseconds while maintaining high accuracy,” highlights Watanabe.

The team also tackled a pressing problem: the limited availability of video datasets of facial movements for training the models. They created an innovative method to simulate high-frame-rate video annotations using existing still image facial datasets. This allowed their algorithms to properly learn motion information at high frame rates.

Finally, the researchers proposed a lens-shift co-axial projector-camera setup to help minimize alignment artifacts. “The lens-shift mechanism incorporated into the camera’s optical system aligns it with the upward projection of the projector's optical system, leading to more accurate coordinate alignment,” explains Watanabe. In this way, the team achieved high optical alignment with only a 1.274-pixel error for users located between 1 m and 2 m depth.

Overall, the various methods developed in this study will help push the field of DFPM forward, leading to more compelling and hyper-realistic effects that will transform performances, fashion shows, and artistic presentations. To gain enhanced comprehension of this research, click here.

## 

About Institute of Science Tokyo (Science Tokyo)

Institute of Science Tokyo (Science Tokyo) was established on October 1, 2024, following the merger between Tokyo Medical and Dental University (TMDU) and Tokyo Institute of Technology (Tokyo Tech), with the mission of “Advancing science and human wellbeing to create value for and with society.”