Estimation of unmasked face images based on voice and 3DMM

During the COVID-19 pandemic, face masks have become common, posing challenges to facial recognition systems used for security and identification. Current techniques to digitally recreate the covered parts of faces often miss important details and identity traits. To overcome this, Mr. Akatsuka and Prof. Sei and their colleagues have introduced a unique method that uses voice data to assist in revealing the covered parts of faces.

Their significant discovery is the inherent connection between speech-related vocal tract structures and certain facial features. Therefore, vocal characteristics can offer valuable insights for rebuilding facial geometry in a combined voice and image approach.

Their research, using a vast collection of matched voice and face data, demonstrates significant improvements in recreating critical facial areas like the mouth, nose, and overall face shape. Both human evaluations and technical accuracy measurements highlight the effectiveness of including voice data in restoring hidden facial features. Additionally, their approach shows notable advancements over traditional methods that do not use voice data.

This inventive use of voice in facial image reconstruction signals exciting new directions in this developing area of research. This research was awarded the IEEE Computer Society Japan Chapter JAWS Young Researcher Award.

Conceptual Diagram of the Proposed Method: The 3D Face Model (3DMM) is created using a masked face image and voice data. The model then assists in digitally painting in the unmasked face.

During the COVID-19 pandemic, face masks have become common, posing challenges to facial recognition systems used for security and identification. Current techniques to digitally recreate the covered parts of faces often miss important details and identity traits. To overcome this, Mr. Akatsuka and Prof. Sei and their colleagues have introduced a unique method that uses voice data to assist in revealing the covered parts of faces.

Their significant discovery is the inherent connection between speech-related vocal tract structures and certain facial features. Therefore, vocal characteristics can offer valuable insights for rebuilding facial geometry in a combined voice and image approach.

Their research, using a vast collection of matched voice and face data, demonstrates significant improvements in recreating critical facial areas like the mouth, nose, and overall face shape. Both human evaluations and technical accuracy measurements highlight the effectiveness of including voice data in restoring hidden facial features. Additionally, their approach shows notable advancements over traditional methods that do not use voice data.

This inventive use of voice in facial image reconstruction signals exciting new directions in this developing area of research. This research was awarded the IEEE Computer Society Japan Chapter JAWS Young Researcher Award.

Conceptual Diagram of the Proposed Method: The 3D Face Model (3DMM) is created using a masked face image and voice data. The model then assists in digitally painting in the unmasked face.

Authors:

Tetsumaru Akatsuka

-- The University of Electro-Communications, Master student

Ryohei Orihara

-- The University of Electro-Communications, Visiting Professor

Yuichi Sei

-- The University of Electro-Communications, Professor

Yasuyuki Tahara

-- The University of Electro-Communications, Associate Professor

Akihiko Ohsuga

-- The University of Electro-Communications, Professor