Walking in a 360° video with foot vibrations for seated observers

Details 

Walking is a basic human activity that improves human physical and mental health. However, physical disabilities and the COVID-19 pandemic have discouraged people from walking outside. To remove these limitations, a research team at Toyohashi University of Technology in collaboration with researchers at the University of Tokyo, has developed a virtual walking system for seated observers with 360° video and scene-congruent foot vibrations. The timing of foot vibrations was automatically calculated from the 360° video. They found that rhythmical scene-congruent vibrations improved the sensation of walking and the perception of ground materials. This system can convert various public and private 360° videos into realistic virtual walking experiences. This study was published in IEEE Access on 20th December, 2021.

Virtual reality (VR) can provide people with walking disabilities or people during the COVID-19 pandemic with realistic walking experiences. However, most virtual walking systems use custom content that is made for a specific system.

Researchers at Toyohashi University of Technology and the University of Tokyo have proposed a virtual walking system that converts 360° videos into virtual walking experiences and provides scene-congruent vibrations on the feet at the appropriate time.

In the experiments, congruent vibration patterns were prepared for four ground scenes. The modulation of walking-related sensations and the perception of ground materials using congruent and incongruent vibrations was verified using psychological measurements. They found that rhythmic foot vibration improved the sensations of self-movement, walking, leg action, and telepresence irrespective of scene–vibration congruency. Moreover, congruent vibrations were better than incongruent vibrations for walking-related sensations and telepresence in indoor corridors and snowy ground scenes. The perception of ground materials was enhanced by scene-congruent vibrations, whereas it was confused by scene-incongruent vibrations.

Development Background:

Junya Nakamura, a graduate student and the first author of the article, said, “I came up with the idea of using existing 360° videos for our virtual walking system, which could provide a variety of travel experiences to people at home. So, I applied visual SLAM (simultaneous localization and mapping) to videos to estimate the motion trajectories of the viewpoint in the movies, and generated appropriately timed foot vibrations.” Professor Michiteru Kitazaki explained, "Another important point of this study is that the congruency of foot vibrations with the ground in scenes improves the perception of ground materials and walking sensations. We can feel soft snowy ground or the hard floor of a corridor. However, we used only four common scenarios and four prepared vibrations in the experiment. We should develop a method for generating congruent vibration patterns from any movie in the future."

Future Outlook:

The developed system can convert various public and private 360° videos into realistic virtual walking experiences and provide virtual travel to various locations around the world. It could also contribute to the improvement of the mental health and well-being of people even people with walking disabilities and people who are unable to walk due to social limitations.

Funding agency:

This research was supported in part by JST ERATO (JPMJER1701) to MK, JSPS KAKENHI JP18H04118 to YI, and JP20H04489 to MK.