image: Proposed "diminished reality" (DR) system. Left: Input image captured by a web camera showing the current situation. Right: DR output image of the future landscape. A moving object (pedestrian) has been virtually removed in real- time along with the immobile building.? view more
Credit: Osaka University
Osaka, Japan - Researchers at Osaka University demonstrated a prototype for real-time augmented reality that can virtually remove both static structures as well as moving objects. This work can be used to visualize future landscapes after urban renovations without interference from passing cars or pedestrians.
Augmented reality, the ability for simulated images to be projected virtually over pictures of actual locations in real time, has been used for both immersive games like Pokémon Go as well as practical applications such as construction planning. As the "augmented" in the name implies, characters or buildings are usually added to scenes, since virtually removing objects to show what lies behind them is a much more computationally intensive task. However, this is exactly what is needed to plan for demolitions, or to see what a landscape would look like after removing distracting moving vehicles or pedestrians from scenes.
Now, researchers at the Division of Sustainable Energy and Environmental Engineering at Osaka University have developed such a system. Their prototype combines deep learning with a video game graphics engine to create real-time landscapes that can virtually remove buildings, trees, or even moving objects, to show the view of the background previously obscured. "Previously, it has not been possible to visually remove moving objects, such as vehicles and pedestrians, from real-time landscape simulations," corresponding author Tomohiro Fukuda says.
To simulate the removal of large immobile structures, like a building to be demolished, the method dynamically blends photographs of the background behind the building based on the current viewpoint of the camera. For moving objects, like passing cars, a machine-learning algorithm can automatically detect and mask the region they obscure, to be replaced with the appropriate current or future scene. To test the prototype system, the team tried it out on the campus of Osaka University. They found it could automatically detect and remove cars and pedestrians from the image.
"Diminished reality can dynamically remove distractions from our landscapes," Fukuda says.
This research may be used for planning purposes whenever renovations become necessary.
###
The article, "Diminished reality system with real-time object detection using deep learning for onsite landscape simulation during redevelopment," was published in Environmental Modelling and Software at DOI: https://doi.org/10.1016/j.envsoft.2020.104759
About Osaka University
Osaka University was founded in 1931 as one of the seven imperial universities of Japan and now has expanded to one of Japan's leading comprehensive universities. The University has now embarked on open research revolution from a position as Japan's most innovative university and among the most innovative institutions in the world according to Reuters 2015 Top 100 Innovative Universities and the Nature Index Innovation 2017. The university's ability to innovate from the stage of fundamental research through the creation of useful technology with economic impact stems from its broad disciplinary spectrum.
Website: https://resou.osaka-u.ac.jp/en/top
Journal
Environmental Modelling & Software