Model for distinguishing materials like a human being does

Overview

A research team including Hideki Tamura, an assistant professor in the Cognitive Neurotechnology Unit of Toyohashi University of Technology's Department of Computer Science and Engineering and Justus Liebig of University Giessen’s Faculty of Psychology has proposed a model that makes it possible to distinguish materials using computer imaging based on judgment criteria similar to the criteria used by humans. This research used two different types of material to be distinguished: a reflective material that reflects its surroundings like a mirror or metal surface, and a transparent material that allows the view of its surroundings to pass through it like glass or ice. The research suggested that humans use the existence of imagery clues when distinguishing between these different materials. The results of the research may have applications in imaging technologies for the accurate depiction of textures at low cost.

Details

Humans have the ability to sense texture to estimate the conditions of a surface or the quality of a material. This enables us to sense rich texture from, for example, the beautiful glitter of a precious metal or the colored, translucent line of light from a jewel. From ancient times to today, humans have always pursued good textures and valued the complicated reflection off the surface of materials and ability to pass through other materials. Against this backdrop, an understanding of the human brain's processing of texture sensing has actively progressed in many different areas of study such as engineering, psychology and neuroscience.

A reflective material like a mirror or polished metal has specular reflections on its surface. Transparent materials like glass or ice allow light to pass through and light refracts inside them. For these two types of material, the image visible on the surface of the material may change in significant and complicated ways depending on the material's surroundings. Regarding this phenomenon, it can be expected that innumerable situations will occur, and little is known about how humans distinguish between the two different types of material.

This led the research team to conduct psychophysical tests to discover how accurately humans distinguish between reflective and transparent materials. The team also verified how accurate convolutional neural networks (CNNs) could be when making the same distinction. The test showed that a human being could distinguish between reflective and transparent materials at 78% accuracy while the CNN could make the same distinction at 94% accuracy, which is considerably more accurate than humans.

According to the above results, the CNN leaves nothing to be desired in terms of the accuracy of distinction, and there are potential industrial applications, where the CNN could be an alternative to a human. However, the question we really want to solve is, “How do humans distinguish between the two different types of material?” The research team concluded that it would be hard to identify the image clues that humans use from the structure or behavior of a model that surpasses humans.

So the team adjusted the CNN so that it would make mistakes like a human and give the same correct answers that a human would. Based on the structure of the model and its similarities with humans, the team sought to verify the things that were being used as clues. As a result, it was suggested that a CNN with a three-layer convolutional structure, which is relatively shallow, could be closest to human and that the model might get clues from changes in the image that appear on the surface of the material. These findings support the insights into human texture sensing reported in previous studies.

This research became the first to successfully structure a model that enables image computing and distinguishes between reflective and transparent materials while imitating a human being’s correct and incorrect answers. Applying this model may make it possible to distinguish between materials and reproduce textures based on summarized data without having to use all of the data in an image. In other words, we can expect that there will be applications for this model in technologies achieving the highly accurate reproduction of textures at low cost.

Reference

Tamura, H., Prokott, K. E., & Fleming, R. W. (2022). Distinguishing mirror from glass: A “big data” approach to material perception. Journal of Vision, 22(4):4, 1-22. https://doi.org/10.1167/jov.22.4.4.

Acknowledgements

Supported by JSPS KAKENHI Grant Numbers JP16J00273 and JP21K21315, the DFG (SFB-TRR-

135: “Cardinal Mechanisms of Perception,” project number 222641018); an ERC Consolidator Award (ERC-2015-CoG-682859: “SHAPE”); and by “The Adaptive Mind,” funded by the Excellence Program of the Hessian Ministry of Higher Education, Science, Research and Art.