Neural decoding of visual information across different neural recording modalities and approaches

Every day, various types of sensory information fromthe external environment are transferred to the brainthrough different modalities and then processed to generate a series of coping behaviors. Among these perceptual modalities, vision is arguably the dominant contributor to the interactions between the external environmentand the brain. Approximately 70 percent of human perception information is derived from vision, far morethan the auditory system, tactile system, and other sensory systems combined. The visual system is the part ofthe central nervous system that is required for visual perception, processing, and interpreting visual information tobuild a representation of the visual environment. It consists of the eye, retina, fibers that conduct visual information to the thalamus, the superior colliculus, and parts ofthe cerebral cortex. Today, researchers can collect neural signals using different recording modalities, e.g., spikes,electroencephalography (EEG), and functional magneticresonance imaging (fMRI), from brain activity in different parts of the visual system, such as the retina, lateralgeniculate nucleus (LGN), and primary visual cortex (V1)cortex, etc. Depending on the corresponding collectingdevices, different recording modalities differ in their invasiveness, scale, and precision.

Neural coding is an important topic for understanding how the brain processes stimuli from the environment. The aim of neural decoding is to read out information embedded in various types of neural signals.As for vision, understanding how neurons perceive and respond to rich natural visual information is a major topicof neural encoding, whereas, the goal of neural decoding of visual information is to restore the original stimulus from neural responses as much as possible. It is also critical to the development of artificialvision used by brain-computer interfaces and virtual reality devices.

Much effort has been made to study the various mechanisms underlying neural decoding in the visual pathwayin recent decades. These mechanisms can be roughlydivided into three categories depending on the decodingtype: 1) Visual stimulus classification, in which a specific stimulus is classified into the best-matched image set;2) Visual stimulus identification, in which the stimulus isidentified with a specific visual object; 3) Visual stimulireconstruction, in which the corresponding visual stimulus is reconstructed in accordance with the resulting neural responses. Most decoding approaches have depended onlinear methods due to their interpretability and computational efficiency. Although linear decoding methods are capable of decoding spatially uniform white noisestimuli and the coarse structure of natural scene stimulifrom neural responses, the recovery of the fine visual details of naturalistic images is difficult for these typesof methods. The most recent decoders utilized nonlinearmethods for the fine decoding of complex visual stimuli.

For instance, optimal Bayesian decoding was leveragedfor white noise stimuli, but achieved limited generalizability to a large neural population. For natural scene image structures, key prior information was used to perform computationally expensive approximations toBayesian inference. Some researchers have combined linear and nonlinear approaches to generate coarsereconstructions of natural stimuli through calcium imaging data. Additionally, many researchers havebegun to successfully use deep learning techniques forvisual neural decoding, leading to the great achievementin artificial vision.

Visual neural decoding is a significant issue that canhelp advance engineering applications such as brain–machine interfaces and a more holistic understanding of thebrain in neuroscience. Considering the rapid developments of related techniques in visual neural decoding,there is a strong demand for a comprehensive and up-to-date review in this field. In this review, researchers sorted out theresearch evolution in visual neural decoding. Variousneural recording modalities are introduced in this review,especially for the emerging calcium imaging data. Researcherssummarized the advantages and disadvantages of different neural decoding methods. In addition, open resources, including public neural data and software toolkits, arealso provided for the convenience of neural decoding research. Finally, they conclude with their perspective on theopen challenges and future directions for the outlook inthis study. Researchers aim to provide a review of neural decoding in visual systems that could serve as an inspiration toboth neuroscience and multidisciplinary researchers looking to understand the state-of-the-art and current problems in neural decoding, especially regarding the development of artificial intelligence and brain-like vision systems.

In this paper, researchers first briefly analyzed the evolution ofdecoding tasks, i.e., classification, identification, and reconstruction, as this research field has developed. Then theyintroduced the main neural recording modalities used invisual neural decoding, including spikes, EEG, fMRI and Calcium imaging signal, and analyzed the characteristics ofthe data they acquire. Figures in this paper show the characteristics of the signals obtained with neural recording modalities, and summarizethe differences in signal types, data structures,and spatial and temporal resolutions.

Then in Section 4 researchers reviewed the main typesof decoding approaches that have been proposed inrecent decades in this field. The first are linear decoding methods. Most early approaches to visual neural decoding havedepended on linear methods due to their interpretabilityand computational efficiency. Nevertheless, the limitedrepresentation power of linear methods makes it difficultto reconstruct the fine visual details of natural images. The second are Bayesian-based decoding methods. Bayesian decoding models usually outperformed simple linear decoding models in visual neural decoding tasks. However, there are also someconstraints for Bayesian decoding methods. The Bayesiandecoding methods usually have to resort to the specificprior information encoded by a specifically designed model. The determination of parameters in the overall decoding process needs to be elaborated. Furthermore, themapping between the visual stimuli and the corresponding brain activity determined by Bayesian methods doesnot typically describe the relationship between these twocross-modal data. Consequently, fine natural image details are difficult to be reconstructed with this type ofmethod. The third are deep neural network methods, which include the introduction of CNN, RNN, generative adversarial networks (GANs), transfer learning and so on.

The open-source nature of datasets with high-qualityneuronal physiological responses is essential for neuroscience research. These open-source data objectively connect different research works through sharing data andplay an important role in forming benchmarks in the fieldof neural decoding. InSection 5, researchers summarize three large-scaleopen neural databases widely used worldwide, as well assome neural analysis software toolboxes. The databases consist of OpenNEURO, the Allen Brain Map, collaborative research in computational neuroscience (CRCNS).

The issue of visual neural decoding has been a topic of interest for decades, with rapid advances in the development of bothbrain-activity recording techniques and neural decodinganalysis methods. In Section 6, researchers of this paper highlight several potential directions and open challenges and hope to provide other researchers with insight into this issue.

The ultimate purpose of visual decoding is to decode the content of people’s experience in the absence of visual input. However, the scarcity of pairwise neurophysiological stimulus datasets and accurate, large-scale recording neural modalities continue to hinder the developmentof this discipline. Nevertheless, the importance of visualneural decoding cannot be understated. The developmentof neural decoding technology will promote the development of neural prostheses and brain-computer interfacedevices. Researchers hope that their brief review will inspire ideasfor future work in the cross-disciplinary field of brain science and neural computing.