New brain mapping technique reveals neural code of images over time

Humans are stepping ever closer to understanding how the brain codes visual information, as researchers have now developed a method that maps time-varying brain responses to images to reveal how the brain processes visual information.  

Colgate University Neuroscience Professor Bruce C. Hansen collaborated with Michelle R. Greene (Bates College), and David J. Field (Cornell University) to introduce dynamic electrode-to-image (DETI) mapping -- an analytical technique that capitalizes on the high temporal resolution of electroencephalography (EEG) to render maps of visual features that are associated with different neural signals over time.  View a real-time example of neural responses mapped to an image.

Read the full journal article published in PLOS Computational Biology, Dynamic Electrode-to-Image (DETI) mapping reveals the human brain’s spatiotemporal code of visual information

“When viewing any environment, our brains code visual information across a large population of neurons in a way that enables a variety of intelligent behaviors. However, the visual code that is used to guide behavior is not steady like a picture but instead evolves over time with different populations of neurons contributing to the code at different points in time. Our DETI mapping technique offers a first glimpse into that time-varying code at every location in images,” said Hansen.

Recent advances in voxel-wise encoding analyses based on functional magnetic resonance imaging (fMRI) have enabled compelling reconstructions of images based on brain data, but are only able to render a single snapshot in time due to fMRI’s limited temporal resolution.  The DETI mapping procedure introduced by Hansen and colleagues is based on EEG signals, which afford an opportunity to map the neural code of images with millisecond precision. 

To successfully map the visual code to images with EEG data, Hansen and colleagues had to overcome a number of methodological challenges.  “The brain signals that are recorded by EEG suffer from interference by the skull as well as different amounts of cancellation due to the folding patterns of the brain.”  Using a biologically plausible encoding model of the brain, Hansen and his team were able to circumvent those problems by measuring the correspondence between encoded pixels across a large number of images and the resulting changes in the neural response.  “One way to think about how the DETI mapping procedure works is by passing an image into the brain and projecting the resulting neural code back onto the image.”  Because EEG can measure neural signals at different scalp locations, DETI mapping produces a multiplexed view of how different populations of neurons code image features at different locations in images over time -- something that was once thought impossible to do with EEG data.

The mapping data produced by the DETI procedure offer new and important insights into how the neural code of images evolves over time. One of the most striking results reported by Hansen and colleagues is that the brain appears to scan images in a way that emphasizes different image regions with different neural populations at different points in time. “Such a scanning procedure likely aids in an early prioritization of the ground plane to support judgments for navigation, with a later emphasis focused on landmark organization.”

These findings lead to new and interesting questions related to how the evolving neural code informs higher level cognitive processes when people are engaged in different tasks.  “We know that the code for visual information is distributed across a large population of neurons, but how that code is distributed depends on the goals of a given task.  What this means is that the brain does not simply create a mental picture based exclusively on the environment, but instead creates a representation that best matches the behavioral goals of the person.”  Fortunately, DETI mapping enables opportunities to explore the neural dynamics of task-based visual codes and how those codes ultimately support task-based decision making.