What if what we see, read or experience could be captured by AI, and then rendered as brief video clips to help scientists understand how the brain processes and encodes information? In the growing landscape of mental health and ageing-related disorders in Singapore and the world, understanding the brain is a key necessity.
With the help of AI and decoding data from brain activity, a team of researchers at the Yong Loo Lin School of Medicine, National University of Singapore (NUS Medicine) is able to construct videos that reflect/recall/mirror the images which study subjects were shown. They say this development/technology can potentially be crucial for the early detection of brain diseases, customised treatments, devices, and learning programmes. The study of brain patterns to figure out how the brain works could entail learning how certain thoughts link to speech and movements. These are beneficial in creating solutions for individuals who suffer from conditions that limit their ability to move or communicate.
Breaking the molds and pushing the boundaries of how AI is used in research, researchers from the Centre for Translational Magnetic Resonance Research at the Yong Loo Lin School of Medicine, National University of Singapore (NUS Medicine), is working on a novel project titled “Mind-Video”, which involves developing an AI model to visualise what individuals see. This model is able to scan and read data obtained from human brains and reconstruct the information as very short video bursts. The study reconstructs videos that participants are shown, using functional magnetic resonance imaging (fMRI) data, which generates representations from brain activity. This form of data provides a dynamic window into individuals’ thought processes that could revolutionise how we interact with, and interpret the human brain.
The study by Associate Professor Helen Zhou from the Centre for Sleep and Cognition in NUS Medicine, and Director, Centre for Translational MR Research at NUS Medicine, Mr Jiaxin Qing, PhD student at Department of Information Engineering, The Chinese University of Hong Kong (CUHK IE), and Ms Chen Zijiao, PhD Student at the Centre for Translational MR Research at NUS Medicine was published in their project page, Mind-Video. 2 Participants were shown videos of various lengths, ranging from two seconds to a few minutes, on moving objects, animals, humans, and more, while the researchers carried out the scanning via fMRI, a non-invasive procedure which helps to visualise the active parts of the brain when it is engaged in different activities. Upon collecting the data, an advanced AI model, Stable Diffusion, was used to decode the brain activity and translate the information into reconstructed videos of about two to three seconds long, as viewed by the participants. The team achieved an impressive accuracy rate of 85%. This is a significant step in bringing forth potentially groundbreaking solutions to treat neuropsychiatric conditions, as well as to help people with conditions that make movement and speech difficult to communicate better. This includes stroke, brain injuries, spinal cord damage, and diseases like Cerebral palsy, ALS and Parkinson’s. This discovery serves as a springboard for the early detection of brain diseases, customized treatments, devices, and learning programmes. For patients, it offers hope for better, more independent lives and effective treatments. “Our work can help to further our understanding of how the brain processes information with an unprecedented degree of detail, while paving the way for a more advanced communication system via technology and brain stimulation strategies. At the same time, we have plans to develop it further with generalisability and interpretability, to set the foundation for future translational work, including helping those individuals with impaired sensory perception or enhancing human potential,” said A/Prof Zhou.
The team will be presenting their work at the 2023 Conference on Neural Information Processing, which will be held this month, in New Orleans, USA.