This work provides support for the notion that distributional methods of representing word meaning from computational linguistics are useful for capturing neural correlates of real life multi-sensory stimuli, where the stimuli ---in this case, a movie being watched by the human subjects--- have been given text annotations. We present an approach to combining sequences of word vectors into a single vector. We also identify a semantically-relevant low-dimensional shared representation of fMRI response in an unsupervised fashion by using views of multiple subjects watching the same natural movie stimulus. Learned orthogonal linear maps between the fMRI and semantic representations allow us to successfully transform fMRI data generated by a natural movie stimulus into semantic vectors representing textual descriptions of the movie. We succeed at a scene classification task with 76% accuracy, over a 20% chance rate. When we selected five brain regions-of-interest (ROIs) and learned distinct maps from these ROIs to the text representations, the Default Mode Network (DMN) supported the highest level of decoding performance.

研究了如何在只有自然语言注释的情况下，将fMRI响应与呈现刺激的含义相对应，并使用Shared Response Model（SRM）等技术提高准确性，可以通过场景分类任务以72％的准确率和场景排名任务在前4％的平均排名中成功实现，并从多个主题聚焦听众学习双向映射。

电影fMRI响应与自然语言注释之间的映射