Developing algorithms for accurate and comprehensive neural decoding of
mental contents is one of the long-cherished goals in the field of neuroscience
and brain-machine interfaces. Previous studies have demonstrated the
feasibility of neural decoding by training machine learning models to map brain
activity patterns into a semantic vector representation of stimuli. These
vectors, hereafter referred as pretrained feature vectors, are usually derived
from semantic spaces based solely on image and/or text features and therefore
they might have a totally different characteristics than how visual stimuli is
represented in the human brain, resulting in limiting the capability of brain
decoders to learn this mapping. To address this issue, we propose a
representation learning framework, termed brain-grounding of semantic vectors,
which fine-tunes pretrained feature vectors to better align with the neural
representation of visual stimuli in the human brain. We trained this model this
model with functional magnetic resonance imaging (fMRI) of 150 different visual
stimuli categories, and then performed zero-shot brain decoding and
identification analyses on 1) fMRI and 2) magnetoencephalography (MEG).
Interestingly, we observed that by using the brain-grounded vectors, the brain
decoding and identification accuracy on brain data from different neuroimaging
modalities increases. These findings underscore the potential of incorporating
a richer array of brain-derived features to enhance performance of brain
decoding algorithms.

提出了一种神经解码的表示学习框架，通过微调预训练特征向量以更好地与人脑中的视觉刺激的神经表示对齐，从而提高大脑解码算法的性能。