How does the brain represent different modes of information? Can we design a
system that automatically understands what the user is thinking? Such questions
can be answered by studying brain recordings like functional magnetic resonance
imaging (fMRI). As a first step, the neuroscience community has contributed
several large cognitive neuroscience datasets related to passive
reading/listening/viewing of concept words, narratives, pictures and movies.
Encoding and decoding models using these datasets have also been proposed in
the past two decades. These models serve as additional tools for basic research
in cognitive science and neuroscience. Encoding models aim at generating fMRI
brain representations given a stimulus automatically. They have several
practical applications in evaluating and diagnosing neurological conditions and
thus also help design therapies for brain damage. Decoding models solve the
inverse problem of reconstructing the stimuli given the fMRI. They are useful
for designing brain-machine or brain-computer interfaces. Inspired by the
effectiveness of deep learning models for natural language processing, computer
vision, and speech, recently several neural encoding and decoding models have
been proposed. In this survey, we will first discuss popular representations of
language, vision and speech stimuli, and present a summary of neuroscience
datasets. Further, we will review popular deep learning based encoding and
decoding architectures and note their benefits and limitations. Finally, we
will conclude with a brief summary and discussion about future trends. Given
the large amount of recently published work in the `computational cognitive
neuroscience' community, we believe that this survey nicely organizes the
plethora of work and presents it as a coherent story.

研究通过功能性磁共振成像对大脑的记录，分析了编码模型和解码模型的应用，尤其关注深度学习算法的效果、好处和限制，并总结了神经科学数据集的代表性研究。

深度神经网络与脑部对齐：脑编码与解码（调研）

Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding  (Survey)

Pretrained language models that have been trained to predict the next word
over billions of text documents have been shown to also significantly predict
brain recordings of people comprehending language. Understanding the reasons
behind the observed similarities between language in machines and language in
the brain can lead to more insight into both systems. Recent works suggest that
the prediction of the next word is a key mechanism that contributes to the
alignment between the two. What is not yet understood is whether prediction of
the next word is necessary for this observed alignment or simply sufficient,
and whether there are other shared mechanisms or information that is similarly
important. In this work, we take a first step towards a better understanding
via two simple perturbations in a popular pretrained language model. The first
perturbation is to improve the model's ability to predict the next word in the
specific naturalistic stimulus text that the brain recordings correspond to. We
show that this indeed improves the alignment with the brain recordings.
However, this improved alignment may also be due to any improved word-level or
multi-word level semantics for the specific world that is described by the
stimulus narrative. We aim to disentangle the contribution of next word
prediction and semantic knowledge via our second perturbation: scrambling the
word order at inference time, which reduces the ability to predict the next
word, but maintains any newly learned word-level semantics. By comparing the
alignment with brain recordings of these differently perturbed models, we show
that improvements in alignment with brain recordings are due to more than
improvements in next word prediction and word-level semantics.

本文探讨使用众多的文本数据训练出来的预训练语言机器，对人脑语言理解的预测能力。研究发现，这两者之间的相似性是由下个词的预测机制引起的。作者的实验发现，对机器的语料库进行特定的自然文本刺激文本预测下个词，可以显著提高他们与大脑记录的对齐程度。然而，他们指出与大脑记录对齐的提高不仅仅是由于词汇级别或多词级别语义的改善，他们通过调整词序进一步证实这一点。