In this work we introduce a self-supervised pretraining framework for
transformers on functional Magnetic Resonance Imaging (fMRI) data. First, we
pretrain our architecture on two self-supervised tasks simultaneously to teach
the model a general understanding of the temporal and spatial dynamics of human
auditory cortex during music listening. Our pretraining results are the first
to suggest a synergistic effect of multitask training on fMRI data. Second, we
finetune the pretrained models and train additional fresh models on a
supervised fMRI classification task. We observe significantly improved accuracy
on held-out runs with the finetuned models, which demonstrates the ability of
our pretraining tasks to facilitate transfer learning. This work contributes to
the growing body of literature on transformer architectures for pretraining and
transfer learning with fMRI data, and serves as a proof of concept for our
pretraining tasks and multitask pretraining on fMRI data.

本文介绍了一种自监督预训练框架，用于基于功能性磁共振（fMRI）数据的变压器。作者在两种自监督任务上预训练模型，以教授模型人类听觉皮层在音乐听取期间的时间和空间动态。作者的预训练结果表明，多任务训练对 fMRI 数据具有协同效应。作者在监督 fMRI 分类任务上发现，微调预先训练的模型和训练额外的新模型都能够显著提高准确性，证明了预训练任务促进迁移学习的能力。

自监督训练配对脑功能核磁共振数据的迁移学习应用于脑解码任务

Self-Supervised Pretraining on Paired Sequences of fMRI Data for  Transfer Learning to Brain Decoding Tasks

Emotion estimation in music listening is confronting challenges to capture
the emotion variation of listeners. Recent years have witnessed attempts to
exploit multimodality fusing information from musical contents and
physiological signals captured from listeners to improve the performance of
emotion recognition. In this paper, we present a study of fusion of signals of
electroencephalogram (EEG), a tool to capture brainwaves at a high-temporal
resolution, and musical features at decision level in recognizing the
time-varying binary classes of arousal and valence. Our empirical results
showed that the fusion could outperform the performance of emotion recognition
using only EEG modality that was suffered from inter-subject variability, and
this suggested the promise of multimodal fusion in improving the accuracy of
music-emotion recognition.

本文研究了如何在音乐欣赏中通过脑电图和音乐特征相结合的多模态融合方法提高情感识别的准确性，结果表明多模态融合方法能够优于单一模态，并提出了未来融合研究的方向。