We present an exploration of machine learning architectures for predicting
brain responses to realistic images on occasion of the Algonauts Challenge
2023. Our research involved extensive experimentation with various pretrained
models. Initially, we employed simpler models to predict brain activity but
gradually introduced more complex architectures utilizing available data and
embeddings generated by large-scale pre-trained models. We encountered typical
difficulties related to machine learning problems, e.g. regularization and
overfitting, as well as issues specific to the challenge, such as difficulty in
combining multiple input encodings, as well as the high dimensionality, unclear
structure, and noisy nature of the output. To overcome these issues we tested
single edge 3D position-based, multi-region of interest (ROI) and hemisphere
predictor models, but we found that employing multiple simple models, each
dedicated to a ROI in each hemisphere of the brain of each subject, yielded the
best results - a single fully connected linear layer with image embeddings
generated by CLIP as input. While we surpassed the challenge baseline, our
results fell short of establishing a robust association with the data.

我们利用预训练模型，探索机器学习架构来预测大规模预训练模型生成的图像对大脑的响应，通过使用单个连接的线性层加上由 CLIP 生成的图像嵌入作为输入，在 Algonauts 挑战赛上取得了相对较好的结果，但未能建立与数据的强有力关联。