We characterized the generalization capabilities of DNN-based encoding models
when predicting neuronal responses from the visual cortex. We collected
\textit{MacaqueITBench}, a large-scale dataset of neural population responses
from the macaque inferior temporal (IT) cortex to over $300,000$ images,
comprising $8,233$ unique natural images presented to seven monkeys over $109$
sessions. Using \textit{MacaqueITBench}, we investigated the impact of
distribution shifts on models predicting neural activity by dividing the images
into Out-Of-Distribution (OOD) train and test splits. The OOD splits included
several different image-computable types including image contrast, hue,
intensity, temperature, and saturation. Compared to the performance on
in-distribution test images -- the conventional way these models have been
evaluated -- models performed worse at predicting neuronal responses to
out-of-distribution images, retaining as little as $20\%$ of the performance on
in-distribution test images. The generalization performance under OOD shifts
can be well accounted by a simple image similarity metric -- the cosine
distance between image representations extracted from a pre-trained object
recognition model is a strong predictor of neural predictivity under different
distribution shifts. The dataset of images, neuronal firing rate recordings,
and computational benchmarks are hosted publicly at: this https URL

使用 DNN 编码模型预测视觉皮层神经元反应时，我们对其泛化能力进行了表征。通过收集来自猕猴颞下皮层的大规模神经群体反应数据集 MacaqueITBench，我们研究了分布转移对预测神经活动模型的影响。结果显示，分布转移对于预测神经元响应的模型性能有很大影响，并且预训练对象识别模型提取的图像表示之间的余弦距离是神经预测性的强预测因子。