Convolutional neural networks (CNNs) have been shown to be the state-of-the-art approach for modeling the transfer functions of visual cortical neurons. Cortical neurons in the primary visual cortex are are sensitive to contextual information mediated by extensive horizontal and feedback connections. Standard CNNs can integrate global spatial image information to model such contextual modulation via two mechanisms: successive rounds of convolutions and a fully connected readout layer. In this paper, we find that non-local networks or self-attention (SA) mechanisms, theoretically related to context-dependent flexible gating mechanisms observed in the primary visual cortex, improve neural response predictions over parameter-matched CNNs in two key metrics: tuning curve correlation and tuning peak. We factorize networks to determine the relative contribution of each context mechanism. This reveals that information in the local receptive field is most important for modeling the overall tuning curve, but surround information is critically necessary for characterizing the tuning peak. We find that self-attention can replace subsequent spatial-integration convolutions when learned in an incremental manner, and is further enhanced in the presence of a fully connected readout layer, suggesting that the two context mechanisms are complementary. Finally, we find that learning a receptive-field-centric model with self-attention, before incrementally learning a fully connected readout, yields a more biologically realistic model in terms of center-surround contributions.

卷积神经网络是建模视觉皮质神经元转移函数的最先进方法。这篇论文发现，与参数匹配的标准卷积神经网络相比，非局部网络或自注意机制在整体调谐曲线相关性和调谐峰值这两个关键度量上可以提高神经响应预测，并且可通过分解网络确定每个上下文机制的相对贡献，揭示了局部感受野中的信息对于模拟整体调谐曲线最为重要，而周围信息对于表征调谐峰值至关重要；并且在学习过程中自注意力可以取代后续的空间整合卷积，与全连接输出层相结合，两种上下文机制相辅相成；最后，学习以接受感受野为中心的自注意模型，在逐步学习全连接输出之前，能够提供更具生物学现实性的中心-周围贡献的模型。

增量学习与自注意机制提升神经系统辨识