Convolutional neural networks (CNNs) have been shown to be the
state-of-the-art approach for modeling the transfer functions of visual
cortical neurons. Cortical neurons in the primary visual cortex are are
sensitive to contextual information mediated by extensive horizontal and
feedback connections. Standard CNNs can integrate global spatial image
information to model such contextual modulation via two mechanisms: successive
rounds of convolutions and a fully connected readout layer. In this paper, we
find that non-local networks or self-attention (SA) mechanisms, theoretically
related to context-dependent flexible gating mechanisms observed in the primary
visual cortex, improve neural response predictions over parameter-matched CNNs
in two key metrics: tuning curve correlation and tuning peak. We factorize
networks to determine the relative contribution of each context mechanism. This
reveals that information in the local receptive field is most important for
modeling the overall tuning curve, but surround information is critically
necessary for characterizing the tuning peak. We find that self-attention can
replace subsequent spatial-integration convolutions when learned in an
incremental manner, and is further enhanced in the presence of a fully
connected readout layer, suggesting that the two context mechanisms are
complementary. Finally, we find that learning a receptive-field-centric model
with self-attention, before incrementally learning a fully connected readout,
yields a more biologically realistic model in terms of center-surround
contributions.

卷积神经网络是建模视觉皮质神经元转移函数的最先进方法。这篇论文发现，与参数匹配的标准卷积神经网络相比，非局部网络或自注意机制在整体调谐曲线相关性和调谐峰值这两个关键度量上可以提高神经响应预测，并且可通过分解网络确定每个上下文机制的相对贡献，揭示了局部感受野中的信息对于模拟整体调谐曲线最为重要，而周围信息对于表征调谐峰值至关重要；并且在学习过程中自注意力可以取代后续的空间整合卷积，与全连接输出层相结合，两种上下文机制相辅相成；最后，学习以接受感受野为中心的自注意模型，在逐步学习全连接输出之前，能够提供更具生物学现实性的中心 - 周围贡献的模型。