Convolutional neural networks (CNNs) have been shown to be the
state-of-the-art approach for modeling the transfer functions of visual
cortical neurons. Cortical neurons in the primary visual cortex are are
sensitive to contextual information mediated by extensive horizontal and
feedback connections. Standard CNNs can integrate global spatial image
information to model such contextual modulation via two mechanisms: successive
rounds of convolutions and a fully connected readout layer. In this paper, we
find that non-local networks or self-attention (SA) mechanisms, theoretically
related to context-dependent flexible gating mechanisms observed in the primary
visual cortex, improve neural response predictions over parameter-matched CNNs
in two key metrics: tuning curve correlation and tuning peak. We factorize
networks to determine the relative contribution of each context mechanism. This
reveals that information in the local receptive field is most important for
modeling the overall tuning curve, but surround information is critically
necessary for characterizing the tuning peak. We find that self-attention can
replace subsequent spatial-integration convolutions when learned in an
incremental manner, and is further enhanced in the presence of a fully
connected readout layer, suggesting that the two context mechanisms are
complementary. Finally, we find that learning a receptive-field-centric model
with self-attention, before incrementally learning a fully connected readout,
yields a more biologically realistic model in terms of center-surround
contributions.

卷积神经网络是建模视觉皮质神经元转移函数的最先进方法。这篇论文发现，与参数匹配的标准卷积神经网络相比，非局部网络或自注意机制在整体调谐曲线相关性和调谐峰值这两个关键度量上可以提高神经响应预测，并且可通过分解网络确定每个上下文机制的相对贡献，揭示了局部感受野中的信息对于模拟整体调谐曲线最为重要，而周围信息对于表征调谐峰值至关重要；并且在学习过程中自注意力可以取代后续的空间整合卷积，与全连接输出层相结合，两种上下文机制相辅相成；最后，学习以接受感受野为中心的自注意模型，在逐步学习全连接输出之前，能够提供更具生物学现实性的中心 - 周围贡献的模型。

增量学习与自注意机制提升神经系统辨识

Incremental Learning and Self-Attention Mechanisms Improve Neural System  Identification

Sparse coding algorithms trained on natural images can accurately predict the
features that excite visual cortical neurons, but it is not known whether such
codes can be learned using biologically realistic plasticity rules. We have
developed a biophysically motivated spiking network, relying solely on
synaptically local information, that can predict the full diversity of V1
simple cell receptive field shapes when trained on natural images. This
represents the first demonstration that sparse coding principles, operating
within the constraints imposed by cortical architecture, can successfully
reproduce these receptive fields. We further prove, mathematically, that
sparseness and decorrelation are the key ingredients that allow for
synaptically local plasticity rules to optimize a cooperative, linear
generative image model formed by the neural representation. Finally, we discuss
several interesting emergent properties of our network, with the intent of
bridging the gap between theoretical and experimental studies of visual cortex.

训练在自然图像上的稀疏编码算法能够准确预测视觉皮层神经元所激发的特征，但不确定是否可以使用生物学真实的可塑性规则来学习这些代码。我们开发了一种生物物理模型的尖峰网络，仅依靠突触局部信息就能预测自然图像上 V1 简单细胞感受野的全部多样性形状，这是首次证明，稀疏编码原则在皮层结构限制下操作时可以成功地再现这些感受野。我们进一步证明，稀疏性和去相关是允许突触局部可塑性规则优化神经表达形成的合作线性生成图像模型的关键因素。最后，我们讨论了我们的网络的几个有趣的新兴属性，以期弥合视觉皮层的理论和实验研究之间的差距。