Recent advances in depthwise-separable convolutional neural networks
(DS-CNNs) have led to novel architectures, that surpass the performance of
classical CNNs, by a considerable scalability and accuracy margin. This paper
reveals another striking property of DS-CNN architectures: discernible and
explainable patterns emerge in their trained depthwise convolutional kernels in
all layers. Through an extensive analysis of millions of trained filters, with
different sizes and from various models, we employed unsupervised clustering
with autoencoders, to categorize these filters. Astonishingly, the patterns
converged into a few main clusters, each resembling the difference of Gaussian
(DoG) functions, and their first and second-order derivatives. Notably, we were
able to classify over 95\% and 90\% of the filters from state-of-the-art
ConvNextV2 and ConvNeXt models, respectively. This finding is not merely a
technological curiosity; it echoes the foundational models neuroscientists have
long proposed for the vision systems of mammals. Our results thus deepen our
understanding of the emergent properties of trained DS-CNNs and provide a
bridge between artificial and biological visual processing systems. More
broadly, they pave the way for more interpretable and biologically-inspired
neural network designs in the future.

最近在深度可分离卷积神经网络（DS-CNNs）方面的进展已经引发了新的架构，通过相当大的可扩展性和准确性优势，超越了经典 CNNs 的性能。本文揭示了 DS-CNN 架构的另一个引人注目的特性：它们训练后的深度卷积核在所有层中出现了明显可辨和可解释的模式。通过对数百万个训练滤波器的广泛分析，我们使用自编码器进行无监督聚类，对这些滤波器进行了分类。令人惊讶的是，这些模式汇聚成了几个主要簇，每个簇都类似于高斯差分函数（DoG）及其一阶和二阶导数。值得注意的是，我们能够将来自最先进的 ConvNextV2 和 ConvNeXt 模型的滤波器分别分类为 95％和 90％以上。这一发现不仅仅是技术上的奇特之处；它回应了神经科学家长期以来对哺乳动物视觉系统的基础模型的提议。因此，我们的结果加深了对训练后 DS-CNN 的出现属性的理解，并在人工和生物视觉处理系统之间提供了一个桥梁。更广泛地说，它们为未来更具可解释性和生物灵感的神经网络设计铺平了道路。

揭示不可见的东西：训练的深度可分卷积核中的可识别聚类

Unveiling the Unseen: Identifiable Clusters in Trained Depthwise  Convolutional Kernels

In this study, we present evidence suggesting that depthwise convolutional
kernels are effectively replicating the structural intricacies of the
biological receptive fields observed in the mammalian retina. We provide
analytics of trained kernels from various state-of-the-art models
substantiating this evidence. Inspired by this intriguing discovery, we propose
an initialization scheme that draws inspiration from the biological receptive
fields. Experimental analysis of the ImageNet dataset with multiple CNN
architectures featuring depthwise convolutions reveals a marked enhancement in
the accuracy of the learned model when initialized with biologically derived
weights. This underlies the potential for biologically inspired computational
models to further our understanding of vision processing systems and to improve
the efficacy of convolutional networks.

我们提供了支持的分析，显示出深度卷积核能够有效地复制哺乳动物视网膜中观察到的生物感受野的结构复杂性。在此基础上，我们提出了一个启发于生物感受野的初始化方案，并利用多个包含深度卷积的 CNN 架构在 ImageNet 数据集上进行实验分析，结果表明使用生物启发的权重初始化可以明显提高学习模型的准确性，这突显了生物启发型计算模型进一步推动视觉处理系统理解和改善卷积网络效果的潜力。