Large-kernel convolutional neural networks (ConvNets) have recently received
extensive research attention, but there are two unresolved and critical issues
that demand further investigation. 1) The architectures of existing
large-kernel ConvNets largely follow the design principles of conventional
ConvNets or transformers, while the architectural design for large-kernel
ConvNets remains under-addressed. 2) As transformers have dominated multiple
modalities, it remains to be investigated whether ConvNets also have a strong
universal perception ability in domains beyond vision. In this paper, we
contribute from two aspects. 1) We propose four architectural guidelines for
designing large-kernel ConvNets, the core of which is to exploit the essential
characteristics of large kernels that distinguish them from small kernels -
they can see wide without going deep. Following such guidelines, our proposed
large-kernel ConvNet shows leading performance in image recognition. For
example, our models achieve an ImageNet accuracy of 88.0%, ADE20K mIoU of
55.6%, and COCO box AP of 56.4%, demonstrating better performance and higher
speed than a number of recently proposed powerful competitors. 2) We discover
that large kernels are the key to unlocking the exceptional performance of
ConvNets in domains where they were originally not proficient. With certain
modality-related preprocessing approaches, the proposed model achieves
state-of-the-art performance on time-series forecasting and audio recognition
tasks even without modality-specific customization to the architecture. Code
and all the models at this https URL

我们提出了四种用于设计大核卷积神经网络的架构指南，并发现大核对于拓展卷积神经网络在非视觉领域具有卓越表现的关键因素。我们的模型在图像识别、时间序列预测和音频识别任务中表现出卓越性能，优于其他竞争模型，并且无需对体系结构进行特定模态的定制化处理。

UniRepLKNet：通用感知大卷积网络用于音频、视频、点云、时间序列和图像识别

UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio,  Video, Point Cloud, Time-Series and Image Recognition

This paper reveals a new appeal of the recently emerged large-kernel
Convolutional Neural Networks (ConvNets): as the teacher in Knowledge
Distillation (KD) for small-kernel ConvNets. While Transformers have led
state-of-the-art (SOTA) performance in various fields with ever-larger models
and labeled data, small-kernel ConvNets are considered more suitable for
resource-limited applications due to the efficient convolution operation and
compact weight sharing. KD is widely used to boost the performance of
small-kernel ConvNets. However, previous research shows that it is not quite
effective to distill knowledge (e.g., global information) from Transformers to
small-kernel ConvNets, presumably due to their disparate architectures. We
hereby carry out a first-of-its-kind study unveiling that modern large-kernel
ConvNets, a compelling competitor to Vision Transformers, are remarkably more
effective teachers for small-kernel ConvNets, due to more similar
architectures. Our findings are backed up by extensive experiments on both
logit-level and feature-level KD ``out of the box", with no dedicated
architectural nor training recipe modifications. Notably, we obtain the
\textbf{best-ever pure ConvNet} under 30M parameters with \textbf{83.1\%} top-1
accuracy on ImageNet, outperforming current SOTA methods including ConvNeXt V2
and Swin V2. We also find that beneficial characteristics of large-kernel
ConvNets, e.g., larger effective receptive fields, can be seamlessly
transferred to students through this large-to-small kernel distillation. Code
is available at: https://github.com/VITA-Group/SLaK.

研究表明，现代大核卷积神经网络对于小核卷积神经网络的教学更加有效，从而更适合于知识蒸馏。在使用大到小核蒸馏的过程中，大卷积神经网络的良好特性，例如更大的有效接收域，可以顺利地传递到学生身上，并且在 ImageNet 上实现了 83.1% 的 top-1 准确度，超过了当前的 SOTA 方法，包括 ConvNeXt V2 和 Swin V2