This paper presents a study on improving human action recognition through the
utilization of knowledge distillation, and the combination of CNN and ViT
models. The research aims to enhance the performance and efficiency of smaller
student models by transferring knowledge from larger teacher models. The
proposed method employs a Transformer vision network as the student model,
while a convolutional network serves as the teacher model. The teacher model
extracts local image features, whereas the student model focuses on global
features using an attention mechanism. The Vision Transformer (ViT)
architecture is introduced as a robust framework for capturing global
dependencies in images. Additionally, advanced variants of ViT, namely PVT,
Convit, MVIT, Swin Transformer, and Twins, are discussed, highlighting their
contributions to computer vision tasks. The ConvNeXt model is introduced as a
teacher model, known for its efficiency and effectiveness in computer vision.
The paper presents performance results for human action recognition on the
Stanford 40 dataset, comparing the accuracy and mAP of student models trained
with and without knowledge distillation. The findings illustrate that the
suggested approach significantly improves the accuracy and mAP when compared to
training networks under regular settings. These findings emphasize the
potential of combining local and global features in action recognition tasks.

通过知识蒸馏以及结合 CNN 和 ViT 模型，本研究旨在改善人体动作识别的性能和效率，从较大的教师模型向较小的学生模型传递知识，并通过引入 Transformer 视觉网络作为学生模型以及卷积网络作为教师模型，扩展了对局部图像特征和全局特征的关注，进而显著提升了动作识别的准确性和平均精度。

从 CNN-Transformer 模型中提取知识，以增强人类动作识别

Distilling Knowledge from CNN-Transformer Models for Enhanced Human  Action Recognition

Haze usually leads to deteriorated images with low contrast, color shift and
structural distortion. We observe that many deep learning based models exhibit
exceptional performance on removing homogeneous haze, but they usually fail to
address the challenge of non-homogeneous dehazing. Two main factors account for
this situation. Firstly, due to the intricate and non uniform distribution of
dense haze, the recovery of structural and chromatic features with high
fidelity is challenging, particularly in regions with heavy haze. Secondly, the
existing small scale datasets for non-homogeneous dehazing are inadequate to
support reliable learning of feature mappings between hazy images and their
corresponding haze-free counterparts by convolutional neural network
(CNN)-based models. To tackle these two challenges, we propose a novel two
branch network that leverages 2D discrete wavelete transform (DWT), fast
Fourier convolution (FFC) residual block and a pretrained ConvNeXt model.
Specifically, in the DWT-FFC frequency branch, our model exploits DWT to
capture more high-frequency features. Moreover, by taking advantage of the
large receptive field provided by FFC residual blocks, our model is able to
effectively explore global contextual information and produce images with
better perceptual quality. In the prior knowledge branch, an ImageNet
pretrained ConvNeXt as opposed to Res2Net is adopted. This enables our model to
learn more supplementary information and acquire a stronger generalization
ability. The feasibility and effectiveness of the proposed method is
demonstrated via extensive experiments and ablation studies. The code is
available at this https URL

该研究提出了一种新的两分支网络，利用 2D 离散小波变换、快速傅里叶卷积残差块和预训练的 ConvNeXt 模型来解决雾霾图像非均质去雾的问题，并进行了大量实验验证其有效性。