We propose Hierarchical ProtoPNet: an interpretable network that explains its
reasoning process by considering the hierarchical relationship between classes.
Different from previous methods that explain their reasoning process by
dissecting the input image and finding the prototypical parts responsible for
the classification, we propose to explain the reasoning process for video
action classification by dissecting the input video frames on multiple levels
of the class hierarchy. The explanations leverage the hierarchy to deal with
uncertainty, akin to human reasoning: When we observe water and human activity,
but no definitive action it can be recognized as the water sports parent class.
Only after observing a person swimming can we definitively refine it to the
swimming action. Experiments on ActivityNet and UCF-101 show performance
improvements while providing multi-level explanations.

本文提出一种可解释的网络模型 Hierarchical ProtoPNet，该模型通过考虑类别之间的层次关系来解释其推理过程，并在视频动作分类领域中应用，提出了一种多级解释的方法。在 ActivityNet 和 UCF-101 数据集上实验证明该方法能够提高性能。

视频动作识别的分层解释

Hierarchical Explanations for Video Action Recognition

The construction of models for video action classification progresses
rapidly. However, the performance of those models can still be easily improved
by ensembling with the same models trained on different modalities (e.g.
Optical flow). Unfortunately, it is computationally expensive to use several
modalities during inference. Recent works examine the ways to integrate
advantages of multi-modality into a single RGB-model. Yet, there is still a
room for improvement. In this paper, we explore the various methods to embed
the ensemble power into a single model. We show that proper initialization, as
well as mutual modality learning, enhances single-modality models. As a result,
we achieve state-of-the-art results in the Something-Something-v2 benchmark.

本研究探讨了将集成学习应用于单个模型的各种方法，通过适当的初始化和相互学习等策略增强单模态模型，达到了在 Something-Something-v2 基准测试中的最优结果。

视频动作分类的互模态学习

Mutual Modality Learning for Video Action Classification

We present Audiovisual SlowFast Networks, an architecture for integrated
audiovisual perception. AVSlowFast has Slow and Fast visual pathways that are
deeply integrated with a Faster Audio pathway to model vision and sound in a
unified representation. We fuse audio and visual features at multiple layers,
enabling audio to contribute to the formation of hierarchical audiovisual
concepts. To overcome training difficulties that arise from different learning
dynamics for audio and visual modalities, we introduce DropPathway, which
randomly drops the Audio pathway during training as an effective regularization
technique. Inspired by prior studies in neuroscience, we perform hierarchical
audiovisual synchronization to learn joint audiovisual features. We report
state-of-the-art results on six video action classification and detection
datasets, perform detailed ablation studies, and show the generalization of
AVSlowFast to learn self-supervised audiovisual features. Code will be made
available at: this https URL

该研究提出了 Audiovisual SlowFast Networks，一种用于集成视听感知的架构，并通过多层次融合视听特征以及采用 DropPathway regularization 技术对音频路径进行正则化来提高训练效果，并通过层次感知来学习联合视听特征，该方法在六个视频动作分类和检测数据集上取得了最先进的结果。