We present Audiovisual SlowFast Networks, an architecture for integrated
audiovisual perception. AVSlowFast has Slow and Fast visual pathways that are
deeply integrated with a Faster Audio pathway to model vision and sound in a
unified representation. We fuse audio and visual features at multiple layers,
enabling audio to contribute to the formation of hierarchical audiovisual
concepts. To overcome training difficulties that arise from different learning
dynamics for audio and visual modalities, we introduce DropPathway, which
randomly drops the Audio pathway during training as an effective regularization
technique. Inspired by prior studies in neuroscience, we perform hierarchical
audiovisual synchronization to learn joint audiovisual features. We report
state-of-the-art results on six video action classification and detection
datasets, perform detailed ablation studies, and show the generalization of
AVSlowFast to learn self-supervised audiovisual features. Code will be made
available at: this https URL

该研究提出了 Audiovisual SlowFast Networks，一种用于集成视听感知的架构，并通过多层次融合视听特征以及采用 DropPathway regularization 技术对音频路径进行正则化来提高训练效果，并通过层次感知来学习联合视听特征，该方法在六个视频动作分类和检测数据集上取得了最先进的结果。