In recent years, researchers combine both audio and video signals to deal
with challenges where actions are not well represented or captured by visual
cues. However, how to effectively leverage the two modalities is still under
development. In this work, we develop a multiscale multimodal Transformer (MMT)
that leverages hierarchical representation learning. Particularly, MMT is
composed of a novel multiscale audio Transformer (MAT) and a multiscale video
Transformer [43]. To learn a discriminative cross-modality fusion, we further
design multimodal supervised contrastive objectives called audio-video
contrastive loss (AVC) and intra-modal contrastive loss (IMC) that robustly
align the two modalities. MMT surpasses previous state-of-the-art approaches by
7.3% and 2.1% on Kinetics-Sounds and VGGSound in terms of the top-1 accuracy
without external training data. Moreover, the proposed MAT significantly
outperforms AST [28] by 22.2%, 4.4% and 4.7% on three public benchmark
datasets, and is about 3% more efficient based on the number of FLOPs and 9.8%
more efficient based on GPU memory usage.

本文介绍了一种多尺度多模态转换器（MMT），利用层次化表示学习，进一步使用音频 - 视频对比损失和同模态对比损失来实现多模态融合，提高动作识别准确率。