In recent years, the deployment of large-scale pre-trained models in
audio-visual downstream tasks has yielded remarkable outcomes. However, these
models, primarily trained on single-modality unconstrained datasets, still
encounter challenges in feature extraction for multi-modal tasks, leading to
suboptimal performance. This limitation arises due to the introduction of
irrelevant modality-specific information during encoding, which adversely
affects the performance of downstream tasks. To address this challenge, this
paper proposes a novel Dual-Guided Spatial-Channel-Temporal (DG-SCT) attention
mechanism. This mechanism leverages audio and visual modalities as soft prompts
to dynamically adjust the parameters of pre-trained models based on the current
multi-modal input features. Specifically, the DG-SCT module incorporates
trainable cross-modal interaction layers into pre-trained audio-visual
encoders, allowing adaptive extraction of crucial information from the current
modality across spatial, channel, and temporal dimensions, while preserving the
frozen parameters of large-scale pre-trained models. Experimental evaluations
demonstrate that our proposed model achieves state-of-the-art results across
multiple downstream tasks, including AVE, AVVP, AVS, and AVQA. Furthermore, our
model exhibits promising performance in challenging few-shot and zero-shot
scenarios. The source code and pre-trained models are available at
this https URL

本论文提出了一种新的双向引导空间 - 通道 - 时间（DG-SCT）注意力机制，通过引入可训练的跨模态交互层，从当前模态跨空间、通道和时间维度自适应地提取关键信息，以解决多模态任务中的特征提取挑战，其结果在多个下游任务中达到最先进水平，并在挑战性的少样本和零样本场景中表现出良好性能。