Existing audio-language task-specific predictive approaches focus on building
complicated late-fusion mechanisms. However, these models are facing challenges
of overfitting with limited labels and low model generalization abilities. In
this paper, we present a Cross-modal Transformer for Audio-and-Language, i.e.,
CTAL, which aims to learn the intra-modality and inter-modality connections
between audio and language through two proxy tasks on a large amount of
audio-and-language pairs: masked language modeling and masked cross-modal
acoustic modeling. After fine-tuning our pre-trained model on multiple
downstream audio-and-language tasks, we observe significant improvements across
various tasks, such as, emotion classification, sentiment analysis, and speaker
verification. On this basis, we further propose a specially-designed fusion
mechanism that can be used in fine-tuning phase, which allows our pre-trained
model to achieve better performance. Lastly, we demonstrate detailed ablation
studies to prove that both our novel cross-modality fusion component and
audio-language pre-training methods significantly contribute to the promising
results.

本论文介绍了一种名为 CTAL 的跨模态转换器，旨在利用大量的音频和语言数据对两个代理任务进行遮蔽语言建模和遮蔽跨模态声学建模，以学习音频和语言之间的内模态和跨模态连接。在多个下游音频和语言任务的微调后，我们观察到在情感分类、情感分析和说话者验证等各种任务中都有显着的改进。最后，我们进行了详细的剖析研究，证明我们的创新跨模态融合组件和音频语言预训练方法显著促成了这一有前途的结果。