This paper proposes a novel framework for multilingual speech and sound
representation learning using contrastive learning. The lack of sizeable
labelled datasets hinders speech-processing research across languages. Recent
advances in contrastive learning provide self-supervised techniques to learn
from unlabelled data. Motivated by reducing data dependence and improving
generalisation across diverse languages and conditions, we develop a
multilingual contrastive framework. This framework enables models to acquire
shared representations across languages, facilitating cross-lingual transfer
with limited target language data.
Additionally, capturing emotional cues within speech is challenging due to
subjective perceptual assessments. By learning expressive representations from
diverse, multilingual data in a self-supervised manner, our approach aims to
develop speech representations that encode emotive dimensions.
Our method trains encoders on a large corpus of multi-lingual audio data.
Data augmentation techniques are employed to expand the dataset. The
contrastive learning approach trains the model to maximise agreement between
positive pairs and minimise agreement between negative pairs. Extensive
experiments demonstrate state-of-the-art performance of the proposed model on
emotion recognition, audio classification, and retrieval benchmarks under
zero-shot and few-shot conditions. This provides an effective approach for
acquiring shared and generalised speech representations across languages and
acoustic conditions while encoding latent emotional dimensions.

提出了一种利用对比学习进行多语言语音和声音表示学习的新框架，旨在通过减少数据依赖性、改善各种语言和条件下的泛化能力，实现多语言共享表示，以便在有限的目标语言数据中促进跨语言转移。通过从多语言数据中自我监督学习情感表示，该方法在情感识别、音频分类和检索基准测试中展示了最先进的性能，为获得跨语言和声学条件下的共享和泛化语音表示提供了一种有效的方法。