Self-supervised learning has attracted plenty of recent research interest.
However, most works for self-supervision in speech are typically unimodal and
there has been limited work that studies the interaction between audio and
visual modalities for cross-modal self-supervision. This work (1) investigates
visual self-supervision via face reconstruction to guide the learning of audio
representations; (2) proposes an audio-only self-supervision approach for
speech representation learning; (3) shows that a multi-task combination of the
proposed visual and audio self-supervision is beneficial for learning richer
features that are more robust in noisy conditions; (4) shows that
self-supervised pretraining can outperform fully supervised training and is
especially useful to prevent overfitting on smaller sized datasets. We evaluate
our learned audio representations for discrete emotion recognition, continuous
affect recognition and automatic speech recognition. We outperform existing
self-supervised methods for all tested downstream tasks. Our results
demonstrate the potential of visual self-supervision for audio feature learning
and suggest that joint visual and audio self-supervision leads to more
informative audio representations for speech and emotion recognition.

本研究探究了面部重建的视觉自监督方法辅助音频表示学习，提出了适用于语音表示学习的仅音频的自监督方法，比较了多任务的视听自监督与单一方式的自监督方法，发现视听自监督对于学习在噪声环境下更具有鲁棒性的特征更有益。同时，本研究发现自监督学习可以胜过全监督学习并防止过拟合，通过情感识别、自然语言识别等测试表明该方法成功提高了语音特征学习的效果。