Self-supervised learning has attracted plenty of recent research interest. However, most works are typically unimodal and there has been limited work that studies the interaction between audio and visual modalities for self-supervised learning. This work (1) investigates visual self-supervision via face reconstruction to guide the learning of audio representations; (2) proposes two audio-only self-supervision approaches for speech representation learning; (3) shows that a multi-task combination of the proposed visual and audio self-supervision is beneficial for learning richer features that are more robust in noisy conditions; (4) shows that self-supervised pretraining leads to a superior weight initialization, which is especially useful to prevent overfitting and lead to faster model convergence on smaller sized datasets. We evaluate our audio representations for emotion and speech recognition, achieving state of the art performance for both problems. Our results demonstrate the potential of visual self-supervision for audio feature learning and suggest that joint visual and audio self-supervision leads to more informative speech representations.

本研究探究了面部重建的视觉自监督方法辅助音频表示学习，提出了适用于语音表示学习的仅音频的自监督方法，比较了多任务的视听自监督与单一方式的自监督方法，发现视听自监督对于学习在噪声环境下更具有鲁棒性的特征更有益。同时，本研究发现自监督学习可以胜过全监督学习并防止过拟合，通过情感识别、自然语言识别等测试表明该方法成功提高了语音特征学习的效果。

视觉自我监督是否提高情感识别的语音表征学习？