Cross-lingual self-supervised learning has been a growing research topic in
the last few years. However, current works only explored the use of audio
signals to create representations. In this work, we study cross-lingual
self-supervised visual representation learning. We use the recently-proposed
Raw Audio-Visual Speech Encoders (RAVEn) framework to pre-train an audio-visual
model with unlabelled multilingual data, and then fine-tune the visual model on
labelled transcriptions. Our experiments show that: (1) multi-lingual models
with more data outperform monolingual ones, but, when keeping the amount of
data fixed, monolingual models tend to reach better performance; (2)
multi-lingual outperforms English-only pre-training; (3) using languages which
are more similar yields better results; and (4) fine-tuning on unseen languages
is competitive to using the target language in the pre-training set. We hope
our study inspires future research on non-English-only speech representation
learning.

本文探讨了跨语言自监督视觉表示学习，使用 Raw Audio-Visual Speech Encoders（RAVEn）框架对无标注的多语言数据进行预训练，然后用标记的转录 fine-tune 视觉模型，实验证明多语言模型优于单语言模型，多语言胜过英文预训练，使用相似的语言效果更好，而无标签的语言进行精调与使用目标语言在预训练集上进行竞争。