This paper investigates the application of voice activity projection (VAP), a
predictive turn-taking model for spoken dialogue, on multilingual data,
encompassing English, Mandarin, and Japanese. The VAP model continuously
predicts the upcoming voice activities of participants in dyadic dialogue,
leveraging a cross-attention Transformer to capture the dynamic interplay
between participants. The results show that a monolingual VAP model trained on
one language does not make good predictions when applied to other languages.
However, a multilingual model, trained on all three languages, demonstrates
predictive performance on par with monolingual models across all languages.
Further analyses show that the multilingual model has learned to discern the
language of the input signal. We also analyze the sensitivity to pitch, a
prosodic cue that is thought to be important for turn-taking. Finally, we
compare two different audio encoders, contrastive predictive coding (CPC)
pre-trained on English, with a recent model based on multilingual wav2vec 2.0
(MMS).

研究报告通过使用音频活动预测模型，在多语言数据中进行了口语对话研究，并对单语和多语模型进行了比较分析，发现多语模型对所有语言的预测表现与单语模型相当，同时还分析了声调敏感性和两种不同的音频编码器。