In this paper, we show how to use audio to supervise the learning of active
speaker detection in video. Voice Activity Detection (VAD) guides the learning
of the vision-based classifier in a weakly supervised manner. The classifier
uses spatio-temporal features to encode upper body motion - facial expressions
and gesticulations associated with speaking. We further improve a generic model
for active speaker detection by learning person specific models. Finally, we
demonstrate the online adaptation of generic models learnt on one dataset, to
previously unseen people in a new dataset, again using audio (VAD) for weak
supervision. The use of temporal continuity overcomes the lack of clean
training data. We are the first to present an active speaker detection system
that learns on one audio-visual dataset and automatically adapts to speakers in
a new dataset. This work can be seen as an example of how the availability of
multi-modal data allows us to learn a model without the need for supervision,
by transferring knowledge from one modality to another.

本文利用音频监督视频中活跃讲话者检测的学习，通过声音活动检测 (VAD) 以弱监督的方式指导基于视觉的分类器学习。使用时空特征进行分类，学习人特定的模型以及在线适应性的改进，利用时间连续性克服了缺乏干净训练数据的问题。通过跨模态学习，不需要监督，将知识从一种模式传递到另一种模式，建立了自学习活跃讲话者检测系统。