In this paper, we present a system that associates faces with voices in a
video by fusing information from the audio and visual signals. The thesis
underlying our work is that an extremely simple approach to generating (weak)
speech clusters can be combined with visual signals to effectively associate
faces and voices by aggregating statistics across a video. This approach does
not need any training data specific to this task and leverages the natural
coherence of information in the audio and visual streams. It is particularly
applicable to tracking speakers in videos on the web where a priori information
about the environment (e.g., number of speakers, spatial signals for
beamforming) is not available. We performed experiments on a real-world dataset
using this analysis framework to determine the speaker in a video. Given a
ground truth labeling determined by human rater consensus, our approach had
~71% accuracy.

本文提出了一种音视频关联系统，将音频和视觉信号的信息融合，通过对视频中的统计数据的聚合实现面部和声音的有效关联，无需针对此任务的具体训练数据，并利用音频和视觉流中信息的自然相干性，特别适用于跟踪网络视频中的说话人，通过对真实数据集的实验表明，该方法的准确率约为 71％。