The goal of this work is Active Speaker Detection (ASD), a task to determine
whether a person is speaking or not in a series of video frames. Previous works
have dealt with the task by exploring network architectures while learning
effective representations has been less explored. In this work, we propose
TalkNCE, a novel talk-aware contrastive loss. The loss is only applied to part
of the full segments where a person on the screen is actually speaking. This
encourages the model to learn effective representations through the natural
correspondence of speech and facial movements. Our loss can be jointly
optimized with the existing objectives for training ASD models without the need
for additional supervision or training data. The experiments demonstrate that
our loss can be easily integrated into the existing ASD frameworks, improving
their performance. Our method achieves state-of-the-art performances on
AVA-ActiveSpeaker and ASW datasets.

通过探索网络架构，以及使用自然语音与面部动作之间的对应进行学习，我们提出了一种新颖的 TalkNCE 对比损失函数，可用于主动说话者检测任务，无需额外的监督或训练数据，可以与现有的主动说话者检测模型在 AVA-ActiveSpeaker 和 ASW 数据集上联合优化，从而提高其性能。