Active speaker detection (ASD) seeks to detect who is speaking in a visual
scene of one or more speakers. The successful ASD depends on accurate
interpretation of short-term and long-term audio and visual information, as
well as audio-visual interaction. Unlike the prior work where systems make
decision instantaneously using short-term features, we propose a novel
framework, named TalkNet, that makes decision by taking both short-term and
long-term features into consideration. TalkNet consists of audio and visual
temporal encoders for feature representation, audio-visual cross-attention
mechanism for inter-modality interaction, and a self-attention mechanism to
capture long-term speaking evidence. The experiments demonstrate that TalkNet
achieves 3.5% and 2.2% improvement over the state-of-the-art systems on the
AVA-ActiveSpeaker dataset and Columbia ASD dataset, respectively. Code has been
made available at: this https URL

本文提出了一个名为 TalkNet 的新框架，用于在视觉场景中识别正在讲话的人并取得了比现有系统更好的效果。TalkNet 的特点是考虑了短期和长期的信息，并具有音频和视频编码器、跨模态的音视频交互机制以及自注意力机制。