Conventional audio-visual approaches for active speaker detection (ASD)
typically rely on visually pre-extracted face tracks and the corresponding
single-channel audio to find the speaker in a video. Therefore, they tend to
fail every time the face of the speaker is not visible. We demonstrate that a
simple audio convolutional recurrent neural network (CRNN) trained with spatial
input features extracted from multichannel audio can perform simultaneous
horizontal active speaker detection and localization (ASDL), independently of
the visual modality. To address the time and cost of generating ground truth
labels to train such a system, we propose a new self-supervised training
pipeline that embraces a ``student-teacher'' learning approach. A conventional
pre-trained active speaker detector is adopted as a ``teacher'' network to
provide the position of the speakers as pseudo-labels. The multichannel audio
``student'' network is trained to generate the same results. At inference, the
student network can generalize and locate also the occluded speakers that the
teacher network is not able to detect visually, yielding considerable
improvements in recall rate. Experiments on the TragicTalkers dataset show that
an audio network trained with the proposed self-supervised learning approach
can exceed the performance of the typical audio-visual methods and produce
results competitive with the costly conventional supervised training. We
demonstrate that improvements can be achieved when minimal manual supervision
is introduced in the learning pipeline. Further gains may be sought with larger
training sets and integrating vision with the multichannel audio system.

通过用从多通道音频中提取的空间输入特征进行训练的简单音频卷积循环神经网络（CRNN），可以独立于视觉模态进行同时水平活动说话者检测和定位（ASDL），超越典型的音频 - 视觉方法的性能并产生竞争性结果的成本昂贵的传统监督式训练。