We introduce a distinctive real-time, causal, neural network-based active
speaker detection system optimized for low-power edge computing. This system
drives a virtual cinematography module and is deployed on a commercial device.
The system uses data originating from a microphone array and a 360-degree
camera. Our network requires only 127 MFLOPs per participant, for a meeting
with 14 participants. Unlike previous work, we examine the error rate of our
network when the computational budget is exhausted, and find that it exhibits
graceful degradation, allowing the system to operate reasonably well even in
this case. Departing from conventional DOA estimation approaches, our network
learns to query the available acoustic data, considering the detected head
locations. We train and evaluate our algorithm on a realistic meetings dataset
featuring up to 14 participants in the same meeting, overlapped speech, and
other challenging scenarios.

在商用设备上部署的低功耗边缘计算优化的实时因果神经网络活动说话人检测系统，通过来自麦克风阵列和 360 度摄像机的数据驱动虚拟电影摄影模块；与以往的研究不同，我们研究了网络在计算预算耗尽时的错误率，并发现它表现出优雅的降级，即使在此情况下系统仍能正常运行；与传统的声源角估计方法不同，我们的网络利用检测到的头部位置学习查询可用的声学数据；我们在一个现实的会议数据集上训练和评估我们的算法，该数据集包含达到 14 个与会者的同一会议、语音重叠和其他具有挑战性的场景。