We introduce a distinctive real-time, causal, neural network-based active
speaker detection system optimized for low-power edge computing. This system
drives a virtual cinematography module and is deployed on a commercial device.
The system uses data originating from a microphone array and a 360-degree
camera. Our network requires only 127 MFLOPs per participant, for a meeting
with 14 participants. Unlike previous work, we examine the error rate of our
network when the computational budget is exhausted, and find that it exhibits
graceful degradation, allowing the system to operate reasonably well even in
this case. Departing from conventional DOA estimation approaches, our network
learns to query the available acoustic data, considering the detected head
locations. We train and evaluate our algorithm on a realistic meetings dataset
featuring up to 14 participants in the same meeting, overlapped speech, and
other challenging scenarios.

在商用设备上部署的低功耗边缘计算优化的实时因果神经网络活动说话人检测系统，通过来自麦克风阵列和 360 度摄像机的数据驱动虚拟电影摄影模块；与以往的研究不同，我们研究了网络在计算预算耗尽时的错误率，并发现它表现出优雅的降级，即使在此情况下系统仍能正常运行；与传统的声源角估计方法不同，我们的网络利用检测到的头部位置学习查询可用的声学数据；我们在一个现实的会议数据集上训练和评估我们的算法，该数据集包含达到 14 个与会者的同一会议、语音重叠和其他具有挑战性的场景。

实时的主动说话者检测系统集成了音频 - 视觉信号和空间查询机制

A Real-Time Active Speaker Detection System Integrating an Audio-Visual  Signal with a Spatial Querying Mechanism

360$^{\circ}$ video requires human viewers to actively control "where" to
look while watching the video. Although it provides a more immersive experience
of the visual content, it also introduces additional burden for viewers;
awkward interfaces to navigate the video lead to suboptimal viewing
experiences. Virtual cinematography is an appealing direction to remedy these
problems, but conventional methods are limited to virtual environments or rely
on hand-crafted heuristics. We propose a new algorithm for virtual
cinematography that automatically controls a virtual camera within a
360$^{\circ}$ video. Compared to the state of the art, our algorithm allows
more general camera control, avoids redundant outputs, and extracts its output
videos substantially more efficiently. Experimental results on over 7 hours of
real "in the wild" video show that our generalized camera control is crucial
for viewing 360$^{\circ}$ video, while the proposed efficient algorithm is
essential for making the generalized control computationally tractable.

在 360 度视频中，虚拟电影技术被用于自动控制镜头，改善手动控制的体验，并通过提出的更高效的算法，使得推广虚拟电影成为可能。