This paper describes a system that generates speaker-annotated transcripts of
meetings by using a microphone array and a 360-degree camera. The hallmark of
the system is its ability to handle overlapped speech, which has been an
unsolved problem in realistic settings for over a decade. We show that this
problem can be addressed by using a continuous speech separation approach. In
addition, we describe an online audio-visual speaker diarization method that
leverages face tracking and identification, sound source localization, speaker
identification, and, if available, prior speaker information for robustness to
various real world challenges. All components are integrated in a meeting
transcription framework called SRD, which stands for "separate, recognize, and
diarize". Experimental results using recordings of natural meetings involving
up to 11 attendees are reported. The continuous speech separation improves a
word error rate (WER) by 16.1% compared with a highly tuned beamformer. When a
complete list of meeting attendees is available, the discrepancy between WER
and speaker-attributed WER is only 1.0%, indicating accurate word-to-speaker
association. This increases marginally to 1.6% when 50% of the attendees are
unknown to the system.

本研究论文介绍了一种使用麦克风阵列和 360 度摄像头生成会议演讲者注释转录的系统，该系统能够处理重叠性语音并使用连续语音分离方法解决该问题。同时，还通过融合脸部跟踪和识别、声源定位、演讲者识别及先前演讲者信息的在线音频视觉演讲者日记化方法，提供了一个名为 SRD 的会议转录框架，并使用 11 名与会者的会议录音的实验结果表明，连续语音分离方法相对于高度调整的波束成形技术可将字词错误率降低 16.1％。当有完整的与会者名单时，字词错误率与演讲者归属字词错误率之间的差异仅为 1.0％，表明字词与演讲者之间的关联准确无误。当 50% 的与会者对系统不知情时，差异略有增加，为 1.6%。