This report describes our approach for the Audio-Visual Diarization (AVD)
task of the Ego4D Challenge 2022. Specifically, we present multiple technical
improvements over the official baselines. First, we improve the detection
performance of the camera wearer's voice activity by modifying the training
scheme of its model. Second, we discover that an off-the-shelf voice activity
detection model can effectively remove false positives when it is applied
solely to the camera wearer's voice activities. Lastly, we show that better
active speaker detection leads to a better AVD outcome. Our final method
obtains 65.9% DER on the test set of Ego4D, which significantly outperforms all
the baselines. Our submission achieved 1st place in the Ego4D Challenge 2022.

报告中介绍了我们在 Ego4D Challenge 2022 中处理 Audio-Visual Diarization 任务的方法，该方法通过对模型的训练计划进行修改改进了声音活动的检测性能，证明了采用现成的语音活动检测模型可以有效地消除误报，而更好的主动说话者检测可提高 AVD 结果。我们的最终方法在 Ego4D 测试集上获得了 65.9% 的 DER，远远优于所有基线，在比赛中取得了第一名。

2022 年 Ego4D 挑战赛中英特尔实验室：音视频对话更优秀的基线

Intel Labs at Ego4D Challenge 2022: A Better Baseline for Audio-Visual  Diarization

We present a Conformer-based end-to-end neural diarization (EEND) model that
uses both acoustic input and features derived from an automatic speech
recognition (ASR) model. Two categories of features are explored: features
derived directly from ASR output (phones, position-in-word and word boundaries)
and features derived from a lexical speaker change detection model, trained by
fine-tuning a pretrained BERT model on the ASR output. Three modifications to
the Conformer-based EEND architecture are proposed to incorporate the features.
First, ASR features are concatenated with acoustic features. Second, we propose
a new attention mechanism called contextualized self-attention that utilizes
ASR features to build robust speaker representations. Finally, multi-task
learning is used to train the model to minimize classification loss for the ASR
features along with diarization loss. Experiments on the two-speaker English
conversations of Switchboard+SRE data sets show that multi-task learning with
position-in-word information is the most effective way of utilizing ASR
features, reducing the diarization error rate (DER) by 20% relative to the
baseline.

本文提出基于 Conformer 的端到端神经口头日记（EEND）模型，该模型利用从自动语音识别（ASR）模型导出的语音输入和特征。通过将 ASR 特征与声学特征相结合，采用新的自注意力机制来建立鲁棒的说话人表示，并使用多任务学习来最小化 ASR 特征的分类损失和日记化损失，将其应用于 Switchboard+SRE 数据集上的两个讲话人英语对话中，相对于基线，利用 ASR 特征的多任务学习最有效，在词汇位置信息下将 DER 降低了 20％。