In this paper, we present a methodology for achieving robust multimodal
person representations optimized for open-set audio-visual speaker
verification. Distance Metric Learning (DML) approaches have typically
dominated this problem space, owing to strong performance on new and unseen
classes. In our work, we explored multitask learning techniques to further
boost performance of the DML approach and show that an auxiliary task with weak
labels can increase the compactness of the learned speaker representation. We
also extend the Generalized end-to-end loss (GE2E) to multimodal inputs and
demonstrate that it can achieve competitive performance in an audio-visual
space. Finally, we introduce a non-synchronous audio-visual sampling random
strategy during training time that has shown to improve generalization. Our
network achieves state of the art performance for speaker verification,
reporting 0.244%, 0.252%, 0.441% Equal Error Rate (EER) on the three official
trial lists of VoxCeleb1-O/E/H, which is to our knowledge, the best published
results on VoxCeleb1-E and VoxCeleb1-H.

本文介绍了一种实现鲁棒的多模式人物表示以优化开放式音视频说话人验证的方法。通过探索多任务学习技术，我们提高了距离度量学习方法的性能，并展示了一个弱标签的辅助任务可以增加学习到的说话人表示的紧凑性。同时，我们将广义端到端损失（GE2E）扩展到多模态输入，并证明其在音视频领域可以达到竞争性的性能。最后，我们引入了一种非同步音视频采样的随机策略，该策略在训练时能改善泛化效果。我们的网络在说话人验证方面达到了最先进的水平，并在 VoxCeleb1-O/E/H 的三个官方试验列表中报告了 0.244%、0.252%、0.441% 的等错误率（EER），据我们所知，这是 VoxCeleb1-E 和 VoxCeleb1-H 的最佳已发表结果。