Multi-modal fusion is proven to be an effective method to improve the accuracy and robustness of speaker tracking, especially in complex scenarios. However, how to combine the heterogeneous information and exploit the complementarity of multi-modal signals remains a challenging issue. In this paper, we propose a novel Multi-modal Perception Tracker (MPT) for speaker tracking using both audio and visual modalities. Specifically, a novel acoustic map based on spatial-temporal Global Coherence Field (stGCF) is first constructed for heterogeneous signal fusion, which employs a camera model to map audio cues to the localization space consistent with the visual cues. Then a multi-modal perception attention network is introduced to derive the perception weights that measure the reliability and effectiveness of intermittent audio and video streams disturbed by noise. Moreover, a unique cross-modal self-supervised learning method is presented to model the confidence of audio and visual observations by leveraging the complementarity and consistency between different modalities. Experimental results show that the proposed MPT achieves 98.6% and 78.3% tracking accuracy on the standard and occluded datasets, respectively, which demonstrates its robustness under adverse conditions and outperforms the current state-of-the-art methods.

本文提出了一种利用声音和视觉模态进行讲话者跟踪的多模态感知跟踪器（MPT），其中包括使用基于空时全局相干字段（stGCF）的声学地图进行异构信号融合，引入多模态感知注意力网络来导出可靠性和效益的知觉权重，以及使用跨模态自我监督学习方法模拟不同模态之间的互补性和一致性。实验结果表明，所提出的 MPT 在标准数据集和遮挡数据集上分别达到了 98.6% 和 78.3% 的跟踪精度，证明了其在不利条件下的鲁棒性优于目前的最新技术。

带自监督学习的多模态感知注意力网络用于音视说话者追踪