Person or identity verification has been recently gaining a lot of attention
using audio-visual fusion as faces and voices share close associations with
each other. Conventional approaches based on audio-visual fusion rely on
score-level or early feature-level fusion techniques. Though existing
approaches showed improvement over unimodal systems, the potential of
audio-visual fusion for person verification is not fully exploited. In this
paper, we have investigated the prospect of effectively capturing both the
intra- and inter-modal relationships across audio and visual modalities, which
can play a crucial role in significantly improving the fusion performance over
unimodal systems. In particular, we introduce a recursive fusion of a joint
cross-attentional model, where a joint audio-visual feature representation is
employed in the cross-attention framework in a recursive fashion to
progressively refine the feature representations that can efficiently capture
the intra-and inter-modal relationships. To further enhance the audio-visual
feature representations, we have also explored BLSTMs to improve the temporal
modeling of audio-visual feature representations. Extensive experiments are
conducted on the Voxceleb1 dataset to evaluate the proposed model. Results
indicate that the proposed model shows promising improvement in fusion
performance by adeptly capturing the intra-and inter-modal relationships across
audio and visual modalities.

通过递归融合的联合交叉关注模型和 BLSTMs，本研究有效地捕捉音频和视觉模态之间的内在和跨模态关系，显著提高了融合性能。