Person or identity verification has been recently gaining a lot of attention
using audio-visual fusion as faces and voices share close associations with
each other. Conventional approaches based on audio-visual fusion rely on
score-level or early feature-level fusion techniques. Though existing
approaches showed improvement over unimodal systems, the potential of
audio-visual fusion for person verification is not fully exploited. In this
paper, we have investigated the prospect of effectively capturing both the
intra- and inter-modal relationships across audio and visual modalities, which
can play a crucial role in significantly improving the fusion performance over
unimodal systems. In particular, we introduce a recursive fusion of a joint
cross-attentional model, where a joint audio-visual feature representation is
employed in the cross-attention framework in a recursive fashion to
progressively refine the feature representations that can efficiently capture
the intra-and inter-modal relationships. To further enhance the audio-visual
feature representations, we have also explored BLSTMs to improve the temporal
modeling of audio-visual feature representations. Extensive experiments are
conducted on the Voxceleb1 dataset to evaluate the proposed model. Results
indicate that the proposed model shows promising improvement in fusion
performance by adeptly capturing the intra-and inter-modal relationships across
audio and visual modalities.

通过递归融合的联合交叉关注模型和 BLSTMs，本研究有效地捕捉音频和视觉模态之间的内在和跨模态关系，显著提高了融合性能。

基于递归融合的联合交叉注意力的音视频人员验证

Audio-Visual Person Verification based on Recursive Fusion of Joint  Cross-Attention

Although person or identity verification has been predominantly explored
using individual modalities such as face and voice, audio-visual fusion has
recently shown immense potential to outperform unimodal approaches. Audio and
visual modalities are often expected to pose strong complementary
relationships, which plays a crucial role in effective audio-visual fusion.
However, they may not always strongly complement each other, they may also
exhibit weak complementary relationships, resulting in poor audio-visual
feature representations. In this paper, we propose a Dynamic Cross-Attention
(DCA) model that can dynamically select the cross-attended or unattended
features on the fly based on the strong or weak complementary relationships,
respectively, across audio and visual modalities. In particular, a conditional
gating layer is designed to evaluate the contribution of the cross-attention
mechanism and choose cross-attended features only when they exhibit strong
complementary relationships, otherwise unattended features. Extensive
experiments are conducted on the Voxceleb1 dataset to demonstrate the
robustness of the proposed model. Results indicate that the proposed model
consistently improves the performance on multiple variants of cross-attention
while outperforming the state-of-the-art methods.

通过动态交叉注意力（DCA）模型，本文提出了一种能够根据音频和视觉模态之间的强弱互补关系动态选择跨模态关注或不关注特性的模型。实验证明该模型在多个跨模态注意力变体上表现出稳健性，同时优于现有方法的性能。

音频 - 视觉个人验证的动态交叉注意力

Dynamic Cross Attention for Audio-Visual Person Verification

The past decade has witnessed the rapid development of feature representation
learning and distance metric learning, whereas the two steps are often
discussed separately. To explore their interaction, this work proposes an
end-to-end learning framework called DARI, i.e. Distance metric And
Representation Integration, and validates the effectiveness of DARI in the
challenging task of person verification. Given the training images annotated
with the labels, we first produce a large number of triplet units, and each one
contains three images, i.e. one person and the matched/mismatch references. For
each triplet unit, the distance disparity between the matched pair and the
mismatched pair tends to be maximized. We solve this objective by building a
deep architecture of convolutional neural networks. In particular, the
Mahalanobis distance matrix is naturally factorized as one top fully-connected
layer that is seamlessly integrated with other bottom layers representing the
image feature. The image feature and the distance metric can be thus
simultaneously optimized via the one-shot backward propagation. On several
public datasets, DARI shows very promising performance on re-identifying
individuals cross cameras against various challenges, and outperforms other
state-of-the-art approaches.

本文提出了名为 DARI 的端到端学习框架，其中将距离度量和特征表示融合在一起。通过使用深度卷积神经网络来解决距离不匹配问题，证明了 DARI 在人员验证等方面的有效性，并在多个公共数据集中实现了最佳性能。