The heterogeneity gap problem is the main challenge in cross-modal retrieval.
Because cross-modal data (e.g. audiovisual) have different distributions and
representations that cannot be directly compared. To bridge the gap between
audiovisual modalities, we learn a common subspace for them by utilizing the
intrinsic correlation in the natural synchronization of audio-visual data with
the aid of annotated labels. TNN-CCCA is the best audio-visual cross-modal
retrieval (AV-CMR) model so far, but the model training is sensitive to hard
negative samples when learning common subspace by applying triplet loss to
predict the relative distance between inputs. In this paper, to reduce the
interference of hard negative samples in representation learning, we propose a
new AV-CMR model to optimize semantic features by directly predicting labels
and then measuring the intrinsic correlation between audio-visual data using
complete cross-triple loss. In particular, our model projects audio-visual
features into label space by minimizing the distance between predicted label
features after feature projection and ground label representations. Moreover,
we adopt complete cross-triplet loss to optimize the predicted label features
by leveraging the relationship between all possible similarity and
dissimilarity semantic information across modalities. The extensive
experimental results on two audio-visual double-checked datasets have shown an
improvement of approximately 2.1% in terms of average MAP over the current
state-of-the-art method TNN-CCCA for the AV-CMR task, which indicates the
effectiveness of our proposed model.

本文提出了一种新的 AV-CMR 模型，通过直接预测标签并使用完整的交叉三元组损失来优化语义特征，从而优化音频 - 视觉数据之间的内在相关性，解决了模型训练敏感性问题和选取困难问题。在两个音频 - 视觉双重检查数据集上的广泛实验结果表明，与现有的 TNN-CCCA 方法相比，平均 MAP 提高了约 2.1％，验证了我们提出的模型的有效性。