Cross-modal contrastive learning has led the recent advances in multimodal
retrieval with its simplicity and effectiveness. In this work, however, we
reveal that cross-modal contrastive learning suffers from incorrect
normalization of the sum retrieval probabilities of each text or video
instance. Specifically, we show that many test instances are either over- or
under-represented during retrieval, significantly hurting the retrieval
performance. To address this problem, we propose Normalized Contrastive
Learning (NCL) which utilizes the Sinkhorn-Knopp algorithm to compute the
instance-wise biases that properly normalize the sum retrieval probabilities of
each instance so that every text and video instance is fairly represented
during cross-modal retrieval. Empirical study shows that NCL brings consistent
and significant gains in text-video retrieval on different model architectures,
with new state-of-the-art multimodal retrieval metrics on the ActivityNet,
MSVD, and MSR-VTT datasets without any architecture engineering.

本研究介绍了一个新的实例归一化方法，通过实例内置偏差计算，实现了跨模态检索中每个实例平等检索与准确表示，提高了文本和视频检索的性能，并在多个数据集上实现了最新的多模态检索指标。