Cross-modal retrieval relies on well-matched large-scale datasets that are
laborious in practice. Recently, to alleviate expensive data collection,
co-occurring pairs from the Internet are automatically harvested for training.
However, it inevitably includes mismatched pairs, \ie, noisy correspondences,
undermining supervision reliability and degrading performance. Current methods
leverage deep neural networks' memorization effect to address noisy
correspondences, which overconfidently focus on \emph{similarity-guided
training with hard negatives} and suffer from self-reinforcing errors. In light
of above, we introduce a novel noisy correspondence learning framework, namely
\textbf{S}elf-\textbf{R}einforcing \textbf{E}rrors \textbf{M}itigation (SREM).
Specifically, by viewing sample matching as classification tasks within the
batch, we generate classification logits for the given sample. Instead of a
single similarity score, we refine sample filtration through energy uncertainty
and estimate model's sensitivity of selected clean samples using swapped
classification entropy, in view of the overall prediction distribution.
Additionally, we propose cross-modal biased complementary learning to leverage
negative matches overlooked in hard-negative training, further improving model
optimization stability and curbing self-reinforcing errors. Extensive
experiments on challenging benchmarks affirm the efficacy and efficiency of
SREM.

通过引入一种名为自我加强误差抑制（SREM）的新型噪声对应关系学习框架，通过将样本匹配视为批处理中的分类任务，我们生成给定样本的分类 logits，通过能量不确定性优化样本筛选，并使用交换分类熵估计所选干净样本的模型灵敏度，从而改善交叉模态检索的准确性和稳定性。同时，我们提出了交叉模态偏向性互补学习，以利用在难负例训练中被忽视的负匹配，进一步提高模型的优化稳定性和抑制自我加强错误。在具有挑战性的基准测试中进行的广泛实验证实了 SREM 的有效性和高效性。