Learning with noisy labels has become an effective strategy for enhancing the
robustness of models, which enables models to better tolerate inaccurate data.
Existing methods either focus on optimizing the loss function to mitigate the
interference from noise, or design procedures to detect potential noise and
correct errors. However, their effectiveness is often compromised in
representation learning due to the dilemma where models overfit to noisy
labels. To address this issue, this paper proposes a relation modeling and
distillation framework that models inter-sample relationships via
self-supervised learning and employs knowledge distillation to enhance
understanding of latent associations, which mitigate the impact of noisy
labels. Specifically, the proposed method, termed RMDNet, includes two main
modules, where the relation modeling (RM) module implements the contrastive
learning technique to learn representations of all data, an unsupervised
approach that effectively eliminates the interference of noisy tags on feature
extraction. The relation-guided representation learning (RGRL) module utilizes
inter-sample relation learned from the RM module to calibrate the
representation distribution for noisy samples, which is capable of improving
the generalization of the model in the inference phase. Notably, the proposed
RMDNet is a plug-and-play framework that can integrate multiple methods to its
advantage. Extensive experiments were conducted on two datasets, including
performance comparison, ablation study, in-depth analysis and case study. The
results show that RMDNet can learn discriminative representations for noisy
data, which results in superior performance than the existing methods.

通过自我监督学习建模样本之间的关系，并采用知识蒸馏的方法增强对潜在关联的理解，从而缓解模糊标签的影响，该研究提出了一个处理噪声标签的关系建模和蒸馏框架，该框架能够学习具有噪声的数据的判别性表示，并取得比现有方法更优越的性能。

关系建模与蒸馏用于带噪声标签的学习

Relation Modeling and Distillation for Learning with Noisy Labels

Text generation models are notoriously vulnerable to errors in the training
data. With the wide-spread availability of massive amounts of web-crawled data
becoming more commonplace, how can we enhance the robustness of models trained
on a massive amount of noisy web-crawled text? In our work, we propose Error
Norm Truncation (ENT), a robust enhancement method to the standard training
objective that truncates noisy data. Compared to methods that only uses the
negative log-likelihood loss to estimate data quality, our method provides a
more accurate estimation by considering the distribution of non-target tokens,
which is often overlooked by previous work. Through comprehensive experiments
across language modeling, machine translation, and text summarization, we show
that equipping text generation models with ENT improves generation quality over
standard training and previous soft and hard truncation methods. Furthermore,
we show that our method improves the robustness of models against two of the
most detrimental types of noise in machine translation, resulting in an
increase of more than 2 BLEU points over the MLE baseline when up to 50% of
noise is added to the data.

我们提出了一种名为 Error Norm Truncation (ENT) 的鲁棒性增强方法，通过截取噪声数据来更准确地评估数据质量，该方法考虑了非目标标记的分布，通过在语言建模、机器翻译和文本摘要等方面的综合实验证明，将 ENT 应用于文本生成模型可以提高生成质量并增强模型的鲁棒性。

误差范数截断：对文本生成模型进行数据噪声下的鲁棒训练

Error Norm Truncation: Robust Training in the Presence of Data Noise for  Text Generation Models

Recently, hashing is widely used in approximate nearest neighbor search for
its storage and computational efficiency. Most of the unsupervised hashing
methods learn to map images into semantic similarity-preserving hash codes by
constructing local semantic similarity structure from the pre-trained model as
the guiding information, i.e., treating each point pair similar if their
distance is small in feature space. However, due to the inefficient
representation ability of the pre-trained model, many false positives and
negatives in local semantic similarity will be introduced and lead to error
propagation during the hash code learning. Moreover, few of the methods
consider the robustness of models, which will cause instability of hash codes
to disturbance. In this paper, we propose a new method named
{\textbf{C}}omprehensive s{\textbf{I}}milarity {\textbf{M}}ining and
c{\textbf{O}}nsistency lear{\textbf{N}}ing (CIMON). First, we use global
refinement and similarity statistical distribution to obtain reliable and
smooth guidance. Second, both semantic and contrastive consistency learning are
introduced to derive both disturb-invariant and discriminative hash codes.
Extensive experiments on several benchmark datasets show that the proposed
method outperforms a wide range of state-of-the-art methods in both retrieval
performance and robustness.

本文提出了一种名为 CIMON 的新方法，通过全局细化和相似性统计分布获得可靠且平滑的指导，并引入语义和对比一致性学习来推导既具有扰动不变性又具有区分性的哈希码，通过在多个基准数据集上进行的大量实验证明，该提出的方法在检索性能和鲁棒性方面均优于一系列现有技术方法。