Reinforcement learning from human feedback (RLHF) has been a central
technique for recent large language model (LLM) alignment. However, its heavy
dependence on costly human or LLM-as-Judge preference feedback could stymie its
wider applications. In this work, we introduce Self-Contrast, a feedback-free
large language model alignment method via exploiting extensive self-generated
negatives. With only supervised fine-tuning (SFT) targets, Self-Contrast
leverages the LLM itself to generate massive diverse candidates, and harnesses
a pre-trained embedding model to filter multiple negatives according to text
similarity. Theoretically, we illustrate that in this setting, merely scaling
negative responses can still effectively approximate situations with more
balanced positive and negative preference annotations. Our experiments with
direct preference optimization (DPO) on three datasets show that, Self-Contrast
could consistently outperform SFT and standard DPO training by large margins.
And as the number of self-generated negatives increases, the performance of
Self-Contrast continues to grow. Code and data are available at
this https URL

通过利用自动生成的负例，自我对比是一种无需依赖人类反馈的大型语言模型对齐方法，仅通过有监督的微调目标，利用语言模型本身生成大量多样化的候选，并根据文本相似性使用预训练的嵌入模型筛选多个负例，实验证明在此设置下，仅通过缩放负响应仍可以有效地近似具有更平衡的正面和负面偏好注释的情况，通过对三个数据集的直接偏好优化实验表明，自我对比可以始终显著优于有监督微调和标准偏好优化训练，当自生成负例的数量增加时，自我对比的性能也在不断提高。

广泛的自对比使得无需反馈的语言模型对准

Extensive Self-Contrast Enables Feedback-Free Language Model Alignment

The reflection capacity of Large Language Model (LLM) has garnered extensive
attention. A post-hoc prompting strategy, e.g., reflexion and self-refine,
refines LLM's response based on self-evaluated or external feedback. However,
recent research indicates without external feedback, LLM's intrinsic reflection
is unstable. Our investigation unveils that the key bottleneck is the quality
of the self-evaluated feedback. We find LLMs often exhibit overconfidence or
high randomness when self-evaluate, offering stubborn or inconsistent feedback,
which causes poor reflection. To remedy this, we advocate Self-Contrast: It
adaptively explores diverse solving perspectives tailored to the request,
contrasts the differences, and summarizes these discrepancies into a checklist
which could be used to re-examine and eliminate discrepancies. Our method
endows LLM with diverse perspectives to alleviate stubborn biases. Moreover,
their discrepancies indicate potential errors or inherent uncertainties that
LLM often overlooks. Reflecting upon these can catalyze more accurate and
stable reflection. Experiments conducted on a series of reasoning and
translation tasks with different LLMs serve to underscore the effectiveness and
generality of our strategy.

我们提出了一种基于自对照的策略，通过适应性地探索多样的解决观点、对比差异并总结出这些差异的清单，为大型语言模型增添了多样的视角，以减轻固执偏见，并且更准确和稳定地进行反思。

自我对比：更好的反射通过不一致的解决视角

Self-Contrast: Better Reflection Through Inconsistent Solving  Perspectives

Graph contrastive learning (GCL) has recently emerged as a promising approach
for graph representation learning. Some existing methods adopt the 1-vs-K
scheme to construct one positive and K negative samples for each graph, but it
is difficult to set K. For those methods that do not use negative samples, it
is often necessary to add additional strategies to avoid model collapse, which
could only alleviate the problem to some extent. All these drawbacks will
undoubtedly have an adverse impact on the generalizability and efficiency of
the model. In this paper, to address these issues, we propose a novel graph
self-contrast framework GraphSC, which only uses one positive and one negative
sample, and chooses triplet loss as the objective. Specifically, self-contrast
has two implications. First, GraphSC generates both positive and negative views
of a graph sample from the graph itself via graph augmentation functions of
various intensities, and use them for self-contrast. Second, GraphSC uses
Hilbert-Schmidt Independence Criterion (HSIC) to factorize the representations
into multiple factors and proposes a masked self-contrast mechanism to better
separate positive and negative samples. Further, Since the triplet loss only
optimizes the relative distance between the anchor and its positive/negative
samples, it is difficult to ensure the absolute distance between the anchor and
positive sample. Therefore, we explicitly reduced the absolute distance between
the anchor and positive sample to accelerate convergence. Finally, we conduct
extensive experiments to evaluate the performance of GraphSC against 19 other
state-of-the-art methods in both unsupervised and transfer learning settings.

本文提出了一种名为 GraphSC 的新型图自我对比框架，它使用图自身通过图增强函数生成正负样本，并使用 Hilbert-Schmidt 独立准则对表示进行分解并引入掩蔽自我对比机制以更好地区分正负样本，同时通过显式减小锚点和正样本之间的绝对距离来加快收敛，并在无监督和迁移学习设置下与其他 19 种最先进方法进行了广泛实验评估。