Recently, contrastive learning has become a key component in fine-tuning code
search models for software development efficiency and effectiveness. It pulls
together positive code snippets while pushing negative samples away given
search queries. Among contrastive learning, InfoNCE is the most widely used
loss function due to its better performance. However, the following problems in
negative samples of InfoNCE may deteriorate its representation learning: 1) The
existence of false negative samples in large code corpora due to duplications.
2). The failure to explicitly differentiate between the potential relevance of
negative samples. As an example, a bubble sorting algorithm example is less
``negative'' than a file saving function for the quick sorting algorithm query.
In this paper, we tackle the above problems by proposing a simple yet effective
Soft-InfoNCE loss that inserts weight terms into InfoNCE. In our proposed loss
function, we apply three methods to estimate the weights of negative pairs and
show that the vanilla InfoNCE loss is a special case of Soft-InfoNCE.
Theoretically, we analyze the effects of Soft-InfoNCE on controlling the
distribution of learnt code representations and on deducing a more precise
mutual information estimation. We furthermore discuss the superiority of
proposed loss functions with other design alternatives. Extensive experiments
demonstrate the effectiveness of Soft-InfoNCE and weights estimation methods
under state-of-the-art code search models on a large-scale public dataset
consisting of six programming languages. Source code is available at
https://github.com/Alex-HaochenLi/Soft-InfoNCE.

在这篇论文中，我们通过提出一种简单但有效的 Soft-InfoNCE 损失函数，通过将权重项插入 InfoNCE 来解决负样本的问题。我们对负样本的权重进行了三种方法的估计，并展示了原始的 InfoNCE 损失是 Soft-InfoNCE 的一种特殊情况。理论上，我们分析了 Soft-InfoNCE 对学习到的代码表示分布的控制效果，并推导出更精确的互信息估计。此外，我们还讨论了提出的损失函数与其他设计替代方案的优越性。通过在一个包含六种编程语言的大规模公共数据集上进行广泛的实验证明了 Soft-InfoNCE 和权重估计方法在最先进的代码搜索模型中的有效性。