Learning scientific document representations can be substantially improved
through contrastive learning objectives, where the challenge lies in creating
positive and negative training samples that encode the desired similarity
semantics. Prior work relies on discrete citation relations to generate
contrast samples. However, discrete citations enforce a hard cut-off to
similarity. This is counter-intuitive to similarity-based learning, and ignores
that scientific papers can be very similar despite lacking a direct citation -
a core problem of finding related research. Instead, we use controlled nearest
neighbor sampling over citation graph embeddings for contrastive learning. This
control allows us to learn continuous similarity, to sample hard-to-learn
negatives and positives, and also to avoid collisions between negative and
positive samples by controlling the sampling margin between them. The resulting
method SciNCL outperforms the state-of-the-art on the SciDocs benchmark.
Furthermore, we demonstrate that it can train (or tune) models
sample-efficiently, and that it can be combined with recent training-efficient
methods. Perhaps surprisingly, even training a general-domain language model
this way outperforms baselines pretrained in-domain.

本文提出了一种通过对比学习来改善科学文档表示的方法 —— 使用引文图嵌入进行所控制的最近邻抽样来学习连续的相似度，内容主要涉及对比学习、科学文档表示、引文图嵌入、相似度语义以及相关研究。