Evaluating the significance of a paper is pivotal yet challenging for the
scientific community. While the citation count is the most commonly used proxy
for this purpose, they are widely criticized for failing to accurately reflect
a paper's true impact. In this work, we propose a causal inference method,
TextMatch, which adapts the traditional matching framework to high-dimensional
text embeddings. Specifically, we encode each paper using the text embeddings
by large language models (LLMs), extract similar samples by cosine similarity,
and synthesize a counterfactual sample by the weighted average of similar
papers according to their similarity values. We apply the resulting metric,
called CausalCite, as a causal formulation of paper citations. We show its
effectiveness on various criteria, such as high correlation with paper impact
as reported by scientific experts on a previous dataset of 1K papers,
(test-of-time) awards for past papers, and its stability across various
sub-fields of AI. We also provide a set of findings that can serve as suggested
ways for future researchers to use our metric for a better understanding of a
paper's quality. Our code and data are at
this https URL

本研究提出了一种叫做 CausalCite 的因果推断方法，通过高维文本嵌入将每篇论文编码，利用余弦相似度提取相似的样本并综合计算出反事实样本，用于衡量论文被引用的因果关系，该方法在科学专家对 1,000 篇论文进行的实验、历史论文的奖项和 AI 各个子领域的稳定性等各种标准中表现出了很高的相关性和有效性，并给出了一些建议供未来研究者利用我们的度量方法以更好地理解论文质量。