Evaluating the significance of a paper is pivotal yet challenging for the
scientific community. While the citation count is the most commonly used proxy
for this purpose, they are widely criticized for failing to accurately reflect
a paper's true impact. In this work, we propose a causal inference method,
TextMatch, which adapts the traditional matching framework to high-dimensional
text embeddings. Specifically, we encode each paper using the text embeddings
by large language models (LLMs), extract similar samples by cosine similarity,
and synthesize a counterfactual sample by the weighted average of similar
papers according to their similarity values. We apply the resulting metric,
called CausalCite, as a causal formulation of paper citations. We show its
effectiveness on various criteria, such as high correlation with paper impact
as reported by scientific experts on a previous dataset of 1K papers,
(test-of-time) awards for past papers, and its stability across various
sub-fields of AI. We also provide a set of findings that can serve as suggested
ways for future researchers to use our metric for a better understanding of a
paper's quality. Our code and data are at
this https URL

本研究提出了一种叫做 CausalCite 的因果推断方法，通过高维文本嵌入将每篇论文编码，利用余弦相似度提取相似的样本并综合计算出反事实样本，用于衡量论文被引用的因果关系，该方法在科学专家对 1,000 篇论文进行的实验、历史论文的奖项和 AI 各个子领域的稳定性等各种标准中表现出了很高的相关性和有效性，并给出了一些建议供未来研究者利用我们的度量方法以更好地理解论文质量。

CausalCite：论文引用的因果表述

CausalCite: A Causal Formulation of Paper Citations

Given the rapid ascent of large language models (LLMs), we study the
question: (How) can large language models help in reviewing of scientific
papers or proposals? We first conduct some pilot studies where we find that (i)
GPT-4 outperforms other LLMs (Bard, Vicuna, Koala, Alpaca, LLaMa, Dolly,
OpenAssistant, StableLM), and (ii) prompting with a specific question (e.g., to
identify errors) outperforms prompting to simply write a review. With these
insights, we study the use of LLMs (specifically, GPT-4) for three tasks:
1. Identifying errors: We construct 13 short computer science papers each
with a deliberately inserted error, and ask the LLM to check for the
correctness of these papers. We observe that the LLM finds errors in 7 of them,
spanning both mathematical and conceptual errors.
2. Verifying checklists: We task the LLM to verify 16 closed-ended checklist
questions in the respective sections of 15 NeurIPS 2022 papers. We find that
across 119 {checklist question, paper} pairs, the LLM had an 86.6% accuracy.
3. Choosing the "better" paper: We generate 10 pairs of abstracts,
deliberately designing each pair in such a way that one abstract was clearly
superior than the other. The LLM, however, struggled to discern these
relatively straightforward distinctions accurately, committing errors in its
evaluations for 6 out of the 10 pairs.
Based on these experiments, we think that LLMs have a promising use as
reviewing assistants for specific reviewing tasks, but not (yet) for complete
evaluations of papers or proposals.

使用 GPT-4 大型语言模型来辅助论文审核的研究发现其可以有效识别大部分错误，然而在挑选更好的论文时还存在一定误差。