Expert feedback lays the foundation of rigorous research. However, the rapid
growth of scholarly production and intricate knowledge specialization challenge
the conventional scientific feedback mechanisms. High-quality peer reviews are
increasingly difficult to obtain. Researchers who are more junior or from
under-resourced settings have especially hard times getting timely feedback.
With the breakthrough of large language models (LLM) such as GPT-4, there is
growing interest in using LLMs to generate scientific feedback on research
manuscripts. However, the utility of LLM-generated feedback has not been
systematically studied. To address this gap, we created an automated pipeline
using GPT-4 to provide comments on the full PDFs of scientific papers. We
evaluated the quality of GPT-4's feedback through two large-scale studies. We
first quantitatively compared GPT-4's generated feedback with human peer
reviewer feedback in 15 Nature family journals (3,096 papers in total) and the
ICLR machine learning conference (1,709 papers). The overlap in the points
raised by GPT-4 and by human reviewers (average overlap 30.85% for Nature
journals, 39.23% for ICLR) is comparable to the overlap between two human
reviewers (average overlap 28.58% for Nature journals, 35.25% for ICLR). The
overlap between GPT-4 and human reviewers is larger for the weaker papers. We
then conducted a prospective user study with 308 researchers from 110 US
institutions in the field of AI and computational biology to understand how
researchers perceive feedback generated by our GPT-4 system on their own
papers. Overall, more than half (57.4%) of the users found GPT-4 generated
feedback helpful/very helpful and 82.4% found it more beneficial than feedback
from at least some human reviewers. While our findings show that LLM-generated
feedback can help researchers, we also identify several limitations.

使用 GPT-4 模型创建了一个自动化平台，通过对科学论文的全文进行评论以评估 GPT-4 生成的反馈质量。 在两项大规模研究中，我们将 GPT-4 生成的反馈与人工同行评审的反馈进行了定量比较，并通过 308 名研究人员的用户研究了解了他们对 GPT-4 生成的反馈的感知。 总体而言，超过半数的用户（57.4%）认为 GPT-4 生成的反馈有所帮助，82.4% 的用户认为它比至少一些人工审稿人的反馈更有益。

大型语言模型能否给研究论文提供有用反馈？一个大规模实证分析

Can large language models provide useful feedback on research papers? A  large-scale empirical analysis

We present PeerSum, a new MDS dataset using peer reviews of scientific
publications. Our dataset differs from the existing MDS datasets in that our
summaries (i.e., the meta-reviews) are highly abstractive and they are real
summaries of the source documents (i.e., the reviews) and it also features
disagreements among source documents. We found that current state-of-the-art
MDS models struggle to generate high-quality summaries for PeerSum, offering
new research opportunities.

本文介绍了 PeerSum，这是一个使用科学出版物的同行评审的新 MDS 数据集。与现有的 MDS 数据集不同，我们的摘要（即元评审）具有高度抽象性，是源文件的真实摘要（即评论），并且还包括源文件之间的争议。我们发现，目前最先进的 MDS 模型在生成 PeerSum 的高质量摘要方面存在困难，提供了新的研究机会。

PeerSum：用于抽象多文档摘要的同行评审数据集

PeerSum: A Peer Review Dataset for Abstractive Multi-document Summarization

Scientific papers are complex and understanding the usefulness of these
papers requires prior knowledge. Peer reviews are comments on a paper provided
by designated experts on that field and hold a substantial amount of
information, not only for the editors and chairs to make the final decision,
but also to judge the potential impact of the paper. In this paper, we propose
to use aspect-based sentiment analysis of scientific reviews to be able to
extract useful information, which correlates well with the accept/reject
decision.
While working on a dataset of close to 8k reviews from ICLR, one of the top
conferences in the field of machine learning, we use an active learning
framework to build a training dataset for aspect prediction, which is further
used to obtain the aspects and sentiments for the entire dataset. We show that
the distribution of aspect-based sentiments obtained from a review is
significantly different for accepted and rejected papers. We use the aspect
sentiments from these reviews to make an intriguing observation, certain
aspects present in a paper and discussed in the review strongly determine the
final recommendation. As a second objective, we quantify the extent of
disagreement among the reviewers refereeing a paper. We also investigate the
extent of disagreement between the reviewers and the chair and find that the
inter-reviewer disagreement may have a link to the disagreement with the chair.
One of the most interesting observations from this study is that reviews, where
the reviewer score and the aspect sentiments extracted from the review text
written by the reviewer are consistent, are also more likely to be concurrent
with the chair's decision.

利用基于方面的情感分析从科学评论中提取有用信息可以更好地判断接受 / 拒绝决策，并且发现审稿人之间的不一致程度可能与主席的意见不一致有关。