Large Language Models (LLMs) have demonstrated wide-ranging applications
across various fields and have shown significant potential in the academic
peer-review process. However, existing applications are primarily limited to
static review generation based on submitted papers, which fail to capture the
dynamic and iterative nature of real-world peer reviews. In this paper, we
reformulate the peer-review process as a multi-turn, long-context dialogue,
incorporating distinct roles for authors, reviewers, and decision makers. We
construct a comprehensive dataset containing over 26,841 papers with 92,017
reviews collected from multiple sources, including the top-tier conference and
prestigious journal. This dataset is meticulously designed to facilitate the
applications of LLMs for multi-turn dialogues, effectively simulating the
complete peer-review process. Furthermore, we propose a series of metrics to
evaluate the performance of LLMs for each role under this reformulated
peer-review setting, ensuring fair and comprehensive evaluations. We believe
this work provides a promising perspective on enhancing the LLM-driven
peer-review process by incorporating dynamic, role-based interactions. It
aligns closely with the iterative and interactive nature of real-world academic
peer review, offering a robust foundation for future research and development
in this area. We open-source the dataset at
this https URL

我们提出了将同行评审过程重新定义为多轮长上下文对话，并在这个重新定义的同行评审框架下开发了一系列用于评估大型语言模型性能的指标，以适应真实世界中的同行评审的迭代和互动特点。

使用基于角色的互动进行多轮和长上下文的同行评审

Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based  Interactions

In this pilot study, we investigate the use of GPT4 to assist in the
peer-review process. Our key hypothesis was that GPT-generated reviews could
achieve comparable helpfulness to human reviewers. By comparing reviews
generated by both human reviewers and GPT models for academic papers submitted
to a major machine learning conference, we provide initial evidence that
artificial intelligence can contribute effectively to the peer-review process.
We also perform robustness experiments with inserted errors to understand which
parts of the paper the model tends to focus on. Our findings open new avenues
for leveraging machine learning tools to address resource constraints in peer
review. The results also shed light on potential enhancements to the review
process and lay the groundwork for further research on scaling oversight in a
domain where human-feedback is increasingly a scarce resource.

本研究旨在探究利用 GPT4 模型协助同行评审过程的可行性，并发现人工智能可以对同行评审过程做出有效的贡献，为解决同行评审中资源限制问题提供了新途径。

GPT4 对同行评审辅助略有帮助：一项试点研究

GPT4 is Slightly Helpful for Peer-Review Assistance: A Pilot Study

Neural Information Processing Systems (NIPS) is a top-tier annual conference
in machine learning. The 2016 edition of the conference comprised more than
2,400 paper submissions, 3,000 reviewers, and 8,000 attendees. This represents
a growth of nearly 40% in terms of submissions, 96% in terms of reviewers, and
over 100% in terms of attendees as compared to the previous year. The massive
scale as well as rapid growth of the conference calls for a thorough quality
assessment of the peer-review process and novel means of improvement. In this
paper, we analyze several aspects of the data collected during the review
process, including an experiment investigating the efficacy of collecting
ordinal rankings from reviewers. Our goal is to check the soundness of the
review process, and provide insights that may be useful in the design of the
review process of subsequent conferences.

本文分析了 Neural Information Processing Systems 会议上关于机器学习的论文审查数据，并测试了从审稿人收集序数排名的效用，旨在检查审查流程的严谨性并提供对后续会议审查流程设计有用的见解。