We present a novel platform for evaluating the capability of Large Language
Models (LLMs) to autonomously compose and critique survey papers spanning a
vast array of disciplines including sciences, humanities, education, and law.
Within this framework, AI systems undertake a simulated peer-review mechanism
akin to traditional scholarly journals, with human organizers serving in an
editorial oversight capacity. Within this framework, we organized a competition
for the AutoML conference 2023. Entrants are tasked with presenting stand-alone
models adept at authoring articles from designated prompts and subsequently
appraising them. Assessment criteria include clarity, reference
appropriateness, accountability, and the substantive value of the content. This
paper presents the design of the competition, including the implementation
baseline submissions and methods of evaluation.

我们提出了一种新颖的平台，用于评估大型语言模型（LLMs）自主撰写和评论跨科学、人文、教育和法律等各个学科的调研论文的能力。这个框架中，人工智能系统通过模拟同行评审机制进行操作，类似于传统学术期刊，人类组织者担任编辑监督职责。在这个框架内，我们为 2023 年 AutoML 会议组织了一次竞赛。参赛者的任务是根据指定提示撰写独立的文章，并对其进行评估。评估标准包括清晰度、参考文献适当性、责任性和内容的实质价值。本文介绍了竞赛的设计，包括实施基线提交和评估方法。