Our study explores how well the state-of-the-art Large Language Models
(LLMs), like GPT-4 and Mistral, can assess the quality of scientific summaries
or, more fittingly, scientific syntheses, comparing their evaluations to those
of human annotators. We used a dataset of 100 research questions and their
syntheses made by GPT-4 from abstracts of five related papers, checked against
human quality ratings. The study evaluates both the closed-source GPT-4 and the
open-source Mistral model's ability to rate these summaries and provide reasons
for their judgments. Preliminary results show that LLMs can offer logical
explanations that somewhat match the quality ratings, yet a deeper statistical
analysis shows a weak correlation between LLM and human ratings, suggesting the
potential and current limitations of LLMs in scientific synthesis evaluation.

研究探讨了大型语言模型（如 GPT-4 和 Mistral）如何评估科学摘要（或更适合称为科学综述）的质量，将它们的评估与人工标注者的评价进行比较。通过使用包含 100 个研究问题及其由 GPT-4 从五篇相关论文摘要生成的综述的数据集，并与人工质量评分进行核查，该研究评估了闭源的 GPT-4 和开源的 Mistral 模型评估这些摘要，并提供其判断的理由。初步结果显示，大型语言模型可以提供与质量评分相对应的逻辑解释，但更深入的统计分析显示大型语言模型与人工评分之间存在较弱的相关性，这表明了大型语言模型在科学综述评估中的潜力和当前的局限性。