As the importance of comprehensive evaluation in workshop courses increases,
there is a growing demand for efficient and fair assessment methods that reduce
the workload for faculty members. This paper presents an evaluation conducted
with Large Language Models (LLMs) using actual student essays in three
scenarios: 1) without providing guidance such as rubrics, 2) with pre-specified
rubrics, and 3) through pairwise comparison of essays. Quantitative analysis of
the results revealed a strong correlation between LLM and faculty member
assessments in the pairwise comparison scenario with pre-specified rubrics,
although concerns about the quality and stability of evaluations remained.
Therefore, we conducted a qualitative analysis of LLM assessment comments,
showing that: 1) LLMs can match the assessment capabilities of faculty members,
2) variations in LLM assessments should be interpreted as diversity rather than
confusion, and 3) assessments by humans and LLMs can differ and complement each
other. In conclusion, this paper suggests that LLMs should not be seen merely
as assistants to faculty members but as partners in evaluation committees and
outlines directions for further research.

本文通过三种场景下的评估：1）不提供指导，2）使用预先规定的评分标准，3）通过论文的两两对比，与实际学生论文一起使用大语言模型（LLM）进行评估，以降低教师的工作量。定量分析结果显示，使用预先规定的评分标准对 LLM 与教师评估之间存在强相关性，尽管存在有关评估质量和稳定性的担忧。因此，对 LLM 的评估意见进行了定性分析，结果表明：1）LLM 可以达到教师的评估能力，2）LLM 评估中的差异应解释为多样性而非混乱，3）人类和 LLM 的评估可以不同且相互补充。综上所述，本文建议将 LLM 视为教师评估委员会的合作伙伴，并为进一步研究提供了方向。

大型语言模型在学生论文评价中的应用

Large Language Models as Partners in Student Essay Evaluation

Workshop courses designed to foster creativity are gaining popularity.
However, achieving a holistic evaluation that accommodates diverse perspectives
is challenging, even for experienced faculty teams. Adequate discussion is
essential to integrate varied assessments, but faculty often lack the time for
such deliberations. Deriving an average score without discussion undermines the
purpose of a holistic evaluation. This paper explores the use of a Large
Language Model (LLM) as a facilitator to integrate diverse faculty assessments.
Scenario-based experiments were conducted to determine if the LLM could
synthesize diverse evaluations and explain the underlying theories to faculty.
The results were noteworthy, showing that the LLM effectively facilitated
faculty discussions. Additionally, the LLM demonstrated the capability to
generalize and create evaluation criteria from a single scenario based on its
learned domain knowledge.

探索利用大型语言模型（LLM）作为促进多样评估的工具，实验表明 LLM 有效地促进师资讨论，具备从单一场景中泛化和创建评估标准的能力。