Although several methods were proposed to address the problem of automated
essay scoring (AES) in the last 50 years, there is still much to desire in
terms of effectiveness. Large Language Models (LLMs) are transformer-based
models that demonstrate extraordinary capabilities on various tasks. In this
paper, we test the ability of LLMs, given their powerful linguistic knowledge,
to analyze and effectively score written essays. We experimented with two
popular LLMs, namely ChatGPT and Llama. We aim to check if these models can do
this task and, if so, how their performance is positioned among the
state-of-the-art (SOTA) models across two levels, holistically and per
individual writing trait. We utilized prompt-engineering tactics in designing
four different prompts to bring their maximum potential to this task. Our
experiments conducted on the ASAP dataset revealed several interesting
observations. First, choosing the right prompt depends highly on the model and
nature of the task. Second, the two LLMs exhibited comparable average
performance in AES, with a slight advantage for ChatGPT. Finally, despite the
performance gap between the two LLMs and SOTA models in terms of predictions,
they provide feedback to enhance the quality of the essays, which can
potentially help both teachers and students.

利用大型语言模型（LLMs）对写作文章进行自动评分的实验表明，虽然适当选择题目对任务和模型性质很重要，但 ChatGPT 相比 Llama 在综合和个体写作特征上的性能稍优。尽管与 SOTA 模型相比预测存在差距，但它们提供了改善文章质量的反馈，为教师和学生提供帮助。