This paper proposes the task of automatic assessment of Sentence Translation
Exercises (STEs), that have been used in the early stage of L2 language
learning. We formalize the task as grading student responses for each rubric
criterion pre-specified by the educators. We then create a dataset for STE
between Japanese and English including 21 questions, along with a total of 3,
498 student responses (167 on average). The answer responses were collected
from students and crowd workers. Using this dataset, we demonstrate the
performance of baselines including finetuned BERT and GPT models with few-shot
in-context learning. Experimental results show that the baseline model with
finetuned BERT was able to classify correct responses with approximately 90% in
F1, but only less than 80% for incorrect responses. Furthermore, the GPT models
with few-shot learning show poorer results than finetuned BERT, indicating that
our newly proposed task presents a challenging issue, even for the
stateof-the-art large language models.

自动评估句子翻译练习（STEs）的任务被提出，旨在为教育人员预设的每个评分标准对学生回答进行评分。使用一个包含 21 个问题和 3498 个学生回答的 STE 日语 - 英语数据集，作者展示了使用微调 BERT 和 GPT 模型的基线性能，结果表明微调 BERT 模型能以约 90% 的 F1 值对正确回答进行分类，但不到 80% 的不正确回答。此外，采用 few-shot 学习的 GPT 模型的结果较微调 BERT 差，表明我们提出的新任务对于最先进的大型语言模型来说是一个具有挑战性的问题。