Automatic evaluation of text generation is essential for improving the
accuracy of generation tasks. In light of the current trend towards
increasingly larger decoder-based language models, we investigate automatic
evaluation methods based on such models for text generation. This paper
compares various methods, including tuning with encoder-based models and large
language models under equal conditions, on two different tasks, machine
translation evaluation and semantic textual similarity, in two languages,
Japanese and English. Experimental results show that compared to the tuned
encoder-based models, the tuned decoder-based models perform poorly. The
analysis of the causes for this suggests that the decoder-based models focus on
surface word sequences and do not capture meaning. It is also revealed that
in-context learning of very large decoder-based models such as ChatGPT makes it
difficult to identify fine-grained semantic differences.

基于大型编码器和解码器语言模型的自动评估方法在文本生成任务中表现比调整后的编码器模型差，研究还发现解码器模型关注于表面字词序列而忽略了意义，同时发现调整后的解码器模型难以识别细粒度语义差异。