Automatic evaluation of various text quality criteria produced by data-driven
intelligent methods is very common and useful because it is cheap, fast, and
usually yields repeatable results. In this paper, we present an attempt to
automate the human likeliness evaluation of the output text samples coming from
natural language generation methods used to solve several tasks. We propose to
use a human likeliness score that shows the percentage of the output samples
from a method that look as if they were written by a human. Instead of having
human participants label or rate those samples, we completely automate the
process by using a discrimination procedure based on large pretrained language
models and their probability distributions. As follow up, we plan to perform an
empirical analysis of human-written and machine-generated texts to find the
optimal setup of this evaluation approach. A validation procedure involving
human participants will also check how the automatic evaluation correlates with
human judgments.

本文提出了一种基于大型预训练语言模型和概率分布的区分程序来自动评估自然语言生成方法产生的文本样本的人类相似度分数，与人类判断的自动评估进行了验证。