Automatic evaluation metrics for generated texts play an important role in
the NLG field, especially with the rapid growth of LLMs. However, existing
metrics are often limited to specific scenarios, making it challenging to meet
the evaluation requirements of expanding LLM applications. Therefore, there is
a demand for new, flexible, and effective metrics. In this study, we introduce
RepEval, the first metric leveraging the projection of LLM representations for
evaluation. RepEval requires minimal sample pairs for training, and through
simple prompt modifications, it can easily transition to various tasks. Results
on ten datasets from three tasks demonstrate the high effectiveness of our
method, which exhibits stronger correlations with human judgments compared to
previous metrics, even outperforming GPT-4. Our work underscores the richness
of information regarding text quality embedded within LLM representations,
offering insights for the development of new metrics.

自动生成文本的自动评估指标在自然语言生成领域中起着重要作用，特别是随着大规模语言模型的快速发展。然而，现有的评估指标通常局限于特定情景，因此需要新的、灵活和有效的指标。本研究介绍了 RepEval，这是第一个利用 LLM 表示的投影进行评估的指标，通过简单的提示修改，能够轻松适应各种任务。在三个任务中的十个数据集上的结果表明我们方法的高效性，与以前的指标相比，甚至超过了 GPT-4，突显了 LLM 表示中嵌入的有关文本质量的丰富信息，为新指标的开发提供了洞见。

RepEval: LLM 表征的有效文本评估

RepEval: Effective Text Evaluation with LLM Representation

Generative language models, such as ChatGPT, have garnered attention for
their ability to generate human-like writing in various fields, including
academic research. The rapid proliferation of generated texts has bolstered the
need for automatic identification to uphold transparency and trust in the
information. However, these generated texts closely resemble human writing and
often have subtle differences in the grammatical structure, tones, and
patterns, which makes systematic scrutinization challenging. In this work, we
attempt to detect the Abstracts generated by ChatGPT, which are much shorter in
length and bounded. We extract the texts semantic and lexical properties and
observe that traditional machine learning models can confidently detect these
Abstracts.

ChatGPT 生成语言模型的研究中，自动生成的文本需要自动识别以保持信息的透明度和可信度。本研究中，我们试图检测 ChatGPT 生成的摘要，通过提取文本的语义和词汇特性，并观察到传统机器学习模型能够自信地检测这些摘要。