Automatic methods for evaluating machine-generated texts hold significant
importance due to the expanding applications of generative systems.
Conventional methods tend to grapple with a lack of explainability, issuing a
solitary numerical score to signify the assessment outcome. Recent advancements
have sought to mitigate this limitation by incorporating large language models
(LLMs) to offer more detailed error analyses, yet their applicability remains
constrained, particularly in industrial contexts where comprehensive error
coverage and swift detection are paramount. To alleviate these challenges, we
introduce DEE, a Dual-stage Explainable Evaluation method for estimating the
quality of text generation. Built upon Llama 2, DEE follows a dual-stage
principle guided by stage-specific instructions to perform efficient
identification of errors in generated texts in the initial stage and
subsequently delves into providing comprehensive diagnostic reports in the
second stage. DEE is fine-tuned on our elaborately assembled dataset AntEval,
which encompasses 15K examples from 4 real-world applications of Alipay that
employ generative systems. The dataset concerns newly emerged issues like
hallucination and toxicity, thereby broadening the scope of DEE's evaluation
criteria. Experimental results affirm that DEE's superiority over existing
evaluation methods, achieving significant improvements in both human
correlation as well as efficiency.

DEE 是一个双阶段解释性评估方法，用于评估文本生成的质量，通过引入大型语言模型在生成文本中进行错误的识别和提供详细的诊断报告，优于现有的评估方法，并在人类相关性和效率方面取得显著改进。

DEE：用于文本生成的双阶段可解释评估方法

DEE: Dual-stage Explainable Evaluation Method for Text Generation

Large language models (LLMs) such as ChatGPT have exhibited remarkable
performance in generating human-like texts. However, machine-generated texts
(MGTs) may carry critical risks, such as plagiarism issues, misleading
information, or hallucination issues. Therefore, it is very urgent and
important to detect MGTs in many situations. Unfortunately, it is challenging
to distinguish MGTs and human-written texts because the distributional
discrepancy between them is often very subtle due to the remarkable performance
of LLMs. In this paper, we seek to exploit \textit{maximum mean discrepancy}
(MMD) to address this issue in the sense that MMD can well identify
distributional discrepancies. However, directly training a detector with MMD
using diverse MGTs will incur a significantly increased variance of MMD since
MGTs may contain \textit{multiple text populations} due to various LLMs. This
will severely impair MMD's ability to measure the difference between two
samples. To tackle this, we propose a novel \textit{multi-population} aware
optimization method for MMD called MMD-MP, which can \textit{avoid variance
increases} and thus improve the stability to measure the distributional
discrepancy. Relying on MMD-MP, we develop two methods for paragraph-based and
sentence-based detection, respectively. Extensive experiments on various LLMs,
\eg, GPT2 and ChatGPT, show superior detection performance of our MMD-MP. The
source code is available at https://github.com/ZSHsh98/MMD-MP.

利用最大均值差异（MMD）来检测生成文本中的多个文本人口，提高测量分布差异的稳定性。

通过多群体感知优化的最大均值差异检测机器生成的文本

Detecting Machine-Generated Texts by Multi-Population Aware Optimization  for Maximum Mean Discrepancy

SemEval-2024 Task 8 introduces the challenge of identifying machine-generated
texts from diverse Large Language Models (LLMs) in various languages and
domains. The task comprises three subtasks: binary classification in
monolingual and multilingual (Subtask A), multi-class classification (Subtask
B), and mixed text detection (Subtask C). This paper focuses on Subtask A & B.
Each subtask is supported by three datasets for training, development, and
testing. To tackle this task, two methods: 1) using traditional machine
learning (ML) with natural language preprocessing (NLP) for feature extraction,
and 2) fine-tuning LLMs for text classification. The results show that
transformer models, particularly LoRA-RoBERTa, exceed traditional ML methods in
effectiveness, with majority voting being particularly effective in
multilingual contexts for identifying machine-generated texts.

SemEval-2024 Task 8 引入了识别多语言和领域中大型语言模型（LLMs）生成的机器文本的挑战。本文关注于二元分类和多类分类的子任务，并通过传统机器学习和自然语言处理进行特征提取以及基于 LLMs 进行文本分类的两种方法进行解决，结果显示变压器模型，尤其是 LoRA-RoBERTa，在多语言环境中使用多数表决方法能够高效地识别机器生成的文本。

用于多生成器、多领域和多语言机器生成文本检测的细调大型语言模型

Fine-tuning Large Language Models for Multigenerator, Multidomain, and  Multilingual Machine-Generated Text Detection

Potential harms of Large Language Models such as mass misinformation and
plagiarism can be partially mitigated if there exists a reliable way to detect
machine generated text. In this paper, we propose a new watermarking method to
detect machine-generated texts. Our method embeds a unique pattern within the
generated text, ensuring that while the content remains coherent and natural to
human readers, it carries distinct markers that can be identified
algorithmically. Specifically, we intervene with the token sampling process in
a way which enables us to trace back our token choices during the detection
phase. We show how watermarking affects textual quality and compare our
proposed method with a state-of-the-art watermarking method in terms of
robustness and detectability. Through extensive experiments, we demonstrate the
effectiveness of our watermarking scheme in distinguishing between watermarked
and non-watermarked text, achieving high detection rates while maintaining
textual quality.

提出了一种新的水印方法来检测机器生成的文本，并通过与一种先进的水印方法进行了比较，证明了该方法在在保持文本质量的同时，能够有效地区分带水印和不带水印的文本，并取得了较高的检测率。