The recent success of Large Language Models (LLMs) has sparked concerns about
their potential to spread misinformation. As a result, there is a pressing need
for tools to identify ``fake arguments'' generated by such models. To create
these tools, examples of texts generated by LLMs are needed. This paper
introduces a methodology to obtain good, bad and ugly arguments from
argumentative essays produced by ChatGPT, OpenAI's LLM. We then describe a
novel dataset containing a set of diverse arguments, ArGPT. We assess the
effectiveness of our dataset and establish baselines for several
argumentation-related tasks. Finally, we show that the artificially generated
data relates well to human argumentation and thus is useful as a tool to train
and test systems for the defined tasks.

本论文介绍了一种从 ChatGPT 生成的论述性文章中提取好的、坏的和丑陋的论点的方法，并描述了一个包含多样化论证的新颖数据集 ArGPT。我们评估了数据集的有效性并为几个与论述相关的任务建立了基线。最后，我们展示人工生成的数据与人类论证之间的关联，证明其可以作为训练和测试系统所定义任务的工具。

评估 ChatGPT 生成的好、坏和丑陋的论证：一个新的数据集、方法论和相关任务

Assessing Good, Bad and Ugly Arguments Generated by ChatGPT: a New  Dataset, its Methodology and Associated Tasks

Background: Recently, ChatGPT and similar generative AI models have attracted
hundreds of millions of users and become part of the public discourse. Many
believe that such models will disrupt society and will result in a significant
change in the education system and information generation in the future. So
far, this belief is based on either colloquial evidence or benchmarks from the
owners of the models -- both lack scientific rigour.
Objective: Through a large-scale study comparing human-written versus
ChatGPT-generated argumentative student essays, we systematically assess the
quality of the AI-generated content.
Methods: A large corpus of essays was rated using standard criteria by a
large number of human experts (teachers). We augment the analysis with a
consideration of the linguistic characteristics of the generated essays.
Results: Our results demonstrate that ChatGPT generates essays that are rated
higher for quality than human-written essays. The writing style of the AI
models exhibits linguistic characteristics that are different from those of the
human-written essays, e.g., it is characterized by fewer discourse and
epistemic markers, but more nominalizations and greater lexical diversity.
Conclusions: Our results clearly demonstrate that models like ChatGPT
outperform humans in generating argumentative essays. Since the technology is
readily available for anyone to use, educators must act immediately. We must
re-invent homework and develop teaching concepts that utilize these AI models
in the same way as math utilized the calculator: teach the general concepts
first and then use AI tools to free up time for other learning objectives.

通过对比 ChatGPT 与人类写作的 argumentative 学生论文，我们系统地评估了 AI 生成内容的质量，结果表明 ChatGPT 生成的论文在质量上要高于人类写作。

人工智能和 ChatGPT 生成文章与人类写作的大规模比较

AI, write an essay for me: A large-scale comparison of human-written  versus ChatGPT-generated essays

This paper surveys and organizes research works in an under-studied area,
which we call automated evaluation for student argumentative writing. Unlike
traditional automated writing evaluation that focuses on holistic essay
scoring, this field is more specific: it focuses on evaluating argumentative
essays and offers specific feedback, including argumentation structures,
argument strength trait score, etc. The focused and detailed evaluation is
useful for helping students acquire important argumentation skill. In this
paper we organize existing works around tasks, data and methods. We further
experiment with BERT on representative datasets, aiming to provide up-to-date
baselines for this field.

这篇论文回顾和整理了一个被少有研究的领域 —— 自动化评估学生的议论写作。与传统的自动化写作评估侧重于整体的论文评分不同，这个领域更具体：它侧重于评估议论性文章，并提供特定的反馈，包括论证结构，论证实力特征分数等。这种聚焦和详细的评估对于帮助学生获得重要的论证技巧非常有用。在本文中，我们根据任务、数据和方法组织现有的工作。我们在代表性数据集上尝试 BERT，旨在为这个领域提供最新的基线。