Recent progress in generative language models has enabled machines to
generate astonishingly realistic texts. While there are many legitimate
applications of such models, there is also a rising need to distinguish
machine-generated texts from human-written ones (e.g., fake news detection).
However, to our best knowledge, there is currently no benchmark environment
with datasets and tasks to systematically study the so-called "Turing Test"
problem for neural text generation methods. In this work, we present the
TuringBench benchmark environment, which is comprised of (1) a dataset with
200K human- or machine-generated samples across 20 labels {Human, GPT-1,
GPT-2_small, GPT-2_medium, GPT-2_large, GPT-2_xl, GPT-2_PyTorch, GPT-3,
GROVER_base, GROVER_large, GROVER_mega, CTRL, XLM, XLNET_base, XLNET_large,
FAIR_wmt19, FAIR_wmt20, TRANSFORMER_XL, PPLM_distil, PPLM_gpt2}, (2) two
benchmark tasks -- i.e., Turing Test (TT) and Authorship Attribution (AA), and
(3) a website with leaderboards. Our preliminary experimental results using
TuringBench show that FAIR_wmt20 and GPT-3 are the current winners, among all
language models tested, in generating the most human-like indistinguishable
texts with the lowest F1 score by five state-of-the-art TT detection models.
The TuringBench is available at: this https URL

该研究提出了 TuringBench 基准环境，旨在解决神经文本生成方法的 “图灵测试” 问题，它包括 200K 个人工或机器生成的样本数据集，分别涵盖 20 个标签，以及两个基准测试任务和网站排行榜，研究初步实验表明，FAIR_wmt20 和 GPT-3 是生成最逼近人类无法辨别的文本的最佳选择。

TURINGBENCH: 面向神经文本生成时代的图灵测试基准环境

TURINGBENCH: A Benchmark Environment for Turing Test in the Age of  Neural Text Generation

Automatically generated fake restaurant reviews are a threat to online review
systems. Recent research has shown that users have difficulties in detecting
machine-generated fake reviews hiding among real restaurant reviews. The method
used in this work (char-LSTM ) has one drawback: it has difficulties staying in
context, i.e. when it generates a review for specific target entity, the
resulting review may contain phrases that are unrelated to the target, thus
increasing its detectability. In this work, we present and evaluate a more
sophisticated technique based on neural machine translation (NMT) with which we
can generate reviews that stay on-topic. We test multiple variants of our
technique using native English speakers on Amazon Mechanical Turk. We
demonstrate that reviews generated by the best variant have almost optimal
undetectability (class-averaged F-score 47%). We conduct a user study with
skeptical users and show that our method evades detection more frequently
compared to the state-of-the-art (average evasion 3.2/4 vs 1.5/4) with
statistical significance, at level {\alpha} = 1% (Section 4.3). We develop very
effective detection tools and reach average F-score of 97% in classifying
these. Although fake reviews are very effective in fooling people, effective
automatic detection is still feasible.

通过神经机器翻译技术，本文提出并评估了一种新的方法，用于生成保持主题一致的餐厅评价，测试表明该方法具有极高的检测难度，同时本研究设计了高效的检测工具，证明虚假评论的自动检测仍然可行。