Large Language Models (LLMs) are increasingly used as "content farm" models
(CFMs), to generate synthetic text that could pass for real news articles. This
is already happening even for languages that do not have high-quality
monolingual LLMs. We show that fine-tuning Llama (v1), mostly trained on
English, on as little as 40K Italian news articles, is sufficient for producing
news-like texts that native speakers of Italian struggle to identify as
synthetic.
We investigate three LLMs and three methods of detecting synthetic texts
(log-likelihood, DetectGPT, and supervised classification), finding that they
all perform better than human raters, but they are all impractical in the real
world (requiring either access to token likelihood information or a large
dataset of CFM texts). We also explore the possibility of creating a proxy CFM:
an LLM fine-tuned on a similar dataset to one used by the real "content farm".
We find that even a small amount of fine-tuning data suffices for creating a
successful detector, but we need to know which base LLM is used, which is a
major challenge.
Our results suggest that there are currently no practical methods for
detecting synthetic news-like texts 'in the wild', while generating them is too
easy. We highlight the urgency of more NLP research on this problem.

大型语言模型（LLM）正在日益被用作 “内容农场” 模型（CFMs），用于生成可以通过真实新闻文章的合成文本。我们展示了只用 40K 份意大利新闻文章对大部分在英语上进行训练的 Llama（v1）进行微调就足以产生意大利本地人难以辨认为合成文本的新闻样本。我们研究了三个 LLM 和三种检测合成文本的方法（对数似然、DetectGPT 和监督分类），发现它们都比人类评比者表现更好，但在实际情况下都不实用（要求对令牌似然信息有访问权或大量 CFM 文本数据集）。我们还探索了创建代理 CFM 的可能性：在与真实 “内容农场” 使用的类似数据集上进行微调的 LLM。我们发现只需少量微调数据就能成功创建一个检测器，但我们需要知道使用了哪个基础 LLM，这是一个重大挑战。我们的结果表明，目前没有实际的方法来 “野外” 检测合成类似新闻的文本，而生成它们太容易。我们强调了对这个问题进行更多自然语言处理研究的紧迫性。