It has become routine to report research results where Large Language Models
(LLMs) outperform average humans in a wide range of language-related tasks, and
creative text writing is no exception. It seems natural, then, to raise the
bid: Are LLMs ready to compete in creative writing skills with a top (rather
than average) novelist? To provide an initial answer for this question, we have
carried out a contest between Patricio Pron (an awarded novelist, considered
one of the best of his generation) and GPT-4 (one of the top performing LLMs),
in the spirit of AI-human duels such as DeepBlue vs Kasparov and AlphaGo vs Lee
Sidol. We asked Pron and GPT-4 to provide thirty titles each, and then to write
short stories for both their titles and their opponent's. Then, we prepared an
evaluation rubric inspired by Boden's definition of creativity, and we
collected 5,400 manual assessments provided by literature critics and scholars.
The results of our experimentation indicate that LLMs are still far from
challenging a top human creative writer, and that reaching such level of
autonomous creative writing skills probably cannot be reached simply with
larger language models.

大型语言模型（LLMs）在创意文学写作方面仍远未能挑战顶级作家。通过与一位获奖小说家进行竞赛，该研究对 GPT-4 进行了评估，结果表明单纯扩大语言模型规模不能实现类似的创造性写作技巧。