Language is the principal tool for human communication, in which humor is one
of the most attractive parts. Producing natural language like humans using
computers, a.k.a, Natural Language Generation (NLG), has been widely used for
dialogue systems, chatbots, machine translation, as well as computer-aid
creation e.g., idea generations, scriptwriting. However, the humor aspect of
natural language is relatively under-investigated, especially in the age of
pre-trained language models. In this work, we aim to preliminarily test whether
NLG can generate humor as humans do. We build a new dataset consisting of
numerous digitized Chinese Comical Crosstalk scripts (called C$^3$ in short),
which is for a popular Chinese performing art called `Xiangsheng' since 1800s.
(For convenience for non-Chinese speakers, we called `crosstalk' for
`Xiangsheng' in this paper.) We benchmark various generation approaches
including training-from-scratch Seq2seq, fine-tuned middle-scale PLMs, and
large-scale PLMs (with and without fine-tuning). Moreover, we also conduct a
human assessment, showing that 1) large-scale pretraining largely improves
crosstalk generation quality; and 2) even the scripts generated from the best
PLM is far from what we expect, with only 65% quality of human-created
crosstalk. We conclude, humor generation could be largely improved using
large-scaled PLMs, but it is still in its infancy.
The data and benchmarking code is publicly available in
https://github.com/anonNo2/crosstalk-generation.

本文旨在测试自然语言生成（NLG）在幽默方面的能力，我们使用各种方法进行生成，包括从头开始训练 Seq2seq、微调中型 PLMs 和大型 PLMs（有和无微调）。我们使用一些由数字化的中国相声剧本组成的新数据集对这些方法进行了基准测试，并进行了人员评估，表明大规模预训练大大提高了相声生成的质量，但我们得出结论：幽默生成目前还处于起步阶段。