Many Natural Language Generation (NLG) tasks aim to generate a single output
text given an input prompt. Other settings require the generation of multiple
texts, e.g., for Synthetic Traffic Generation (STG). This generation task is
crucial for training and evaluating QA systems as well as conversational
agents, where the goal is to generate multiple questions or utterances
resembling the linguistic variability of real users. In this paper, we show
that common NLG metrics, like BLEU, are not suitable for evaluating STG. We
propose and evaluate several metrics designed to compare the generated traffic
to the distribution of real user texts. We validate our metrics with an
automatic procedure to verify whether they capture different types of quality
issues of generated data; we also run human annotations to verify the
correlation with human judgements. Experiments on three tasks, i.e., Shopping
Utterance Generation, Product Question Generation and Query Auto Completion,
demonstrate that our metrics are effective for evaluating STG tasks, and
improve the agreement with human judgement up to 20% with respect to common NLG
metrics. We believe these findings can pave the way towards better solutions
for estimating the representativeness of synthetic text data.

本文提出并评估了几种用于比较生成的流量与真实用户文本分布的度量标准，证明了常见的自然语言生成度量指标对于评估合成流量生成任务不适用。通过在三个任务上的实验验证，即购物话语生成、产品问题生成和查询自动完成，我们得出结论：我们的度量标准对于评估合成流量生成任务是有效的，且与人工判断的一致性相比普通的自然语言生成度量指标提高了 20%。我们相信这些研究结果可以为更好地估计合成文本数据的代表性提供解决方案。