Despite growing interest in natural language generation (NLG) models that
produce diverse outputs, there is currently no principled method for evaluating
the diversity of an NLG system. In this work, we propose a framework for
evaluating diversity metrics. The framework measures the correlation between a
proposed diversity metric and a diversity parameter, a single parameter that
controls some aspect of diversity in generated text. For example, a diversity
parameter might be a binary variable used to instruct crowdsourcing workers to
generate text with either low or high content diversity. We demonstrate the
utility of our framework by: (a) establishing best practices for eliciting
diversity judgments from humans, (b) showing that humans substantially
outperform automatic metrics in estimating content diversity, and (c)
demonstrating that existing methods for controlling diversity by tuning a
"decoding parameter" mostly affect form but not meaning. Our framework can
advance the understanding of different diversity metrics, an essential step on
the road towards better NLG systems.

本文提出了一种通过建立多样性度量指标及其多样性参数之间关系的系统来评估自然语言生成（NLG）系统多样性的方法，并通过对人类和自动度量，解码参数调整等方面的实验展示了这个框架的实用性。