Natural language processing researchers have identified limitations of
evaluation methodology for generation tasks, with new questions raised about
the validity of automatic metrics and of crowdworker judgments. Meanwhile,
efforts to improve generation models tend to depend on simple n-gram overlap
metrics (e.g., BLEU, ROUGE). We argue that new advances on models and metrics
should each more directly benefit and inform the other. We therefore propose a
generalization of leaderboards, bidimensional leaderboards (Billboards), that
simultaneously tracks progress in language generation models and metrics for
their evaluation. Unlike conventional unidimensional leaderboards that sort
submitted systems by predetermined metrics, a Billboard accepts both generators
and evaluation metrics as competing entries. A Billboard automatically creates
an ensemble metric that selects and linearly combines a few metrics based on a
global analysis across generators. Further, metrics are ranked based on their
correlation with human judgments. We release four Billboards for machine
translation, summarization, and image captioning. We demonstrate that a linear
ensemble of a few diverse metrics sometimes substantially outperforms existing
metrics in isolation. Our mixed-effects model analysis shows that most
automatic metrics, especially the reference-based ones, overrate machine over
human generation, demonstrating the importance of updating metrics as
generation models become stronger (and perhaps more similar to humans) in the
future. Our project website is available at
this https URL

提出了一种新的比较模型，即 Bidimensional Leaderboards，这种模型同时跟踪语言生成模型的进展和评价指标，通过人类评价，对评价指标进行排名和选择，以模型和评价指标为竞争方，最后得出一个集成评价指标。