We now have a rich and growing set of modeling tools and algorithms for
inducing linguistic structure from text that is less than fully annotated. In
this paper, we discuss some of the weaknesses of our current methodology. We
present a new abstract framework for evaluating natural language processing
(NLP) models in general and unsupervised NLP models in particular. The central
idea is to make explicit certain adversarial roles among researchers, so that
the different roles in an evaluation are more clearly defined and performers of
all roles are offered ways to make measurable contributions to the larger goal.
Adopting this approach may help to characterize model successes and failures by
encouraging earlier consideration of error analysis. The framework can be
instantiated in a variety of ways, simulating some familiar intrinsic and
extrinsic evaluations as well as some new evaluations.

本文提出了一种用于评估自然语言处理模型的新的抽象框架，通过明确研究人员之间的某些对抗角色，这有助于定义不同角色在评估中的贡献，并鼓励更早的错误分析，这个框架可以用多种方式实例化，并模拟一些熟悉的内部和外部评估以及一些新的评估。