Over the last decade there has been increasing concern about the biases
embodied in traditional evaluation methods for Natural Language
Processing/Learning, particularly methods borrowed from Information Retrieval.
Without knowledge of the Bias and Prevalence of the contingency being tested,
or equivalently the expectation due to chance, the simple conditional
probabilities Recall, Precision and Accuracy are not meaningful as evaluation
measures, either individually or in combinations such as F-factor. The
existence of bias in NLP measures leads to the 'improvement' of systems by
increasing their bias, such as the practice of improving tagging and parsing
scores by using most common value (e.g. water is always a Noun) rather than the
attempting to discover the correct one. The measures Cohen Kappa and Powers
Informedness are discussed as unbiased alternative to Recall and related to the
psychologically significant measure DeltaP. In this paper we will analyze both
biased and unbiased measures theoretically, characterizing the precise
relationship between all these measures as well as evaluating the evaluation
measures themselves empirically using a Monte Carlo simulation.

通过理论分析和 Monte Carlo 模拟，对传统评估方法中存在的偏见进行了讨论，提出了一些不受偏见影响的替代评估方法，如 Cohen Kappa，并进行了实证评估。