Context: Software engineering has a problem in that when we empirically
evaluate competing prediction systems we obtain conflicting results. Objective:
To reduce the inconsistency amongst validation study results and provide a more
formal foundation to interpret results with a particular focus on continuous
prediction systems. Method: A new framework is proposed for evaluating
competing prediction systems based upon (1) an unbiased statistic, Standardised
Accuracy, (2) testing the result likelihood relative to the baseline technique
of random 'predictions', that is guessing, and (3) calculation of effect sizes.
Results: Previously published empirical evaluations of prediction systems are
re-examined and the original conclusions shown to be unsafe. Additionally, even
the strongest results are shown to have no more than a medium effect size
relative to random guessing. Conclusions: Biased accuracy statistics such as
MMRE are deprecated. By contrast this new empirical validation framework leads
to meaningful results. Such steps will assist in performing future
meta-analyses and in providing more robust and usable recommendations to
practitioners.

提出一个新的框架来进行预测系统的评估，并检验了现有的统计指标的有效性，呈现出可靠的实验结论。

评估软件项目估算中的预测系统

Evaluating prediction systems in software project estimation

Statistical significance testing centered on p-values is commonly used to
compare NLP system performance, but p-values alone are insufficient because
statistical significance differs from practical significance. The latter can be
measured by estimating effect size. In this paper, we propose a three-stage
procedure for comparing NLP system performance and provide a toolkit,
NLPStatTest, that automates the process. Users can upload NLP system evaluation
scores and the toolkit will analyze these scores, run appropriate significance
tests, estimate effect size, and conduct power analysis to estimate Type II
error. The toolkit provides a convenient and systematic way to compare NLP
system performance that goes beyond statistical significance testing

本文提出了一种基于效应大小估计的三阶段比较 NLP 系统性能的方法，并提供了一个工具包 NLPStatTest，该工具包可以自动化处理上传的 NLP 系统评估分数，进行合适的显著性检验和效应大小估计，并进行功效分析来估计 II 型错误。这个工具包提供了一个便捷和系统的方法来比较 NLP 系统性能，超越了统计显著性检验。