Statistical significance testing is used in natural language processing (NLP)
to determine whether the results of a study or experiment are likely to be due
to chance or if they reflect a genuine relationship. A key step in significance
testing is the estimation of confidence interval which is a function of sample
variance. Sample variance calculation is straightforward when evaluating
against ground truth. However, in many cases, a metric model is often used for
evaluation. For example, to compare toxicity of two large language models, a
toxicity classifier is used for evaluation. Existing works usually do not
consider the variance change due to metric model errors, which can lead to
wrong conclusions. In this work, we establish the mathematical foundation of
significance testing for model-based metrics. With experiments on public
benchmark datasets and a production system, we show that considering metric
model errors to calculate sample variances for model-based metrics changes the
conclusions in certain experiments.

统计学意义检验用于自然语言处理（NLP）中，用于确定研究或实验的结果是否可能是由于偶然性还是反映了真实关系。我们在模型基于度量的统计显著性测试中，考虑模型误差对样本方差的影响，并通过公共基准数据集和生产系统的实验，展示了在某些实验中，考虑模型误差计算样本方差对模型基于度量的结果产生了改变。

模型基准度量方法的忠实模型评估

Faithful Model Evaluation for Model-Based Metrics

Accurate automatic evaluation metrics for open-domain dialogs are in high
demand. Existing model-based metrics for system response evaluation are trained
on human annotated data, which is cumbersome to collect. In this work, we
propose to use information that can be automatically extracted from the next
user utterance, such as its sentiment or whether the user explicitly ends the
conversation, as a proxy to measure the quality of the previous system
response. This allows us to train on a massive set of dialogs with weak
supervision, without requiring manual system turn quality annotations.
Experiments show that our model is comparable to models trained on human
annotated data. Furthermore, our model generalizes across both spoken and
written open-domain dialog corpora collected from real and paid users.

本文提出使用自动提取的信息作为衡量已有机器人回答质量的替代方法，从而避免了对人工标注数据的依赖，实现对弱监督数据的训练，并添加了对口语和书面语的支持。