We introduce a new framework for sample-efficient model evaluation that we
call active testing. While approaches like active learning reduce the number of
labels needed for model training, existing literature largely ignores the cost
of labeling test data, typically unrealistically assuming large test sets for
model evaluation. This creates a disconnect to real applications, where test
labels are important and just as expensive, e.g. for optimizing
hyperparameters. Active testing addresses this by carefully selecting the test
points to label, ensuring model evaluation is sample-efficient. To this end, we
derive theoretically-grounded and intuitive acquisition strategies that are
specifically tailored to the goals of active testing, noting these are distinct
to those of active learning. As actively selecting labels introduces a bias; we
further show how to remove this bias while reducing the variance of the
estimator at the same time. Active testing is easy to implement and can be
applied to any supervised machine learning method. We demonstrate its
effectiveness on models including WideResNets and Gaussian processes on
datasets including Fashion-MNIST and CIFAR-100.

本文介绍了一种新的样本效率模型评估框架 —— 主动测试，通过精心选择要标记的测试点，以充分利用样本，解决了现有文献大都忽略了标记测试数据的成本，从而导致模型评估与实际应用的脱节的问题，在理论上建立了基于目标的获取策略并做了进一步的改进，以消除引入的偏差并在相同时间内降低估计方差。

主动测试：高效模型评估

Active Testing: Sample-Efficient Model Evaluation

Distant supervision has been a widely used method for neural relation
extraction for its convenience of automatically labeling datasets. However,
existing works on distantly supervised relation extraction suffer from the low
quality of test set, which leads to considerable biased performance evaluation.
These biases not only result in unfair evaluations but also mislead the
optimization of neural relation extraction. To mitigate this problem, we
propose a novel evaluation method named active testing through utilizing both
the noisy test set and a few manual annotations. Experiments on a widely used
benchmark show that our proposed approach can yield approximately unbiased
evaluations for distantly supervised relation extractors.

为了解决现有远程监督关系提取方法中测试集质量低导致的评估偏差问题，本文提出了一种名为主动测试的新型评估方法，通过利用嘈杂的测试集和一些人工注释，可以为远程监督关系提取器提供几乎无偏的评估。在广泛使用的基准测试中，实验表明我们提出的方法可以为远程监督关系提取器提供几乎无偏的评估。