Much recent work on visual recognition aims to scale up learning to massive,
noisily-annotated datasets. We address the problem of scaling- up the
evaluation of such models to large-scale datasets with noisy labels. Current
protocols for doing so require a human user to either vet (re-annotate) a small
fraction of the test set and ignore the rest, or else correct errors in
annotation as they are found through manual inspection of results. In this
work, we re-formulate the problem as one of active testing, and examine
strategies for efficiently querying a user so as to obtain an accu- rate
performance estimate with minimal vetting. We demonstrate the effectiveness of
our proposed active testing framework on estimating two performance metrics,
Precision@K and mean Average Precision, for two popular computer vision tasks,
multi-label classification and instance segmentation. We further show that our
approach is able to save significant human annotation effort and is more robust
than alternative evaluation protocols.

本研究致力于解决在大型带噪数据集上扩大模型评估规模的问题，并采用主动测试框架，有效查询用户，以最小化审核，测试两种常见计算机视觉任务的性能指标，并表明该方法比其他评估协议更具稳健性，并能节省显着的人工标注工作量。