Creating large-scale high-quality labeled datasets is a major bottleneck in
supervised machine learning workflows. Auto-labeling systems are a promising
way to reduce reliance on manual labeling for dataset construction.
Threshold-based auto-labeling, where validation data obtained from humans is
used to find a threshold for confidence above which the data is
machine-labeled, is emerging as a popular solution used widely in practice.
Given the long shelf-life and diverse usage of the resulting datasets,
understanding when the data obtained by such auto-labeling systems can be
relied on is crucial. In this work, we analyze threshold-based auto-labeling
systems and derive sample complexity bounds on the amount of human-labeled
validation data required for guaranteeing the quality of machine-labeled data.
Our results provide two insights. First, reasonable chunks of the unlabeled
data can be automatically and accurately labeled by seemingly bad models.
Second, a hidden downside of threshold-based auto-labeling systems is
potentially prohibitive validation data usage. Together, these insights
describe the promise and pitfalls of using such systems. We validate our
theoretical guarantees with simulations and study the efficacy of
threshold-based auto-labeling on real datasets.

本文分析了基于阈值的自动标注系统，推导了保证机器标注数据质量所需的人工标注验证数据量的样本复杂度上限，并通过模拟和实际数据集研究了基于阈值的自动标注在真实数据上的有效性和使用限制。