In this paper, we introduce the FOCAL (Ford-OLIVES Collaboration on Active
Learning) dataset which enables the study of the impact of annotation-cost
within a video active learning setting. Annotation-cost refers to the time it
takes an annotator to label and quality-assure a given video sequence. A
practical motivation for active learning research is to minimize
annotation-cost by selectively labeling informative samples that will maximize
performance within a given budget constraint. However, previous work in video
active learning lacks real-time annotation labels for accurately assessing cost
minimization and instead operates under the assumption that annotation-cost
scales linearly with the amount of data to annotate. This assumption does not
take into account a variety of real-world confounding factors that contribute
to a nonlinear cost such as the effect of an assistive labeling tool and the
variety of interactions within a scene such as occluded objects, weather, and
motion of objects. FOCAL addresses this discrepancy by providing real
annotation-cost labels for 126 video sequences across 69 unique city scenes
with a variety of weather, lighting, and seasonal conditions. We also introduce
a set of conformal active learning algorithms that take advantage of the
sequential structure of video data in order to achieve a better trade-off
between annotation-cost and performance while also reducing floating point
operations (FLOPS) overhead by at least 77.67%. We show how these approaches
better reflect how annotations on videos are done in practice through a
sequence selection framework. We further demonstrate the advantage of these
approaches by introducing two performance-cost metrics and show that the best
conformal active learning method is cheaper than the best traditional active
learning method by 113 hours.

本文介绍了 FOCAL（Ford-OLIVES 协作主动学习）数据集，该数据集用于研究视频主动学习环境中标注成本的影响。通过提供真实的标注成本标签，以及引入一系列的符合性主动学习算法，本研究展示了这些方法如何更好地反映了视频标注的实际操作，并且证明了最佳的符合性主动学习方法比传统的主动学习方法更便宜 113 个小时。