Large-scale datasets are essential to modern day deep learning. Advocates
argue that understanding these methods requires dataset transparency (e.g.
"dataset curation, motivation, composition, collection process, etc...").
However, almost no one has suggested the release of the detailed definitions
and visual category examples provided to annotators - information critical to
understanding the structure of the annotations present in each dataset. These
labels are at the heart of public datasets, yet few datasets include the
instructions that were used to generate them. We introduce a new task, Labeling
Instruction Generation, to address missing publicly available labeling
instructions. In Labeling Instruction Generation, we take a reasonably
annotated dataset and: 1) generate a set of examples that are visually
representative of each category in the dataset; 2) provide a text label that
corresponds to each of the examples. We introduce a framework that requires no
model training to solve this task and includes a newly created rapid retrieval
system that leverages a large, pre-trained vision and language model. This
framework acts as a proxy to human annotators that can help to both generate a
final labeling instruction set and evaluate its quality. Our framework
generates multiple diverse visual and text representations of dataset
categories. The optimized instruction set outperforms our strongest baseline
across 5 folds by 7.06 mAP for NuImages and 12.9 mAP for COCO.

我们引入了一个新的任务，标签指令生成，旨在解决缺少公开可用的标注说明的问题，我们介绍了一个无需模型训练的框架，并使用一个新创建的快速检索系统，利用大型预训练视觉和语言模型，生成数据集类别的多个不同的视觉和文本表示，我们优化后的标注指令集在 5 个折叠中比 NuImages 高出 7.06 mAP，比 COCO 高出 12.9 mAP。