In the era of big data and large models, automatic annotating functions for
multi-modal data are of great significance for real-world AI-driven
applications, such as autonomous driving and embodied AI. Unlike traditional
closed-set annotation, open-vocabulary annotation is essential to achieve
human-level cognition capability. However, there are few open-vocabulary
auto-labeling systems for multi-modal 3D data. In this paper, we introduce
OpenAnnotate3D, an open-source open-vocabulary auto-labeling system that can
automatically generate 2D masks, 3D masks, and 3D bounding box annotations for
vision and point cloud data. Our system integrates the chain-of-thought
capabilities of Large Language Models (LLMs) and the cross-modality
capabilities of vision-language models (VLMs). To the best of our knowledge,
OpenAnnotate3D is one of the pioneering works for open-vocabulary multi-modal
3D auto-labeling. We conduct comprehensive evaluations on both public and
in-house real-world datasets, which demonstrate that the system significantly
improves annotation efficiency compared to manual annotation while providing
accurate open-vocabulary auto-annotating results.

在大数据和大型模型的时代，用于自动标注多模态数据的功能对于自动驾驶和具身化人工智能等实际应用具有重要意义。本文介绍了 OpenAnnotate3D，一种开源开词汇库自动标注系统，可以为视觉和点云数据自动生成 2D 掩模、3D 掩模和 3D 边界框标注，并整合了大型语言模型的思维链和视觉 - 语言模型的跨模态能力，旨在改善标注效率并提供准确的开词汇库自动标注结果。