A vision model with general-purpose object-level 3D understanding should be
capable of inferring both 2D (e.g., class name and bounding box) and 3D
information (e.g., 3D location and 3D viewpoint) for arbitrary rigid objects in
natural images. This is a challenging task, as it involves inferring 3D
information from 2D signals and most importantly, generalizing to rigid objects
from unseen categories. However, existing datasets with object-level 3D
annotations are often limited by the number of categories or the quality of
annotations. Models developed on these datasets become specialists for certain
categories or domains, and fail to generalize. In this work, we present
ImageNet3D, a large dataset for general-purpose object-level 3D understanding.
ImageNet3D augments 200 categories from the ImageNet dataset with 2D bounding
box, 3D pose, 3D location annotations, and image captions interleaved with 3D
information. With the new annotations available in ImageNet3D, we could (i)
analyze the object-level 3D awareness of visual foundation models, and (ii)
study and develop general-purpose models that infer both 2D and 3D information
for arbitrary rigid objects in natural images, and (iii) integrate unified 3D
models with large language models for 3D-related reasoning.. We consider two
new tasks, probing of object-level 3D awareness and open vocabulary pose
estimation, besides standard classification and pose estimation. Experimental
results on ImageNet3D demonstrate the potential of our dataset in building
vision models with stronger general-purpose object-level 3D understanding.

通过与 ImageNet 数据集相结合，ImageNet3D 数据集提供了 200 个类别的 2D 和 3D 信息，从而为构建具有更强的通用性目标级三维理解的视觉模型提供了潜力。