A vision model with general-purpose object-level 3D understanding should be
capable of inferring both 2D (e.g., class name and bounding box) and 3D
information (e.g., 3D location and 3D viewpoint) for arbitrary rigid objects in
natural images. This is a challenging task, as it involves inferring 3D
information from 2D signals and most importantly, generalizing to rigid objects
from unseen categories. However, existing datasets with object-level 3D
annotations are often limited by the number of categories or the quality of
annotations. Models developed on these datasets become specialists for certain
categories or domains, and fail to generalize. In this work, we present
ImageNet3D, a large dataset for general-purpose object-level 3D understanding.
ImageNet3D augments 200 categories from the ImageNet dataset with 2D bounding
box, 3D pose, 3D location annotations, and image captions interleaved with 3D
information. With the new annotations available in ImageNet3D, we could (i)
analyze the object-level 3D awareness of visual foundation models, and (ii)
study and develop general-purpose models that infer both 2D and 3D information
for arbitrary rigid objects in natural images, and (iii) integrate unified 3D
models with large language models for 3D-related reasoning.. We consider two
new tasks, probing of object-level 3D awareness and open vocabulary pose
estimation, besides standard classification and pose estimation. Experimental
results on ImageNet3D demonstrate the potential of our dataset in building
vision models with stronger general-purpose object-level 3D understanding.

通过与 ImageNet 数据集相结合，ImageNet3D 数据集提供了 200 个类别的 2D 和 3D 信息，从而为构建具有更强的通用性目标级三维理解的视觉模型提供了潜力。

ImageNet3D：面向通用对象级别 3D 理解

ImageNet3D: Towards General-Purpose Object-Level 3D Understanding

Molecular representation learning has attracted much attention recently. A
molecule can be viewed as a 2D graph with nodes/atoms connected by edges/bonds,
and can also be represented by a 3D conformation with 3-dimensional coordinates
of all atoms. We note that most previous work handles 2D and 3D information
separately, while jointly leveraging these two sources may foster a more
informative representation. In this work, we explore this appealing idea and
propose a new representation learning method based on a unified 2D and 3D
pre-training. Atom coordinates and interatomic distances are encoded and then
fused with atomic representations through graph neural networks. The model is
pre-trained on three tasks: reconstruction of masked atoms and coordinates, 3D
conformation generation conditioned on 2D graph, and 2D graph generation
conditioned on 3D conformation. We evaluate our method on 11 downstream
molecular property prediction tasks: 7 with 2D information only and 4 with both
2D and 3D information. Our method achieves state-of-the-art results on 10
tasks, and the average improvement on 2D-only tasks is 8.3%. Our method also
achieves significant improvement on two 3D conformation generation tasks.

本文基于统一的 2D 和 3D 预训练，提出了一种新的基于图神经网络的表示学习方法，将原子坐标和原子间距编码，并通过图神经网络将其与原子表示融合。在 11 个下游分子属性预测任务中进行评估，其在 10 项任务中都获得了最先进的结果，2D 任务平均改进了 8.3％，同时在 2 个三维构象生成任务中也取得了显着的改进。