Scaling up representations for images or text has been extensively
investigated in the past few years and has led to revolutions in learning
vision and language. However, scalable representation for 3D objects and scenes
is relatively unexplored. In this work, we present Uni3D, a 3D foundation model
to explore the unified 3D representation at scale. Uni3D uses a 2D initialized
ViT end-to-end pretrained to align the 3D point cloud features with the
image-text aligned features. Via the simple architecture and pretext task,
Uni3D can leverage abundant 2D pretrained models as initialization and
image-text aligned models as the target, unlocking the great potential of 2D
models and scaling-up strategies to the 3D world. We efficiently scale up Uni3D
to one billion parameters, and set new records on a broad range of 3D tasks,
such as zero-shot classification, few-shot classification, open-world
understanding and part segmentation. We show that the strong Uni3D
representation also enables applications such as 3D painting and retrieval in
the wild. We believe that Uni3D provides a new direction for exploring both
scaling up and efficiency of the representation in 3D domain.

通过 Uni3D，我们提出了一种可扩展的 3D 对象和场景表示方法，利用 2D 模型作为初始化，并通过图像 - 文本对齐模型作为目标，解锁了 2D 模型和扩展策略在 3D 世界中的潜力，并在许多 3D 任务中取得了新的成绩，如零样本分类、少样本分类、开放世界理解和部分分割，并展示了 Uni3D 在野外环境中进行的 3D 绘画和检索等应用。我们相信，Uni3D 为探索 3D 领域的表示扩展和效率提供了新的方向。

Uni3D: 在规模上探索统一的三维表示

Uni3D: Exploring Unified 3D Representation at Scale

Current 3D object detection models follow a single dataset-specific training
and testing paradigm, which often faces a serious detection accuracy drop when
they are directly deployed in another dataset. In this paper, we study the task
of training a unified 3D detector from multiple datasets. We observe that this
appears to be a challenging task, which is mainly due to that these datasets
present substantial data-level differences and taxonomy-level variations caused
by different LiDAR types and data acquisition standards. Inspired by such
observation, we present a Uni3D which leverages a simple data-level correction
operation and a designed semantic-level coupling-and-recoupling module to
alleviate the unavoidable data-level and taxonomy-level differences,
respectively. Our method is simple and easily combined with many 3D object
detection baselines such as PV-RCNN and Voxel-RCNN, enabling them to
effectively learn from multiple off-the-shelf 3D datasets to obtain more
discriminative and generalizable representations. Experiments are conducted on
many dataset consolidation settings including Waymo-nuScenes, nuScenes-KITTI,
Waymo-KITTI, and Waymo-nuScenes-KITTI consolidations. Their results demonstrate
that Uni3D exceeds a series of individual detectors trained on a single
dataset, with a 1.04x parameter increase over a selected baseline detector. We
expect this work will inspire the research of 3D generalization since it will
push the limits of perceptual performance.

该论文研究了如何训练一个来自多个数据集的统一 3D 检测器，提出了一种名为 Uni3D 的方法来解决数据级别和分类学级别的差异，证明了该方法的有效性并对进一步的 3D 泛化研究具有启发意义。