Recent advances in Vision and Language Models (VLMs) have improved open-world
3D representation, facilitating 3D zero-shot capability in unseen categories.
Existing open-world methods pre-train an extra 3D encoder to align features
from 3D data (e.g., depth maps or point clouds) with CAD-rendered images and
corresponding texts. However, the limited color and texture variations in CAD
images can compromise the alignment robustness. Furthermore, the volume
discrepancy between pre-training datasets of the 3D encoder and VLM leads to
sub-optimal 2D to 3D knowledge transfer. To overcome these issues, we propose
OpenDlign, a novel framework for learning open-world 3D representations, that
leverages depth-aligned images generated from point cloud-projected depth maps.
Unlike CAD-rendered images, our generated images provide rich, realistic color
and texture diversity while preserving geometric and semantic consistency with
the depth maps. OpenDlign also optimizes depth map projection and integrates
depth-specific text prompts, improving 2D VLM knowledge adaptation for 3D
learning efficient fine-tuning. Experimental results show that OpenDlign
significantly outperforms existing benchmarks in zero-shot and few-shot 3D
tasks, exceeding prior scores by 8.0% on ModelNet40 and 16.4% on OmniObject3D
with just 6 million tuned parameters. Moreover, integrating generated
depth-aligned images into existing 3D learning pipelines consistently improves
their performance.

OpenDlign 是一个新颖的框架，利用点云投影的深度映射生成的深度对齐图像来学习开放世界的 3D 表示，通过丰富多样的颜色和纹理差异以及保持几何和语义一致性，优化了 2D 到 3D 知识迁移，从而显著提高了零样本和少样本 3D 任务的性能。