We introduce Duoduo CLIP, a model for 3D representation learning that learns
shape encodings from multi-view images instead of point-clouds. The choice of
multi-view images allows us to leverage 2D priors from off-the-shelf CLIP
models to facilitate fine-tuning with 3D data. Our approach not only shows
better generalization compared to existing point cloud methods, but also
reduces GPU requirements and training time. In addition, we modify the model
with cross-view attention to leverage information across multiple frames of the
object which further boosts performance. Compared to the current SOTA point
cloud method that requires 480 A100 hours to train 1 billion model parameters
we only require 57 A5000 hours and 87 million parameters. Multi-view images
also provide more flexibility in use cases compared to point clouds. This
includes being able to encode objects with a variable number of images, with
better performance when more views are used. This is in contrast to point cloud
based methods, where an entire scan or model of an object is required. We
showcase this flexibility with object retrieval from images of real-world
objects. Our model also achieves better performance in more fine-grained text
to shape retrieval, demonstrating better text-and-shape alignment than point
cloud based models.

推出了 Duoduo CLIP 模型，用于从多视图图像中学习形状表示，相比基于点云的方法，它展示了更好的泛化性能、减少的 GPU 要求和训练时间，以及利用多帧对象的交叉视图注意力进一步提升的性能。多视图图像还提供了与基于点云方法相比更大的灵活性，在实际世界对象的图像分类和文本到形状检索中展示了更好的性能。