Large pre-trained models have had a significant impact on computer vision by
enabling multi-modal learning, where the CLIP model has achieved impressive
results in image classification, object detection, and semantic segmentation.
However, the model's performance on 3D point cloud processing tasks is limited
due to the domain gap between depth maps from 3D projection and training images
of CLIP. This paper proposes DiffCLIP, a new pre-training framework that
incorporates stable diffusion with ControlNet to minimize the domain gap in the
visual branch. Additionally, a style-prompt generation module is introduced for
few-shot tasks in the textual branch. Extensive experiments on the ModelNet10,
ModelNet40, and ScanObjectNN datasets show that DiffCLIP has strong abilities
for 3D understanding. By using stable diffusion and style-prompt generation,
DiffCLIP achieves an accuracy of 43.2\% for zero-shot classification on OBJ\_BG
of ScanObjectNN, which is state-of-the-art performance, and an accuracy of
80.6\% for zero-shot classification on ModelNet10, which is comparable to
state-of-the-art performance.

本文提出了一种新的预训练框架 DiffCLIP，其结合稳定扩散和 ControlNet，以减小视觉分支中的域间差异，并引入了一种样式提示生成模块，用于少样本任务，在 ModelNet10、ModelNet40 和 ScanObjectNN 数据集上进行广泛实验，表明 DiffCLIP 具有强大的 3D 理解能力。使用稳定扩散和样式提示生成，DiffCLIP 在 ScanObjectNN 的 OBJ_BG 数据集上实现了 43.2％的零样本分类精度，这是最先进的技术水平，并且在 ModelNet10 上实现了 80.6％的零样本分类精度，这与最先进的技术水平相当。