This paper focuses on the recently popular task of point cloud completion
guided by multimodal information. Although existing methods have achieved
excellent performance by fusing auxiliary images, there are still some
deficiencies, including the poor generalization ability of the model and
insufficient fine-grained semantic information for extracted features. In this
work, we propose a novel multimodal fusion network for point cloud completion,
which can simultaneously fuse visual and textual information to predict the
semantic and geometric characteristics of incomplete shapes effectively.
Specifically, to overcome the lack of prior information caused by the
small-scale dataset, we employ a pre-trained vision-language model that is
trained with a large amount of image-text pairs. Therefore, the textual and
visual encoders of this large-scale model have stronger generalization ability.
Then, we propose a multi-stage feature fusion strategy to fuse the textual and
visual features into the backbone network progressively. Meanwhile, to further
explore the effectiveness of fine-grained text descriptions for point cloud
completion, we also build a text corpus with fine-grained descriptions, which
can provide richer geometric details for 3D shapes. The rich text descriptions
can be used for training and evaluating our network. Extensive quantitative and
qualitative experiments demonstrate the superior performance of our method
compared to state-of-the-art point cloud completion networks.

本文提出了一种新颖的多模态融合网络用于点云补全，该网络可以同时融合视觉和文本信息，有效预测不完整形状的语义和几何特征，并通过多阶段特征融合策略将文本和视觉特征渐进地融入骨干网络，同时借助精细化的文本描述进一步探索点云补全的有效性，通过广泛的定量和定性实验证明我们方法相比最先进的点云补全网络具有更优越的性能。