Large-scale vision 2D vision language models, such as CLIP can be aligned
with a 3D encoder to learn generalizable (open-vocabulary) 3D vision models.
However, current methods require supervised pre-training for such alignment,
and the performance of such 3D zero-shot models remains sub-optimal for
real-world adaptation. In this work, we propose an optimization framework:
Cross-MoST: Cross-Modal Self-Training, to improve the label-free classification
performance of a zero-shot 3D vision model by simply leveraging unlabeled 3D
data and their accompanying 2D views. We propose a student-teacher framework to
simultaneously process 2D views and 3D point clouds and generate joint pseudo
labels to train a classifier and guide cross-model feature alignment. Thereby
we demonstrate that 2D vision language models such as CLIP can be used to
complement 3D representation learning to improve classification performance
without the need for expensive class annotations. Using synthetic and
real-world 3D datasets, we further demonstrate that Cross-MoST enables
efficient cross-modal knowledge exchange resulting in both image and point
cloud modalities learning from each other's rich representations.

通过使用 Cross-MoST 优化框架，结合 CLIP 等 2D 视觉语言模型，可以提高无需标签的零样本 3D 视觉模型的分类性能，并且实现图像和点云模态之间的跨模态知识交流。

跨模态自训练：无需标签的图像与点云对齐学习分类

Cross-Modal Self-Training: Aligning Images and Pointclouds to Learn  Classification without Labels

Correspondences emerge from large-scale vision models trained for generative
and discriminative tasks. This has been revealed and benchmarked by computing
correspondence maps between pairs of images, using nearest neighbors on the
feature grids. Existing work has attempted to improve the quality of these
correspondence maps by carefully mixing features from different sources, such
as by combining the features of different layers or networks. We point out that
a better correspondence strategy is available, which directly imposes structure
on the correspondence field: the functional map. Wielding this simple
mathematical tool, we lift the correspondence problem from the pixel space to
the function space and directly optimize for mappings that are globally
coherent. We demonstrate that our technique yields correspondences that are not
only smoother but also more accurate, with the possibility of better reflecting
the knowledge embedded in the large-scale vision models that we are studying.
Our approach sets a new state-of-the-art on various dense correspondence tasks.
We also demonstrate our effectiveness in keypoint correspondence and affordance
map transfer.

使用功能映射方法来提高大规模视觉模型训练中的对应关系地图的质量，并在各种密集对应任务中取得了最新的最佳性能。

零样本图像特征一致性与深度函数图

Zero-Shot Image Feature Consensus with Deep Functional Maps

Significant progress has recently been made in creative applications of large
pre-trained models for downstream tasks in 3D vision, such as text-to-shape
generation. This motivates our investigation of how these pre-trained models
can be used effectively to generate 3D shapes from sketches, which has largely
remained an open challenge due to the limited sketch-shape paired datasets and
the varying level of abstraction in the sketches. We discover that conditioning
a 3D generative model on the features (obtained from a frozen large pre-trained
vision model) of synthetic renderings during training enables us to effectively
generate 3D shapes from sketches at inference time. This suggests that the
large pre-trained vision model features carry semantic signals that are
resilient to domain shifts, i.e., allowing us to use only RGB renderings, but
generalizing to sketches at inference time. We conduct a comprehensive set of
experiments investigating different design factors and demonstrate the
effectiveness of our straightforward approach for generation of multiple 3D
shapes per each input sketch regardless of their level of abstraction without
requiring any paired datasets during training.

This paper explores the use of pre-trained models and synthetic renderings to generate 3D shapes from sketches without the need for paired datasets, demonstrating the effectiveness of the approach for generating multiple 3D shapes per input sketch regardless of their level of abstraction.