Contrastive image-to-LiDAR knowledge transfer, commonly used for learning 3D
representations with synchronized images and point clouds, often faces a
self-conflict dilemma. This issue arises as contrastive losses unintentionally
dissociate features of unmatched points and pixels that share semantic labels,
compromising the integrity of learned representations. To overcome this, we
harness Visual Foundation Models (VFMs), which have revolutionized the
acquisition of pixel-level semantics, to enhance 3D representation learning.
Specifically, we utilize off-the-shelf VFMs to generate semantic labels for
weakly-supervised pixel-to-point contrastive distillation. Additionally, we
employ von Mises-Fisher distributions to structure the feature space, ensuring
semantic embeddings within the same class remain consistent across varying
inputs. Furthermore, we adapt sampling probabilities of points to address
imbalances in spatial distribution and category frequency, promoting
comprehensive and balanced learning. Extensive experiments demonstrate that our
approach mitigates the challenges posed by traditional methods and consistently
surpasses existing image-to-LiDAR contrastive distillation methods in
downstream tasks. The source code is available at
\href{https://github.com/Eaphan/OLIVINE.}{\color{black}this https URL}.

通过利用 VFMs 的像素级语义增强三维表示学习，采用 von Mises-Fisher 分布对特征空间进行结构化，以解决对手法的挑战并在下游任务中始终优于现有的图像到 LiDAR 对比蒸馏方法。

细粒度图像到 LiDAR 对比蒸馏与视觉基础模型

Fine-grained Image-to-LiDAR Contrastive Distillation with Visual  Foundation Models

Proxy-based Deep Metric Learning (DML) learns deep representations by
embedding images close to their class representatives (proxies), commonly with
respect to the angle between them. However, this disregards the embedding norm,
which can carry additional beneficial context such as class- or image-intrinsic
uncertainty. In addition, proxy-based DML struggles to learn class-internal
structures. To address both issues at once, we introduce non-isotropic
probabilistic proxy-based DML. We model images as directional von Mises-Fisher
(vMF) distributions on the hypersphere that can reflect image-intrinsic
uncertainties. Further, we derive non-isotropic von Mises-Fisher (nivMF)
distributions for class proxies to better represent complex class-specific
variances. To measure the proxy-to-image distance between these models, we
develop and investigate multiple distribution-to-point and
distribution-to-distribution metrics. Each framework choice is motivated by a
set of ablational studies, which showcase beneficial properties of our
probabilistic approach to proxy-based DML, such as uncertainty-awareness,
better-behaved gradients during training, and overall improved generalization
performance. The latter is especially reflected in the competitive performance
on the standard DML benchmarks, where our approach compares favorably,
suggesting that existing proxy-based DML can significantly benefit from a more
probabilistic treatment. Code is available at
github.com/ExplainableML/Probabilistic_Deep_Metric_Learning.

该研究提出了基于概率的代理深度度量学习方法，使用方向 von Mises-Fisher（vMF）分布对图像进行建模，以更好地表示类特定差异，并比较了多个分布到点和分布到分布度量之间的代理 - 图像距离，表现出具有不确定性感知能力，更好的梯度性能和整体改进的泛化性能。

基于代理的非各向同性概率深度度量学习

A Non-isotropic Probabilistic Take on Proxy-based Deep Metric Learning

We propose a method that efficiently learns distributions over articulation
model parameters directly from depth images without the need to know
articulation model categories a priori. By contrast, existing methods that
learn articulation models from raw observations typically only predict point
estimates of the model parameters, which are insufficient to guarantee the safe
manipulation of articulated objects. Our core contributions include a novel
representation for distributions over rigid body transformations and
articulation model parameters based on screw theory, von Mises-Fisher
distributions, and Stiefel manifolds. Combining these concepts allows for an
efficient, mathematically sound representation that implicitly satisfies the
constraints that rigid body transformations and articulations must adhere to.
Leveraging this representation, we introduce a novel deep learning based
approach, DUST-net, that performs category-independent articulation model
estimation while also providing model uncertainties. We evaluate our approach
on several benchmarking datasets and real-world objects and compare its
performance with two current state-of-the-art methods. Our results demonstrate
that DUST-net can successfully learn distributions over articulation models for
novel objects across articulation model categories, which generate point
estimates with better accuracy than state-of-the-art methods and effectively
capture the uncertainty over predicted model parameters due to noisy inputs.
Project webpage: this https URL

提出了一种基于深度图像学习关节模型分布的方法，通过螺旋理论、von Mises-Fisher 分布和 Stiefel 流形的数学结构，提供一种有效的、数学上严谨的表示形式，并结合 DUST-net 神经网络提供模型的不确定性评估。