Matching cross-modality features between images and point clouds is a
fundamental problem for image-to-point cloud registration. However, due to the
modality difference between images and points, it is difficult to learn robust
and discriminative cross-modality features by existing metric learning methods
for feature matching. Instead of applying metric learning on cross-modality
data, we propose to unify the modality between images and point clouds by
pretrained large-scale models first, and then establish robust correspondence
within the same modality. We show that the intermediate features, called
diffusion features, extracted by depth-to-image diffusion models are
semantically consistent between images and point clouds, which enables the
building of coarse but robust cross-modality correspondences. We further
extract geometric features on depth maps produced by the monocular depth
estimator. By matching such geometric features, we significantly improve the
accuracy of the coarse correspondences produced by diffusion features.
Extensive experiments demonstrate that without any task-specific training,
direct utilization of both features produces accurate image-to-point cloud
registration. On three public indoor and outdoor benchmarks, the proposed
method averagely achieves a 20.6 percent improvement in Inlier Ratio, a
three-fold higher Inlier Number, and a 48.6 percent improvement in Registration
Recall than existing state-of-the-arts.

图像与点云之间的特征匹配问题是图像到点云配准的基本问题。通过预训练的大规模模型来统一图像和点云之间的模态，并在同一模态内建立可靠的对应关系。我们利用深度到图像扩散模型提取的中间特征即扩散特征在图像和点云之间具有语义一致性，这使得粗糙但稳健的跨模态对应关系的建立成为可能。在由单目深度估计器产生的深度图上进一步提取几何特征，通过匹配这些几何特征，显著提高了扩散特征产生的粗糙对应关系的准确性。在三个公共室内外基准测试上，通过直接利用这两种特征，该方法平均提高了 20.6％的内点比率、三倍的内点数目和 48.6％的配准召回率相比现有最先进方法。

FreeReg: 集成预训练扩散模型和单目深度估计的图像到点云注册

FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained  Diffusion Models and Monocular Depth Estimators

Salient object detection (SOD) on RGB and depth images has attracted more and
more research interests, due to its effectiveness and the fact that depth cues
can now be conveniently captured. Existing RGB-D SOD models usually adopt
different fusion strategies to learn a shared representation from the two
modalities (\ie, RGB and depth), while few methods explicitly consider how to
preserve modality-specific characteristics. In this study, we propose a novel
framework, termed SPNet} (Specificity-preserving network), which benefits SOD
performance by exploring both the shared information and modality-specific
properties (\eg, specificity). Specifically, we propose to adopt two
modality-specific networks and a shared learning network to generate individual
and shared saliency prediction maps, respectively. To effectively fuse
cross-modal features in the shared learning network, we propose a
cross-enhanced integration module (CIM) and then propagate the fused feature to
the next layer for integrating cross-level information. Moreover, to capture
rich complementary multi-modal information for boosting the SOD performance, we
propose a multi-modal feature aggregation (MFA) module to integrate the
modality-specific features from each individual decoder into the shared
decoder. By using a skip connection, the hierarchical features between the
encoder and decoder layers can be fully combined. Extensive experiments
demonstrate that our~\ours~outperforms cutting-edge approaches on six popular
RGB-D SOD and three camouflaged object detection benchmarks. The project is
publicly available at: this https URL

该研究提出了一种称为 SPNet 的新框架，它通过探索共享信息和特定性质（如特定性）来受益于 SOD 性能，并采用双模态特定网络和共同学习网络来生成单独的和共享的显着性预测地图，分别。此外，为了捕获丰富的互补多模态信息以提高 SOD 性能，该研究还提出了一种多模态特征聚合（MFA）模块。

保留特异性的 RGB-D 显著性检测

Specificity-preserving RGB-D Saliency Detection

This paper proposes a new method called Multimodal RNNs for RGB-D scene
semantic segmentation. It is optimized to classify image pixels given two input
sources: RGB color channels and Depth maps. It simultaneously performs training
of two recurrent neural networks (RNNs) that are crossly connected through
information transfer layers, which are learnt to adaptively extract relevant
cross-modality features. Each RNN model learns its representations from its own
previous hidden states and transferred patterns from the other RNNs previous
hidden states; thus, both model-specific and crossmodality features are
retained. We exploit the structure of quad-directional 2D-RNNs to model the
short and long range contextual information in the 2D input image. We carefully
designed various baselines to efficiently examine our proposed model structure.
We test our Multimodal RNNs method on popular RGB-D benchmarks and show how it
outperforms previous methods significantly and achieves competitive results
with other state-of-the-art works.

提出了一种新的多模态递归神经网络方法 (Multimodal RNNs) 用于 RGB-D 场景语义分割，它同时优化两个递归神经网络模型的训练，各自从自己以及另外一个模型的传递模式中学习特征，以提取相关的跨模态信息特征，实现了在 RGB 和深度图像数据上的显著优化和竞争性结果。