3D single object tracking plays a crucial role in computer vision. Mainstream
methods mainly rely on point clouds to achieve geometry matching between target
template and search area. However, textureless and incomplete point clouds make
it difficult for single-modal trackers to distinguish objects with similar
structures. To overcome the limitations of geometry matching, we propose a
Multi-modal Multi-level Fusion Tracker (MMF-Track), which exploits the image
texture and geometry characteristic of point clouds to track 3D target.
Specifically, we first propose a Space Alignment Module (SAM) to align RGB
images with point clouds in 3D space, which is the prerequisite for
constructing inter-modal associations. Then, in feature interaction level, we
design a Feature Interaction Module (FIM) based on dual-stream structure, which
enhances intra-modal features in parallel and constructs inter-modal semantic
associations. Meanwhile, in order to refine each modal feature, we introduce a
Coarse-to-Fine Interaction Module (CFIM) to realize the hierarchical feature
interaction at different scales. Finally, in similarity fusion level, we
propose a Similarity Fusion Module (SFM) to aggregate geometry and texture
clues from the target. Experiments show that our method achieves
state-of-the-art performance on KITTI (39% Success and 42% Precision gains
against previous multi-modal method) and is also competitive on NuScenes.

提出 MMF-Track，一种多模态多级融合跟踪器，利用点云的几何和图像纹理特征跟踪 3D 目标，在 KITTI 上实现了最先进的性能表现，并在 NuScenes 上具有竞争力。

多模态多层次融合的三维单目标跟踪

Multi-modal Multi-level Fusion for 3D Single Object Tracking

Matching local geometric features on real-world depth images is a challenging
task due to the noisy, low-resolution, and incomplete nature of 3D scan data.
These difficulties limit the performance of current state-of-art methods, which
are typically based on histograms over geometric properties. In this paper, we
present 3DMatch, a data-driven model that learns a local volumetric patch
descriptor for establishing correspondences between partial 3D data. To amass
training data for our model, we propose a self-supervised feature learning
method that leverages the millions of correspondence labels found in existing
RGB-D reconstructions. Experiments show that our descriptor is not only able to
match local geometry in new scenes for reconstruction, but also generalize to
different tasks and spatial scales (e.g. instance-level object model alignment
for the Amazon Picking Challenge, and mesh surface correspondence). Results
show that 3DMatch consistently outperforms other state-of-the-art approaches by
a significant margin. Code, data, benchmarks, and pre-trained models are
available online at this http URL

本文提出了一种名为 3DMatch 的数据驱动模型，它可以学习用于建立局部 3D 数据对应关系的局部体积块描述符，且利用自我监督特征学习方法来汇集训练数据。实验证明，该描述符不仅可以用于重构新场景的局部几何形状的匹配，而且可以推广到不同的任务和空间尺度。