Multimodal Visual Object Tracking (VOT) has recently gained significant
attention due to its robustness. Early research focused on fully fine-tuning
RGB-based trackers, which was inefficient and lacked generalized representation
due to the scarcity of multimodal data. Therefore, recent studies have utilized
prompt tuning to transfer pre-trained RGB-based trackers to multimodal data.
However, the modality gap limits pre-trained knowledge recall, and the
dominance of the RGB modality persists, preventing the full utilization of
information from other modalities. To address these issues, we propose a novel
symmetric multimodal tracking framework called SDSTrack. We introduce
lightweight adaptation for efficient fine-tuning, which directly transfers the
feature extraction ability from RGB to other domains with a small number of
trainable parameters and integrates multimodal features in a balanced,
symmetric manner. Furthermore, we design a complementary masked patch
distillation strategy to enhance the robustness of trackers in complex
environments, such as extreme weather, poor imaging, and sensor failure.
Extensive experiments demonstrate that SDSTrack outperforms state-of-the-art
methods in various multimodal tracking scenarios, including RGB+Depth,
RGB+Thermal, and RGB+Event tracking, and exhibits impressive results in extreme
conditions. Our source code is available at this https URL

我们提出了一种新的对称多模态跟踪框架 SDSTrack，通过轻量级适应和补充掩膜路径蒸馏策略，提高了跟踪器在复杂环境中的鲁棒性，并在各种多模态跟踪场景中表现出优异结果。

SDSTrack: 自蒸馏对称适配器学习用于多模态视觉目标跟踪

SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal  Visual Object Tracking

Most existing RGB-based trackers target low frame rate benchmarks of around
30 frames per second. This setting restricts the tracker's functionality in the
real world, especially for fast motion. Event-based cameras as bioinspired
sensors provide considerable potential for high frame rate tracking due to
their high temporal resolution. However, event-based cameras cannot offer
fine-grained texture information like conventional cameras. This unique
complementarity motivates us to combine conventional frames and events for high
frame rate object tracking under various challenging conditions. Inthispaper,
we propose an end-to-end network consisting of multi-modality alignment and
fusion modules to effectively combine meaningful information from both
modalities at different measurement rates. The alignment module is responsible
for cross-style and cross-frame-rate alignment between frame and event
modalities under the guidance of the moving cues furnished by events. While the
fusion module is accountable for emphasizing valuable features and suppressing
noise information by the mutual complement between the two modalities.
Extensive experiments show that the proposed approach outperforms
state-of-the-art trackers by a significant margin in high frame rate tracking.
With the FE240hz dataset, our approach achieves high frame rate tracking up to
240Hz.

该论文提出了使用多模方式结合 RGB-based trackers 和 event-based cameras 进行 high frame rate tracking 的方法，并应用 multi-modality alignment 和 fusion modules 对两种模式的信息进行融合。在 FE240hz 数据集上，该方法表现优异，实现了高达 240Hz 的高帧率追踪。