Visual tracking often faces challenges such as invalid targets and decreased
performance in low-light conditions when relying solely on RGB image sequences.
While incorporating additional modalities like depth and infrared data has
proven effective, existing multi-modal imaging platforms are complex and lack
real-world applicability. In contrast, near-infrared (NIR) imaging, commonly
used in surveillance cameras, can switch between RGB and NIR based on light
intensity. However, tracking objects across these heterogeneous modalities
poses significant challenges, particularly due to the absence of modality
switch signals during tracking. To address these challenges, we propose an
adaptive cross-modal object tracking algorithm called Modality-Aware Fusion
Network (MAFNet). MAFNet efficiently integrates information from both RGB and
NIR modalities using an adaptive weighting mechanism, effectively bridging the
appearance gap and enabling a modality-aware target representation. It consists
of two key components: an adaptive weighting module and a modality-specific
representation module......

提出了一种自适应跨模态目标跟踪算法 Modality-Aware Fusion Network（MAFNet），它通过自适应加权机制有效地整合 RGB 和近红外模态的信息，弥合了外观差距，实现了模态感知的目标表示。

跨模式目标跟踪通过模态感知融合网络和大规模数据集

Cross-Modal Object Tracking via Modality-Aware Fusion Network and A  Large-Scale Dataset

Cross-modal object tracking is an important research topic in the field of
information fusion, and it aims to address imaging limitations in challenging
scenarios by integrating switchable visible and near-infrared modalities.
However, existing tracking methods face some difficulties in adapting to
significant target appearance variations in the presence of modality switch.
For instance, model update based tracking methods struggle to maintain stable
tracking results during modality switching, leading to error accumulation and
model drift. Template based tracking methods solely rely on the template
information from first frame and/or last frame, which lacks sufficient
representation ability and poses challenges in handling significant target
appearance changes. To address this problem, we propose a prototype-based
cross-modal object tracker called ProtoTrack, which introduces a novel
prototype learning scheme to adapt to significant target appearance variations,
for cross-modal object tracking. In particular, we design a multi-modal
prototype to represent target information by multi-kind samples, including a
fixed sample from the first frame and two representative samples from different
modalities. Moreover, we develop a prototype generation algorithm based on two
new modules to ensure the prototype representative in different
challenges......

我们提出了一种基于原型的跨模式对象跟踪器 ProtoTrack，通过引入新的原型学习方案来适应显著的目标外观变化，解决了现有跟踪方法在模态切换时的困难。

基于原型的跨模态物体跟踪

Prototype-based Cross-Modal Object Tracking

This paper addresses the problem of cross-modal object tracking from RGB
videos and event data. Rather than constructing a complex cross-modal fusion
network, we explore the great potential of a pre-trained vision Transformer
(ViT). Particularly, we delicately investigate plug-and-play training
augmentations that encourage the ViT to bridge the vast distribution gap
between the two modalities, enabling comprehensive cross-modal information
interaction and thus enhancing its ability. Specifically, we propose a mask
modeling strategy that randomly masks a specific modality of some tokens to
enforce the interaction between tokens from different modalities interacting
proactively. To mitigate network oscillations resulting from the masking
strategy and further amplify its positive effect, we then theoretically propose
an orthogonal high-rank loss to regularize the attention matrix. Extensive
experiments demonstrate that our plug-and-play training augmentation techniques
can significantly boost state-of-the-art one-stream and twostream trackers to a
large extent in terms of both tracking precision and success rate. Our new
perspective and findings will potentially bring insights to the field of
leveraging powerful pre-trained ViTs to model cross-modal data. The code will
be publicly available.

本研究针对 RGB 视频和事件数据的跨模态物体追踪问题，通过利用预先训练的视觉变换器 (ViT) 的巨大潜力，探索了构建复杂的跨模态融合网络的可能性。特别地，我们精心研究了一种插件式训练增强方法，以鼓励 ViT 填补两种模态之间巨大分布差异，并增强其相互作用，从而提高其能力。具体而言，我们提出了一种掩码建模策略，随机屏蔽一些标记，增加了交互效果，使用高阶模型进行正则化。通过广泛的实验验证，我们的插件式训练增强技术在跟踪精度和成功率等方面极大提升了最先进的单流和双流跟踪器，有望为跨模态数据建模的领域带来新的见解。代码将公开提供。