Multi-modal object tracking (MMOT) is an emerging field that combines data
from various modalities, \eg vision (RGB), depth, thermal infrared, event,
language and audio, to estimate the state of an arbitrary object in a video
sequence. It is of great significance for many applications such as autonomous
driving and intelligent surveillance. In recent years, MMOT has received more
and more attention. However, existing MMOT algorithms mainly focus on two
modalities (\eg RGB+depth, RGB+thermal infrared, and RGB+language). To leverage
more modalities, some recent efforts have been made to learn a unified visual
object tracking model for any modality. Additionally, some large-scale
multi-modal tracking benchmarks have been established by simultaneously
providing more than two modalities, such as vision-language-audio (\eg
WebUAV-3M) and vision-depth-language (\eg UniMod1K). To track the latest
progress in MMOT, we conduct a comprehensive investigation in this report.
Specifically, we first divide existing MMOT tasks into five main categories,
\ie RGBL tracking, RGBE tracking, RGBD tracking, RGBT tracking, and
miscellaneous (RGB+X), where X can be any modality, such as language, depth,
and event. Then, we analyze and summarize each MMOT task, focusing on widely
used datasets and mainstream tracking algorithms based on their technical
paradigms (\eg self-supervised learning, prompt learning, knowledge
distillation, generative models, and state space models). Finally, we maintain
a continuously updated paper list for MMOT at
this https URL

多模态目标跟踪是一个新兴领域，它利用来自各种模态的数据（比如视觉、深度、红外热像、事件、语言和音频）来估计视频序列中任意物体的状态。本研究综述了现有多模态目标跟踪任务，并分析总结了各个任务的常用数据集和主流的跟踪算法，重点关注了它们基于自监督学习、提示学习、知识蒸馏、生成模型和状态空间模型等技术范式。同时，在此网址上维护了一个持续更新的多模态目标跟踪论文清单。