Tracking using bio-inspired event cameras has drawn more and more attention
in recent years. Existing works either utilize aligned RGB and event data for
accurate tracking or directly learn an event-based tracker. The first category
needs more cost for inference and the second one may be easily influenced by
noisy events or sparse spatial resolution. In this paper, we propose a novel
hierarchical knowledge distillation framework that can fully utilize
multi-modal / multi-view information during training to facilitate knowledge
transfer, enabling us to achieve high-speed and low-latency visual tracking
during testing by using only event signals. Specifically, a teacher
Transformer-based multi-modal tracking framework is first trained by feeding
the RGB frame and event stream simultaneously. Then, we design a new
hierarchical knowledge distillation strategy which includes pairwise
similarity, feature representation, and response maps-based knowledge
distillation to guide the learning of the student Transformer network.
Moreover, since existing event-based tracking datasets are all low-resolution
($346 \times 260$), we propose the first large-scale high-resolution ($1280
\times 720$) dataset named EventVOT. It contains 1141 videos and covers a wide
range of categories such as pedestrians, vehicles, UAVs, ping pongs, etc.
Extensive experiments on both low-resolution (FE240hz, VisEvent, COESOT), and
our newly proposed high-resolution EventVOT dataset fully validated the
effectiveness of our proposed method. The dataset, evaluation toolkit, and
source code are available on
https://github.com/Event-AHU/EventVOT_Benchmark

通过多模态跟踪和知识蒸馏方法，在高分辨率事件信号下实现了高速低延迟的视觉跟踪的研究。

基于事件流的视觉目标跟踪：一个高分辨率基准数据集和一种新的基线模型

Event Stream-based Visual Object Tracking: A High-Resolution Benchmark  Dataset and A Novel Baseline

Generative models (GMs) have received increasing research interest for their
remarkable capacity to achieve comprehensive understanding. However, their
potential application in the domain of multi-modal tracking has remained
relatively unexplored. In this context, we seek to uncover the potential of
harnessing generative techniques to address the critical challenge, information
fusion, in multi-modal tracking. In this paper, we delve into two prominent GM
techniques, namely, Conditional Generative Adversarial Networks (CGANs) and
Diffusion Models (DMs). Different from the standard fusion process where the
features from each modality are directly fed into the fusion block, we
condition these multi-modal features with random noise in the GM framework,
effectively transforming the original training samples into harder instances.
This design excels at extracting discriminative clues from the features,
enhancing the ultimate tracking performance. To quantitatively gauge the
effectiveness of our approach, we conduct extensive experiments across two
multi-modal tracking tasks, three baseline methods, and three challenging
benchmarks. The experimental results demonstrate that the proposed
generative-based fusion mechanism achieves state-of-the-art performance,
setting new records on LasHeR and RGBD1K.

通过使用生成模型的融合机制，本研究在多模态跟踪中展示了生成模型的潜力，并通过实验证明该方法取得了最新的性能记录。

基于生成的融合机制的多模态追踪

Generative-based Fusion Mechanism for Multi-Modal Tracking

Visible-modal object tracking gives rise to a series of downstream
multi-modal tracking tributaries. To inherit the powerful representations of
the foundation model, a natural modus operandi for multi-modal tracking is full
fine-tuning on the RGB-based parameters. Albeit effective, this manner is not
optimal due to the scarcity of downstream data and poor transferability, etc.
In this paper, inspired by the recent success of the prompt learning in
language models, we develop Visual Prompt multi-modal Tracking (ViPT), which
learns the modal-relevant prompts to adapt the frozen pre-trained foundation
model to various downstream multimodal tracking tasks. ViPT finds a better way
to stimulate the knowledge of the RGB-based model that is pre-trained at scale,
meanwhile only introducing a few trainable parameters (less than 1% of model
parameters). ViPT outperforms the full fine-tuning paradigm on multiple
downstream tracking tasks including RGB+Depth, RGB+Thermal, and RGB+Event
tracking. Extensive experiments show the potential of visual prompt learning
for multi-modal tracking, and ViPT can achieve state-of-the-art performance
while satisfying parameter efficiency. Code and models are available at
this https URL

本文提出了一种基于视觉提示学习的多模态跟踪方法，通过学习模态相关的提示来适应预训练好的基础模型，在 RGB + 深度、RGB + 热等多个跟踪任务中表现优秀，实现了参数效率和最先进性的平衡。

视觉提示的多模式跟踪

Visual Prompt Multi-Modal Tracking

Multi-modal tracking gains attention due to its ability to be more accurate
and robust in complex scenarios compared to traditional RGB-based tracking. Its
key lies in how to fuse multi-modal data and reduce the gap between modalities.
However, multi-modal tracking still severely suffers from data deficiency, thus
resulting in the insufficient learning of fusion modules. Instead of building
such a fusion module, in this paper, we provide a new perspective on
multi-modal tracking by attaching importance to the multi-modal visual prompts.
We design a novel multi-modal prompt tracker (ProTrack), which can transfer the
multi-modal inputs to a single modality by the prompt paradigm. By best
employing the tracking ability of pre-trained RGB trackers learning at scale,
our ProTrack can achieve high-performance multi-modal tracking by only altering
the inputs, even without any extra training on multi-modal data. Extensive
experiments on 5 benchmark datasets demonstrate the effectiveness of the
proposed ProTrack.

本研究提出一种基于 prompt 范式的多模态 prompt 跟踪器 ProTrack，通过最大程度地利用 RGB 跟踪器预训练的跟踪能力，仅通过修改输入即可实现高性能的多模态跟踪，对 5 个基准数据集进行的大量实验证明了 ProTrack 的有效性。