Existing datasets for RGB-DVS tracking are collected with DVS346 camera and
their resolution ($346 \times 260$) is low for practical applications.
Actually, only visible cameras are deployed in many practical systems, and the
newly designed neuromorphic cameras may have different resolutions. The latest
neuromorphic sensors can output high-definition event streams, but it is very
difficult to achieve strict alignment between events and frames on both spatial
and temporal views. Therefore, how to achieve accurate tracking with unaligned
neuromorphic and visible sensors is a valuable but unresearched problem. In
this work, we formally propose the task of object tracking using unaligned
neuromorphic and visible cameras. We build the first unaligned frame-event
dataset CRSOT collected with a specially built data acquisition system, which
contains 1,030 high-definition RGB-Event video pairs, 304,974 video frames. In
addition, we propose a novel unaligned object tracking framework that can
realize robust tracking even using the loosely aligned RGB-Event data.
Specifically, we extract the template and search regions of RGB and Event data
and feed them into a unified ViT backbone for feature embedding. Then, we
propose uncertainty perception modules to encode the RGB and Event features,
respectively, then, we propose a modality uncertainty fusion module to
aggregate the two modalities. These three branches are jointly optimized in the
training phase. Extensive experiments demonstrate that our tracker can
collaborate the dual modalities for high-performance tracking even without
strictly temporal and spatial alignment. The source code, dataset, and
pre-trained models will be released at
this https URL

使用不对齐的神经形态和可见光相机进行目标跟踪，提出了一种新的跟踪框架，能够通过提取 RGB 和 Event 数据的特征，并利用不确定性感知模块和融合模块实现高性能的跟踪。

CRSOT: 使用非对准帧和事件相机进行跨分辨率目标跟踪

CRSOT: Cross-Resolution Object Tracking using Unaligned Frame and Event  Cameras

Most existing RGB-T tracking networks extract modality features in a separate
manner, which lacks interaction and mutual guidance between modalities. This
limits the network's ability to adapt to the diverse dual-modality appearances
of targets and the dynamic relationships between the modalities. Additionally,
the three-stage fusion tracking paradigm followed by these networks
significantly restricts the tracking speed. To overcome these problems, we
propose a unified single-stage Transformer RGB-T tracking network, namely
USTrack, which unifies the above three stages into a single ViT (Vision
Transformer) backbone with a dual embedding layer through self-attention
mechanism. With this structure, the network can extract fusion features of the
template and search region under the mutual interaction of modalities.
Simultaneously, relation modeling is performed between these features,
efficiently obtaining the search region fusion features with better
target-background discriminability for prediction. Furthermore, we introduce a
novel feature selection mechanism based on modality reliability to mitigate the
influence of invalid modalities for prediction, further improving the tracking
performance. Extensive experiments on three popular RGB-T tracking benchmarks
demonstrate that our method achieves new state-of-the-art performance while
maintaining the fastest inference speed 84.2FPS. In particular, MPR/MSR on the
short-term and long-term subsets of VTUAV dataset increased by
11.1$\%$/11.7$\%$ and 11.3$\%$/9.7$\%$.

我们提出了一个统一的单阶段 Transformer RGB-T 跟踪网络，名为 USTrack，它通过自注意机制将上述三个阶段统一到一个 ViT（Vision Transformer）主干中，并利用模态之间的相互作用提取融合特征，增强预测的目标 - 背景区分度，同时通过模态可靠性的特征选择机制改善跟踪性能。通过在三个流行的 RGB-T 跟踪基准上进行广泛实验，证明我们的方法在保持最快推理速度 84.2FPS 的同时，实现了新的最先进性能，特别是在 VTUAV 数据集的短期和长期子集上，MPR/MSR 分别增加了 11.1% 和 11.3%。