Most existing RGB-T tracking networks extract modality features in a separate
manner, which lacks interaction and mutual guidance between modalities. This
limits the network's ability to adapt to the diverse dual-modality appearances
of targets and the dynamic relationships between the modalities. Additionally,
the three-stage fusion tracking paradigm followed by these networks
significantly restricts the tracking speed. To overcome these problems, we
propose a unified single-stage Transformer RGB-T tracking network, namely
USTrack, which unifies the above three stages into a single ViT (Vision
Transformer) backbone with a dual embedding layer through self-attention
mechanism. With this structure, the network can extract fusion features of the
template and search region under the mutual interaction of modalities.
Simultaneously, relation modeling is performed between these features,
efficiently obtaining the search region fusion features with better
target-background discriminability for prediction. Furthermore, we introduce a
novel feature selection mechanism based on modality reliability to mitigate the
influence of invalid modalities for prediction, further improving the tracking
performance. Extensive experiments on three popular RGB-T tracking benchmarks
demonstrate that our method achieves new state-of-the-art performance while
maintaining the fastest inference speed 84.2FPS. In particular, MPR/MSR on the
short-term and long-term subsets of VTUAV dataset increased by
11.1$\%$/11.7$\%$ and 11.3$\%$/9.7$\%$.

我们提出了一个统一的单阶段 Transformer RGB-T 跟踪网络，名为 USTrack，它通过自注意机制将上述三个阶段统一到一个 ViT（Vision Transformer）主干中，并利用模态之间的相互作用提取融合特征，增强预测的目标 - 背景区分度，同时通过模态可靠性的特征选择机制改善跟踪性能。通过在三个流行的 RGB-T 跟踪基准上进行广泛实验，证明我们的方法在保持最快推理速度 84.2FPS 的同时，实现了新的最先进性能，特别是在 VTUAV 数据集的短期和长期子集上，MPR/MSR 分别增加了 11.1% 和 11.3%。

高效 RGB-T 跟踪的统一单级 Transformer 网络

Unified Single-Stage Transformer Network for Efficient RGB-T Tracking

RGB-T semantic segmentation has been widely adopted to handle hard scenes
with poor lighting conditions by fusing different modality features of RGB and
thermal images. Existing methods try to find an optimal fusion feature for
segmentation, resulting in sensitivity to modality noise, class-imbalance, and
modality bias. To overcome the problems, this paper proposes a novel
Variational Probabilistic Fusion Network (VPFNet), which regards fusion
features as random variables and obtains robust segmentation by averaging
segmentation results under multiple samples of fusion features. The random
samples generation of fusion features in VPFNet is realized by a novel
Variational Feature Fusion Module (VFFM) designed based on variation attention.
To further avoid class-imbalance and modality bias, we employ the weighted
cross-entropy loss and introduce prior information of illumination and category
to control the proposed VFFM. Experimental results on MFNet and PST900 datasets
demonstrate that the proposed VPFNet can achieve state-of-the-art segmentation
performance.

本研究提出了一种基于变分推断的概率融合网络（VPFNet），采用随机特征样本平均，引入加权交叉熵损失函数、光照和类别先验信息，应对 RGB 和热红外图像融合语义分割中的模态噪声、类不平衡和模态偏差问题，在 MFNet 和 PST900 数据集上实验结果显示该网络取得了最佳的分割性能。