We propose an end-to-end tracking framework for fusing the RGB and TIR
modalities in RGB-T tracking. Our baseline tracker is DiMP (Discriminative
Model Prediction), which employs a carefully designed target prediction network
trained end-to-end using a discriminative loss. We analyze the effectiveness of
modality fusion in each of the main components in DiMP, i.e. feature extractor,
target estimation network, and classifier. We consider several fusion
mechanisms acting at different levels of the framework, including pixel-level,
feature-level and response-level. Our tracker is trained in an end-to-end
manner, enabling the components to learn how to fuse the information from both
modalities. As data to train our model, we generate a large-scale RGB-T dataset
by considering an annotated RGB tracking dataset (GOT-10k) and synthesizing
paired TIR images using an image-to-image translation approach. We perform
extensive experiments on VOT-RGBT2019 dataset and RGBT210 dataset, evaluating
each type of modality fusing on each model component. The results show that the
proposed fusion mechanisms improve the performance of the single modality
counterparts. We obtain our best results when fusing at the feature-level on
both the IoU-Net and the model predictor, obtaining an EAO score of 0.391 on
VOT-RGBT2019 dataset. With this fusion mechanism we achieve the
state-of-the-art performance on RGBT210 dataset.

提出了一种融合红外和可见光图像的端到端相似跟踪框架，这个框架通过特征级别的融合机制，在特征提取、目标估计网络和分类器三个主要部件中提高了单模态组件的表现表现，并在 VOT-RGBT2019 数据集上获得了 0.391 的准确率，进而取得 RGBT210 数据集的最佳表现。