The target representation learned by convolutional neural networks plays an
important role in Thermal Infrared (TIR) tracking. Currently, most of the
top-performing TIR trackers are still employing representations learned by the
model trained on the RGB data. However, this representation does not take into
account the information in the TIR modality itself, limiting the performance of
TIR tracking. To solve this problem, we propose to distill representations of
the TIR modality from the RGB modality with Cross-Modal Distillation (CMD) on a
large amount of unlabeled paired RGB-TIR data. We take advantage of the
two-branch architecture of the baseline tracker, i.e. DiMP, for cross-modal
distillation working on two components of the tracker. Specifically, we use one
branch as a teacher module to distill the representation learned by the model
into the other branch. Benefiting from the powerful model in the RGB modality,
the cross-modal distillation can learn the TIR-specific representation for
promoting TIR tracking. The proposed approach can be incorporated into
different baseline trackers conveniently as a generic and independent
component. Furthermore, the semantic coherence of paired RGB and TIR images is
utilized as a supervised signal in the distillation loss for cross-modal
knowledge transfer. In practice, three different approaches are explored to
generate paired RGB-TIR patches with the same semantics for training in an
unsupervised way. It is easy to extend to an even larger scale of unlabeled
training data. Extensive experiments on the LSOTB-TIR dataset and PTB-TIR
dataset demonstrate that our proposed cross-modal distillation method
effectively learns TIR-specific target representations transferred from the RGB
modality. Our tracker outperforms the baseline tracker by achieving absolute
gains of 2.3% Success, 2.7% Precision, and 2.5% Normalized Precision
respectively.

本研究提出利用跨模态蒸馏技术从大量无标签 RGB-TIR 数据中提炼 TIR 模态的目标表示，作为热红外追踪的优化方法，并利用 RGB 和 TIR 图像中的语义信息进行 “蒸馏损失” 的监督学习，成功地推广到 LSOTB-TIR 和 PTB-TIR 数据集中，比基线跟踪器的绝对增益显著。

热红外跟踪的无监督跨模态蒸馏

Unsupervised Cross-Modal Distillation for Thermal Infrared Tracking

Most thermal infrared (TIR) tracking methods are discriminative, treating the
tracking problem as a classification task. However, the objective of the
classifier (label prediction) is not coupled to the objective of the tracker
(location estimation). The classification task focuses on the between-class
difference of the arbitrary objects, while the tracking task mainly deals with
the within-class difference of the same objects. In this paper, we cast the TIR
tracking problem as a similarity verification task, which is coupled well to
the objective of the tracking task. We propose a TIR tracker via a Hierarchical
Spatial-aware Siamese Convolutional Neural Network (CNN), named HSSNet. To
obtain both spatial and semantic features of the TIR object, we design a
Siamese CNN that coalesces the multiple hierarchical convolutional layers.
Then, we propose a spatial-aware network to enhance the discriminative ability
of the coalesced hierarchical feature. Subsequently, we train this network end
to end on a large visible video detection dataset to learn the similarity
between paired objects before we transfer the network into the TIR domain.
Next, this pre-trained Siamese network is used to evaluate the similarity
between the target template and target candidates. Finally, we locate the
candidate that is most similar to the tracked target. Extensive experimental
results on the benchmarks VOT-TIR 2015 and VOT-TIR 2016 show that our proposed
method achieves favourable performance compared to the state-of-the-art
methods.

本文提出了一种热红外跟踪器，命名为 HSSNet，它使用层次空间感知 Siamese 卷积神经网络来验证相似性并生成更强的可辨别能力，与其他方法相比，在 VOT-TIR 2015 和 VOT-TIR 2016 基准测试上取得了较好的性能表现。