Current RGBT tracking researches mainly focus on the modality-complete
scenarios, overlooking the modality-missing challenge in real-world scenes. In
this work, we comprehensively investigate the impact of modality-missing
challenge in RGBT tracking and propose a novel invertible prompt learning
approach, which integrates the content-preserving prompts into a well-trained
tracking model to adapt to various modality-missing scenarios, for
modality-missing RGBT tracking. In particular, given one modality-missing
scenario, we propose to utilize the available modality to generate the prompt
of the missing modality to adapt to RGBT tracking model. However, the
cross-modality gap between available and missing modalities usually causes
semantic distortion and information loss in prompt generation. To handle this
issue, we propose the invertible prompt learning scheme by incorporating the
full reconstruction of the input available modality from the prompt in prompt
generation model. Considering that there lacks a modality-missing RGBT tracking
dataset and many modality-missing scenarios are difficult to capture, we design
a high-quality data simulation method based on hierarchical combination schemes
to generate real-world modality-missing data. Extensive experiments on three
modality-missing datasets show that our method achieves significant performance
improvements compared with state-of-the-art methods. We will release the code
and simulation dataset.

在当前 RGBT 跟踪研究中，主要关注完整模态的场景，忽视了现实场景中模态缺失的挑战。本文综合研究了模态缺失挑战对 RGBT 跟踪的影响，并提出了一种新的可逆提示学习方法，将保留内容的提示集成到训练充分的跟踪模型中，以适应各种模态缺失场景，用于模态缺失的 RGBT 跟踪。

通过可逆提示学习和高质量数据模拟的遗失模态 RGBT 跟踪

Modality-missing RGBT Tracking via Invertible Prompt Learning and A  High-quality Data Simulation Method

In this paper, we address multi-modal pretraining of product data in the
field of E-commerce. Current multi-modal pretraining methods proposed for image
and text modalities lack robustness in the face of modality-missing and
modality-noise, which are two pervasive problems of multi-modal product data in
real E-commerce scenarios. To this end, we propose a novel method, K3M, which
introduces knowledge modality in multi-modal pretraining to correct the noise
and supplement the missing of image and text modalities. The modal-encoding
layer extracts the features of each modality. The modal-interaction layer is
capable of effectively modeling the interaction of multiple modalities, where
an initial-interactive feature fusion model is designed to maintain the
independence of image modality and text modality, and a structure aggregation
module is designed to fuse the information of image, text, and knowledge
modalities. We pretrain K3M with three pretraining tasks, including masked
object modeling (MOM), masked language modeling (MLM), and link prediction
modeling (LPM). Experimental results on a real-world E-commerce dataset and a
series of product-based downstream tasks demonstrate that K3M achieves
significant improvements in performances than the baseline and state-of-the-art
methods when modality-noise or modality-missing exists.

该论文提出一种名为 K3M 的新方法，通过引入知识模态来进行多模态预训练，以解决实际 E-commerce 场景中的产品数据中存在的多模态噪声和缺失问题，并在真实世界的 E-commerce 数据集和一系列基于产品的下游任务上显示出明显的性能提升。