With the emergence of a single large model capable of successfully solving a
multitude of tasks in NLP, there has been growing research interest in
achieving similar goals in computer vision. On the one hand, most of these
generic models, referred to as generalist vision models, aim at producing
unified outputs serving different tasks. On the other hand, some existing
models aim to combine different input types (aka data modalities), which are
then processed by a single large model. Yet, this step of combination remains
specialized, which falls short of serving the initial ambition. In this paper,
we showcase that such specialization (during unification) is unnecessary, in
the context of RGB-X video object tracking. Our single model tracker, termed
XTrack, can remain blind to any modality X during inference time. Our tracker
employs a mixture of modal experts comprising those dedicated to shared
commonality and others capable of flexibly performing reasoning conditioned on
input modality. Such a design ensures the unification of input modalities
towards a common latent space, without weakening the modality-specific
information representation. With this idea, our training process is extremely
simple, integrating multi-label classification loss with a routing function,
thereby effectively aligning and unifying all modalities together, even from
only paired data. Thus, during inference, we can adopt any modality without
relying on the inductive bias of the modal prior and achieve generalist
performance. Without any bells and whistles, our generalist and blind tracker
can achieve competitive performance compared to well-established modal-specific
models on 5 benchmarks across 3 auxiliary modalities, covering commonly used
depth, thermal, and event data.

我们展示了在 RGB-X 视频目标跟踪上，在统一化时的专门化不必要，通过使用一种称为 XTrack 的单一模型跟踪器，在推理过程中对任何模态 X 保持盲目，以实现输入模态的统一，并实现专门模态的信息表示，从只有配对数据开始，通过简单的训练过程有效地整合了多标签分类损失和路由函数，从而达到了与模态特定模型相媲美的性能。