The ability to detect and track the dynamic objects in different scenes is
fundamental to real-world applications, e.g., autonomous driving and robot
navigation. However, traditional Multi-Object Tracking (MOT) is limited to
tracking objects belonging to the pre-defined closed-set categories. Recently,
Open-Vocabulary MOT (OVMOT) and Generic MOT (GMOT) are proposed to track
interested objects beyond pre-defined categories with the given text prompt and
template image. However, the expensive well pre-trained (vision-)language model
and fine-grained category annotations are required to train OVMOT models. In
this paper, we focus on GMOT and propose a simple but effective method,
Siamese-DETR, for GMOT. Only the commonly used detection datasets (e.g., COCO)
are required for training. Different from existing GMOT methods, which train a
Single Object Tracking (SOT) based detector to detect interested objects and
then apply a data association based MOT tracker to get the trajectories, we
leverage the inherent object queries in DETR variants. Specifically: 1) The
multi-scale object queries are designed based on the given template image,
which are effective for detecting different scales of objects with the same
category as the template image; 2) A dynamic matching training strategy is
introduced to train Siamese-DETR on commonly used detection datasets, which
takes full advantage of provided annotations; 3) The online tracking pipeline
is simplified through a tracking-by-query manner by incorporating the tracked
boxes in previous frame as additional query boxes. The complex data association
is replaced with the much simpler Non-Maximum Suppression (NMS). Extensive
experimental results show that Siamese-DETR surpasses existing MOT methods on
GMOT-40 dataset by a large margin.

本文针对 Generic MOT 提出了一种简单而有效的方法 Siamese-DETR，通过基于给定模板图像设计多尺度的对象查询，引入动态匹配训练策略，并通过在前一帧中添加跟踪框作为额外的查询框，简化了在线跟踪流程，将复杂的数据关联替换为更简单的非极大值抑制方法。广泛的实验证明，Siamese-DETR 在 GMOT-40 数据集上远远超过现有的 MOT 方法。

用于通用多目标跟踪的孪生 DETR

Siamese-DETR for Generic Multi-Object Tracking

The ability to recognize, localize and track dynamic objects in a scene is
fundamental to many real-world applications, such as self-driving and robotic
systems. Yet, traditional multiple object tracking (MOT) benchmarks rely only
on a few object categories that hardly represent the multitude of possible
objects that are encountered in the real world. This leaves contemporary MOT
methods limited to a small set of pre-defined object categories. In this paper,
we address this limitation by tackling a novel task, open-vocabulary MOT, that
aims to evaluate tracking beyond pre-defined training categories. We further
develop OVTrack, an open-vocabulary tracker that is capable of tracking
arbitrary object classes. Its design is based on two key ingredients: First,
leveraging vision-language models for both classification and association via
knowledge distillation; second, a data hallucination strategy for robust
appearance feature learning from denoising diffusion probabilistic models. The
result is an extremely data-efficient open-vocabulary tracker that sets a new
state-of-the-art on the large-scale, large-vocabulary TAO benchmark, while
being trained solely on static images. Project page:
this https URL

本研究解决了传统多目标跟踪方法只针对少数预定义对象类型的固有限制，并提出了一种新的任务 Open-vocabulary MOT，进一步开发出一种数据效率优异的开放词汇跟踪器 OVTrack，通过知识蒸馏和数据幻觉策略有效提升图像分类和关联准确性，最终在大规模的 TAO 基准测试上取得了最新的最优效果。