Existing referring understanding tasks tend to involve the detection of a single text-referred object. In this paper, we propose a new and general referring understanding task, termed referring multi-object tracking (RMOT). Its core idea is to employ a language expression as a semantic cue to guide the prediction of multi-object tracking. To the best of our knowledge, it is the first work to achieve an arbitrary number of referent object predictions in videos. To push forward RMOT, we construct one benchmark with scalable expressions based on KITTI, named Refer-KITTI. Specifically, it provides 18 videos with 818 expressions, and each expression in a video is annotated with an average of 10.7 objects. Further, we develop a transformer-based architecture TransRMOT to tackle the new task in an online manner, which achieves impressive detection performance and outperforms other counterparts.

本文提出了一个新的、通用的指代多目标跟踪（RMOT）任务，其核心思想是利用语言表达作为语义线索来引导多目标跟踪的预测。作者构建了一个基于KITTI的可扩展表达基准Refer-KITTI，并开发了一个基于变压器的TransRMOT架构来解决该任务，在线方式取得了令人瞩目的检测性能，并超越了其他对手。

多目标引用跟踪