This paper extends the popular task of multi-object tracking to multi-object tracking and segmentation (MOTS). Towards this goal, we create dense pixel-level annotations for two existing tracking datasets using a semi-automatic annotation procedure. Our new annotations comprise 70,430 pixel masks for 1,084 distinct objects (cars and pedestrians) in 10,870 video frames. For evaluation, we extend existing multi-object tracking metrics to this new task. Moreover, we propose a new baseline method which jointly addresses detection, tracking, and segmentation with a single convolutional network. We demonstrate the value of our datasets by achieving improvements in performance when training on MOTS annotations. We believe that our datasets, metrics and baseline will become a valuable resource towards developing multi-object tracking approaches that go beyond 2D bounding boxes.

本论文将多目标跟踪扩展到多目标跟踪和分割（MOTS）。为此，我们使用半自动注释程序为两个现有的跟踪数据集创建密集的像素级注释。通过我们的新注释，我们提出了一个新的基线方法，该方法使用单个卷积网络共同处理检测、跟踪和分割。我们演示了我们数据集的价值，在 MOTS 注释上训练可以提高性能，我们相信我们的数据集、指标和基线方法将成为开发超越 2D 边界框的多目标跟踪方法的宝贵资源。

MOTS: 多目标跟踪和分割