In this work, we present SeqFormer for video instance segmentation. SeqFormer
follows the principle of vision transformer that models instance relationships
among video frames. Nevertheless, we observe that a stand-alone instance query
suffices for capturing a time sequence of instances in a video, but attention
mechanisms shall be done with each frame independently. To achieve this,
SeqFormer locates an instance in each frame and aggregates temporal information
to learn a powerful representation of a video-level instance, which is used to
predict the mask sequences on each frame dynamically. Instance tracking is
achieved naturally without tracking branches or post-processing. On
YouTube-VIS, SeqFormer achieves 47.4 AP with a ResNet-50 backbone and 49.0 AP
with a ResNet-101 backbone without bells and whistles. Such achievement
significantly exceeds the previous state-of-the-art performance by 4.6 and 4.4,
respectively. In addition, integrated with the recently-proposed Swin
transformer, SeqFormer achieves a much higher AP of 59.3. We hope SeqFormer
could be a strong baseline that fosters future research in video instance
segmentation, and in the meantime, advances this field with a more robust,
accurate, neat model. The code is available at
this https URL

本文介绍了 SeqFormer，一种基于视频实例分割的模型，通过注意力机制捕捉视频帧之间的关系，定位每帧的实例并聚合时间信息，从而实现自然的实例跟踪并预测动态遮罩序列，结合 Swin Transformer 可获得更高的 AP，是视频实例分割领域的强基线模型。