Frame quality deterioration is one of the main challenges in the field of
video understanding. To compensate for the information loss caused by
deteriorated frames, recent approaches exploit transformer-based integration
modules to obtain spatio-temporal information. However, these integration
modules are heavy and complex. Furthermore, each integration module is
specifically tailored for its target task, making it difficult to generalise to
multiple tasks. In this paper, we present a neat and unified framework, called
Spatio-Temporal Prompting Network (STPN). It can efficiently extract robust and
accurate video features by dynamically adjusting the input features in the
backbone network. Specifically, STPN predicts several video prompts containing
spatio-temporal information of neighbour frames. Then, these video prompts are
prepended to the patch embeddings of the current frame as the updated input for
video feature extraction. Moreover, STPN is easy to generalise to various video
tasks because it does not contain task-specific modules. Without bells and
whistles, STPN achieves state-of-the-art performance on three widely-used
datasets for different video understanding tasks, i.e., ImageNetVID for video
object detection, YouTubeVIS for video instance segmentation, and GOT-10k for
visual object tracking. Code is available at
this https URL

帧质量下降是视频理解领域中的主要挑战之一。为了弥补由于帧质量下降而引起的信息损失，最近的方法利用基于 Transformer 的集成模块来获得时空信息。然而，这些集成模块过于复杂和繁重。在本文中，我们提出了一个简洁且统一的框架，称为时空提示网络 (STPN)。它通过动态调整骨干网络中的输入特征，可以高效地提取稳健准确的视频特征。此外，STPN 易于推广到各种视频任务，因为它不包含任务特定的模块。没有花哨的设计，STPN 在三个广泛使用的数据集上取得了最先进的性能，涵盖了不同的视频理解任务，例如用于视频对象检测的 ImageNetVID，用于视频实例分割的 YouTubeVIS 以及用于视觉目标跟踪的 GOT-10k。