We introduce Spatial-Temporal Memory Networks (STMN) for video object detection. At its core, we propose a novel Spatial-Temporal Memory module (STMM) as the recurrent computation unit to model long-term temporal appearance and motion dynamics. The STMM's design enables the integration of ImageNet pre-trained backbone CNN weights for both the feature stack as well as the prediction head, which we find to be critical for accurate detection. Furthermore, in order to tackle object motion in videos, we propose a novel MatchTrans module to align the spatial-temporal memory from frame to frame. We compare our method to state-of-the-art detectors on ImageNet VID, and conduct ablative studies to dissect the contribution of our different design choices. We obtain state-of-the-art results with the VGG backbone, and competitive results with the ResNet backbone. To our knowledge, this is the first video object detector that is equipped with an explicit memory mechanism to model long-term temporal dynamics.

本文介绍了一种用于视频目标检测的空间-时间记忆网络，其核心是一种新颖的空间-时间记忆模块，用于模拟长期的时间外观和动态运动，并提出了一种新的MatchTrans模块来解决视频中的物体运动问题。作者的方法在benchmark ImageNet VID数据集上取得了最先进的结果，并且文章的实验表明了不同设计选择的贡献。

基于空间-时间对齐记忆的视频物体检测