With recent video object segmentation (VOS) benchmarks evolving to
challenging scenarios, we revisit a simple but overlooked strategy: restricting
the size of memory banks. This diverges from the prevalent practice of
expanding memory banks to accommodate extensive historical information. Our
specially designed "memory deciphering" study offers a pivotal insight
underpinning such a strategy: expanding memory banks, while seemingly
beneficial, actually increases the difficulty for VOS modules to decode
relevant features due to the confusion from redundant information. By
restricting memory banks to a limited number of essential frames, we achieve a
notable improvement in VOS accuracy. This process balances the importance and
freshness of frames to maintain an informative memory bank within a bounded
capacity. Additionally, restricted memory banks reduce the training-inference
discrepancy in memory lengths compared with continuous expansion. This fosters
new opportunities in temporal reasoning and enables us to introduce the
previously overlooked "temporal positional embedding." Finally, our insights
are embodied in "RMem" ("R" for restricted), a simple yet effective VOS
modification that excels at challenging VOS scenarios and establishes new state
of the art for object state changes (on the VOST dataset) and long videos (on
the Long Videos dataset). Our code and demo are available at
this https URL

通过限制记忆库的规模，我们在视频目标分割模块（VOS）中实现了显著的提高。这个过程平衡了帧的重要性和新鲜度，以维持一个有限容量的信息性记忆库，并降低了训练 - 推理过程中记忆长度的差异，从而在时间推理方面开辟了新的机遇。最终，我们通过 “RMem”（即 “R” 表示限制）将这些见解体现出来，这是一个简单而有效的在具有挑战性的 VOS 场景中取得卓越表现的 VOS 修改方法，并在目标状态变化方面（在 VOST 数据集上）和长视频方面（在长视频数据集上）建立了新的最佳性能。