Recently, transformer-based approaches have shown promising results for semi-supervised video object segmentation. However, these approaches typically struggle on long videos due to increased GPU memory demands, as they frequently expand the memory bank every few frames. We propose a transformer-based approach, named MAVOS, that introduces an optimized and dynamic long-term modulated cross-attention (MCA) memory to model temporal smoothness without requiring frequent memory expansion. The proposed MCA effectively encodes both local and global features at various levels of granularity while efficiently maintaining consistent speed regardless of the video length. Extensive experiments on multiple benchmarks, LVOS, Long-Time Video, and DAVIS 2017, demonstrate the effectiveness of our proposed contributions leading to real-time inference and markedly reduced memory demands without any degradation in segmentation accuracy on long videos. Compared to the best existing transformer-based approach, our MAVOS increases the speed by 7.6x, while significantly reducing the GPU memory by 87% with comparable segmentation performance on short and long video datasets. Notably on the LVOS dataset, our MAVOS achieves a J&F score of 63.3% while operating at 37 frames per second (FPS) on a single V100 GPU. Our code and models will be publicly available at: https://github.com/Amshaker/MAVOS.

提出了一种名为MAVOS的基于变压器的方法，通过引入优化和动态的长期调制交叉注意力(MCA)存储器，对长视频进行模型化学处理，同时在不需要频繁扩展内存的情况下实现跨帧的时间平滑。在多个基准测试中，包括LVOS、Long-Time Video和DAVIS 2017，大量实验证明了我们提出的改进的有效性，实现了实时推理，并在长视频上显著降低了内存需求，而不降低分割精度。与现有的最佳基于变压器的方法相比，我们的MAVOS在可比较的短视频和长视频数据集上将速度提高了7.6倍，同时将GPU内存降低了87%。值得注意的是，在LVOS数据集上，我们的MAVOS在单个V100 GPU上以每秒37帧的速度实现了63.3%的J&F分数。

通过调控交叉注意力记忆实现高效视频对象分割