Video Semantic Segmentation (VSS) involves assigning a semantic label to each
pixel in a video sequence. Prior work in this field has demonstrated promising
results by extending image semantic segmentation models to exploit temporal
relationships across video frames; however, these approaches often incur
significant computational costs. In this paper, we propose an efficient mask
propagation framework for VSS, called MPVSS. Our approach first employs a
strong query-based image segmentor on sparse key frames to generate accurate
binary masks and class predictions. We then design a flow estimation module
utilizing the learned queries to generate a set of segment-aware flow maps,
each associated with a mask prediction from the key frame. Finally, the
mask-flow pairs are warped to serve as the mask predictions for the non-key
frames. By reusing predictions from key frames, we circumvent the need to
process a large volume of video frames individually with resource-intensive
segmentors, alleviating temporal redundancy and significantly reducing
computational costs. Extensive experiments on VSPW and Cityscapes demonstrate
that our mask propagation framework achieves SOTA accuracy and efficiency
trade-offs. For instance, our best model with Swin-L backbone outperforms the
SOTA MRCFA using MiT-B5 by 4.0% mIoU, requiring only 26% FLOPs on the VSPW
dataset. Moreover, our framework reduces up to 4x FLOPs compared to the
per-frame Mask2Former baseline with only up to 2% mIoU degradation on the
Cityscapes validation set. Code is available at
this https URL

视频语义分割（VSS）通过为视频序列中的每个像素分配语义标签来完成。本文提出了一种名为 MPVSS 的高效掩码传播框架，通过利用学习到的查询生成一组与关键帧的掩码预测相关的面向片段的流图，并将这些掩码 - 流对变换为非关键帧的掩码预测，从而在保证精度的同时大幅降低计算成本。经过 VSPW 和 Cityscapes 等数据集的大量实验证明，我们的框架在准确性和效率方面取得了 SOTA 水平的折衷，最佳模型在 VSPW 数据集上使用 Swin-L 骨干网络相比于 SOTA MRCFA（使用 MiT-B5）的 mIoU 提高了 4.0%，只需要 26% 的 FLOPs。此外，与单帧 Mask2Former 基线相比，我们的框架在 Cityscapes 验证集上降低了高达 4 倍的 FLOPs，仅引起 2% 的 mIoU 降低。