In recent years, video semantic segmentation has made great progress with
advanced deep neural networks. However, there still exist two main challenges
\ie, information inconsistency and computation cost. To deal with the two
difficulties, we propose a novel motion-state alignment framework for video
semantic segmentation to keep both motion and state consistency. In the
framework, we first construct a motion alignment branch armed with an efficient
decoupled transformer to capture dynamic semantics, guaranteeing region-level
temporal consistency. Then, a state alignment branch composed of a stage
transformer is designed to enrich feature spaces for the current frame to
extract static semantics and achieve pixel-level state consistency. Next, by a
semantic assignment mechanism, the region descriptor of each semantic category
is gained from dynamic semantics and linked with pixel descriptors from static
semantics. Benefiting from the alignment of these two kinds of effective
information, the proposed method picks up dynamic and static semantics in a
targeted way, so that video semantic regions are consistently segmented to
obtain precise locations with low computational complexity. Extensive
experiments on Cityscapes and CamVid datasets show that the proposed approach
outperforms state-of-the-art methods and validates the effectiveness of the
motion-state alignment framework.

本研究提出了一种基于运动状态对齐的视频语义分割方法，该方法解决了信息不一致性和计算成本等难点，实现了像素级状态一致性和区域级时间一致性，能够以低计算成本高精度地分割出视频语义区域，并在 Cityscapes 和 CamVid 数据集上验证了其优越性。