Video semantic segmentation (VSS) is a computationally expensive task due to
the per-frame prediction for videos of high frame rates. In recent work,
compact models or adaptive network strategies have been proposed for efficient
VSS. However, they did not consider a crucial factor that affects the
computational cost from the input side: the input resolution. In this paper, we
propose an altering resolution framework called AR-Seg for compressed videos to
achieve efficient VSS. AR-Seg aims to reduce the computational cost by using
low resolution for non-keyframes. To prevent the performance degradation caused
by downsampling, we design a Cross Resolution Feature Fusion (CReFF) module,
and supervise it with a novel Feature Similarity Training (FST) strategy.
Specifically, CReFF first makes use of motion vectors stored in a compressed
video to warp features from high-resolution keyframes to low-resolution
non-keyframes for better spatial alignment, and then selectively aggregates the
warped features with local attention mechanism. Furthermore, the proposed FST
supervises the aggregated features with high-resolution features through an
explicit similarity loss and an implicit constraint from the shared decoding
layer. Extensive experiments on CamVid and Cityscapes show that AR-Seg achieves
state-of-the-art performance and is compatible with different segmentation
backbones. On CamVid, AR-Seg saves 67% computational cost (measured in GFLOPs)
with the PSPNet18 backbone while maintaining high segmentation accuracy. Code:
this https URL

本文提出了一种称为 AR-Seg 的改变分辨率框架，旨在通过使用低分辨率来减少非关键帧的计算成本，同时避免降采样引起的性能下降，CReFF 模块能够更好地进行空间对齐和聚合，FST 策略能够通过显式相似性损失和隐式约束提高分割准确性，CamVid 和 Cityscapes 等多个数据集上，AR-Seg 在不同分割骨干网的情况下实现了最先进的性能。