Real-time semantic video segmentation is a challenging task due to the strict
requirements of inference speed. Recent approaches mainly devote great efforts
to reducing the model size for high efficiency. In this paper, we rethink this
problem from a different viewpoint: using knowledge contained in compressed
videos. We propose a simple and effective framework, dubbed TapLab, to tap into
resources from the compressed domain. Specifically, we design a fast feature
warping module using motion vectors for acceleration. To reduce the noise
introduced by motion vectors, we design a residual-guided correction module and
a residual-guided frame selection module using residuals. TapLab significantly
reduces redundant computations of the state-of-the-art fast semantic image
segmentation models, running 3 to 10 times faster with controllable accuracy
degradation. The experimental results show that TapLab achieves 70.6% mIoU on
the Cityscapes dataset at 99.8 FPS with a single GPU card for the 1024x2048
videos. A high-speed version even reaches the speed of 160+ FPS. Codes will be
available soon at this https URL

本文提出了一个名为 TapLab 的简单有效的框架，通过利用压缩视频中的知识来加速实时语义视频分割。通过使用运动矢量进行快速特征变形，以及使用残差解决因运动矢量引入的噪声，TapLab 将现有技术的冗余计算量减少了 3 到 10 倍，并在可控的精度下降的条件下实现了 99.8 FPS。