Real-time video segmentation is a crucial task for many real-world
applications such as autonomous driving and robot control. Since
state-of-the-art semantic segmentation models are often too heavy for real-time
applications despite their impressive performance, researchers have proposed
lightweight architectures with speed-accuracy trade-offs, achieving real-time
speed at the expense of reduced accuracy. In this paper, we propose a novel
framework to speed up any architecture with skip-connections for real-time
vision tasks by exploiting the temporal locality in videos. Specifically, at
the arrival of each frame, we transform the features from the previous frame to
reuse them at specific spatial bins. We then perform partial computation of the
backbone network on the regions of the current frame that captures temporal
differences between the current and previous frame. This is done by dynamically
dropping out residual blocks using a gating mechanism which decides which
blocks to drop based on inter-frame distortion. We validate our
Spatial-Temporal Mask Generator (STMG) on video semantic segmentation
benchmarks with multiple backbone networks, and show that our method largely
speeds up inference with minimal loss of accuracy.

本文提出了一个新的框架，利用视频中的时空局部性来加速实时视觉任务中带有跳跃连接的任何结构，通过基于失真的门控机制动态删除残差块来在当前帧的区域上执行骨干网络的部分计算。