For semantic segmentation, most existing real-time deep models trained with
each frame independently may produce inconsistent results for a video sequence.
Advanced methods take into considerations the correlations in the video
sequence, e.g., by propagating the results to the neighboring frames using
optical flow, or extracting the frame representations with other frames, which
may lead to inaccurate results or unbalanced latency. In this work, we process
efficient semantic video segmentation in a per-frame fashion during the
inference process. Different from previous per-frame models, we explicitly
consider the temporal consistency among frames as extra constraints during the
training process and embed the temporal consistency into the segmentation
network. Therefore, in the inference process, we can process each frame
independently with no latency, and improve the temporal consistency with no
extra computational cost and post-processing. We employ compact models for
real-time execution. To narrow the performance gap between compact models and
large models, new knowledge distillation methods are designed. Our results
outperform previous keyframe based methods with a better trade-off between the
accuracy and the inference speed on popular benchmarks, including the
Cityscapes and Camvid. The temporal consistency is also improved compared with
corresponding baselines which are trained with each frame independently. Code
is available at: this https URL

本文提出了一种有效的语义视频分割方法，借鉴于现有方法中将结果传播到相邻帧或使用其他帧提取帧表示时可能会导致不准确的结果或不平衡的延迟问题，我们在推理过程中采用逐帧方式处理，其中显式考虑帧与帧之间的时间一致性，并将此一致性嵌入到语义分割网络中。通过新的知识蒸馏方法，我们成功地缩小了紧凑型模型与大型模型之间的性能差距。我们的结果在 Cityscapes 和 Camvid 等流行基准测试中优于先前的基于关键帧的方法，并提高了与独立训练每帧的对应基线的时间一致性。