The essence of video semantic segmentation (VSS) is how to leverage temporal
information for prediction. Previous efforts are mainly devoted to developing
new techniques to calculate the cross-frame affinities such as optical flow and
attention. Instead, this paper contributes from a different angle by mining
relations among cross-frame affinities, upon which better temporal information
aggregation could be achieved. We explore relations among affinities in two
aspects: single-scale intrinsic correlations and multi-scale relations.
Inspired by traditional feature processing, we propose Single-scale Affinity
Refinement (SAR) and Multi-scale Affinity Aggregation (MAA). To make it
feasible to execute MAA, we propose a Selective Token Masking (STM) strategy to
select a subset of consistent reference tokens for different scales when
calculating affinities, which also improves the efficiency of our method. At
last, the cross-frame affinities strengthened by SAR and MAA are adopted for
adaptively aggregating temporal information. Our experiments demonstrate that
the proposed method performs favorably against state-of-the-art VSS methods.
The code is publicly available at this https URL

本文从挖掘帧间关联的角度出发，提出了一种使用 “Single-scale Affinity Refinement” 和 “Multi-scale Affinity Aggregation” 加强帧间亲和力的方法，该方法在视频语义分割任务中性能表现优异。