Modern approaches have proved the huge potential of addressing semantic
segmentation as a mask classification task which is widely used in
instance-level segmentation. This paradigm trains models by assigning part of
object queries to ground truths via conventional one-to-one matching. However,
we observe that the popular video semantic segmentation (VSS) dataset has
limited categories per video, meaning less than 10% of queries could be matched
to receive meaningful gradient updates during VSS training. This inefficiency
limits the full expressive potential of all queries.Thus, we present a novel
solution THE-Mask for VSS, which introduces temporal-aware hierarchical object
queries for the first time. Specifically, we propose to use a simple two-round
matching mechanism to involve more queries matched with minimal cost during
training while without any extra cost during inference. To support our
more-to-one assignment, in terms of the matching results, we further design a
hierarchical loss to train queries with their corresponding hierarchy of
primary or secondary. Moreover, to effectively capture temporal information
across frames, we propose a temporal aggregation decoder that fits seamlessly
into the mask-classification paradigm for VSS. Utilizing temporal-sensitive
multi-level queries, our method achieves state-of-the-art performance on the
latest challenging VSS benchmark VSPW without bells and whistles.

我们提出了一个用于视频语义分割（VSS）的新解决方案 THE-Mask，首次引入了时态感知的分层物体查询，并利用简单的两轮匹配机制，在训练过程中以最小代价匹配更多的查询对象，同时在推理过程中无需任何额外代价。为了支持多对一的分配，我们设计了一种分层损失来训练带有相应主次层级关系的查询。此外，为了有效捕捉帧间的时态信息，我们提出了一个时态聚合解码器，以无缝融入 VSS 的掩码分类模型，利用时态敏感的多级查询方法，在最新的具有挑战性的 VSS 基准 VSPW 上取得了最先进的性能。