The successful application of semantic segmentation technology in the real
world has been among the most exciting achievements in the computer vision
community over the past decade. Although the long-tailed phenomenon has been
investigated in many fields, e.g., classification and object detection, it has
not received enough attention in semantic segmentation and has become a
non-negligible obstacle to applying semantic segmentation technology in
autonomous driving and virtual reality. Therefore, in this work, we focus on a
relatively under-explored task setting, long-tailed semantic segmentation
(LTSS). We first establish three representative datasets from different
aspects, i.e., scene, object, and human. We further propose a dual-metric
evaluation system and construct the LTSS benchmark to demonstrate the
performance of semantic segmentation methods and long-tailed solutions. We also
propose a transformer-based algorithm to improve LTSS, frequency-based matcher,
which solves the oversuppression problem by one-to-many matching and
automatically determines the number of matching queries for each class. Given
the comprehensiveness of this work and the importance of the issues revealed,
this work aims to promote the empirical study of semantic segmentation tasks.
Our datasets, codes, and models will be publicly available.

本文关注于长尾语义分割 (LTSS) 这一相对未经深入研究的任务设置，建立了三个代表性数据集，并提出了基于 Transformer 的改进算法，用于解决应用语义分割技术在自动驾驶和虚拟现实中遇到的长尾问题，并通过 LTSS 评估系统和基准进行性能展示，旨在推动对语义分割任务的实证研究。