We explore semantic segmentation beyond the conventional, single-dataset
homogeneous training and bring forward the problem of Heterogeneous Training of
Semantic Segmentation (HTSS). HTSS involves simultaneous training on multiple
heterogeneous datasets, i.e. datasets with conflicting label spaces and
different (weak) annotation types from the perspective of semantic
segmentation. The HTSS formulation exposes deep networks to a larger and
previously unexplored aggregation of information that can potentially enhance
semantic segmentation in three directions: i) performance: increased
segmentation metrics on seen datasets, ii) generalization: improved
segmentation metrics on unseen datasets, and iii) knowledgeability: increased
number of recognizable semantic concepts. To research these benefits of HTSS,
we propose a unified framework, that incorporates heterogeneous datasets in a
single-network training pipeline following the established FCN standard. Our
framework first curates heterogeneous datasets to bring them into a common
format and then trains a single-backbone FCN on all of them simultaneously. To
achieve this, it transforms weak annotations, which are incompatible with
semantic segmentation, to per-pixel labels, and hierarchizes their label spaces
into a universal taxonomy. The trained HTSS models demonstrate performance and
generalization gains over a wide range of datasets and extend the inference
label space entailing hundreds of semantic classes.

我们提出了一个新的训练框架，用于解决 Heterogeneous Training of Semantic Segmentation (HTSS) 问题，可以同时训练多个具有不同标签空间和标注类型的数据集，以提高 Semantic Segmentation 在性能、泛化和可识别的语义概念等方面的表现。