Recently, open-vocabulary learning has emerged to accomplish segmentation for
arbitrary categories of text-based descriptions, which popularizes the
segmentation system to more general-purpose application scenarios. However,
existing methods devote to designing specialized architectures or parameters
for specific segmentation tasks. These customized design paradigms lead to
fragmentation between various segmentation tasks, thus hindering the uniformity
of segmentation models. Hence in this paper, we propose FreeSeg, a generic
framework to accomplish Unified, Universal and Open-Vocabulary Image
Segmentation. FreeSeg optimizes an all-in-one network via one-shot training and
employs the same architecture and parameters to handle diverse segmentation
tasks seamlessly in the inference procedure. Additionally, adaptive prompt
learning facilitates the unified model to capture task-aware and
category-sensitive concepts, improving model robustness in multi-task and
varied scenarios. Extensive experimental results demonstrate that FreeSeg
establishes new state-of-the-art results in performance and generalization on
three segmentation tasks, which outperforms the best task-specific
architectures by a large margin: 5.5% mIoU on semantic segmentation, 17.6% mAP
on instance segmentation, 20.1% PQ on panoptic segmentation for the unseen
class on COCO.

本文提出了一种通用框架 FreeSeg，通过一次训练优化全一体网络，并采用相同的架构和参数在推理过程中无缝处理各种分割任务。此外，自适应提示学习有助于统一的模型捕捉任务感知和类别敏感概念，在多任务和各种场景中提高模型鲁棒性。大量实验结果表明，在三个分割任务上，FreeSeg 在性能和泛化方面建立了新的最先进结果，比最好的任务特定架构高出很大的幅度：在语义分割、实例分割和 COCO 上的全景分割中，未见类别的 PQ 分别达到 5.5％，17.6％和 20.1％。