In recent years, open-vocabulary (OV) dense visual prediction (such as OV
object detection, semantic, instance and panoptic segmentations) has attracted
increasing research attention. However, most of existing approaches are
task-specific and individually tackle each task. In this paper, we propose a
Unified Open-Vocabulary Network (UOVN) to jointly address four common dense
prediction tasks. Compared with separate models, a unified network is more
desirable for diverse industrial applications. Moreover, OV dense prediction
training data is relatively less. Separate networks can only leverage
task-relevant training data, while a unified approach can integrate diverse
training data to boost individual tasks. We address two major challenges in
unified OV prediction. Firstly, unlike unified methods for fixed-set
predictions, OV networks are usually trained with multi-modal data. Therefore,
we propose a multi-modal, multi-scale and multi-task (MMM) decoding mechanism
to better leverage multi-modal data. Secondly, because UOVN uses data from
different tasks for training, there are significant domain and task gaps. We
present a UOVN training mechanism to reduce such gaps. Experiments on four
datasets demonstrate the effectiveness of our UOVN.

本文提出了统一开放词汇网络（UOVN），用于联合解决四个常见的密集预测任务，通过与单独模型相比较，统一网络更适用于不同的工业应用，并能集成多样化的训练数据来提升个别任务的效果。本文解决了统一 OV 预测面临的两个主要挑战：多模态多尺度和多任务解码机制，以及不同任务和域之间存在的巨大差异。实验表明，UOVN 的有效性。