Semantic segmentation and semantic image synthesis are two representative
tasks in visual perception and generation. While existing methods consider them
as two distinct tasks, we propose a unified diffusion-based framework (SemFlow)
and model them as a pair of reverse problems. Specifically, motivated by
rectified flow theory, we train an ordinary differential equation (ODE) model
to transport between the distributions of real images and semantic masks. As
the training object is symmetric, samples belonging to the two distributions,
images and semantic masks, can be effortlessly transferred reversibly. For
semantic segmentation, our approach solves the contradiction between the
randomness of diffusion outputs and the uniqueness of segmentation results. For
image synthesis, we propose a finite perturbation approach to enhance the
diversity of generated results without changing the semantic categories.
Experiments show that our SemFlow achieves competitive results on semantic
segmentation and semantic image synthesis tasks. We hope this simple framework
will motivate people to rethink the unification of low-level and high-level
vision. Project page: this https URL

基于扩散的统一框架（SemFlow）将语义分割和语义图像合成作为一对相反问题，通过训练一个常微分方程（ODE）模型在真实图像和语义掩码的分布之间进行传输，解决了扩散输出的随机性与分割结果唯一性之间的矛盾，同时提升生成结果的多样性，该框架在语义分割和语义图像合成任务上取得了有竞争力的结果，希望能激发人们重新思考低级和高级视觉的统一。