Diffusion-model-based text-guided image generation has recently made
astounding progress, producing fascinating results in open-domain image
manipulation tasks. Few models, however, currently have complete zero-shot
capabilities for both global and local image editing due to the complexity and
diversity of image manipulation tasks. In this work, we propose a method with a
mixture-of-expert (MOE) controllers to align the text-guided capacity of
diffusion models with different kinds of human instructions, enabling our model
to handle various open-domain image manipulation tasks with natural language
instructions. First, we use large language models (ChatGPT) and conditional
image synthesis models (ControlNet) to generate a large number of global image
transfer dataset in addition to the instruction-based local image editing
dataset. Then, using an MOE technique and task-specific adaptation training on
a large-scale dataset, our conditional diffusion model can edit images globally
and locally. Extensive experiments demonstrate that our approach performs
surprisingly well on various image manipulation tasks when dealing with
open-domain images and arbitrary human instructions. Please refer to our
project page: [this https URL]

基于扩散模型的文字引导图像生成在开放领域图像处理任务中取得了令人惊讶的进展，本研究提出了一种混合专家控制器的方法，以将不同类型的人类指令与不同的扩散模型的文字引导能力相结合，实现对各种开放领域图像处理任务的处理。