Text-to-image generation has witnessed great progress, especially with the
recent advancements in diffusion models. Since texts cannot provide detailed
conditions like object appearance, reference images are usually leveraged for
the control of objects in the generated images. However, existing methods still
suffer limited accuracy when the relationship between the foreground and
background is complicated. To address this issue, we develop a framework termed
Mask-ControlNet by introducing an additional mask prompt. Specifically, we
first employ large vision models to obtain masks to segment the objects of
interest in the reference image. Then, the object images are employed as
additional prompts to facilitate the diffusion model to better understand the
relationship between foreground and background regions during image generation.
Experiments show that the mask prompts enhance the controllability of the
diffusion model to maintain higher fidelity to the reference image while
achieving better image quality. Comparison with previous text-to-image
generation methods demonstrates our method's superior quantitative and
qualitative performance on the benchmark datasets.

通过引入额外的掩码提示 Mask-ControlNet，本研究在文本到图像生成方面取得了较好的性能，改善了前景与背景之间复杂关系的准确性，提升了生成图像的质量。