Advancements in generative models have sparked significant interest in
generating images while adhering to specific structural guidelines. Scene graph
to image generation is one such task of generating images which are consistent
with the given scene graph. However, the complexity of visual scenes poses a
challenge in accurately aligning objects based on specified relations within
the scene graph. Existing methods approach this task by first predicting a
scene layout and generating images from these layouts using adversarial
training. In this work, we introduce a novel approach to generate images from
scene graphs which eliminates the need of predicting intermediate layouts. We
leverage pre-trained text-to-image diffusion models and CLIP guidance to
translate graph knowledge into images. Towards this, we first pre-train our
graph encoder to align graph features with CLIP features of corresponding
images using a GAN based training. Further, we fuse the graph features with
CLIP embedding of object labels present in the given scene graph to create a
graph consistent CLIP guided conditioning signal. In the conditioning input,
object embeddings provide coarse structure of the image and graph features
provide structural alignment based on relationships among objects. Finally, we
fine tune a pre-trained diffusion model with the graph consistent conditioning
signal with reconstruction and CLIP alignment loss. Elaborate experiments
reveal that our method outperforms existing methods on standard benchmarks of
COCO-stuff and Visual Genome dataset.

利用预训练的文本到图像扩散模型和 CLIP 引导，我们的方法通过消除中间布局的预测，将图形知识转化为图像，从而生成与给定场景图一致的图像。实验证明，我们的方法在 COCO-stuff 和 Visual Genome 数据集的标准基准上优于现有方法。