Despite significant recent progress on generative models, controlled
generation of images depicting multiple and complex object layouts is still a
difficult problem. Among the core challenges are the diversity of appearance a
given object may possess and, as a result, exponential set of images consistent
with a specified layout. To address these challenges, we propose a novel
approach for layout-based image generation; we call it Layout2Im. Given the
coarse spatial layout (bounding boxes + object categories), our model can
generate a set of realistic images which have the correct objects in the
desired locations. The representation of each object is disentangled into a
specified/certain part (category) and an unspecified/uncertain part
(appearance). The category is encoded using a word embedding and the appearance
is distilled into a low-dimensional vector sampled from a normal distribution.
Individual object representations are composed together using convolutional
LSTM, to obtain an encoding of the complete layout, and then decoded to an
image. Several loss terms are introduced to encourage accurate and diverse
generation. The proposed Layout2Im model significantly outperforms the previous
state of the art, boosting the best reported inception score by 24.66% and
28.57% on the very challenging COCO-Stuff and Visual Genome datasets,
respectively. Extensive experiments also demonstrate our method's ability to
generate complex and diverse images with multiple objects.

Layout2Im 是一种基于布局的图像生成方法，将物体表示分解为指定和不确定部分，并使用卷积 LSTM 编码并解码整个布局，以生成准确且多样化的图像，提高了最佳 Inception 分数 24.66％至 28.57％。

基于版面的图像生成

Image Generation from Layout

Spatial relationships between objects provide important information for
text-based image retrieval. As users are more likely to describe a scene from a
real world perspective, using 3D spatial relationships rather than 2D
relationships that assume a particular viewing direction, one of the main
challenges is to infer the 3D structure that bridges images with users' text
descriptions. However, direct inference of 3D structure from images requires
learning from large scale annotated data. Since interactions between objects
can be reduced to a limited set of atomic spatial relations in 3D, we study the
possibility of inferring 3D structure from a text description rather than an
image, applying physical relation models to synthesize holistic 3D abstract
object layouts satisfying the spatial constraints present in a textual
description. We present a generic framework for retrieving images from a
textual description of a scene by matching images with these generated abstract
object layouts. Images are ranked by matching object detection outputs
(bounding boxes) to 2D layout candidates (also represented by bounding boxes)
which are obtained by projecting the 3D scenes with sampled camera directions.
We validate our approach using public indoor scene datasets and show that our
method outperforms baselines built upon object occurrence histograms and
learned 2D pairwise relations.

使用物理关系模型，通过将虚构的抽象物体布局与文本描述中存在的空间约束相匹配来从文本描述中推断 3D 结构，并通过将对象检测输出与表示为边界框的 2D 布局候选项进行匹配来评定图像排序，从而检索与场景的文本描述相匹配的图像，其性能优于基于对象出现直方图和学习的 2D 成对关系的基线方法。