Recognizing visual entities in a natural language sentence and arranging them
in a 2D spatial layout require a compositional understanding of language and
space. This task of layout prediction is valuable in text-to-image synthesis as
it allows localized and controlled in-painting of the image. In this
comparative study it is shown that we can predict layouts from language
representations that implicitly or explicitly encode sentence syntax, if the
sentences mention similar entity-relationships to the ones seen during
training. To test compositional understanding, we collect a test set of
grammatically correct sentences and layouts describing compositions of entities
and relations that unlikely have been seen during training. Performance on this
test set substantially drops, showing that current models rely on correlations
in the training data and have difficulties in understanding the structure of
the input sentences. We propose a novel structural loss function that better
enforces the syntactic structure of the input sentence and show large
performance gains in the task of 2D spatial layout prediction conditioned on
text. The loss has the potential to be used in other generation tasks where a
tree-like structure underlies the conditioning modality. Code, trained models
and the USCOCO evaluation set will be made available via github.

通过自然语言句子中的识别视觉实体和将其按照二维空间布局排列，需要对语言和空间进行构造性理解。本研究通过对比显示，如果句子提到了在训练期间看到的类似实体关系，则可以从隐式或显式编码句子句法的语言表示中预测布局。为了测试构造性理解，我们收集了一组语法正确的测试句子和布局，它们描述了在训练期间很可能没有见过的实体和关系的组合。结果显示，在这个测试集上的表现大幅下降，表明当前模型依赖于训练数据中的相关性，并且在理解输入句子的结构方面存在困难。我们提出了一种新颖的结构损失函数，更好地强制执行输入句子的句法结构，并在以文本为条件的 2D 空间布局预测任务中取得了显著的性能提升。该损失函数有潜力在其他生成任务中使用，其中基础的调节方式是树状结构。代码、训练模型和 USCOCO 评估集将通过 GitHub 提供。