Recognizing visual entities in a natural language sentence and arranging them
in a 2D spatial layout require a compositional understanding of language and
space. This task of layout prediction is valuable in text-to-image synthesis as
it allows localized and controlled in-painting of the image. In this
comparative study it is shown that we can predict layouts from language
representations that implicitly or explicitly encode sentence syntax, if the
sentences mention similar entity-relationships to the ones seen during
training. To test compositional understanding, we collect a test set of
grammatically correct sentences and layouts describing compositions of entities
and relations that unlikely have been seen during training. Performance on this
test set substantially drops, showing that current models rely on correlations
in the training data and have difficulties in understanding the structure of
the input sentences. We propose a novel structural loss function that better
enforces the syntactic structure of the input sentence and show large
performance gains in the task of 2D spatial layout prediction conditioned on
text. The loss has the potential to be used in other generation tasks where a
tree-like structure underlies the conditioning modality. Code, trained models
and the USCOCO evaluation set will be made available via github.

通过自然语言句子中的识别视觉实体和将其按照二维空间布局排列，需要对语言和空间进行构造性理解。本研究通过对比显示，如果句子提到了在训练期间看到的类似实体关系，则可以从隐式或显式编码句子句法的语言表示中预测布局。为了测试构造性理解，我们收集了一组语法正确的测试句子和布局，它们描述了在训练期间很可能没有见过的实体和关系的组合。结果显示，在这个测试集上的表现大幅下降，表明当前模型依赖于训练数据中的相关性，并且在理解输入句子的结构方面存在困难。我们提出了一种新颖的结构损失函数，更好地强制执行输入句子的句法结构，并在以文本为条件的 2D 空间布局预测任务中取得了显著的性能提升。该损失函数有潜力在其他生成任务中使用，其中基础的调节方式是树状结构。代码、训练模型和 USCOCO 评估集将通过 GitHub 提供。

显式表示语法提升预测意外情境下的句子布局

Explicitly Representing Syntax Improves Sentence-to-layout Prediction of  Unexpected Situations

Salient object detection, which aims to identify and locate the most salient
pixels or regions in images, has been attracting more and more interest due to
its various real-world applications. However, this vision task is quite
challenging, especially under complex image scenes. Inspired by the intrinsic
reflection of natural images, in this paper we propose a novel feature learning
framework for large-scale salient object detection. Specifically, we design a
symmetrical fully convolutional network (SFCN) to learn complementary saliency
features under the guidance of lossless feature reflection. The location
information, together with contextual and semantic information, of salient
objects are jointly utilized to supervise the proposed network for more
accurate saliency predictions. In addition, to overcome the blurry boundary
problem, we propose a new structural loss function to learn clear object
boundaries and spatially consistent saliency. The coarse prediction results are
effectively refined by these structural information for performance
improvements. Extensive experiments on seven saliency detection datasets
demonstrate that our approach achieves consistently superior performance and
outperforms the very recent state-of-the-art methods.

本研究提出了基于对称全卷积网络的独特特征学习框架，该方法将无损特征反射的指导下，结合显著目标的位置信息、语境和语义信息，用于更精确地进行显著性预测，并使用新的结构损失函数来辅助预测边缘和保证空间一致性。实验表明，该方法在七个显著性检测数据集上均取得了优异的性能，超过了最近的先进方法。