Solving grounded language tasks often requires reasoning about relationships
between objects in the context of a given task. For example, to answer the
question "What color is the mug on the plate?" we must check the color of the
specific mug that satisfies the "on" relationship with respect to the plate.
Recent work has proposed various methods capable of complex relational
reasoning. However, most of their power is in the inference structure, while
the scene is represented with simple local appearance features. In this paper,
we take an alternate approach and build contextualized representations for
objects in a visual scene to support relational reasoning. We propose a general
framework of Language-Conditioned Graph Networks (LCGN), where each node
represents an object, and is described by a context-aware representation from
related objects through iterative message passing conditioned on the textual
input. E.g., conditioning on the "on" relationship to the plate, the object
"mug" gathers messages from the object "plate" to update its representation to
"mug on the plate", which can be easily consumed by a simple classifier for
answer prediction. We experimentally show that our LCGN approach effectively
supports relational reasoning and improves performance across several tasks and
datasets. Our code is available at this http URL

本文提出了一种基于场景上下文化表征的图网络模型，通过文本输入的迭代信息传递来构建物体的语境化表达，以支持关系推理，实验结果表明该方法有效地提高了多项任务和数据集的表现。