We propose associating language utterances to 3D visual abstractions of the
scene they describe. The 3D visual abstractions are encoded as 3-dimensional
visual feature maps. We infer these 3D visual scene feature maps from RGB
images of the scene via view prediction: when the generated 3D scene feature
map is neurally projected from a camera viewpoint, it should match the
corresponding RGB image. We present generative models that condition on the
dependency tree of an utterance and generate a corresponding visual 3D feature
map as well as reason about its plausibility, and detector models that
condition on both the dependency tree of an utterance and a related image and
localize the object referents in the 3D feature map inferred from the image.
Our model outperforms models of language and vision that associate language
with 2D CNN activations or 2D images by a large margin in a variety of tasks,
such as, classifying plausibility of utterances, detecting referential
expressions, and supplying rewards for trajectory optimization of object
placement policies from language instructions. We perform numerous ablations
and show the improved performance of our detectors is due to its better
generalization across camera viewpoints and lack of object interferences in the
inferred 3D feature space, and the improved performance of our generators is
due to their ability to spatially reason about objects and their configurations
in 3D when mapping from language to scenes.

提出结合语言表述和三维可视化的方法，通过生成模型和检测模型等工具，可以从图像中推理出三维可视特征图，并进一步实现语言方面的任务，如检测引用表达和物体放置策略。这种方法可以更好地进行全景视角和空间推理。