In-depth scene descriptions and question answering tasks have greatly increased the scope of today's definition of scene understanding. While such tasks are in principle open ended, current formulations primarily focus on describing only the current state of the scenes under consideration. In contrast, in this paper, we focus on the future states of the scenes which are also conditioned on actions. We posit this as a question answering task, where an answer has to be given about a future scene state, given observations of the current scene, and a question that includes a hypothetical action. Our solution is a hybrid model which integrates a physics engine into a question answering architecture in order to anticipate future scene states resulting from object-object interactions caused by an action. We demonstrate first results on this challenging new problem and compare to baselines, where we outperform fully data-driven end-to-end learning approaches.

本文以场景理解为主题，提出了一种融合物理引擎和问答结构的混合模型，用于预测场景中受到行为影响的未来状态。通过与端到端学习方法进行比较，展示了这种具有挑战性的新问题的初步结果。

回答视觉假设问题：从行为到预测场景描述