Despite rapid progress in Visual question answering (VQA), existing datasets
and models mainly focus on testing reasoning in 2D. However, it is important
that VQA models also understand the 3D structure of visual scenes, for example
to support tasks like navigation or manipulation. This includes an
understanding of the 3D object pose, their parts and occlusions. In this work,
we introduce the task of 3D-aware VQA, which focuses on challenging questions
that require a compositional reasoning over the 3D structure of visual scenes.
We address 3D-aware VQA from both the dataset and the model perspective. First,
we introduce Super-CLEVR-3D, a compositional reasoning dataset that contains
questions about object parts, their 3D poses, and occlusions. Second, we
propose PO3D-VQA, a 3D-aware VQA model that marries two powerful ideas:
probabilistic neural symbolic program execution for reasoning and deep neural
networks with 3D generative representations of objects for robust visual
recognition. Our experimental results show our model PO3D-VQA outperforms
existing methods significantly, but we still observe a significant performance
gap compared to 2D VQA benchmarks, indicating that 3D-aware VQA remains an
important open research area.

通过引入 3D 感知的视觉问答（3D-aware VQA）任务以及相应的数据集 Super-CLEVR-3D 和模型 PO3D-VQA，本论文探索了在视觉场景的三维结构上进行合成推理的挑战性问题，实验证明 PO3D-VQA 模型在 3D 感知的视觉问答上取得了显著进展，但与 2D 视觉问答基准相比仍存在明显的性能差距，显示出 3D 感知的视觉问答依然是一个重要的开放研究领域。