We propose a new 3D spatial understanding task of 3D Question Answering
(3D-QA). In the 3D-QA task, models receive visual information from the entire
3D scene of the rich RGB-D indoor scan and answer the given textual questions
about the 3D scene. Unlike the 2D-question answering of VQA, the conventional
2D-QA models suffer from problems with spatial understanding of object
alignment and directions and fail the object identification from the textual
questions in 3D-QA. We propose a baseline model for 3D-QA, named ScanQA model,
where the model learns a fused descriptor from 3D object proposals and encoded
sentence embeddings. This learned descriptor correlates the language
expressions with the underlying geometric features of the 3D scan and
facilitates the regression of 3D bounding boxes to determine described objects
in textual questions and outputs correct answers. We collected human-edited
question-answer pairs with free-form answers that are grounded to 3D objects in
each 3D scene. Our new ScanQA dataset contains over 40K question-answer pairs
from the 800 indoor scenes drawn from the ScanNet dataset. To the best of our
knowledge, the proposed 3D-QA task is the first large-scale effort to perform
object-grounded question-answering in 3D environments.

通过学习语言表达与三维场景的地理特征相关的学习描述符，我们提出了一种基线模型（ScanQA），用于在三维环境中执行基于对象的问题回答，并构建了一个新的 ScanQA 数据集，其中包含来自 800 个室内场景的 40,000 个问题答案对。