As interest in "reformulating" the 3D Visual Question Answering (VQA) problem
in the context of foundation models grows, it is imperative to assess how these
new paradigms influence existing closed-vocabulary datasets. In this case
study, we evaluate the zero-shot performance of foundational models (GPT-4
Vision and GPT-4) on well-established 3D VQA benchmarks, namely 3D-VQA and
ScanQA. We provide an investigation to contextualize the performance of
GPT-based agents relative to traditional modeling approaches. We find that
GPT-based agents without any fine-tuning perform on par with the closed
vocabulary approaches. Our findings corroborate recent results that "blind"
models establish a surprisingly strong baseline in closed-vocabulary settings.
We demonstrate that agents benefit significantly from scene-specific vocabulary
via in-context textual grounding. By presenting a preliminary comparison with
previous baselines, we hope to inform the community's ongoing efforts to refine
multi-modal 3D benchmarks.

我们在 3D VQA 基准测试中评估了 GPT-4 Vision 和 GPT-4 等基础模型的零样本性能，发现没有任何微调的 GPT-based agents 在封闭词汇环境中表现与传统建模方法相当，我们还通过与以前的基准测试进行初步比较，希望为改进多模态 3D 基准测试的努力提供信息。