This work addresses the 3D situated reasoning task which aims to answer
questions given egocentric observations in a 3D environment. The task remains
challenging as it requires comprehensive 3D perception and complex reasoning
skills. End-to-end models trained on supervised data for 3D situated reasoning
suffer from data scarcity and generalization ability. Inspired by the recent
success of leveraging large language models (LLMs) for visual reasoning, we
propose LLM-TPC, a novel framework that leverages the planning, tool usage, and
reflection capabilities of LLMs through a ThinkProgram-reCtify loop. The Think
phase first decomposes the compositional question into a sequence of steps, and
then the Program phase grounds each step to a piece of code and calls carefully
designed 3D visual perception modules. Finally, the Rectify phase adjusts the
plan and code if the program fails to execute. Experiments and analysis on the
SQA3D benchmark demonstrate the effectiveness, interpretability and robustness
of our method. Our code is publicly available at
this https URL

该研究通过 LLM-TPC 框架，利用大型语言模型，解决了 3D 环境中基于自我的观察进行问题回答的挑战，该方法在 SQA3D 基准测试上表现出了有效性、可解释性和鲁棒性。