Embodied reasoning systems integrate robotic hardware and cognitive processes to perform complex tasks typically in response to a natural language query about a specific physical environment. This usually involves changing the belief about the scene or physically interacting and changing the scene (e.g. 'Sort the objects from lightest to heaviest'). In order to facilitate the development of such systems we introduce a new simulating environment that makes use of MuJoCo physics engine and high-quality renderer Blender to provide realistic visual observations that are also accurate to the physical state of the scene. Together with the simulator we propose a new benchmark composed of 10 classes of multi-step reasoning scenarios that require simultaneous visual and physical measurements. Finally, we develop a new modular Closed Loop Interactive Reasoning (CLIER) approach that takes into account the measurements of non-visual object properties, changes in the scene caused by external disturbances as well as uncertain outcomes of robotic actions. We extensively evaluate our reasoning approach in simulation and in the real world manipulation tasks with a success rate above 76% and 64%, respectively.

通过使用MuJoCo物理引擎和高质量渲染器Blender，我们引入一个新的模拟环境，以提供既逼真又准确反映物理环境状态的视觉观察。除此之外，我们还提出了一个由10类多步骤推理场景构成的新基准，并开发了一种新的模块化闭环交互式推理（CLIER）方法，考虑到非视觉对象属性的测量、外部干扰引起的场景变化以及机器人动作的不确定结果。我们在模拟环境和实际操作任务中对我们的推理方法进行了广泛评估，并取得了分别高达76%和64%的成功率。

闭环交互式具身推理的机器人操控