In this paper we present a neuro-symbolic (hybrid) compositional reasoning
model for coupling language-guided visual reasoning with robot manipulation. A
non-expert human user can prompt the robot agent using natural language,
providing a referring expression, a question or a grasp action instruction. The
model tackles all cases in a task-agnostic fashion through the utilization of a
shared library of primitive skills. Each primitive handles an independent
sub-task, such as reasoning about visual attributes, spatial relation
comprehension, logic and enumeration, as well as arm control. A language parser
maps the input query to an executable program composed of such primitives
depending on the context. While some primitives are purely symbolic operations
(e.g. counting), others are trainable neural functions (e.g. image/word
grounding), therefore marrying the interpretability and systematic
generalization benefits of discrete symbolic approaches with the scalability
and representational power of deep networks. We generate a synthetic dataset of
tabletop scenes to train our approach and perform several evaluation
experiments for visual reasoning. Results show that the proposed method
achieves very high accuracy while being transferable to real-world scenes with
few-shot visual fine-tuning. Finally, we integrate our method with a robot
framework and demonstrate how it can serve as an interpretable solution for an
interactive object picking task, both in simulation and with a real robot.
Supplementary material is available in this https URL.

本文介绍了一种神经符号 (混合) 组合推理模型，以将语言引导的视觉推理与机器人操作相结合。该模型通过使用共享的原始技能库以任务非特定的方式处理所有情况。通过语言解析器将输入查询映射为由这些原语组成的可执行程序取决于上下文。结果表明，该方法达到了非常高的准确性，同时可以进行少量的视觉微调，从而实现了真实场景的可转移性。