Humans, even at a very early age, can learn visual concepts and understand
geometry and layout through active interaction with the environment, and
generalize their compositions to complete tasks described by natural languages
in novel scenes. To mimic such capability, we propose Embodied Concept Learner
(ECL) in an interactive 3D environment. Specifically, a robot agent can ground
visual concepts, build semantic maps and plan actions to complete tasks by
learning purely from human demonstrations and language instructions, without
access to ground-truth semantic and depth supervisions from simulations. ECL
consists of: (i) an instruction parser that translates the natural languages
into executable programs; (ii) an embodied concept learner that grounds visual
concepts based on language descriptions; (iii) a map constructor that estimates
depth and constructs semantic maps by leveraging the learned concepts; and (iv)
a program executor with deterministic policies to execute each program. ECL has
several appealing benefits thanks to its modularized design. Firstly, it
enables the robotic agent to learn semantics and depth unsupervisedly acting
like babies, e.g., ground concepts through active interaction and perceive
depth by disparities when moving forward. Secondly, ECL is fully transparent
and step-by-step interpretable in long-term planning. Thirdly, ECL could be
beneficial for the embodied instruction following (EIF), outperforming previous
works on the ALFRED benchmark when the semantic label is not provided. Also,
the learned concept can be reused for other downstream tasks, such as reasoning
of object states. Project page: this http URL

通过交互式三维环境中的体验式概念学习者，机器人可以通过学习自然语言指令和人类示范，理解视觉概念、建立语义地图并规划行动，实现长期计划，且无需通过模拟数据获得地基准真的语义深度监督。