Relying on large language models (LLMs), embodied robots could perform
complex multimodal robot manipulation tasks from visual observations with
powerful generalization ability. However, most visual behavior-cloning agents
suffer from manipulation performance degradation and skill knowledge forgetting
when adapting into a series of challenging unseen tasks. We here investigate
the above challenge with NBCagent in embodied robots, a pioneering
language-conditioned Never-ending Behavior-Cloning agent, which can continually
learn observation knowledge of novel robot manipulation skills from
skill-specific and skill-shared attributes. Specifically, we establish a
skill-specific evolving planner to perform knowledge decoupling, which can
continually embed novel skill-specific knowledge in our NBCagent agent from
latent and low-rank space. Meanwhile, we propose a skill-shared semantics
rendering module and a skill-shared representation distillation module to
effectively transfer anti-forgetting skill-shared knowledge, further tackling
catastrophic forgetting on old skills from semantics and representation
aspects. Finally, we design a continual embodied robot manipulation benchmark,
and several expensive experiments demonstrate the significant performance of
our method. Visual results, code, and dataset are provided at:
this https URL

通过大型语言模型，我们研究了在具有高度泛化能力的视觉行为克隆代理中，从视觉观察中进行复杂多模态机器人操作任务。我们提出了 NBCagent，一个语言条件下的持续学习行为克隆代理，在面对一系列具有挑战性的未知任务时，可以不断学习新的机器人操作技能的观察知识。我们设计了一个特定于技能的进化规划器来进行知识解耦，可以将新的技能特定知识不断嵌入到我们的 NBCagent 代理中。同时，我们还提出了一个技能共享的语义渲染模块和技能共享的表示蒸馏模块，有效地在语义和表示方面传递抗遗忘的技能共享知识，进一步解决了旧技能的灾难性遗忘问题。最后，我们设计了一个持续的机器人操作基准，并进行了一些昂贵的实验，证明了我们方法的显著性能。