Demonstrations and natural language instructions are two common ways to
specify and teach robots novel tasks. However, for many complex tasks, a
demonstration or language instruction alone contains ambiguities, preventing
tasks from being specified clearly. In such cases, a combination of both a
demonstration and an instruction more concisely and effectively conveys the
task to the robot than either modality alone. To instantiate this problem
setting, we train a single multi-task policy on a few hundred challenging
robotic pick-and-place tasks and propose DeL-TaCo (Joint Demo-Language Task
Conditioning), a method for conditioning a robotic policy on task embeddings
comprised of two components: a visual demonstration and a language instruction.
By allowing these two modalities to mutually disambiguate and clarify each
other during novel task specification, DeL-TaCo (1) substantially decreases the
teacher effort needed to specify a new task and (2) achieves better
generalization performance on novel objects and instructions over previous
task-conditioning methods. To our knowledge, this is the first work to show
that simultaneously conditioning a multi-task robotic manipulation policy on
both demonstration and language embeddings improves sample efficiency and
generalization over conditioning on either modality alone. See additional
materials at this https URL

通过将图像演示和语言指令互相协作，同时在多任务机器人操作中对这两种模态进行调节可以有效降低任务教师的工作量，并提高对新物体和指令的泛化能力。