Natural language is perhaps the most flexible and intuitive way for humans to
communicate tasks to a robot. Prior work in imitation learning typically
requires each task be specified with a task id or goal image -- something that
is often impractical in open-world environments. On the other hand, previous
approaches in instruction following allow agent behavior to be guided by
language, but typically assume structure in the observations, actuators, or
language that limit their applicability to complex settings like robotics. In
this work, we present a method for incorporating free-form natural language
conditioning into imitation learning. Our approach learns perception from
pixels, natural language understanding, and multitask continuous control
end-to-end as a single neural network. Unlike prior work in imitation learning,
our method is able to incorporate unlabeled and unstructured demonstration data
(i.e. no task or language labels). We show this dramatically improves language
conditioned performance, while reducing the cost of language annotation to less
than 1% of total data. At test time, a single language conditioned visuomotor
policy trained with our method can perform a wide variety of robotic
manipulation skills in a 3D environment, specified only with natural language
descriptions of each task (e.g. "open the drawer...now pick up the block...now
press the green button..."). To scale up the number of instructions an agent
can follow, we propose combining text conditioned policies with large
pretrained neural language models. We find this allows a policy to be robust to
many out-of-distribution synonym instructions, without requiring new
demonstrations. See videos of a human typing live text commands to our agent at
language-play.github.io

通过自然语言条件化的模拟学习方法，结合像素感知、自然语言理解和多任务连续控制的神经网络，可以在无需任务或语言标签的情况下，显著提高任务成果，同时将语言注释成本降低到总数据量的不到 1％。