Natural language is one of the most intuitive ways to express human intent.
However, translating instructions and commands towards robotic motion
generation and deployment in the real world is far from being an easy task. The
challenge of combining a robot's inherent low-level geometric and kinodynamic
constraints with a human's high-level semantic instructions traditionally is
solved using task-specific solutions with little generalizability between
hardware platforms, often with the use of static sets of target actions and
commands. This work instead proposes a flexible language-based framework that
allows a user to modify generic robotic trajectories. Our method leverages
pre-trained language models (BERT and CLIP) to encode the user's intent and
target objects directly from a free-form text input and scene images, fuses
geometrical features generated by a transformer encoder network, and finally
outputs trajectories using a transformer decoder, without the need of priors
related to the task or robot information. We significantly extend our own
previous work presented in Bucker et al. by expanding the trajectory
parametrization space to 3D and velocity as opposed to just XY movements. In
addition, we now train the model to use actual images of the objects in the
scene for context (as opposed to textual descriptions), and we evaluate the
system in a diverse set of scenarios beyond manipulation, such as aerial and
legged robots. Our simulated and real-life experiments demonstrate that our
transformer model can successfully follow human intent, modifying the shape and
speed of trajectories within multiple environments. Codebase available at:
this https URL

该研究提出了一个灵活的基于语言的框架，使用自然语言输入和场景图像处理机器人运动指令，通过预先训练的语言模型和变压器编码器网络生成目标几何特征，然后使用变压器解码器生成轨迹，无需任何先验知识。在复杂环境下，仿真和实际实验表明，该系统可以成功地跟随人类意图，修改轨迹的形状和速度。