Natural language is the most intuitive medium for us to interact with other people when expressing commands and instructions. However, using language is seldom an easy task when humans need to express their intent towards robots, since most of the current language interfaces require rigid templates with a static set of action targets and commands. In this work, we provide a flexible language-based interface for human-robot collaboration, which allows a user to reshape existing trajectories for an autonomous agent. We take advantage of recent advancements in the field of large language models (BERT and CLIP) to encode the user command, and then combine these features with trajectory information using multi-modal attention transformers. We train the model using imitation learning over a dataset containing robot trajectories modified by language commands, and treat the trajectory generation process as a sequence prediction problem, analogously to how language generation architectures operate. We evaluate the system in multiple simulated trajectory scenarios, and show a significant performance increase of our model over baseline approaches. In addition, our real-world experiments with a robot arm show that users significantly prefer our natural language interface over traditional methods such as kinesthetic teaching or cost-function programming. Our study shows how the field of robotics can take advantage of large pre-trained language models towards creating more intuitive interfaces between robots and machines. Project webpage: https://arthurfenderbucker.github.io/NL_trajectory_reshaper/

本文提出了一种基于自然语言的灵活性极高的人机协作接口，将大型语言模型(BERT和CLIP)的优势与轨迹信息相结合，通过多模态注意力变换实现对用户命令的编码，并将其与轨迹信息相结合。在包含机器人轨迹的数据集上进行模仿学习，并将轨迹生成过程视为序列预测问题，通过仿真轨迹实验和真实机器人实验验证了该系统的效果优异，并证明了自然语言接口在人机协作中的优越性。

使用自然语言命令重塑机器人轨迹: 多模态数据对齐的变压器研究