Robots operating in the real world require both rich manipulation skills as
well as the ability to semantically reason about when to apply those skills.
Towards this goal, recent works have integrated semantic representations from
large-scale pretrained vision-language (VL) models into manipulation models,
imparting them with more general reasoning capabilities. However, we show that
the conventional pretraining-finetuning pipeline for integrating such
representations entangles the learning of domain-specific action information
and domain-general visual information, leading to less data-efficient training
and poor generalization to unseen objects and tasks. To this end, we propose
ProgramPort, a modular approach to better leverage pretrained VL models by
exploiting the syntactic and semantic structures of language instructions. Our
framework uses a semantic parser to recover an executable program, composed of
functional modules grounded on vision and action across different modalities.
Each functional module is realized as a combination of deterministic
computation and learnable neural networks. Program execution produces
parameters to general manipulation primitives for a robotic end-effector. The
entire modular network can be trained with end-to-end imitation learning
objectives. Experiments show that our model successfully disentangles action
and perception, translating to improved zero-shot and compositional
generalization in a variety of manipulation behaviors. Project webpage at:
https://progport.github.io.

通过提出一种利用语言指令的可执行程序来更好地利用预训练的 VL 模型的模块化方法，我们可以改善在任务和物体未知情况下的运用和组合泛化能力。

基于程序的、组合通用的机器人操作

Programmatically Grounded, Compositionally Generalizable Robotic  Manipulation

We present an end-to-end imitation learning system for agile, off-road
autonomous driving using only low-cost sensors. By imitating a model predictive
controller equipped with advanced sensors, we train a deep neural network
control policy to map raw, high-dimensional observations to continuous steering
and throttle commands. Compared with recent approaches to similar tasks, our
method requires neither state estimation nor on-the-fly planning to navigate
the vehicle. Our approach relies on, and experimentally validates, recent
imitation learning theory. Empirically, we show that policies trained with
online imitation learning overcome well-known challenges related to covariate
shift and generalize better than policies trained with batch imitation
learning. Built on these insights, our autonomous driving system demonstrates
successful high-speed off-road driving, matching the state-of-the-art
performance.

我们提出了一种使用低成本传感器进行敏捷越野自主驾驶的端到端模仿学习系统，通过模仿配备高级传感器的模型预测控制器，我们训练了一个深度神经网络控制策略，将原始的高维观测映射到连续的转向和油门命令，这个方法不需要状态估计或实时规划来导航车辆，并通过在线模仿学习克服了协变量漂移方面的挑战，实现了成功的高速越野行驶，匹配了最新的性能水平。