Demonstrations and natural language instructions are two common ways to
specify and teach robots novel tasks. However, for many complex tasks, a
demonstration or language instruction alone contains ambiguities, preventing
tasks from being specified clearly. In such cases, a combination of both a
demonstration and an instruction more concisely and effectively conveys the
task to the robot than either modality alone. To instantiate this problem
setting, we train a single multi-task policy on a few hundred challenging
robotic pick-and-place tasks and propose DeL-TaCo (Joint Demo-Language Task
Conditioning), a method for conditioning a robotic policy on task embeddings
comprised of two components: a visual demonstration and a language instruction.
By allowing these two modalities to mutually disambiguate and clarify each
other during novel task specification, DeL-TaCo (1) substantially decreases the
teacher effort needed to specify a new task and (2) achieves better
generalization performance on novel objects and instructions over previous
task-conditioning methods. To our knowledge, this is the first work to show
that simultaneously conditioning a multi-task robotic manipulation policy on
both demonstration and language embeddings improves sample efficiency and
generalization over conditioning on either modality alone. See additional
materials at this https URL

通过将图像演示和语言指令互相协作，同时在多任务机器人操作中对这两种模态进行调节可以有效降低任务教师的工作量，并提高对新物体和指令的泛化能力。

同时使用演示和语言指令高效学习机器人任务

Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Tasks

We study the problem of policy optimization (PO) with linear temporal logic
(LTL) constraints. The language of LTL allows flexible description of tasks
that may be unnatural to encode as a scalar cost function. We consider
LTL-constrained PO as a systematic framework, decoupling task specification
from policy selection, and as an alternative to the standard of cost shaping.
With access to a generative model, we develop a model-based approach that
enjoys a sample complexity analysis for guaranteeing both task satisfaction and
cost optimality (through a reduction to a reachability problem). Empirically,
our algorithm can achieve strong performance even in low-sample regimes.

本文研究了带线性时间逻辑约束的策略优化问题。通过使用生成模型和 LTL 语言，提出了一种基于模型的方法，用于在低样本数据情形下保证任务满足和成本最优的算法。

线性时态逻辑约束下的策略优化

Policy Optimization with Linear Temporal Logic Constraints

Task specification is at the core of programming autonomous robots. A
low-effort modality for task specification is critical for engagement of
non-expert end-users and ultimate adoption of personalized robot agents. A
widely studied approach to task specification is through goals, using either
compact state vectors or goal images from the same robot scene. The former is
hard to interpret for non-experts and necessitates detailed state estimation
and scene understanding. The latter requires the generation of desired goal
image, which often requires a human to complete the task, defeating the purpose
of having autonomous robots. In this work, we explore alternate and more
general forms of goal specification that are expected to be easier for humans
to specify and use such as images obtained from the internet, hand sketches
that provide a visual description of the desired task, or simple language
descriptions. As a preliminary step towards this, we investigate the
capabilities of large scale pre-trained models (foundation models) for
zero-shot goal specification, and find promising results in a collection of
simulated robot manipulation tasks and real-world datasets.

探索了任务规格的另一种目标，如来自互联网的图像，提供所需任务的视觉描述的手绘图或简单的语言描述，并研究了大规模预训练模型（基础模型）的能力，以实现零样本目标规格，结果表明在模拟机器人操作任务和真实数据集中具有良好的表现。

基础模型能否对机器人操作执行零样本任务规范？

Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?

We study the problem of learning a range of vision-based manipulation tasks
from a large offline dataset of robot interaction. In order to accomplish this,
humans need easy and effective ways of specifying tasks to the robot. Goal
images are one popular form of task specification, as they are already grounded
in the robot's observation space. However, goal images also have a number of
drawbacks: they are inconvenient for humans to provide, they can over-specify
the desired behavior leading to a sparse reward signal, or under-specify task
information in the case of non-goal reaching tasks. Natural language provides a
convenient and flexible alternative for task specification, but comes with the
challenge of grounding language in the robot's observation space. To scalably
learn this grounding we propose to leverage offline robot datasets (including
highly sub-optimal, autonomously collected data) with crowd-sourced natural
language labels. With this data, we learn a simple classifier which predicts if
a change in state completes a language instruction. This provides a
language-conditioned reward function that can then be used for offline
multi-task RL. In our experiments, we find that on language-conditioned
manipulation tasks our approach outperforms both goal-image specifications and
language conditioned imitation techniques by more than 25%, and is able to
perform visuomotor tasks from natural language, such as "open the right drawer"
and "move the stapler", on a Franka Emika Panda robot.

本文研究了使用自然语言标签并结合机器人交互数据集，来学习规划机器人视觉操作任务的问题，并发现此方法在具有一定自由度的语言规划任务中表现更优秀，成功地完成了使用自然语言描述的物品移动任务。