Generalization remains one of the most important desiderata for robust robot
learning systems. While recently proposed approaches show promise in
generalization to novel objects, semantic concepts, or visual distribution
shifts, generalization to new tasks remains challenging. For example, a
language-conditioned policy trained on pick-and-place tasks will not be able to
generalize to a folding task, even if the arm trajectory of folding is similar
to pick-and-place. Our key insight is that this kind of generalization becomes
feasible if we represent the task through rough trajectory sketches. We propose
a policy conditioning method using such rough trajectory sketches, which we
call RT-Trajectory, that is practical, easy to specify, and allows the policy
to effectively perform new tasks that would otherwise be challenging to
perform. We find that trajectory sketches strike a balance between being
detailed enough to express low-level motion-centric guidance while being coarse
enough to allow the learned policy to interpret the trajectory sketch in the
context of situational visual observations. In addition, we show how trajectory
sketches can provide a useful interface to communicate with robotic policies:
they can be specified through simple human inputs like drawings or videos, or
through automated methods such as modern image-generating or
waypoint-generating methods. We evaluate RT-Trajectory at scale on a variety of
real-world robotic tasks, and find that RT-Trajectory is able to perform a
wider range of tasks compared to language-conditioned and goal-conditioned
policies, when provided the same training data.

该研究论文探讨了使用粗略轨迹草图作为任务表示的方法，称为 RT-Trajectory，该方法通过有效地执行新任务，克服了现有语言条件和目标条件策略的局限性，具有广泛的泛化能力。

RT - 轨迹：通过事后轨迹草图实现机器人任务的泛化

RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory  Sketches

Recent works have shown that attaching prompts to the input is effective at
conditioning Language Models (LM) to perform specific tasks. However, prompts
are always included in the input text during inference, thus incurring
substantial computational and memory overhead. Also, there is currently no
straightforward method of utilizing prompts that are longer than the maximum
input length of the LMs without incurring additional costs during inference. We
propose Prompt Injection (PI), a novel formulation of injecting the prompt into
the parameters of an LM to be an efficient alternative to attaching fixed
prompts to the input. We show that in scenarios with long fixed prompts, PI can
be up to 280 times more efficient in terms of total FLOPs than previous
approaches. We further explore methodologies for PI and show promising results
in persona-dependent conversation, semantic parsing, and zero-shot learning
with task instructions. Through these explorations, we show that PI can be a
promising direction for conditioning language models, especially in scenarios
with long and fixed prompts.

本文提出了一种名为 Prompt Injection（PI）的新方法，通过将提示注入 LM 的参数中，来代替在输入中添加固定提示的方法，以提高其在特定任务上的性能。在实现固定提示较长的情况下，PI 的操作效率比之前的方法高达 280 倍。我们进一步探究了 PI 的方法学，并在特定任务下展示了其良好的效果，结果表明 PI 在特定任务下是一种很有前途的方向。

Prompt Injection: 固定输入参数化

Prompt Injection: Parameterization of Fixed Inputs

Learning task-agnostic dynamics models in high-dimensional observation spaces
can be challenging for model-based RL agents. We propose a novel way to learn
latent world models by learning to predict sequences of future actions
conditioned on task completion. These task-conditioned models adaptively focus
modeling capacity on task-relevant dynamics, while simultaneously serving as an
effective heuristic for planning with sparse rewards. We evaluate our method on
challenging visual goal completion tasks and show a substantial increase in
performance compared to prior model-free approaches.

提出了一种新的方法来学习依赖任务完成的未来动作序列预测的潜在世界模型，该模型适应地关注任务相关的动态学习，并同时充当稀疏奖励下计划的有效启发式方法，通过挑战性的视觉目标完成任务的评估，我们发现该方法较之前的无模型方法在性能上有了显著提高。