Successfully addressing a wide variety of tasks is a core ability of
autonomous agents, which requires flexibly adapting the underlying
decision-making strategies and, as we argue in this work, also adapting the
underlying perception modules. An analogical argument would be the human visual
system, which uses top-down signals to focus attention determined by the
current task. Similarly, in this work, we adapt pre-trained large vision models
conditioned on specific downstream tasks in the context of multi-task policy
learning. We introduce task-conditioned adapters that do not require finetuning
any pre-trained weights, combined with a single policy trained with behavior
cloning and capable of addressing multiple tasks. We condition the policy and
visual adapters on task embeddings, which can be selected at inference if the
task is known, or alternatively inferred from a set of example demonstrations.
To this end, we propose a new optimization-based estimator. We evaluate the
method on a wide variety of tasks of the CortexBench benchmark and show that,
compared to existing work, it can be addressed with a single policy. In
particular, we demonstrate that adapting visual features is a key design choice
and that the method generalizes to unseen tasks given visual demonstrations.

在这项工作中，我们证明自适应底层感知模块对于自主代理在各种任务中的成功是至关重要的，我们引入了任务条件适配器的概念，并展示了它可以通过单一策略来解决多个任务，同时证明了方法可以在未见过的任务中使用视觉示范进行泛化。