Skill-based reinforcement learning (RL) approaches have shown considerable
promise, especially in solving long-horizon tasks via hierarchical structures.
These skills, learned task-agnostically from offline datasets, can accelerate
the policy learning process for new tasks. Yet, the application of these skills
in different domains remains restricted due to their inherent dependency on the
datasets, which poses a challenge when attempting to learn a skill-based policy
via RL for a target domain different from the datasets' domains. In this paper,
we present a novel offline skill learning framework DuSkill which employs a
guided Diffusion model to generate versatile skills extended from the limited
skills in datasets, thereby enhancing the robustness of policy learning for
tasks in different domains. Specifically, we devise a guided diffusion-based
skill decoder in conjunction with the hierarchical encoding to disentangle the
skill embedding space into two distinct representations, one for encapsulating
domain-invariant behaviors and the other for delineating the factors that
induce domain variations in the behaviors. Our DuSkill framework enhances the
diversity of skills learned offline, thus enabling to accelerate the learning
procedure of high-level policies for different domains. Through experiments, we
show that DuSkill outperforms other skill-based imitation learning and RL
algorithms for several long-horizon tasks, demonstrating its benefits in
few-shot imitation and online RL.

通过离线数据集学习的、能够在不同领域中应用的多功能技能是一项全新的离线技能学习框架 DuSkill 的核心，通过引导式扩散模型生成可以应用于任务的多功能技能，从而增加不同领域中策略学习的稳健性。

离线技能扩散的稳健策略学习

Robust Policy Learning via Offline Skill Diffusion

Skill-based reinforcement learning (RL) has emerged as a promising strategy
to leverage prior knowledge for accelerated robot learning. Skills are
typically extracted from expert demonstrations and are embedded into a latent
space from which they can be sampled as actions by a high-level RL agent.
However, this skill space is expansive, and not all skills are relevant for a
given robot state, making exploration difficult. Furthermore, the downstream RL
agent is limited to learning structurally similar tasks to those used to
construct the skill space. We firstly propose accelerating exploration in the
skill space using state-conditioned generative models to directly bias the
high-level agent towards only sampling skills relevant to a given state based
on prior experience. Next, we propose a low-level residual policy for
fine-grained skill adaptation enabling downstream RL agents to adapt to unseen
task variations. Finally, we validate our approach across four challenging
manipulation tasks that differ from those used to build the skill space,
demonstrating our ability to learn across task variations while significantly
accelerating exploration, outperforming prior works. Code and videos are
available on our project website: this https URL.

通过使用状态条件生成模型在技能空间中加速探索，同时提出低层次的剩余策略来适应未知的任务变化，从而在与先前工作的比较中显着加快了探索速度，并跨四个不同于那些用于构建技能空间的具有挑战性的操作任务进行了验证。