Large language models (LLMs) have recently been used for sequential decision
making in interactive environments. However, leveraging environment reward
signals for continual LLM actor improvement is not straightforward. We propose
Skill Set Optimization (SSO) for improving LLM actor performance through
constructing and refining sets of transferable skills. SSO constructs skills by
extracting common subtrajectories with high rewards and generating subgoals and
instructions to represent each skill. These skills are provided to the LLM
actor in-context to reinforce behaviors with high rewards. Then, SSO further
refines the skill set by pruning skills that do not continue to result in high
rewards. We evaluate our method in the classic videogame NetHack and the text
environment ScienceWorld to demonstrate SSO's ability to optimize a set of
skills and perform in-context policy improvement. SSO outperforms baselines by
40% in our custom NetHack task and outperforms the previous state-of-the-art in
ScienceWorld by 35%.

通过构建和优化一组可迁移技能，我们提出了技能集优化（Skill Set Optimization，SSO）来提高大型语言模型在交互环境中的决策性能。在这种方法中，SSO 通过提取具有高奖励的共同子轨迹，并生成子目标和指示来构建技能。这些技能在上下文中提供给大型语言模型以加强具有高奖励的行为，并通过删除不再产生高奖励的技能来进一步优化技能集。我们通过在经典游戏 NetHack 和文本环境 ScienceWorld 中的实证评估来展示 SSO 优化技能集并提升在上下文中策略改进的能力。在我们自定义的 NetHack 任务中，SSO 的性能比基准提升了 40%，并在 ScienceWorld 中超过了先前的最先进方法达到 35% 的改进。

技能集优化：通过可转移技能强化语言模型行为

Skill Set Optimization: Reinforcing Language Model Behavior via  Transferable Skills

We propose a novel parameterized skill-learning algorithm that aims to learn
transferable parameterized skills and synthesize them into a new action space
that supports efficient learning in long-horizon tasks. We propose to leverage
off-policy Meta-RL combined with a trajectory-centric smoothness term to learn
a set of parameterized skills. Our agent can use these learned skills to
construct a three-level hierarchical framework that models a
Temporally-extended Parameterized Action Markov Decision Process. We
empirically demonstrate that the proposed algorithms enable an agent to solve a
set of difficult long-horizon (obstacle-course and robot manipulation) tasks.

本研究提出了一种新的参数化技能学习算法，此算法可以学习可转移的参数化技能并将其综合到新的行动空间中，从而支持长视深远任务的高效学习。经实证表明，这种方法可以使代理人解决一组困难的长视深远任务。