Large language models (LLMs) have recently been used for sequential decision
making in interactive environments. However, leveraging environment reward
signals for continual LLM actor improvement is not straightforward. We propose
Skill Set Optimization (SSO) for improving LLM actor performance through
constructing and refining sets of transferable skills. SSO constructs skills by
extracting common subtrajectories with high rewards and generating subgoals and
instructions to represent each skill. These skills are provided to the LLM
actor in-context to reinforce behaviors with high rewards. Then, SSO further
refines the skill set by pruning skills that do not continue to result in high
rewards. We evaluate our method in the classic videogame NetHack and the text
environment ScienceWorld to demonstrate SSO's ability to optimize a set of
skills and perform in-context policy improvement. SSO outperforms baselines by
40% in our custom NetHack task and outperforms the previous state-of-the-art in
ScienceWorld by 35%.

通过构建和优化一组可迁移技能，我们提出了技能集优化（Skill Set Optimization，SSO）来提高大型语言模型在交互环境中的决策性能。在这种方法中，SSO 通过提取具有高奖励的共同子轨迹，并生成子目标和指示来构建技能。这些技能在上下文中提供给大型语言模型以加强具有高奖励的行为，并通过删除不再产生高奖励的技能来进一步优化技能集。我们通过在经典游戏 NetHack 和文本环境 ScienceWorld 中的实证评估来展示 SSO 优化技能集并提升在上下文中策略改进的能力。在我们自定义的 NetHack 任务中，SSO 的性能比基准提升了 40%，并在 ScienceWorld 中超过了先前的最先进方法达到 35% 的改进。