Reinforcement learning (RL) trains agents to accomplish complex tasks through
environmental interaction data, but its capacity is also limited by the scope
of the available data. To obtain a knowledgeable agent, a promising approach is
to leverage the knowledge from large language models (LLMs). Despite previous
studies combining LLMs with RL, seamless integration of the two components
remains challenging due to their semantic gap. This paper introduces a novel
method, Knowledgeable Agents from Language Model Rollouts (KALM), which
extracts knowledge from LLMs in the form of imaginary rollouts that can be
easily learned by the agent through offline reinforcement learning methods. The
primary challenge of KALM lies in LLM grounding, as LLMs are inherently limited
to textual data, whereas environmental data often comprise numerical vectors
unseen to LLMs. To address this, KALM fine-tunes the LLM to perform various
tasks based on environmental data, including bidirectional translation between
natural language descriptions of skills and their corresponding rollout data.
This grounding process enhances the LLM's comprehension of environmental
dynamics, enabling it to generate diverse and meaningful imaginary rollouts
that reflect novel skills. Initial empirical evaluations on the CLEVR-Robot
environment demonstrate that KALM enables agents to complete complex
rephrasings of task goals and extend their capabilities to novel tasks
requiring unprecedented optimal behaviors. KALM achieves a success rate of 46%
in executing tasks with unseen goals, substantially surpassing the 26% success
rate achieved by baseline methods. Furthermore, KALM effectively enables the
LLM to comprehend environmental dynamics, resulting in the generation of
meaningful imaginary rollouts that reflect novel skills and demonstrate the
seamless integration of large language models and reinforcement learning.

基于大型语言模型的知识型代理系统能够通过想象中的轨迹从环境交互数据中提取知识并与强化学习进行无缝集成。