While reinforcement learning (RL) shows remarkable success in decision-making
problems, it often requires a lot of interactions with the environment, and in
sparse-reward environments, it is challenging to learn meaningful policies.
Large Language Models (LLMs) can potentially provide valuable guidance to
agents in learning policies, thereby enhancing the performance of RL algorithms
in such environments. However, LLMs often encounter difficulties in
understanding downstream tasks, which hinders their ability to optimally assist
agents in these tasks. A common approach to mitigating this issue is to
fine-tune the LLMs with task-related data, enabling them to offer useful
guidance for RL agents. However, this approach encounters several difficulties,
such as inaccessible model weights or the need for significant computational
resources, making it impractical. In this work, we introduce RLAdapter, a
framework that builds a better connection between RL algorithms and LLMs by
incorporating an adapter model. Within the RLAdapter framework, fine-tuning a
lightweight language model with information generated during the training
process of RL agents significantly aids LLMs in adapting to downstream tasks,
thereby providing better guidance for RL agents. We conducted experiments to
evaluate RLAdapter in the Crafter environment, and the results show that
RLAdapter surpasses the SOTA baselines. Furthermore, agents under our framework
exhibit common-sense behaviors that are absent in baseline models.

RLAdapter 通过引入一个适配器模型，在强化学习和大型语言模型之间建立更好的联系，通过在 RL 代理的训练过程中生成的信息来微调轻量级语言模型，从而在适应下游任务方面提供更好的指导，并在 Crafter 环境中实验表明 RLAdapter 超过了基线模型，且我们的框架下的代理展现出了常识行为。