There has been a growing interest in developing learner models to enhance
learning and teaching experiences in educational environments. However,
existing works have primarily focused on structured environments relying on
meticulously crafted representations of tasks, thereby limiting the agent's
ability to generalize skills across tasks. In this paper, we aim to enhance the
generalization capabilities of agents in open-ended text-based learning
environments by integrating Reinforcement Learning (RL) with Large Language
Models (LLMs). We investigate three types of agents: (i) RL-based agents that
utilize natural language for state and action representations to find the best
interaction strategy, (ii) LLM-based agents that leverage the model's general
knowledge and reasoning through prompting, and (iii) hybrid LLM-assisted RL
agents that combine these two strategies to improve agents' performance and
generalization. To support the development and evaluation of these agents, we
introduce PharmaSimText, a novel benchmark derived from the PharmaSim virtual
pharmacy environment designed for practicing diagnostic conversations. Our
results show that RL-based agents excel in task completion but lack in asking
quality diagnostic questions. In contrast, LLM-based agents perform better in
asking diagnostic questions but fall short of completing the task. Finally,
hybrid LLM-assisted RL agents enable us to overcome these limitations,
highlighting the potential of combining RL and LLMs to develop high-performing
agents for open-ended learning environments.

通过将强化学习与大型语言模型结合，研究了在开放性文本学习环境中增强代理的泛化能力，提出了三种代理类型：基于强化学习的代理、基于大型语言模型的代理和融合两者的混合代理，以提高代理的性能和泛化能力，并通过 PharmaSimText 提供的基准测试验证了研究成果。结果表明，基于强化学习的代理在任务完成方面表现出色，但在提问诊断问题方面有所欠缺；相反，基于大型语言模型的代理在提问诊断问题方面表现较好，但在完成任务方面表现较差；而混合的大型语言模型辅助强化学习代理能够克服这些限制，凸显了将强化学习和大型语言模型相结合在开放性学习环境中开发高性能代理的潜力。

面向基于文本的教育环境的通用智能体: RL 与 LLMs 的整合研究

Towards Generalizable Agents in Text-Based Educational Environments: A  Study of Integrating RL with LLMs

Student modeling is central to many educational technologies as it enables
the prediction of future learning outcomes and targeted instructional
strategies. However, open-ended learning environments pose challenges for
accurately modeling students due to the diverse behaviors exhibited by students
and the absence of a well-defined set of learning skills. To approach these
challenges, we explore the application of Large Language Models (LLMs) for
in-context student modeling in open-ended learning environments. We introduce a
novel framework, LLM-SS, that leverages LLMs for synthesizing student's
behavior. More concretely, given a particular student's solving attempt on a
reference task as observation, the goal is to synthesize the student's attempt
on a target task. Our framework can be combined with different LLMs; moreover,
we fine-tune LLMs using domain-specific expertise to boost their understanding
of domain background and student behaviors. We evaluate several concrete
methods based on LLM-SS using the StudentSyn benchmark, an existing student's
attempt synthesis benchmark in visual programming. Experimental results show a
significant improvement compared to baseline methods included in the StudentSyn
benchmark. Furthermore, our method using the fine-tuned Llama2-70B model
improves noticeably compared to using the base model and becomes on par with
using the state-of-the-art GPT-4 model.

利用大型语言模型在开放式学习环境中进行学生建模，引入了一种新的框架，LLM-SS，该框架利用 LLMs 来合成学生的行为，并使用领域专业知识对 LLMs 进行微调，以提高对领域背景和学生行为的理解，实验证明在学生尝试合成基准中与基线方法相比有显著改进，并且使用经过微调的 Llama2-70B 模型的方法与使用最先进的 GPT-4 模型相当。