Can emergent language models faithfully model the intelligence of
decision-making agents? Though modern language models exhibit already some
reasoning ability, and theoretically can potentially express any probable
distribution over tokens, it remains underexplored how the world knowledge
these pretrained models have memorized can be utilized to comprehend an agent's
behaviour in the physical world. This study empirically examines, for the first
time, how well large language models (LLMs) can build a mental model of agents,
termed agent mental modelling, by reasoning about an agent's behaviour and its
effect on states from agent interaction history. This research may unveil the
potential of leveraging LLMs for elucidating RL agent behaviour, addressing a
key challenge in eXplainable reinforcement learning (XRL). To this end, we
propose specific evaluation metrics and test them on selected RL task datasets
of varying complexity, reporting findings on agent mental model establishment.
Our results disclose that LLMs are not yet capable of fully mental modelling
agents through inference alone without further innovations. This work thus
provides new insights into the capabilities and limitations of modern LLMs.

大语言模型是否能够忠实地模拟决策智能体的智能能力，本研究通过实证方法首次探讨了大语言模型在通过推理构建决策智能体的心智模型方面的能力，并提供了关于其对强化学习智能体行为的解释性约束方面的新见解。

强化学习智能体的心智模型构建：基于语言模型的研究

Mental Modeling of Reinforcement Learning Agents by Language Models

Explainable Reinforcement Learning (XRL) can provide transparency into the
decision-making process of a Deep Reinforcement Learning (DRL) model and
increase user trust and adoption in real-world use cases. By utilizing XRL
techniques, researchers can identify potential vulnerabilities within a trained
DRL model prior to deployment, therefore limiting the potential for mission
failure or mistakes by the system. This paper introduces the ARLIN (Assured RL
Model Interrogation) Toolkit, an open-source Python library that identifies
potential vulnerabilities and critical points within trained DRL models through
detailed, human-interpretable explainability outputs. To illustrate ARLIN's
effectiveness, we provide explainability visualizations and vulnerability
analysis for a publicly available DRL model. The open-source code repository is
available for download at this https URL

通过使用 ARLIN 工具包，本文介绍了一种基于可解释强化学习（XRL）的方法，通过提供人类可解释的输出，帮助识别培训过的深度强化学习（DRL）模型中的潜在漏洞和关键问题，以增加用户在实际应用中的信任和采用。