In contrast to classical cognitive science which studied brains in isolation,
ecological approaches focused on the role of the body and environment in
shaping cognition. Similarly, in this thesis we adopt an ecological approach to
grounded natural language understanding (NLU) research. Grounded language
understanding studies language understanding systems situated in the context of
events, actions and precepts in naturalistic/simulated virtual environments.
Where classic research tends to focus on designing new models and optimization
methods while treating environments as given, we explore the potential of
environment design for improving data collection and model development. We
developed novel training and annotation approaches for procedural text
understanding based on text-based game environments. We also drew upon embodied
cognitive linguistics literature to propose a roadmap for grounded NLP
research, and to inform the development of a new benchmark for measuring the
progress of large language models on challenging commonsense reasoning tasks.
We leveraged the richer supervision provided by text-based game environments to
develop Breakpoint Transformers, a novel approach to modeling intermediate
semantic information in long narrative or procedural texts. Finally, we
integrated theories on the role of environments in collective human
intelligence to propose a design for AI-augmented "social thinking
environments" for knowledge workers like scientists.

通过生态学方法研究自然语言理解，利用基于基于文本游戏环境的训练和注释方法来改进数据收集和模型开发，提出了一种新的测试基准来衡量大型语言模型在常识推理任务中的进展，并提出了 AI 辅助的 “社交思维环境” 设计用于科学家等知识工作者。

探索环境在基于语境的自然语言理解中的作用

"What's my model inside of?": Exploring the role of environments for  grounded natural language understanding

One major challenge in reinforcement learning (RL) is the large amount of
steps for the RL agent needs to converge in the training process and learn the
optimal policy, especially in text-based game environments where the action
space is extensive. However, non-player characters (NPCs) sometimes hold some
key information about the game, which can potentially help to train RL agents
faster. Thus, this paper explores how to interact and converse with NPC agents
to get the key information using large language models (LLMs), as well as
incorporate this information to speed up RL agent's training using knowledge
graphs (KGs) and Story Shaping.

使用大型语言模型与知识图谱相结合，通过与非玩家角色的互动交流获取关键信息，以加快强化学习代理的训练。

对话塑造：通过 NPC 互动赋能代理

Dialogue Shaping: Empowering Agents through NPC Interaction

Text-based game environments are challenging because agents must deal with
long sequences of text, execute compositional actions using text and learn from
sparse rewards. We address these challenges by proposing Long-Context Language
Decision Transformers (LLDTs), a framework that is based on long transformer
language models and decision transformers (DTs). LLDTs extend DTs with 3
components: (1) exponential tilt to guide the agent towards high obtainable
goals, (2) novel goal conditioning methods yielding significantly better
results than the traditional return-to-go (sum of all future rewards), and (3)
a model of future observations. Our ablation results show that predicting
future observations improves agent performance. To the best of our knowledge,
LLDTs are the first to address offline RL with DTs on these challenging games.
Our experiments show that LLDTs achieve the highest scores among many different
types of agents on some of the most challenging Jericho games, such as
Enchanter.

通过提出一种基于长 Transformer 语言模型和决策 Transformer 的 Long-Context Language Decision Transformers（LLDT）框架，成功应对了基于文本的游戏环境中智能体处理长序列文本、使用文本执行组合动作和从稀疏奖励中学习的挑战，并在传统奖励方法之外引入了三个组成部分，即指导代理朝着高可获得目标的指数倾斜、新颖的目标调节方法及对未来观测的模型，获得了在一些最具挑战性的 Jericho 游戏中比其他各种类型代理都要高的得分。

长文本语言决策 Transformer 和指数倾斜在交互式文本环境中的应用

Long-Context Language Decision Transformers and Exponential Tilt for Interactive Text Environments

We propose a recurrent RL agent with an episodic exploration mechanism that
helps discovering good policies in text-based game environments. We show
promising results on a set of generated text-based games of varying difficulty
where the goal is to collect a coin located at the end of a chain of rooms. In
contrast to previous text-based RL approaches, we observe that our agent learns
policies that generalize to unseen games of greater difficulty.

本论文提出了一种带有片段式探索机制的循环强化学习代理，在文本游戏环境中发现良好策略。我们在一系列生成的文本游戏中展示了有希望的结果，游戏难度各异，目标是在一系列房间的末尾收集硬币。与以往的文本强化学习方法相比，我们发现我们的代理学习到可以泛化到更难的未见过游戏的策略。