Navigation is a complex skill with a long history of research in animals and
humans. In this work, we simulate the Morris Water Maze in 2D to train deep
reinforcement learning agents. We perform automatic classification of
navigation strategies, analyze the distribution of strategies used by
artificial agents, and compare them with experimental data to show similar
learning dynamics as those seen in humans and rodents. We develop
environment-specific auxiliary tasks and examine factors affecting their
usefulness. We suggest that the most beneficial tasks are potentially more
biologically feasible for real agents to use. Lastly, we explore the
development of internal representations in the activations of artificial agent
neural networks. These representations resemble place cells and head-direction
cells found in mouse brains, and their presence has correlation to the
navigation strategies that artificial agents employ.

本研究使用 2D 模拟 Morris 水迷宫来训练深度强化学习代理程序，并自动分类了导航策略， 将人工智能的学习动力学与动物实验数据进行比较。通过开发环境特定的辅助任务来研究影响其有用性的因素， 最后研究了人工代理神经网络中激活的内部表征的发展，这些表征类似于老鼠大脑中发现的 place cells 和 head-direction cells，并且它们的存在与人工代理采用的导航策略有相关性。

通过深度强化学习探究 Morris 水迷宫中的导航策略

Investigating Navigation Strategies in the Morris Water Maze through  Deep Reinforcement Learning

Partially observed control problems are a challenging aspect of reinforcement
learning. We extend two related, model-free algorithms for continuous control
-- deterministic policy gradient and stochastic value gradient -- to solve
partially observed domains using recurrent neural networks trained with
backpropagation through time.
We demonstrate that this approach, coupled with long-short term memory is
able to solve a variety of physical control problems exhibiting an assortment
of memory requirements. These include the short-term integration of information
from noisy sensors and the identification of system parameters, as well as
long-term memory problems that require preserving information over many time
steps. We also demonstrate success on a combined exploration and memory problem
in the form of a simplified version of the well-known Morris water maze task.
Finally, we show that our approach can deal with high-dimensional observations
by learning directly from pixels.
We find that recurrent deterministic and stochastic policies are able to
learn similarly good solutions to these tasks, including the water maze where
the agent must learn effective search strategies.

本研究利用 RNN 与反向传播算法，扩展了两个连续控制的无模型算法，能够成功地解决大量的物理控制问题，包括部分受到噪声干扰而需要信息短暂整合的问题，以及需要在多个时间步骤中保留信息的长期记忆问题，并且使用简化版本的 Morris 水迷宫任务进行了探索与记忆结合问题的研究。同时，通过直接从像素中学习，能够处理高维度的观测问题。