Animals often demonstrate a remarkable ability to adapt to their environments
during their lifetime. They do so partly due to the evolution of morphological
and neural structures. These structures capture features of environments shared
between generations to bias and speed up lifetime learning. In this work, we
propose a computational model for studying a mechanism that can enable such a
process. We adopt a computational framework based on meta reinforcement
learning as a model of the interplay between evolution and development. At the
evolutionary scale, we evolve reservoirs, a family of recurrent neural networks
that differ from conventional networks in that one optimizes not the weight
values but hyperparameters of the architecture: the later control macro-level
properties, such as memory and dynamics. At the developmental scale, we employ
these evolved reservoirs to facilitate the learning of a behavioral policy
through Reinforcement Learning (RL). Within an RL agent, a reservoir encodes
the environment state before providing it to an action policy. We evaluate our
approach on several 2D and 3D simulated environments. Our results show that the
evolution of reservoirs can improve the learning of diverse challenging tasks.
We study in particular three hypotheses: the use of an architecture combining
reservoirs and reinforcement learning could enable (1) solving tasks with
partial observability, (2) generating oscillatory dynamics that facilitate the
learning of locomotion tasks, and (3) facilitating the generalization of
learned behaviors to new tasks unknown during the evolution phase.

通过进化和发展之间的相互作用，本文提出了一种计算模型来研究能够使生物适应环境的机制，并使用元强化学习作为计算框架。通过进化递归神经网络架构的超参数而非权重值来生成的水库，可以促进复杂任务的学习、局部可观测任务的解决、促进运动任务学习中的振荡动力学以及学习行为的泛化能力。