Reinforcement learning (RL) agents are commonly trained and evaluated in the
same environment. In contrast, humans often train in a specialized environment
before being evaluated, such as studying a book before taking an exam. The
potential of such specialized training environments is still vastly
underexplored, despite their capacity to dramatically speed up training.
The framework of synthetic environments takes a first step in this direction
by meta-learning neural network-based Markov decision processes (MDPs). The
initial approach was limited to toy problems and produced environments that did
not transfer to unseen RL algorithms. We extend this approach in three ways:
Firstly, we modify the meta-learning algorithm to discover environments
invariant towards hyperparameter configurations and learning algorithms.
Secondly, by leveraging hardware parallelism and introducing a curriculum on an
agent's evaluation episode horizon, we can achieve competitive results on
several challenging continuous control problems. Thirdly, we surprisingly find
that contextual bandits enable training RL agents that transfer well to their
evaluation environment, even if it is a complex MDP. Hence, we set up our
experiments to train synthetic contextual bandits, which perform on par with
synthetic MDPs, yield additional insights into the evaluation environment, and
can speed up downstream applications.

通过元学习神经网络马尔可夫决策过程，我们发现专门的训练环境对于训练强化学习智能体具有潜在的速度提升能力，并且发现上下文为基的赌博机能够实现良好的评估环境转移，从而加速下游应用。