We propose a joint simulation and real-world learning framework for mapping
navigation instructions and raw first-person observations to continuous
control. Our model estimates the need for environment exploration, predicts the
likelihood of visiting environment positions during execution, and controls the
agent to both explore and visit high-likelihood positions. We introduce
Supervised Reinforcement Asynchronous Learning (SuReAL). Learning uses both
simulation and real environments without requiring autonomous flight in the
physical environment during training, and combines supervised learning for
predicting positions to visit and reinforcement learning for continuous
control. We evaluate our approach on a natural language instruction-following
task with a physical quadcopter, and demonstrate effective execution and
exploration behavior.

提出一种联合仿真和现实学习框架，将导航指令和第一人称视角的原始观察结果映射到连续控制上，并采用监督式加强异步学习算法，将仿真和真实环境结合起来进行训练，最终在一个自然语言指令遵从的物理四轴飞行器任务中验证其有效性。