The ability to converse with humans and follow commands in natural language
is crucial for intelligent unmanned aerial vehicles (a.k.a. drones). It can
relieve people's burden of holding a controller all the time, allow
multitasking, and make drone control more accessible for people with
disabilities or with their hands occupied. To this end, we introduce Aerial
Vision-and-Dialog Navigation (AVDN), to navigate a drone via natural language
conversation. We build a drone simulator with a continuous photorealistic
environment and collect a new AVDN dataset of over 3k recorded navigation
trajectories with asynchronous human-human dialogs between commanders and
followers. The commander provides initial navigation instruction and further
guidance by request, while the follower navigates the drone in the simulator
and asks questions when needed. During data collection, followers' attention on
the drone's visual observation is also recorded. Based on the AVDN dataset, we
study the tasks of aerial navigation from (full) dialog history and propose an
effective Human Attention Aided (HAA) baseline model, which learns to predict
both navigation waypoints and human attention.

本文提出 Aerial Vision-and-Dialog Navigation (AVDN) 技术，利用自然语言对话方式实现无人机的导航，研究使用 AVDN 数据集实现从全对话历史中进行航行的任务，并提出一种有效的 Human Attention Aided (HAA) 基线模型，通过预测导航航点和人类注意力实现自动导航。