Autonomous drones can operate in remote and unstructured environments,
enabling various real-world applications. However, the lack of effective
vision-based algorithms has been a stumbling block to achieving this goal.
Existing systems often require hand-engineered components for state estimation,
planning, and control. Such a sequential design involves laborious tuning,
human heuristics, and compounding delays and errors. This paper tackles the
vision-based autonomous-drone-racing problem by learning deep sensorimotor
policies. We use contrastive learning to extract robust feature representations
from the input images and leverage a two-stage learning-by-cheating framework
for training a neural network policy. The resulting policy directly infers
control commands with feature representations learned from raw images, forgoing
the need for globally-consistent state estimation, trajectory planning, and
handcrafted control design. Our experimental results indicate that our
vision-based policy can achieve the same level of racing performance as the
state-based policy while being robust against different visual disturbances and
distractors. We believe this work serves as a stepping-stone toward developing
intelligent vision-based autonomous systems that control the drone purely from
image inputs, like human pilots.

通过学习深度的感知动作策略，使用对比学习从输入图像中提取固定特征表示，通过两阶段的作弊式学习框架训练神经网络策略，将视觉驱动的自主无人机竞赛问题转化为了提取原始图像的特征表示进行控制命令推断，无需全局一致的状态估计、轨迹规划和手工控制设计。该方法不仅可使控制策略更具有鲁棒性，而且可以实现与状态法相同的赛车性能，为开发纯靠图像输入控制无人机的智能视觉自主系统铺平了道路。

基于视觉的自主无人机赛车的深度感觉运动策略学习

Learning Deep Sensorimotor Policies for Vision-based Autonomous Drone Racing

Teleoperation offers the possibility of imparting robotic systems with
sophisticated reasoning skills, intuition, and creativity to perform tasks.
However, current teleoperation solutions for high degree-of-actuation (DoA),
multi-fingered robots are generally cost-prohibitive, while low-cost offerings
usually provide reduced degrees of control. Herein, a low-cost, vision based
teleoperation system, DexPilot, was developed that allows for complete control
over the full 23 DoA robotic system by merely observing the bare human hand.
DexPilot enables operators to carry out a variety of complex manipulation tasks
that go beyond simple pick-and-place operations. This allows for collection of
high dimensional, multi-modality, state-action data that can be leveraged in
the future to learn sensorimotor policies for challenging manipulation tasks.
The system performance was measured through speed and reliability metrics
across two human demonstrators on a variety of tasks. The videos of the
experiments can be found at this https URL

本研究设计了一个低成本的基于视觉的遥操作系统 DexPilot，透过观察裸露的人手简单操作即可完全控制拥有 23 个 DoA 的机器人系统，能够执行各种复杂的操作，收集高维度、多模态的状态 - 行动数据，为未来学习挑战性操作任务的感觉运动策略奠定基础。