Addressing decision-making problems using sequence modeling to predict future
trajectories shows promising results in recent years. In this paper, we take a
step further to leverage the sequence predictive method in wider areas such as
long-term planning, vision-based control, and multi-task decision-making. To
this end, we propose a method to utilize a diffusion-based generative sequence
model to plan a series of milestones in a latent space and to have an agent to
follow the milestones to accomplish a given task. The proposed method can learn
control-relevant, low-dimensional latent representations of milestones, which
makes it possible to efficiently perform long-term planning and vision-based
control. Furthermore, our approach exploits generation flexibility of the
diffusion model, which makes it possible to plan diverse trajectories for
multi-task decision-making. We demonstrate the proposed method across offline
reinforcement learning (RL) benchmarks and an visual manipulation environment.
The results show that our approach outperforms offline RL methods in solving
long-horizon, sparse-reward tasks and multi-task problems, while also achieving
the state-of-the-art performance on the most challenging vision-based
manipulation benchmark.

利用序列建模方法来预测未来轨迹的决策问题在近年来取得了良好的结果。本文进一步利用这种序列预测方法在长期规划、基于视觉的控制和多任务决策等更广泛的领域中进行研究，提出了一种基于扩散模型的生成序列模型的方法，在潜空间内计划一系列里程碑，并使代理人遵循这些里程碑完成给定任务。该方法可以学习控制相关的低维潜变量表示，从而能够高效进行长期规划和基于视觉的控制。此外，我们的方法利用了扩散模型的生成灵活性，可以为多任务决策制定多样的轨迹。我们在离线强化学习（RL）基准测试和视觉操作环境中对该方法进行了验证，结果表明我们的方法在解决长远视角、稀疏奖励任务和多任务问题方面优于离线 RL 方法，并在最具挑战性的基于视觉的操作基准上实现了最先进的性能。

扩散式任务无关里程碑规划器

Diffused Task-Agnostic Milestone Planner

Deep reinforcement learning provides a promising approach for vision-based
control of real-world robots. However, the generalization of such models
depends critically on the quantity and variety of data available for training.
This data can be difficult to obtain for some types of robotic systems, such as
fragile, small-scale quadrotors. Simulated rendering and physics can provide
for much larger datasets, but such data is inherently of lower quality: many of
the phenomena that make the real-world autonomous flight problem challenging,
such as complex physics and air currents, are modeled poorly or not at all, and
the systematic differences between simulation and the real world are typically
impossible to eliminate. In this work, we investigate how data from both
simulation and the real world can be combined in a hybrid deep reinforcement
learning algorithm. Our method uses real-world data to learn about the dynamics
of the system, and simulated data to learn a generalizable perception system
that can enable the robot to avoid collisions using only a monocular camera. We
demonstrate our approach on a real-world nano aerial vehicle collision
avoidance task, showing that with only an hour of real-world data, the
quadrotor can avoid collisions in new environments with various lighting
conditions and geometry. Code, instructions for building the aerial vehicles,
and videos of the experiments can be found at github.com/gkahn13/GtS

本文提出一种深度强化学习算法，将模拟数据和真实世界数据相结合，通过学习机器的动力学和建立一个可扩展的感知系统，让四旋翼机器人仅凭单目相机避免碰撞，并在各种光线和几何条件下，用一个小时的真实世界数据就能在新环境中避免碰撞。

通过模拟实现泛化：将模拟和真实数据集成到基于视觉的无人机深度强化学习中

Generalization through Simulation: Integrating Simulated and Real Data  into Deep Reinforcement Learning for Vision-Based Autonomous Flight

In this paper, we study the problem of learning vision-based dynamic
manipulation skills using a scalable reinforcement learning approach. We study
this problem in the context of grasping, a longstanding challenge in robotic
manipulation. In contrast to static learning behaviors that choose a grasp
point and then execute the desired grasp, our method enables closed-loop
vision-based control, whereby the robot continuously updates its grasp strategy
based on the most recent observations to optimize long-horizon grasp success.
To that end, we introduce QT-Opt, a scalable self-supervised vision-based
reinforcement learning framework that can leverage over 580k real-world grasp
attempts to train a deep neural network Q-function with over 1.2M parameters to
perform closed-loop, real-world grasping that generalizes to 96% grasp success
on unseen objects. Aside from attaining a very high success rate, our method
exhibits behaviors that are quite distinct from more standard grasping systems:
using only RGB vision-based perception from an over-the-shoulder camera, our
method automatically learns regrasping strategies, probes objects to find the
most effective grasps, learns to reposition objects and perform other
non-prehensile pre-grasp manipulations, and responds dynamically to
disturbances and perturbations.

本文提出了一种基于 QT-Opt 的可扩展自监督视觉强化学习框架，该框架能够利用超过 580k 的真实抓取尝试来训练一个具有超过 1.2M 个参数的深度神经网络 Q 函数，实现闭环实际抓取并能够推广到 96% 的机器人抓取任务中，而且不仅实现了极高的成功率，而且通过 RGB 视觉感知和操纵，自动学习了重抓策略，动态响应干扰与扰动，并能够重新定位物品和执行其他非抓取前的操作。