Reinforcement learning (RL) for motion planning of multi-degree-of-freedom
robots still suffers from low efficiency in terms of slow training speed and
poor generalizability. In this paper, we propose a novel RL-based robot motion
planning framework that uses implicit behavior cloning (IBC) and dynamic
movement primitive (DMP) to improve the training speed and generalizability of
an off-policy RL agent. IBC utilizes human demonstration data to leverage the
training speed of RL, and DMP serves as a heuristic model that transfers motion
planning into a simpler planning space. To support this, we also create a human
demonstration dataset using a pick-and-place experiment that can be used for
similar studies. Comparison studies in simulation reveal the advantage of the
proposed method over the conventional RL agents with faster training speed and
higher scores. A real-robot experiment indicates the applicability of the
proposed method to a simple assembly task. Our work provides a novel
perspective on using motion primitives and human demonstration to leverage the
performance of RL for robot applications.

提出了一种基于强化学习的机器人运动规划框架，使用隐式行为克隆和动态运动原语来提高离线强化学习代理的训练速度和泛化能力，同时提供了人类示范数据集，实验结果表明了该方法在模拟环境下的优势以及实际装配任务的适用性。

利用隐式行为克隆和动态运动基元促进机器人动作规划的强化学习

Using Implicit Behavior Cloning and Dynamic Movement Primitive to  Facilitate Reinforcement Learning for Robot Motion Planning

Imitation learning has been applied to a range of robotic tasks, but can
struggle when (1) robots encounter edge cases that are not represented in the
training data (distribution shift) or (2) the human demonstrations are
heterogeneous: taking different paths around an obstacle, for instance
(multimodality). Interactive fleet learning (IFL) mitigates distribution shift
by allowing robots to access remote human teleoperators during task execution
and learn from them over time, but is not equipped to handle multimodality.
Recent work proposes Implicit Behavior Cloning (IBC), which is able to
represent multimodal demonstrations using energy-based models (EBMs). In this
work, we propose addressing both multimodality and distribution shift with
Implicit Interactive Fleet Learning (IIFL), the first extension of implicit
policies to interactive imitation learning (including the single-robot,
single-human setting). IIFL quantifies uncertainty using a novel application of
Jeffreys divergence to EBMs. While IIFL is more computationally expensive than
explicit methods, results suggest that IIFL achieves 4.5x higher return on
human effort in simulation experiments and an 80% higher success rate in a
physical block pushing task over (Explicit) IFL, IBC, and other baselines when
human supervision is heterogeneous.

本文提出了解决 Interactive Fleet Learning (分布转移) 和 Implicit Behavior Cloning (多模态) 问题的 Implicit Interactive Fleet Learning 方法，并通过模拟实验和物理实验表明其在人类辅助下具有更高的执行成功率和更高的回报。