We present a policy search method for learning complex feedback control
policies that map from high-dimensional sensory inputs to motor torques, for
manipulation tasks with discontinuous contact dynamics. We build on a prior
technique called guided policy search (GPS), which iteratively optimizes a set
of local policies for specific instances of a task, and uses these to train a
complex, high-dimensional global policy that generalizes across task instances.
We extend GPS in the following ways: (1) we propose the use of a model-free
local optimizer based on path integral stochastic optimal control (PI2), which
enables us to learn local policies for tasks with highly discontinuous contact
dynamics; and (2) we enable GPS to train on a new set of task instances in
every iteration by using on-policy sampling: this increases the diversity of
the instances that the policy is trained on, and is crucial for achieving good
generalization. We show that these contributions enable us to learn deep neural
network policies that can directly perform torque control from visual input. We
validate the method on a challenging door opening task and a pick-and-place
task, and we demonstrate that our approach substantially outperforms the prior
LQR-based local policy optimizer on these tasks. Furthermore, we show that
on-policy sampling significantly increases the generalization ability of these
policies.

本文提出了一种通过策略搜索学习复杂的反馈控制策略的方法，该策略可将高维度感知输入映射到电机扭矩以执行具有不连续性接触动力学的操纵任务，该方法在使用先前的技术基础上进行了改进，使用了基于 PI2 的无模型本地优化器和使用 on-policy 抽样来训练针对一系列任务实例的复合全局策略，从而实现了直接从视觉输入执行扭矩控制的深度神经网络策略。