Reinforcement learning provides a powerful and flexible framework for
automated acquisition of robotic motion skills. However, applying reinforcement
learning requires a sufficiently detailed representation of the state,
including the configuration of task-relevant objects. We present an approach
that automates state-space construction by learning a state representation
directly from camera images. Our method uses a deep spatial autoencoder to
acquire a set of feature points that describe the environment for the current
task, such as the positions of objects, and then learns a motion skill with
these feature points using an efficient reinforcement learning method based on
local linear models. The resulting controller reacts continuously to the
learned feature points, allowing the robot to dynamically manipulate objects in
the world with closed-loop control. We demonstrate our method with a PR2 robot
on tasks that include pushing a free-standing toy block, picking up a bag of
rice using a spatula, and hanging a loop of rope on a hook at various
positions. In each task, our method automatically learns to track task-relevant
objects and manipulate their configuration with the robot's arm.

该研究采用增强学习的方法，通过深度空间自编码器从摄像头图像中直接学习状态表征，以实现对环境进行特征点提取和控制的闭环控制，展示了 PR2 机器人在推动自由玩具方块、使用铲子拿起一袋米和在不同位置将绳环悬挂在钩子上等任务中自动学习控制的成功案例。