We study a security threat to batch reinforcement learning and control where
the attacker aims to poison the learned policy. The victim is a reinforcement
learner / controller which first estimates the dynamics and the rewards from a
batch data set, and then solves for the optimal policy with respect to the
estimates. The attacker can modify the data set slightly before learning
happens, and wants to force the learner into learning a target policy chosen by
the attacker. We present a unified framework for solving batch policy poisoning
attacks, and instantiate the attack on two standard victims: tabular certainty
equivalence learner in reinforcement learning and linear quadratic regulator in
control. We show that both instantiation result in a convex optimization
problem on which global optimality is guaranteed, and provide analysis on
attack feasibility and attack cost. Experiments show the effectiveness of
policy poisoning attacks.

本研究针对批量强化学习和控制中的安全威胁进行了探讨，该攻击旨在污染所学策略。案例针对强化学习中的表格确定等价学习器和控制中的线性二次调节器进行了实例化，并表明了两个实例均可实现全局最优解。