Reinforcement learning (RL) with continuous state and action spaces remains one of the most challenging problems within the field. Most current learning methods focus on integral identities such as value functions to derive an optimal strategy for the learning agent. In this paper, we instead study the dual form of the original RL formulation to propose the first differential RL framework that can handle settings with limited training samples and short-length episodes. Our approach introduces Differential Policy Optimization (DPO), a pointwise and stage-wise iteration method that optimizes policies encoded by local-movement operators. We prove a pointwise convergence estimate for DPO and provide a regret bound comparable with current theoretical works. Such pointwise estimate ensures that the learned policy matches the optimal path uniformly across different steps. We then apply DPO to a class of practical RL problems which search for optimal configurations with Lagrangian rewards. DPO is easy to implement, scalable, and shows competitive results on benchmarking experiments against several popular RL methods.

提出了第一个可以处理拥有有限训练样本和较短长度回合的差分强化学习框架，命名为差分策略优化（DPO）。DPO是一种点对点和阶段对阶段迭代方法，通过本地运动算子编码的策略进行优化，具有可扩展性，且在基准实验中与几种流行的强化学习方法相比展现出有竞争力的结果。

DPO: 差分强化学习及其在最优配置搜索中的应用