We develop a parameterized Primal-Dual $\pi$ Learning method based on deep
neural networks for Markov decision process with large state space and
off-policy reinforcement learning. In contrast to the popular Q-learning and
actor-critic methods that are based on successive approximations to the
nonlinear Bellman equation, our method makes primal-dual updates to the policy
and value functions utilizing the fundamental linear Bellman duality. Naive
parametrization of the primal-dual $\pi$ learning method using deep neural
networks would encounter two major challenges: (1) each update requires
computing a probability distribution over the state space and is intractable;
(2) the iterates are unstable since the parameterized Lagrangian function is no
longer linear. We address these challenges by proposing a relaxed Lagrangian
formulation with a regularization penalty using the advantage function. We show
that the dual policy update step in our method is equivalent to the policy
gradient update in the actor-critic method in some special case, while the
value updates differ substantially. The main advantage of the primal-dual $\pi$
learning method lies in that the value and policy updates are closely coupled
together using the Bellman duality and therefore more informative. Experiments
on a simple cart-pole problem show that the algorithm significantly outperforms
the one-step temporal-difference actor-critic method, which is the most
relevant benchmark method to compare with. We believe that the primal-dual
updates to the value and policy functions would expedite the learning process.
The proposed methods might open a door to more efficient algorithms and sharper
theoretical analysis.

基于深度神经网络的参数 Primal-Dual pi 学习方法，旨在解决马尔可夫决策过程中状态空间大和策略离线学习问题，通过基本线性 Bellman 方法对价值和策略函数进行原始对偶更新，从而更加有效地进行价值和策略更新，在与同类方法比较的测试中表现明显优于最相关的基准方法