In many real-world applications of reinforcement learning (RL), interactions
with the environment are limited due to cost or feasibility. This presents a
challenge to traditional RL algorithms since the max-return objective involves
an expectation over on-policy samples. We introduce a new formulation of
max-return optimization that allows the problem to be re-expressed by an
expectation over an arbitrary behavior-agnostic and off-policy data
distribution. We first derive this result by considering a regularized version
of the dual max-return objective before extending our findings to unregularized
objectives through the use of a Lagrangian formulation of the linear
programming characterization of Q-values. We show that, if auxiliary dual
variables of the objective are optimized, then the gradient of the off-policy
objective is exactly the on-policy policy gradient, without any use of
importance weighting. In addition to revealing the appealing theoretical
properties of this approach, we also show that it delivers good practical
performance.

本文提出一种新颖的强化学习算法，利用双重变量不等式技术，通过辅助最优化问题得到二阶鞍点，无需使用重要性加权，实现了从任意行为策略上学习并优化目标策略。

AlgaeDICE: 来自任意经验的策略梯度

AlgaeDICE: Policy Gradient from Arbitrary Experience

We propose a new algorithm for minimizing regularized empirical loss:
Stochastic Dual Newton Ascent (SDNA). Our method is dual in nature: in each
iteration we update a random subset of the dual variables. However, unlike
existing methods such as stochastic dual coordinate ascent, SDNA is capable of
utilizing all curvature information contained in the examples, which leads to
striking improvements in both theory and practice - sometimes by orders of
magnitude. In the special case when an L2-regularizer is used in the primal,
the dual problem is a concave quadratic maximization problem plus a separable
term. In this regime, SDNA in each step solves a proximal subproblem involving
a random principal submatrix of the Hessian of the quadratic function; whence
the name of the method. If, in addition, the loss functions are quadratic, our
method can be interpreted as a novel variant of the recently introduced
Iterative Hessian Sketch.

通过 Stochastic Dual Newton Ascent 算法，我们提出一种新的途径最小化正则化经验损失，该方法更新了随机子集的对偶变量，可以利用模型中所有曲率信息，实践中有着明显的提高，特别对于二次损失函数。