Gradient-based methods for value estimation in reinforcement learning have
favorable stability properties, but they are typically much slower than
Temporal Difference (TD) learning methods. We study the root causes of this
slowness and show that Mean Square Bellman Error (MSBE) is an ill-conditioned
loss function in the sense that its Hessian has large condition-number. To
resolve the adverse effect of poor conditioning of MSBE on gradient based
methods, we propose a low complexity batch-free proximal method that
approximately follows the Gauss-Newton direction and is asymptotically robust
to parameterization. Our main algorithm, called RANS, is efficient in the sense
that it is significantly faster than the residual gradient methods while having
almost the same computational complexity, and is competitive with TD on the
classic problems that we tested.

本文研究了值估计的梯度方法在强化学习中速度较慢的原因，提出了一种基于 Gauss-Newton 方向的低复杂度非批量化近端方法，并介绍了主要算法 RANS 及其在经典问题中的表现。

基于梯度的价值估计的高效实现

Toward Efficient Gradient-Based Value Estimation

We propose a novel sparse constrained formulation and from it derive a
real-time optimization method for 3D human pose and shape estimation. Our
optimization method, SCOPE (Sparse Constrained Optimization for 3D human Pose
and shapE estimation), is orders of magnitude faster (avg. 4 ms convergence)
than existing optimization methods, while being mathematically equivalent to
their dense unconstrained formulation under mild assumptions. We achieve this
by exploiting the underlying sparsity and constraints of our formulation to
efficiently compute the Gauss-Newton direction. We show that this computation
scales linearly with the number of joints and measurements of a complex 3D
human model, in contrast to prior work where it scales cubically due to their
dense unconstrained formulation. Based on our optimization method, we present a
real-time motion capture framework that estimates 3D human poses and shapes
from a single image at over 30 FPS. In benchmarks against state-of-the-art
methods on multiple public datasets, our framework outperforms other
optimization methods and achieves competitive accuracy against regression
methods. Project page with code and videos:
this https URL

该研究提出了一种新颖的基于稀疏约束的 3D 人体姿态和形状估计方法，通过高效计算 Gauss-Newton 方向并利用优化问题的稀疏约束性质，将优化时间缩短到 4ms 左右，并成功地将其应用于单张图像的实时运动捕捉领域。