Ever since the concepts of dynamic programming were introduced, one of the
most difficult challenges has been to adequately address high-dimensional
control problems. With growing dimensionality, the utilisation of Deep Neural
Networks promises to circumvent the issue of an otherwise exponentially
increasing complexity. The paper specifically investigates the sampling issues
the Deep Galerkin Method is subjected to. It proposes a drift relaxation-based
sampling approach to alleviate the symptoms of high-variance policy
approximations. This is validated on mean-field control problems; namely, the
variations of the opinion dynamics presented by the Sznajd and the
Hegselmann-Krause model. The resulting policies induce a significant cost
reduction over manually optimised control functions and show improvements on
the Linear-Quadratic Regulator problem over the Deep FBSDE approach.

通过采用基于漂移放松的采样方法，本文研究了 Deep Galerkin 方法所面临的采样问题，通过验证 Sznajd 和 Hegselmann-Krause 模型中的意见动态变化的多场控制问题，得出的策略在手动优化控制函数上实现了显著成本降低，并在 Deep FBSDE 方法上改进了线性 - 二次调节器问题。

基于深度 Galerkin 反馈法的智能体动力学最优控制

Optimal Control of Agent-Based Dynamics under Deep Galerkin Feedback  Laws

Policy gradient methods have enjoyed great success in deep reinforcement
learning but suffer from high variance of gradient estimates. The high variance
problem is particularly exasperated in problems with long horizons or
high-dimensional action spaces. To mitigate this issue, we derive a bias-free
action-dependent baseline for variance reduction which fully exploits the
structural form of the stochastic policy itself and does not make any
additional assumptions about the MDP. We demonstrate and quantify the benefit
of the action-dependent baseline through both theoretical analysis as well as
numerical results, including an analysis of the suboptimality of the optimal
state-dependent baseline. The result is a computationally efficient policy
gradient algorithm, which scales to high-dimensional control problems, as
demonstrated by a synthetic 2000-dimensional target matching task. Our
experimental results indicate that action-dependent baselines allow for faster
learning on standard reinforcement learning benchmarks and high-dimensional
hand manipulation and synthetic tasks. Finally, we show that the general idea
of including additional information in baselines for improved variance
reduction can be extended to partially observed and multi-agent tasks.

本研究提出了一种无偏差的基于动作的基线方法，该方法可以减少深度增强学习中梯度估计的高方差问题，最终实现高维控制问题的策略梯度算法。此外，还证明了该方法的有效性并扩展到部分观察和多智能体任务。