In this paper, we present a new policy gradient (PG) methods, namely the
block policy mirror descent (BPMD) method for solving a class of regularized
reinforcement learning (RL) problems with (strongly)-convex regularizers.
Compared to the traditional PG methods with a batch update rule, which visits
and updates the policy for every state, BPMD method has cheap per-iteration
computation via a partial update rule that performs the policy update on a
sampled state. Despite the nonconvex nature of the problem and a partial update
rule, we provide a unified analysis for several sampling schemes, and show that
BPMD achieves fast linear convergence to the global optimality. In particular,
uniform sampling leads to comparable worst-case total computational complexity
as batch PG methods. A necessary and sufficient condition for convergence with
on-policy sampling is also identified. With a hybrid sampling scheme, we
further show that BPMD enjoys potential instance-dependent acceleration,
leading to improved dependence on the state space and consequently
outperforming batch PG methods. We then extend BPMD methods to the stochastic
setting, by utilizing stochastic first-order information constructed from
samples. With a generative model, $\tilde{\mathcal{O}}(\left\lvert
\mathcal{S}\right\rvert \left\lvert \mathcal{A}\right\rvert /\epsilon)$ (resp.
$\tilde{\mathcal{O}}(\left\lvert \mathcal{S}\right\rvert \left\lvert
\mathcal{A} \right\rvert /\epsilon^2)$) sample complexities are established for
the strongly-convex (resp. non-strongly-convex) regularizers, where $\epsilon$
denotes the target accuracy. To the best of our knowledge, this is the first
time that block coordinate descent methods have been developed and analyzed for
policy optimization in reinforcement learning, which provides a new perspective
on solving large-scale RL problems.

本文提出了一种新的策略梯度方法，即基于块的策略镜像下降（BPMD）方法，用于解决一类带有（强）凸正则化器的强化学习（RL）问题，通过部分更新规则执行已采样状态上的策略更新，从而实现了每次迭代计算代价的降低，并且在分析多种采样方案时达到快速的线性收敛。