Infinite-state Markov Decision Processes (MDPs) are essential in modeling and
optimizing a wide variety of engineering problems. In the reinforcement
learning (RL) context, a variety of algorithms have been developed to learn and
optimize these MDPs. At the heart of many popular policy-gradient based
learning algorithms, such as natural actor-critic, TRPO, and PPO, lies the
Natural Policy Gradient (NPG) algorithm. Convergence results for these RL
algorithms rest on convergence results for the NPG algorithm. However, all
existing results on the convergence of the NPG algorithm are limited to
finite-state settings.
We prove the first convergence rate bound for the NPG algorithm for
infinite-state average-reward MDPs, proving a $O(1/\sqrt{T})$ convergence rate,
if the NPG algorithm is initialized with a good initial policy. Moreover, we
show that in the context of a large class of queueing MDPs, the MaxWeight
policy suffices to satisfy our initial-policy requirement and achieve a
$O(1/\sqrt{T})$ convergence rate. Key to our result are state-dependent bounds
on the relative value function achieved by the iterate policies of the NPG
algorithm.

该研究证明了自然策略梯度算法在无限状态的平均奖励马尔可夫决策过程中的收敛速度，如果采用良好的初始策略进行初始化，则收敛速度为 O (1/√T)。此外，针对大类排队马尔可夫决策过程，最大权重策略足以满足我们的初始策略要求并实现 O (1/√T) 的收敛速度。关键是根据 NPG 算法的迭代策略所达到的相对值函数，我们得出了这一结果。

自然策略梯度在无限状态平均奖励马尔可夫决策过程上的收敛性

Convergence for Natural Policy Gradient on Infinite-State Average-Reward  Markov Decision Processes

We develop several new algorithms for learning Markov Decision Processes in
an infinite-horizon average-reward setting with linear function approximation.
Using the optimism principle and assuming that the MDP has a linear structure,
we first propose a computationally inefficient algorithm with optimal
$\widetilde{O}(\sqrt{T})$ regret and another computationally efficient variant
with $\widetilde{O}(T^{3/4})$ regret, where $T$ is the number of interactions.
Next, taking inspiration from adversarial linear bandits, we develop yet
another efficient algorithm with $\widetilde{O}(\sqrt{T})$ regret under a
different set of assumptions, improving the best existing result by Hao et al.
(2020) with $\widetilde{O}(T^{2/3})$ regret. Moreover, we draw a connection
between this algorithm and the Natural Policy Gradient algorithm proposed by
Kakade (2002), and show that our analysis improves the sample complexity bound
recently given by Agarwal et al. (2020).

开发多种学习用于 Markov Decision Processes 的无限时间平均奖励设置和线性函数逼近的算法，使用乐观原则和假设 MDP 具有线性结构，提出具有优化的计算效率的算法，并展开了详细的分析，改进了现有最佳结果。

使用线性函数逼近学习无限时间平均回报马尔可夫决策过程

Learning Infinite-horizon Average-reward MDPs with Linear Function  Approximation

Reinforcement learning has emerged as a promising methodology for training
robot controllers. However, most results have been limited to simulation due to
the need for a large number of samples and the lack of automated-yet-safe data
collection methods. Model-based reinforcement learning methods provide an
avenue to circumvent these challenges, but the traditional concern has been the
mismatch between the simulator and the real world. Here, we show that control
policies learned in simulation can successfully transfer to a physical system,
composed of three Phantom robots pushing an object to various desired target
positions. We use a modified form of the natural policy gradient algorithm for
learning, applied to a carefully identified simulation model. The resulting
policies, trained entirely in simulation, work well on the physical system
without additional training. In addition, we show that training with an
ensemble of models makes the learned policies more robust to modeling errors,
thus compensating for difficulties in system identification.

本研究使用一种修改自然策略梯度算法的模型，通过模拟学习并训练，成功将自主控制策略从虚拟系统转移到由三个机器人组成的物理系统中，并证明使用多个模型训练可以使学习到的策略更加稳健，从而弥补了系统识别的困难。