Calculating optimal policies is known to be computationally difficult for
Markov decision processes (MDPs) with Borel state and action spaces. This paper
studies finite-state approximations of discrete time Markov decision processes
with Borel state and action spaces, for both discounted and average costs
criteria. The stationary policies thus obtained are shown to approximate the
optimal stationary policy with arbitrary precision under quite general
conditions for discounted cost and more restrictive conditions for average
cost. For compact-state MDPs, we obtain explicit rate of convergence bounds
quantifying how the approximation improves as the size of the approximating
finite state space increases. Using information theoretic arguments, the order
optimality of the obtained convergence rates is established for a large class
of problems. We also show that, as a pre-processing step the action space can
also be finitely approximated with sufficiently large number points; thereby,
well known algorithms, such as value or policy iteration, Q-learning, etc., can
be used to calculate near optimal policies.

本文针对局限于有限状态下的马尔可夫决策过程，对于包括折扣和平均成本标准在内的情况进行了研究，获得了近似最优策略，使用预处理步骤将操作空间有限近似，可以使用众所周知的算法计算近似最优政策。

带 Borel 空间的马尔科夫决策过程的有限逼近的渐近最优性

Asymptotic Optimality of Finite Approximations to Markov Decision  Processes with Borel Spaces

We consider the problem of controlling a Markov decision process (MDP) with a
large state space, so as to minimize average cost. Since it is intractable to
compete with the optimal policy for large scale problems, we pursue the more
modest goal of competing with a low-dimensional family of policies. We use the
dual linear programming formulation of the MDP average cost problem, in which
the variable is a stationary distribution over state-action pairs, and we
consider a neighborhood of a low-dimensional subset of the set of stationary
distributions (defined in terms of state-action features) as the comparison
class. We propose two techniques, one based on stochastic convex optimization,
and one based on constraint sampling. In both cases, we give bounds that show
that the performance of our algorithms approaches the best achievable by any
policy in the comparison class. Most importantly, these results depend on the
size of the comparison class, but not on the size of the state space.
Preliminary experiments show the effectiveness of the proposed algorithms in a
queuing application.

本文考虑了控制具有大状态空间的马尔可夫决策过程以最小化平均成本的问题，并使用线性规划和两种方法，即基于随机凸优化和基于约束采样的方法，将性能提高到与在低维策略类中的任何策略相比的最佳水平。