Model-based offline reinforcement learning methods (RL) have achieved
state-of-the-art performance in many decision-making problems thanks to their
sample efficiency and generalizability. Despite these advancements, existing
model-based offline RL approaches either focus on theoretical studies without
developing practical algorithms or rely on a restricted parametric policy
space, thus not fully leveraging the advantages of an unrestricted policy space
inherent to model-based methods. To address this limitation, we develop MoMA, a
model-based mirror ascent algorithm with general function approximations under
partial coverage of offline data. MoMA distinguishes itself from existing
literature by employing an unrestricted policy class. In each iteration, MoMA
conservatively estimates the value function by a minimization procedure within
a confidence set of transition models in the policy evaluation step, then
updates the policy with general function approximations instead of
commonly-used parametric policy classes in the policy improvement step. Under
some mild assumptions, we establish theoretical guarantees of MoMA by proving
an upper bound on the suboptimality of the returned policy. We also provide a
practically implementable, approximate version of the algorithm. The
effectiveness of MoMA is demonstrated via numerical studies.

利用模型为基础的镜像爬升算法 (MoMA) 在部分离线数据覆盖下，采用通用函数逼近能力，通过在策略评估步骤中的转移模型置信区间内的最小化过程来保守估计值函数，然后在策略改进步骤中使用通用函数逼近而不是常用的参数策略类，从而充分利用模型为基础方法中固有的无限制策略空间，并且通过对返回策略的次优性建立理论保证。