We propose a nonparametric additive model for estimating interpretable value
functions in reinforcement learning. Learning effective adaptive clinical
interventions that rely on digital phenotyping features is a major for concern
medical practitioners. With respect to spine surgery, different post-operative
recovery recommendations concerning patient mobilization can lead to
significant variation in patient recovery. While reinforcement learning has
achieved widespread success in domains such as games, recent methods heavily
rely on black-box methods, such neural networks. Unfortunately, these methods
hinder the ability of examining the contribution each feature makes in
producing the final suggested decision. While such interpretations are easily
provided in classical algorithms such as Least Squares Policy Iteration, basic
linearity assumptions prevent learning higher-order flexible interactions
between features. In this paper, we present a novel method that offers a
flexible technique for estimating action-value functions without making
explicit parametric assumptions regarding their additive functional form. This
nonparametric estimation strategy relies on incorporating local kernel
regression and basis expansion to obtain a sparse, additive representation of
the action-value function. Under this approach, we are able to locally
approximate the action-value function and retrieve the nonlinear, independent
contribution of select features as well as joint feature pairs. We validate the
proposed approach with a simulation study, and, in an application to spine
disease, uncover recovery recommendations that are inline with related clinical
knowledge.

我们提出了一种非参数的加性模型，用于估计强化学习中可解释的价值函数。该方法通过使用局部核回归和基扩展来获取动作价值函数的稀疏、加性表示，从而实现局部逼近和提取特征的非线性、独立贡献以及特征对的共同贡献。我们通过模拟研究验证了该方法，并在脊椎疾病的应用中揭示了与相关临床知识一致的康复建议。