Programmatically Interpretable Reinforcement Learning (PIRL) encodes policies in human-readable computer programs. Novel algorithms were recently introduced with the goal of handling the lack of gradient signal to guide the search in the space of programmatic policies. Most of such PIRL algorithms first train a neural policy that is used as an oracle to guide the search in the programmatic space. In this paper, we show that such PIRL-specific algorithms are not needed, depending on the language used to encode the programmatic policies. This is because one can use actor-critic algorithms to directly obtain a programmatic policy. We use a connection between ReLU neural networks and oblique decision trees to translate the policy learned with actor-critic algorithms into programmatic policies. This translation from ReLU networks allows us to synthesize policies encoded in programs with if-then-else structures, linear transformations of the input values, and PID operations. Empirical results on several control problems show that this translation approach is capable of learning short and effective policies. Moreover, the translated policies are at least competitive and often far superior to the policies PIRL algorithms synthesize.

在这篇论文中，我们展示了使用actor-critic算法将从actor-critic算法学习到的策略转化为以程序形式编码的策略的连接，以此避免了需要使用特定于PIRL的算法的问题。实证结果表明，这种转化方法能够学习出简短而有效的策略，并且这些转化后的策略至少具有与PIRL算法相竞争的水平，往往更优秀。

使用演员-评论算法和ReLU网络合成程序策略