Posterior sampling allows the exploitation of prior knowledge of the environment's transition dynamics to improve the sample efficiency of reinforcement learning. The prior is typically specified as a class of parametric distributions, a task that can be cumbersome in practice, often resulting in the choice of uninformative priors. In this work, we propose a novel posterior sampling approach in which the prior is given as a (partial) causal graph over the environment's variables. The latter is often more natural to design, such as listing known causal dependencies between biometric features in a medical treatment study. Specifically, we propose a hierarchical Bayesian procedure, called C-PSRL, simultaneously learning the full causal graph at the higher level and the parameters of the resulting factored dynamics at the lower level. For this procedure, we provide an analysis of its Bayesian regret, which explicitly connects the regret rate with the degree of prior knowledge. Our numerical evaluation conducted in illustrative domains confirms that C-PSRL strongly improves the efficiency of posterior sampling with an uninformative prior while performing close to posterior sampling with the full causal graph.

我们提出了一种新的后验抽样方法，其中先验以环境变量之间的（部分）因果图形式给出，该方法称为C-PSRL，在同时学习更高层次的完整因果图和更低层次的分解动力学参数时，明确了其贝叶斯遗失与先验知识的程度之间的关系。我们在说明性领域进行的数值评估证实了C-PSRL在先验信息不足的情况下极大提高了后验抽样的效率，并且性能接近使用完整因果图进行后验抽样。

利用后验抽样开发因果图先验的强化学习