Learning efficiently a causal model of the environment is a key challenge of
model-based RL agents operating in POMDPs. We consider here a scenario where
the learning agent has the ability to collect online experiences through direct
interactions with the environment (interventional data), but has also access to
a large collection of offline experiences, obtained by observing another agent
interacting with the environment (observational data). A key ingredient, that
makes this situation non-trivial, is that we allow the observed agent to
interact with the environment based on hidden information, which is not
observed by the learning agent. We then ask the following questions: can the
online and offline experiences be safely combined for learning a causal model ?
And can we expect the offline experiences to improve the agent's performances ?
To answer these questions, we import ideas from the well-established causal
framework of do-calculus, and we express model-based reinforcement learning as
a causal inference problem. Then, we propose a general yet simple methodology
for leveraging offline data during learning. In a nutshell, the method relies
on learning a latent-based causal transition model that explains both the
interventional and observational regimes, and then using the recovered latent
variable to infer the standard POMDP transition model via deconfounding. We
prove our method is correct and efficient in the sense that it attains better
generalization guarantees due to the offline data (in the asymptotic case), and
we illustrate its effectiveness empirically on synthetic toy problems. Our
contribution aims at bridging the gap between the fields of reinforcement
learning and causality.

本文探讨了利用 Causal Inference 理论和 Latent-based causal transition model 在 offline data 的基础上，使用 observational data 可以有效地提高 model-based RL agents 的 generalization guarantees 问题。