This paper addresses a key limitation in existing counterfactual inference methods for Markov Decision Processes (MDPs). Current approaches assume a specific causal model to make counterfactuals identifiable. However, there are usually many causal models that align with the observational and interventional distributions of an MDP, each yielding different counterfactual distributions, so fixing a particular causal model limits the validity (and usefulness) of counterfactual inference. We propose a novel non-parametric approach that computes tight bounds on counterfactual transition probabilities across all compatible causal models. Unlike previous methods that require solving prohibitively large optimisation problems (with variables that grow exponentially in the size of the MDP), our approach provides closed-form expressions for these bounds, making computation highly efficient and scalable for non-trivial MDPs. Once such an interval counterfactual MDP is constructed, our method identifies robust counterfactual policies that optimise the worst-case reward w.r.t. the uncertain interval MDP probabilities. We evaluate our method on various case studies, demonstrating improved robustness over existing methods.

本文解决了现有马尔可夫决策过程（MDP）反事实推断方法中的关键局限，现有方法依赖特定的因果模型，而此模型限制了反事实推断的有效性。我们提出了一种新的非参数方法，通过计算所有兼容因果模型的反事实转移概率的紧界，为反事实推断提供了高效和可扩展的解决方案。研究结果表明，我们的方法在各种案例研究中展现了比现有方法更强的稳健性。

马尔可夫决策过程中的稳健反事实推断