In the world of stochastic control, especially in economics and engineering,
Markov Decision Processes (MDPs) can effectively model various stochastic
decision processes, from asset management to transportation optimization. These
underlying MDPs, upon closer examination, often reveal a specifically
constrained causal structure concerning the transition and reward dynamics. By
exploiting this structure, we can obtain a reduction in the causal
representation of the problem setting, allowing us to solve of the optimal
value function more efficiently. This work defines an MDP framework, the
\texttt{SD-MDP}, where we disentangle the causal structure of MDPs' transition
and reward dynamics, providing distinct partitions on the temporal causal
graph. With this stochastic reduction, the \texttt{SD-MDP} reflects a general
class of resource allocation problems. This disentanglement further enables us
to derive theoretical guarantees on the estimation error of the value function
under an optimal policy by allowing independent value estimation from Monte
Carlo sampling. Subsequently, by integrating this estimator into well-known
Monte Carlo planning algorithms, such as Monte Carlo Tree Search (MCTS), we
derive bounds on the simple regret of the algorithm. Finally, we quantify the
policy improvement of MCTS under the \texttt{SD-MDP} framework by demonstrating
that the MCTS planning algorithm achieves higher expected reward (lower costs)
under a constant simulation budget, on a tangible economic example based on
maritime refuelling.

在随机控制的领域中，尤其是在经济学和工程学中，马尔可夫决策过程（MDPs）能够有效地建模各种随机决策过程，从资产管理到运输优化。本文定义了一个 MDP 框架，SD-MDP，通过解开 MDPs 的转移和奖励动态的因果结构，提供了时间因果图上的不同分区。通过将这个估计器集成到著名的蒙特卡洛规划算法中，如蒙特卡洛树搜索（MCTS），我们还得出了算法的简单遗憾界限。最后，我们通过在基于海上加油的实际经济示例中展示 MCTS 规划算法在 SD-MDP 框架下取得更高预期奖励（更低成本）的政策改进。