Off-policy evaluation (OPE) aims to estimate the benefit of following a
counterfactual sequence of actions, given data collected from executed
sequences. However, existing OPE estimators often exhibit high bias and high
variance in problems involving large, combinatorial action spaces. We
investigate how to mitigate this issue using factored action spaces i.e.
expressing each action as a combination of independent sub-actions from smaller
action spaces. This approach facilitates a finer-grained analysis of how
actions differ in their effects. In this work, we propose a new family of
"decomposed" importance sampling (IS) estimators based on factored action
spaces. Given certain assumptions on the underlying problem structure, we prove
that the decomposed IS estimators have less variance than their original
non-decomposed versions, while preserving the property of zero bias. Through
simulations, we empirically verify our theoretical results, probing the
validity of various assumptions. Provided with a technique that can derive the
action space factorisation for a given problem, our work shows that OPE can be
improved "for free" by utilising this inherent problem structure.

通过利用分解行动空间的技术，我们提出了一种新的家族式重要性采样估计器，以减少现有 OPE 估计器的偏差和方差问题，同时保持零偏差的特性。

利用分解的动作空间进行脱机策略评估

Leveraging Factored Action Spaces for Off-Policy Evaluation

Many reinforcement learning (RL) applications have combinatorial action
spaces, where each action is a composition of sub-actions. A standard RL
approach ignores this inherent factorization structure, resulting in a
potential failure to make meaningful inferences about rarely observed
sub-action combinations; this is particularly problematic for offline settings,
where data may be limited. In this work, we propose a form of linear Q-function
decomposition induced by factored action spaces. We study the theoretical
properties of our approach, identifying scenarios where it is guaranteed to
lead to zero bias when used to approximate the Q-function. Outside the regimes
with theoretical guarantees, we show that our approach can still be useful
because it leads to better sample efficiency without necessarily sacrificing
policy optimality, allowing us to achieve a better bias-variance trade-off.
Across several offline RL problems using simulators and real-world datasets
motivated by healthcare, we demonstrate that incorporating factored action
spaces into value-based RL can result in better-performing policies. Our
approach can help an agent make more accurate inferences within underexplored
regions of the state-action space when applying RL to observational datasets.

本文研究了如何在强化学习的组合行为空间中通过线性 Q 函数分解来更好地处理少见子行动组合的情况，并对该方法进行了理论分析和实验评估，证明了它可以提高数据效率和策略优化的性能。