We establish a connection between the importance sampling estimators typically used for off-policy policy evaluation in reinforcement learning and the extended conditional Monte Carlo method. We show with some examples that in the finite horizon case there is no strict ordering in general between the variance of such conditional importance sampling estimators: the variance of the per-decision or stationary variants may, in fact, be higher than that of the crude importance sampling estimator. We also provide sufficient conditions for the finite horizon case under which the per-decision or stationary estimators can reduce the variance. We then develop an asymptotic analysis and derive sufficient conditions under which there exists an exponential v.s. polynomial gap (in terms of horizon $T$) between the variance of importance sampling and that of the per-decision or stationary estimators.

分析了新型IS方法中最流行的方法，通过有限MDP的条件蒙特卡罗视角发现，在有限MDP中，每决策的IS或固定采样IS的严格方差减少与普通IS相比没有优势。然后，提供了充分的条件，即决策或稳态估计将在有限的时间内在方差上超越IS。针对渐近时间，提出了上下界并得出充分的条件，允许普通IS和每个决策或稳态估计之间存在指数v.s.多项式差异。通过这些结果，有助于推进我们对新型IS估计器何时提高离线策略评估准确性的理解。

通过条件重要性抽样理解离线策略评估中视野理论的作用