This paper offers a detailed investigation of switchback designs in A/B
testing, which alternate between baseline and new policies over time. Our aim
is to thoroughly evaluate the effects of these designs on the accuracy of their
resulting average treatment effect (ATE) estimators. We propose a novel "weak
signal analysis" framework, which substantially simplifies the calculations of
the mean squared errors (MSEs) of these ATEs in Markov decision process
environments. Our findings suggest that (i) when the majority of reward errors
are positively correlated, the switchback design is more efficient than the
alternating-day design which switches policies in a daily basis. Additionally,
increasing the frequency of policy switches tends to reduce the MSE of the ATE
estimator. (ii) When the errors are uncorrelated, however, all these designs
become asymptotically equivalent. (iii) In cases where the majority of errors
are negative correlated, the alternating-day design becomes the optimal choice.
These insights are crucial, offering guidelines for practitioners on designing
experiments in A/B testing. Our analysis accommodates a variety of policy value
estimators, including model-based estimators, least squares temporal difference
learning estimators, and double reinforcement learning estimators, thereby
offering a comprehensive understanding of optimal design strategies for policy
evaluation in reinforcement learning.

该研究详细调查了 A/B 测试中的切换设计，通过在基准和新策略之间进行时间交替以评估其对结果平均处理效应（ATE）估计器准确性的影响。研究表明，当奖励误差大多正相关时，切换设计比每日轮换策略的交替日设计更有效；增加策略切换频率往往会降低 ATE 估计器的均方误差（MSE）；然而，当误差不相关时，所有设计都变得渐近等价；而当大多数误差为负相关时，交替日设计成为最优选择。这些洞见为 A/B 测试中实验设计提供重要的指导，同时为强化学习中政策评估的最优设计策略提供了全面的理解。