The selection of the assumed effect size (AES) critically determines the
duration of an experiment, and hence its accuracy and efficiency.
Traditionally, experimenters determine AES based on domain knowledge. However,
this method becomes impractical for online experimentation services managing
numerous experiments, and a more automated approach is hence of great demand.
We initiate the study of data-driven AES selection in for online
experimentation services by introducing two solutions. The first employs a
three-layer Gaussian Mixture Model considering the heteroskedasticity across
experiments, and it seeks to estimate the true expected effect size among
positive experiments. The second method, grounded in utility theory, aims to
determine the optimal effect size by striking a balance between the
experiment's cost and the precision of decision-making. Through comparisons
with baseline methods using both simulated and real data, we showcase the
superior performance of the proposed approaches.

基于数据驱动的在线实验，提出了两种方法：使用考虑实验差异性的三层高斯混合模型来估计期望效应大小，以及基于效用理论来确定最佳效应大小，通过与基准方法的比较，表明了这些方法的卓越性能。