In online randomized experiments or A/B tests, accurate predictions of
participant inclusion rates are of paramount importance. These predictions not
only guide experimenters in optimizing the experiment's duration but also
enhance the precision of treatment effect estimates. In this paper we present a
novel, straightforward, and scalable Bayesian nonparametric approach for
predicting the rate at which individuals will be exposed to interventions
within the realm of online A/B testing. Our approach stands out by offering
dual prediction capabilities: it forecasts both the quantity of new customers
expected in future time windows and, unlike available alternative methods, the
number of times they will be observed. We derive closed-form expressions for
the posterior distributions of the quantities needed to form predictions about
future user activity, thereby bypassing the need for numerical algorithms such
as Markov chain Monte Carlo. After a comprehensive exposition of our model, we
test its performance on experiments on real and simulated data, where we show
its superior performance with respect to existing alternatives in the
literature.

在线随机实验和 A/B 测试中，参与者包含率的准确预测是至关重要的，本文提出了一种新颖、直接且可扩展的贝叶斯非参数方法，用于预测在线 A/B 测试中个体接受干预的比率，并展示了其在实验和模拟数据上相对于现有方法的卓越性能。

在线 A/B 测试中未来用户活动的改进预测

Improved prediction of future user activity in online A/B testing

We propose a nonparametric sequential test that aims to address two practical
problems pertinent to online randomized experiments: (i) how to do a hypothesis
test for complex metrics; (ii) how to prevent type $1$ error inflation under
continuous monitoring. The proposed test does not require knowledge of the
underlying probability distribution generating the data. We use the bootstrap
to estimate the likelihood for blocks of data followed by mixture sequential
probability ratio test. We validate this procedure on data from a major online
e-commerce website. We show that the proposed test controls type $1$ error at
any time, has good power, is robust to misspecification in the distribution
generating the data, and allows quick inference in online randomized
experiments.

提出了一种非参数顺序检验方法，可解决在线随机实验中出现的复杂度量的假设检验以及在持续监控下如何防止类型 I 错误膨胀的问题。该方法无需了解数据生成的概率分布，利用 Bootstrap 估计数据块的似然，然后采用混合序列概率比检验。在来自一家重要的在线电商网站的数据上验证了该过程。结果表明，该检验在任何时候都能控制 I 型错误，在在线随机实验中具有良好的功率，对于数据生成的分布错误具有鲁棒性，并且允许快速推断。