In many randomized experiments, the treatment effect of the long-term metric
(i.e. the primary outcome of interest) is often difficult or infeasible to
measure. Such long-term metrics are often slow to react to changes and
sufficiently noisy they are challenging to faithfully estimate in short-horizon
experiments. A common alternative is to measure several short-term proxy
metrics in the hope they closely track the long-term metric -- so they can be
used to effectively guide decision-making in the near-term. We introduce a new
statistical framework to both define and construct an optimal proxy metric for
use in a homogeneous population of randomized experiments. Our procedure first
reduces the construction of an optimal proxy metric in a given experiment to a
portfolio optimization problem which depends on the true latent treatment
effects and noise level of experiment under consideration. We then denoise the
observed treatment effects of the long-term metric and a set of proxies in a
historical corpus of randomized experiments to extract estimates of the latent
treatment effects for use in the optimization problem. One key insight derived
from our approach is that the optimal proxy metric for a given experiment is
not apriori fixed; rather it should depend on the sample size (or effective
noise level) of the randomized experiment for which it is deployed. To
instantiate and evaluate our framework, we employ our methodology in a large
corpus of randomized experiments from an industrial recommendation system and
construct proxy metrics that perform favorably relative to several baselines.

在大量随机实验中，通过生成决策指南，我们引入了一种新的统计框架来定义并构建适用于同质人群的最佳代理度量标准，该度量标准能够与长期度量密切跟踪，并在噪声较高的情况下进行优化问题的解决。通过运用这种方法，我们在一个工业推荐系统的大量随机实验中应用我们的方法进行实例化和评估，并构建了相对于几个基准值表现优异的代理度量标准。

选择过去实验中的代理指标

Choosing a Proxy Metric from Past Experiments

North star metrics and online experimentation play a central role in how
technology companies improve their products. In many practical settings,
however, evaluating experiments based on the north star metric directly can be
difficult. The two most significant issues are 1) low sensitivity of the north
star metric and 2) differences between the short-term and long-term impact on
the north star metric. A common solution is to rely on proxy metrics rather
than the north star in experiment evaluation and launch decisions. Existing
literature on proxy metrics concentrates mainly on the estimation of the
long-term impact from short-term experimental data. In this paper, instead, we
focus on the trade-off between the estimation of the long-term impact and the
sensitivity in the short term. In particular, we propose the Pareto optimal
proxy metrics method, which simultaneously optimizes prediction accuracy and
sensitivity. In addition, we give an efficient multi-objective optimization
algorithm that outperforms standard methods. We applied our methodology to
experiments from a large industrial recommendation system, and found proxy
metrics that are eight times more sensitive than the north star and
consistently moved in the same direction, increasing the velocity and the
quality of the decisions to launch new features.

论文提出 Pareto 最优代理指标方法，该方法同时优化了预测精度和灵敏度，并给出了一种有效的多目标优化算法用于实验评估和决策，大幅提高了工业推荐系统启动新功能的决策速度和质量。

帕累托最优代理指标

Pareto optimal proxy metrics

Recent work has addressed textual reasoning tasks by prompting large language
models with explanations via the chain-of-thought paradigm. However, subtly
different explanations can yield widely varying downstream task accuracy, so
explanations that have not been "tuned" for a task, such as off-the-shelf
explanations written by nonexperts, may lead to mediocre performance. This
paper tackles the problem of how to optimize explanation-infused prompts in a
black-box fashion. We first generate sets of candidate explanations for each
example in the prompt using a leave-one-out scheme. We then use a two-stage
framework where we first evaluate explanations for each in-context example in
isolation according to proxy metrics. Finally, we search over sets of
explanations to find a set which yields high performance against a
silver-labeled development set, drawing inspiration from recent work on
bootstrapping language models on unlabeled data. Across four textual reasoning
tasks spanning question answering, mathematical reasoning, and natural language
inference, results show that our proxy metrics correlate with ground truth
accuracy and our overall method can effectively improve prompts over
crowdworker annotations and naive search strategies.

本文提出了一种黑盒优化解释注入的提示方法，以提高文本推理任务的性能，在问题回答、数学推理和自然语言推理等四个文本推理任务中实现了有效的促进效果。