Because high-quality data is like oxygen for AI systems, effectively
eliciting information from crowdsourcing workers has become a first-order
problem for developing high-performance machine learning algorithms. Two
prevalent paradigms, spot-checking and peer prediction, enable the design of
mechanisms to evaluate and incentivize high-quality data from human labelers.
So far, at least three metrics have been proposed to compare the performances
of these techniques [33, 8, 3]. However, different metrics lead to divergent
and even contradictory results in various contexts. In this paper, we harmonize
these divergent stories, showing that two of these metrics are actually the
same within certain contexts and explain the divergence of the third. Moreover,
we unify these different contexts by introducing \textit{Spot Check
Equivalence}, which offers an interpretable metric for the effectiveness of a
peer prediction mechanism. Finally, we present two approaches to compute spot
check equivalence in various contexts, where simulation results verify the
effectiveness of our proposed metric.

通过协调不同的度量标准，本论文将展示两种度量标准事实上在某些情况下相同，并解释第三种度量标准的差异。此外，引入 “Spot Check Equivalence” 以统一这些不同的上下文，并提出了计算其效果的两种方法。模拟结果验证了我们提出度量标准的有效性。

快速检查等效性：信息引导机制的可解释度度量

Spot Check Equivalence: an Interpretable Metric for Information  Elicitation Mechanisms

Peer prediction mechanisms are often adopted to elicit truthful contributions
from crowd workers when no ground-truth verification is available. Recently,
mechanisms of this type have been developed to incentivize effort exertion, in
addition to truthful elicitation. In this paper, we study a sequential peer
prediction problem where a data requester wants to dynamically determine the
reward level to optimize the trade-off between the quality of information
elicited from workers and the total expected payment. In this problem, workers
have homogeneous expertise and heterogeneous cost for exerting effort, both
unknown to the requester. We propose a sequential posted-price mechanism to
dynamically learn the optimal reward level from workers' contributions and to
incentivize effort exertion and truthful reporting. We show that (1) in our
mechanism, workers exerting effort according to a non-degenerate threshold
policy and then reporting truthfully is an equilibrium that returns highest
utility for each worker, and (2) The regret of our learning mechanism w.r.t.
offering the optimal reward (price) is upper bounded by $\tilde{O}(T^{3/4})$
where $T$ is the learning horizon. We further show the power of our learning
approach when the reports of workers do not necessarily follow the
game-theoretic equilibrium.

研究了顺序同行预测问题，提出了顺序张贴价格机制，通过动态学习最优奖励水平来激励工人付出努力和真实报告信息，并探讨了参数调优与博弈理论平衡之间的权衡。

序列式同侪预测：利用公开定价学习诱发努力

Sequential Peer Prediction: Learning to Elicit Effort using Posted  Prices

The problem of peer prediction is to elicit information from agents in
settings without any objective ground truth against which to score reports.
Peer prediction mechanisms seek to exploit correlations between signals to
align incentives with truthful reports. A long-standing concern has been the
possibility of uninformative equilibria. For binary signals, a multi-task
mechanism [Dasgupta-Ghosh '13] achieves strong truthfulness, so that the
truthful equilibrium strictly maximizes payoff. We characterize conditions on
the signal distribution for which this mechanism remains strongly-truthful with
non-binary signals, also providing a greatly simplified proof. We introduce the
Correlated Agreement (CA) mechanism, which handles multiple signals and
provides informed truthfulness: no strategy profile provides more payoff in
equilibrium than truthful reporting, and the truthful equilibrium is strictly
better than any uninformed strategy (where an agent avoids the effort of
obtaining a signal). The CA mechanism is maximally strongly truthful, in that
no mechanism in a broad class of mechanisms is strongly truthful on a larger
family of signal distributions. We also give a detail-free version of the
mechanism that removes any knowledge requirements on the part of the designer,
using reports on many tasks to learn statistics while retaining
epsilon-informed truthfulness.

研究内容为解决缺乏客观基本事实的情况下从多个代理人中收集信息的问题，介绍了一种有效的多任务机制和相关协议机制，并在不同信号分布条件下证明了其在保持强真实性方面的可行性。