We present a prototype hybrid prediction market and demonstrate the avenue it
represents for meaningful human-AI collaboration. We build on prior work
proposing artificial prediction markets as a novel machine-learning algorithm.
In an artificial prediction market, trained AI agents buy and sell outcomes of
future events. Classification decisions can be framed as outcomes of future
events, and accordingly, the price of an asset corresponding to a given
classification outcome can be taken as a proxy for the confidence of the system
in that decision. By embedding human participants in these markets alongside
bot traders, we can bring together insights from both. In this paper, we detail
pilot studies with prototype hybrid markets for the prediction of replication
study outcomes. We highlight challenges and opportunities, share insights from
semi-structured interviews with hybrid market participants, and outline a
vision for ongoing and future work.

本文介绍了一个混合预测市场原型，并展示了它为有意义的人工智能协作提供的途径。我们以前提出的人工预测市场作为新的机器学习算法进行建设。通过将人类参与者与机器人交易者嵌入到这些市场中，我们可以汇集两者的见解。本文详细介绍了用于预测复制研究结果的混合市场模型的试点研究，强调了挑战和机会，并分享了与混合市场参与者进行的半结构化访谈的见解，并概述了未来的工作愿景。

一个评估已发表作品可复制性的混合预测市场的原型

A prototype hybrid prediction market for estimating replicability of published work

Large language models are increasingly capable of generating fluent-appearing
text with relatively little task-specific supervision. But can these models
accurately explain classification decisions? We consider the task of generating
free-text explanations using human-written examples in a few-shot manner. We
find that (1) authoring higher quality prompts results in higher quality
generations; and (2) surprisingly, in a head-to-head comparison, crowdworkers
often prefer explanations generated by GPT-3 to crowdsourced explanations in
existing datasets. Our human studies also show, however, that while models
often produce factual, grammatical, and sufficient explanations, they have room
to improve along axes such as providing novel information and supporting the
label. We create a pipeline that combines GPT-3 with a supervised filter that
incorporates binary acceptability judgments from humans in the loop. Despite
the intrinsic subjectivity of acceptability judgments, we demonstrate that
acceptability is partially correlated with various fine-grained attributes of
explanations. Our approach is able to consistently filter GPT-3-generated
explanations deemed acceptable by humans.

本文探讨利用人类书写的样例以几乎无监督的方式创作自由文本解释的任务，发现高质量的提示有助于提高语言模型的生成效果，同时人类研究表明 GPT-3 生成的解释在某些情况下能够胜过人工生成的解释；作者还结合 GPT-3 与学习自评价的筛选器对生成的解释进行过滤，结果表明这一方法能够实现较高水平的解释过滤。