Explainably estimating confidence in published scholarly work offers
opportunity for faster and more robust scientific progress. We develop a
synthetic prediction market to assess the credibility of published claims in
the social and behavioral sciences literature. We demonstrate our system and
detail our findings using a collection of known replication projects. We
suggest that this work lays the foundation for a research agenda that
creatively uses AI for peer review.

本研究开发了一种合成预测市场来评估社会与行为科学文献中已发表声明的可信度，并使用一系列已知的复制项目来展示该系统，并建议此项工作为使用人工智能进行同行评审奠定了基础。

一项用于估计发表作品置信度的合成预测市场

A Synthetic Prediction Market for Estimating Confidence in Published Work

The last decade saw the emergence of systematic large-scale replication
projects in the social and behavioral sciences, (Camerer et al., 2016, 2018;
Ebersole et al., 2016; Klein et al., 2014, 2018; Collaboration, 2015). These
projects were driven by theoretical and conceptual concerns about a high
fraction of "false positives" in the scientific publications (Ioannidis, 2005)
(and a high prevalence of "questionable research practices" (Simmons, Nelson,
and Simonsohn, 2011). Concerns about the credibility of research findings are
not unique to the behavioral and social sciences; within Computer Science,
Artificial Intelligence (AI) and Machine Learning (ML) are areas of particular
concern (Lucic et al., 2018; Freire, Bonnet, and Shasha, 2012; Gundersen and
Kjensmo, 2018; Henderson et al., 2018). Given the pioneering role of the
behavioral and social sciences in the promotion of novel methodologies to
improve the credibility of research, it is a promising approach to analyze the
lessons learned from this field and adjust strategies for Computer Science, AI
and ML In this paper, we review approaches used in the behavioral and social
sciences and in the DARPA SCORE project. We particularly focus on the role of
human forecasting of replication outcomes, and how forecasting can leverage the
information gained from relatively labor and resource-intensive replications.
We will discuss opportunities and challenges of using these approaches to
monitor and improve the credibility of research areas in Computer Science, AI,
and ML.

本文回顾了行为和社会科学领域以及 DARPA SCORE 项目中使用的方法，特别关注人类预测复制结果的作用，以及预测如何利用相对劳动力和资源密集的复制所获得的信息。探讨了使用这些方法来监测和改进计算机科学，人工智能和机器学习领域中研究可信度的机会和挑战。