The last decade saw the emergence of systematic large-scale replication
projects in the social and behavioral sciences, (Camerer et al., 2016, 2018;
Ebersole et al., 2016; Klein et al., 2014, 2018; Collaboration, 2015). These
projects were driven by theoretical and conceptual concerns about a high
fraction of "false positives" in the scientific publications (Ioannidis, 2005)
(and a high prevalence of "questionable research practices" (Simmons, Nelson,
and Simonsohn, 2011). Concerns about the credibility of research findings are
not unique to the behavioral and social sciences; within Computer Science,
Artificial Intelligence (AI) and Machine Learning (ML) are areas of particular
concern (Lucic et al., 2018; Freire, Bonnet, and Shasha, 2012; Gundersen and
Kjensmo, 2018; Henderson et al., 2018). Given the pioneering role of the
behavioral and social sciences in the promotion of novel methodologies to
improve the credibility of research, it is a promising approach to analyze the
lessons learned from this field and adjust strategies for Computer Science, AI
and ML In this paper, we review approaches used in the behavioral and social
sciences and in the DARPA SCORE project. We particularly focus on the role of
human forecasting of replication outcomes, and how forecasting can leverage the
information gained from relatively labor and resource-intensive replications.
We will discuss opportunities and challenges of using these approaches to
monitor and improve the credibility of research areas in Computer Science, AI,
and ML.

本文回顾了行为和社会科学领域以及 DARPA SCORE 项目中使用的方法，特别关注人类预测复制结果的作用，以及预测如何利用相对劳动力和资源密集的复制所获得的信息。探讨了使用这些方法来监测和改进计算机科学，人工智能和机器学习领域中研究可信度的机会和挑战。