In recent years, funding agencies and journals increasingly advocate for open
science practices (e.g. data and method sharing) to improve the transparency,
access, and reproducibility of science. However, quantifying these practices at
scale has proven difficult. In this work, we leverage a large-scale dataset of
1.1M papers from arXiv that are representative of the fields of physics, math,
and computer science to analyze the adoption of data and method link-sharing
practices over time and their impact on article reception. To identify links to
data and methods, we train a neural text classification model to automatically
classify URL types based on contextual mentions in papers. We find evidence
that the practice of link-sharing to methods and data is spreading as more
papers include such URLs over time. Reproducibility efforts may also be
spreading because the same links are being increasingly reused across papers
(especially in computer science); and these links are increasingly concentrated
within fewer web domains (e.g. Github) over time. Lastly, articles that share
data and method links receive increased recognition in terms of citation count,
with a stronger effect when the shared links are active (rather than defunct).
Together, these findings demonstrate the increased spread and perceived value
of data and method sharing practices in open science.

近年来，资助机构和期刊越来越倡导开放科学实践（如数据和方法共享），以提高科学的透明度、可获取性和可重复性。本研究利用来自 arXiv 的 110 万篇论文的大规模数据集，代表了物理、数学和计算机科学领域，分析了随时间推移数据和方法链接共享实践的采用情况以及对文章接受程度的影响。我们通过训练神经文本分类模型来识别数据和方法链接，根据论文中的上下文提及自动分类 URL 类型。研究发现，随着时间推移，链接方法和数据的共享实践正在扩散，越来越多的论文包含此类 URL。可重复性努力也可能在扩散，因为相同的链接越来越多地在论文中被重复使用（特别是计算机科学领域），并且这些链接随时间越来越集中在更少的网域（如 Github）中。最后，分享数据和方法链接的文章在引用次数方面获得了增加的承认，当链接是有效的时候效果更强。这些发现共同证明了数据和方法共享实践在开放科学中的扩展和受到的重视。

开放科学的兴起：追踪数据和方法共享实践的演变和感知价值

The Rise of Open Science: Tracking the Evolution and Perceived Value of  Data and Methods Link-Sharing Practices

As natural language processing (NLP) has recently seen an unprecedented level
of excitement, and more people are eager to enter the field, it is unclear
whether current research reproducibility efforts are sufficient for this group
of beginners to apply the latest developments. To understand their needs, we
conducted a study with 93 students in an introductory NLP course, where
students reproduced the results of recent NLP papers. Surprisingly, we find
that their programming skill and comprehension of research papers have a
limited impact on their effort spent completing the exercise. Instead, we find
accessibility efforts by research authors to be the key to success, including
complete documentation, better coding practice, and easier access to data
files. Going forward, we recommend that NLP researchers pay close attention to
these simple aspects of open-sourcing their work, and use insights from
beginners' feedback to provide actionable ideas on how to better support them.

本文为了解当下自然语言处理领域新手对当前研究重复性工作的需求，对 93 名 introductory NLP 课程学生开展了研究。结果表明，编程技能和对研究论文的理解与其完成任务的时间无明显相关性，而研究作者开源工作的可访问性、文档完整性、良好编程规范和易于获取数据文件等均对成功至关重要。此外，作者建议自然语言处理研究者关注这些简单但关键的方面并充分利用初学者的反馈来提供更好的支持和改进。