Natural Language Processing (NLP), a cornerstone field within artificial
intelligence, has been increasingly utilized in the field of materials science
literature. Our study conducts a reproducibility analysis of two pioneering
works within this domain: "Machine-learned and codified synthesis parameters of
oxide materials" by Kim et al., and "Unsupervised word embeddings capture
latent knowledge from materials science literature" by Tshitoyan et al. We aim
to comprehend these studies from a reproducibility perspective, acknowledging
their significant influence on the field of materials informatics, rather than
critiquing them. Our study indicates that both papers offered thorough
workflows, tidy and well-documented codebases, and clear guidance for model
evaluation. This makes it easier to replicate their results successfully and
partially reproduce their findings. In doing so, they set commendable standards
for future materials science publications to aspire to. However, our analysis
also highlights areas for improvement such as to provide access to training
data where copyright restrictions permit, more transparency on model
architecture and the training process, and specifications of software
dependency versions. We also cross-compare the word embedding models between
papers, and find that some key differences in reproducibility and
cross-compatibility are attributable to design choices outside the bounds of
the models themselves. In summary, our study appreciates the benchmark set by
these seminal papers while advocating for further enhancements in research
reproducibility practices in the field of NLP for materials science. This
balance of understanding and continuous improvement will ultimately propel the
intersecting domains of NLP and materials science literature into a future of
exciting discoveries.

我们的研究对两篇杰出的物质科学文献领域的先驱作品进行了重现性分析：金和安迪的 “氧化物材料机器学习和编码合成参数” 和西托扬的 “无监督词嵌入从物质科学文献中捕捉潜在知识”。我们的研究发现这两篇论文的工作流程严谨，代码可复现，获得了明确的模型评估指导；然而，还存在改进的空间，如尽可能提供训练数据的访问权限，提供更多关于模型架构和训练过程的透明度，以及规定软件依赖版本。

材料科学中的自复现经验教训

Lessons in Reproducibility: Insights from NLP Studies in Materials  Science

This article offers a short guide to the steps scientists can take to ensure
that their data and associated analyses continue to be of value and to be
recognized. In just the past few years, hundreds of scholarly papers and
reports have been written on questions of data sharing, data provenance,
research reproducibility, licensing, attribution, privacy, and more, but our
goal here is not to review that literature. Instead, we present a short guide
intended for researchers who want to know why it is important to "care for and
feed" data, with some practical advice on how to do that.

本文为科学家提供了一份简短指南，介绍了他们应该采取的步骤，以确保其数据和相关分析持续有价值并得到认可。该指南旨在帮助研究人员了解为何 “关心和管理” 数据的重要性以及如何实现这一目标。