Over the past few years, deep learning methods have been applied for a wide
range of Software Engineering (SE) tasks, including in particular for the
important task of automatically predicting and localizing faults in software.
With the rapid adoption of increasingly complex machine learning models, it
however becomes more and more difficult for scholars to reproduce the results
that are reported in the literature. This is in particular the case when the
applied deep learning models and the evaluation methodology are not properly
documented and when code and data are not shared. Given some recent -- and very
worrying -- findings regarding reproducibility and progress in other areas of
applied machine learning, the goal of this work is to analyze to what extent
the field of software engineering, in particular in the area of software fault
prediction, is plagued by similar problems. We have therefore conducted a
systematic review of the current literature and examined the level of
reproducibility of 56 research articles that were published between 2019 and
2022 in top-tier software engineering conferences. Our analysis revealed that
scholars are apparently largely aware of the reproducibility problem, and about
two thirds of the papers provide code for their proposed deep learning models.
However, it turned out that in the vast majority of cases, crucial elements for
reproducibility are missing, such as the code of the compared baselines, code
for data pre-processing or code for hyperparameter tuning. In these cases, it
therefore remains challenging to exactly reproduce the results in the current
research literature. Overall, our meta-analysis therefore calls for improved
research practices to ensure the reproducibility of machine-learning based
research.

分析软件工程领域的软件错误预测是否存在与其他机器学习领域相似的可复现性问题，发现大部分研究文章缺乏关键的复现元素，呼吁改进研究实践以确保基于机器学习的研究的可复现性。

深度学习软件故障预测的可复现性研究

Investigating Reproducibility in Deep Learning-Based Software Fault  Prediction

This evidence-based position paper critiques current research practices
within the language model pre-training literature. Despite rapid recent
progress afforded by increasingly better pre-trained language models (PLMs),
current PLM research practices often conflate different possible sources of
model improvement, without conducting proper ablation studies and principled
comparisons between different models under comparable conditions. These
practices (i) leave us ill-equipped to understand which pre-training approaches
should be used under what circumstances; (ii) impede reproducibility and credit
assignment; and (iii) render it difficult to understand: "How exactly does each
factor contribute to the progress that we have today?" We provide a case in
point by revisiting the success of BERT over its baselines, ELMo and GPT-1, and
demonstrate how -- under comparable conditions where the baselines are tuned to
a similar extent -- these baselines (and even-simpler variants thereof) can, in
fact, achieve competitive or better performance than BERT. These findings
demonstrate how disentangling different factors of model improvements can lead
to valuable new insights. We conclude with recommendations for how to encourage
and incentivize this line of work, and accelerate progress towards a better and
more systematic understanding of what factors drive the progress of our
foundation models today.

本文对在语言模型预训练文献中的当前研究方法进行了基于证据的批判。尽管越来越好的预训练语言模型（PLMs）已经取得了快速进展，但当前的 PLM 研究实践通常混淆不同可能的模型改进来源，没有在可比条件下进行适当的消融研究和原则性比较。