Data drifts pose a critical challenge in the lifecycle of machine learning
(ML) models, affecting their performance and reliability. In response to this
challenge, we present a microbenchmark study, called D3Bench, which evaluates
the efficacy of open-source drift detection tools. D3Bench examines the
capabilities of Evidently AI, NannyML, and Alibi-Detect, leveraging real-world
data from two smart building use cases.We prioritize assessing the functional
suitability of these tools to identify and analyze data drifts. Furthermore, we
consider a comprehensive set of non-functional criteria, such as the
integrability with ML pipelines, the adaptability to diverse data types,
user-friendliness, computational efficiency, and resource demands. Our findings
reveal that Evidently AI stands out for its general data drift detection,
whereas NannyML excels at pinpointing the precise timing of shifts and
evaluating their consequent effects on predictive accuracy.

数据漂移对机器学习模型的生命周期造成重大挑战。本研究通过微基准研究 D3Bench 评估了开源漂移检测工具的效能，评估了 Evidently AI、NannyML 和 Alibi-Detect 在两个智能建筑使用案例中的能力。我们重点评估这些工具在识别和分析数据漂移方面的功能适应性，并考虑了一系列非功能性标准，如与机器学习流水线的集成性、对不同数据类型的适应性、用户友好性、计算效率和资源需求。研究结果表明，Evidently AI 在一般数据漂移检测方面表现出色，而 NannyML 在准确定位漂移发生的时间和评估其对预测准确性的影响方面表现出色。