We present a machine learning-based anomaly detection product, AI Detect and
Respond (AIDR), that monitors Walmart's business and system health in
real-time. During the validation over 3 months, the product served predictions
from over 3000 models to more than 25 application, platform, and operation
teams, covering 63\% of major incidents and reducing the mean-time-to-detect
(MTTD) by more than 7 minutes. Unlike previous anomaly detection methods, our
solution leverages statistical, ML and deep learning models while continuing to
incorporate rule-based static thresholds to incorporate domain-specific
knowledge. Both univariate and multivariate ML models are deployed and
maintained through distributed services for scalability and high availability.
AIDR has a feedback loop that assesses model quality with a combination of
drift detection algorithms and customer feedback. It also offers
self-onboarding capabilities and customizability. AIDR has achieved success
with various internal teams with lower time to detection and fewer false
positives than previous methods. As we move forward, we aim to expand incident
coverage and prevention, reduce noise, and integrate further with root cause
recommendation (RCR) to enable an end-to-end AIDR experience.

我们提出了一种基于机器学习的异常检测产品 ——AI Detect and Respond (AIDR)，它能够实时监测沃尔玛的业务和系统健康状况。在验证期间，该产品使用超过 3000 个模型为 25 个应用程序、平台和运营团队提供了预测，覆盖了 63% 的重大事件，并将平均检测时间 (MTTD) 缩短了 7 分钟以上。与以往的异常检测方法不同，我们的解决方案利用统计学、机器学习和深度学习模型，同时继续使用基于规则的静态阈值来整合领域特定知识。单变量和多变量机器学习模型通过分布式服务部署和维护，以实现可伸缩性和高可用性。AIDR 具有反馈循环，借助漂移检测算法和用户反馈来评估模型质量。它还提供自助入门功能和可定制性。与以前的方法相比，AIDR 在各个内部团队中都取得了成功，其检测时间更短，误报较少。在未来，我们的目标是扩大事故覆盖和预防范围，减少噪音，并与根本原因推荐 (RCR) 进一步整合，实现端到端的 AIDR 体验。