The complexity and scale of IT systems are increasing dramatically, posing
many challenges to real-world anomaly detection. Deep learning anomaly
detection has emerged, aiming at feature learning and anomaly scoring, which
has gained tremendous success. However, little work has been done on the
thresholding problem despite it being a critical factor for the effectiveness
of anomaly detection. In this paper, we model thresholding in anomaly detection
as a Markov Decision Process and propose an agent-based dynamic thresholding
(ADT) framework based on a deep Q-network. The proposed method can be
integrated into many systems that require dynamic thresholding. An auto-encoder
is utilized in this study to obtain feature representations and produce anomaly
scores for complex input data. ADT can adjust thresholds adaptively by
utilizing the anomaly scores from the auto-encoder and significantly improve
anomaly detection performance. The properties of ADT are studied through
experiments on three real-world datasets and compared with benchmarks, hence
demonstrating its thresholding capability, data-efficient learning, stability,
and robustness. Our study validates the effectiveness of reinforcement learning
in optimal thresholding control in anomaly detection.

本研究通过将阈值问题建模为马尔可夫决策过程，并基于深度 Q 网络提出了一种基于代理的动态阈值 (ADT) 框架，该框架可以用于需要动态阈值的多个系统。通过利用自编码器从复杂输入数据中获取特征表示并生成异常得分，ADT 可以自适应地调整阈值，并显著改善异常检测性能。通过在三个真实数据集上进行实验证明了 ADT 的阈值能力、数据高效学习、稳定性和鲁棒性，并验证了强化学习在异常检测中的最优阈值控制的有效性。