Due to the scale and complexity of cloud systems, a system failure would
trigger an "alert storm", i.e., massive correlated alerts. Although these
alerts can be traced back to a few root causes, the overwhelming number makes
it infeasible for manual handling. Alert aggregation is thus critical to help
engineers concentrate on the root cause and facilitate failure resolution.
Existing methods typically utilize semantic similarity-based methods or
statistical methods to aggregate alerts. However, semantic similarity-based
methods overlook the causal rationale of alerts, while statistical methods can
hardly handle infrequent alerts.
To tackle these limitations, we introduce leveraging external knowledge,
i.e., Standard Operation Procedure (SOP) of alerts as a supplement. We propose
COLA, a novel hybrid approach based on correlation mining and LLM (Large
Language Model) reasoning for online alert aggregation. The correlation mining
module effectively captures the temporal and spatial relations between alerts,
measuring their correlations in an efficient manner. Subsequently, only
uncertain pairs with low confidence are forwarded to the LLM reasoning module
for detailed analysis. This hybrid design harnesses both statistical evidence
for frequent alerts and the reasoning capabilities of computationally intensive
LLMs, ensuring the overall efficiency of COLA in handling large volumes of
alerts in practical scenarios. We evaluate COLA on three datasets collected
from the production environment of a large-scale cloud platform. The
experimental results show COLA achieves F1-scores from 0.901 to 0.930,
outperforming state-of-the-art methods and achieving comparable efficiency. We
also share our experience in deploying COLA in our real-world cloud system,
Cloud X.

通过利用预先制定的操作规程 (SOP)，我们提出了 COLA，一种基于相关挖掘和 LLM 推理的新型混合方法，用于在线警报聚合。实验证明，COLA 在处理大规模云平台的大量警报时，具有可比较的效率，且优于最先进的方法。