Network Intrusion Detection Systems (NIDS) are a fundamental tool in
cybersecurity. Their ability to generalize across diverse networks is a
critical factor in their effectiveness and a prerequisite for real-world
applications. In this study, we conduct a comprehensive analysis on the
generalization of machine-learning-based NIDS through an extensive
experimentation in a cross-dataset framework. We employ four machine learning
classifiers and utilize four datasets acquired from different networks:
CIC-IDS-2017, CSE-CIC-IDS2018, LycoS-IDS2017, and LycoS-Unicas-IDS2018.
Notably, the last dataset is a novel contribution, where we apply corrections
based on LycoS-IDS2017 to the well-known CSE-CIC-IDS2018 dataset. The results
show nearly perfect classification performance when the models are trained and
tested on the same dataset. However, when training and testing the models in a
cross-dataset fashion, the classification accuracy is largely commensurate with
random chance except for a few combinations of attacks and datasets. We employ
data visualization techniques in order to provide valuable insights on the
patterns in the data. Our analysis unveils the presence of anomalies in the
data that directly hinder the classifiers capability to generalize the learned
knowledge to new scenarios. This study enhances our comprehension of the
generalization capabilities of machine-learning-based NIDS, highlighting the
significance of acknowledging data heterogeneity.

通过对网络入侵检测系统（NIDS）的泛化能力进行全面分析，本研究采用跨数据集框架的广泛实验，使用四个数据集和四种机器学习分类器，发现当模型在同一数据集上进行训练和测试时，几乎可以完美分类攻击。然而，在跨数据集训练和测试模型时，除了少数攻击和数据集的组合外，分类准确率与随机机会基本持平。本研究的数据可视化技术揭示了数据中的异常存在，直接影响分类器将学习的知识泛化到新场景的能力。这一研究拓宽了我们对基于机器学习的 NIDS 泛化能力的理解，凸显了认识数据异质性的重要性。