Large language models (LLMs) can respond to human language queries and have
shown powerful potential applications in network operations (NetOps). Thanks to
the large amount of commonsense knowledge inherent, LLMs achieve much better
inference accuracy than traditional models and emerge with strong abilities in
generalization, reasoning, and code generation. These abilities may have a
crucial boost to automated and intelligent NetOps. However, it remains
under-explored how well LLMs perform in various NetOps tasks. In this work, we
make a systematic assessment of the capabilities, strengths, and limitations of
selected LLMs in the field of NetOps. The evaluation is conducted on a
collection of 5,732 questions about NetOps, encompassing 26 publicly available
general-domain LLMs, including ChatGPT, LLaMA, Falcon, etc. We also finetune
some of these LLMs with our collected NetOps corpus and evaluate the resulting
models. The evaluation method follows the widely adopted benchmarks for
general-domain LLMs, combined with Chain-of-Thought Prompts and
Retrieval-Augmented Generation. The results show that only GPT-4 achieves high
accuracy equivalent to passing the NetOps certification exam for humans, while
all the other LLMs have much lower accuracy. However, some open models like
LLaMA 2 still demonstrate significant potential. Furthermore, we evaluate the
impact of factors such as model parameters, prompt engineering, instruction
fine-tuning etc. This work shall be treated as the initial effort to systematic
evaluation of LLMs in NetOps, and a more rigorous study is required for
production use. The evaluation code and dataset will be released to benefit
future research.

通过系统评估选择的大型语言模型（LLMs）在网络运维（NetOps）领域的能力、优势和局限性，本研究发现只有 GPT-4 能够达到与人类通过网络运维认证考试的高精度等级。