Misconfigurations are the major causes of software failures. Existing
configuration validation techniques rely on manually written rules or test
cases, which are expensive to implement and maintain, and are hard to be
comprehensive. Leveraging machine learning (ML) and natural language processing
(NLP) for configuration validation is considered a promising direction, but has
been facing challenges such as the need of not only large-scale configuration
data, but also system-specific features and models which are hard to
generalize. Recent advances in Large Language Models (LLMs) show the promises
to address some of the long-lasting limitations of ML/NLP-based configuration
validation techniques. In this paper, we present an exploratory analysis on the
feasibility and effectiveness of using LLMs like GPT and Codex for
configuration validation. Specifically, we take a first step to empirically
evaluate LLMs as configuration validators without additional fine-tuning or
code generation. We develop a generic LLM-based validation framework, named
Ciri, which integrates different LLMs. Ciri devises effective prompt
engineering with few-shot learning based on both valid configuration and
misconfiguration data. Ciri also validates and aggregates the outputs of LLMs
to generate validation results, coping with known hallucination and
nondeterminism of LLMs. We evaluate the validation effectiveness of Ciri on
five popular LLMs using configuration data of six mature, widely deployed
open-source systems. Our analysis (1) confirms the potential of using LLMs for
configuration validation, (2) understands the design space of LLMbased
validators like Ciri, especially in terms of prompt engineering with few-shot
learning, and (3) reveals open challenges such as ineffectiveness in detecting
certain types of misconfigurations and biases to popular configuration
parameters.

使用大型语言模型进行配置验证的可行性和有效性的初步研究，发展了集成不同语言模型的通用验证框架 Ciri，并评估了其在六个成熟、广泛部署的开源系统的配置数据上的验证效果。