Large Language Models (LLMs) changed the way we design and interact with
software systems. Their ability to process and extract information from text
has drastically improved productivity in a number of routine tasks. Developers
that want to include these models in their software stack, however, face a
dreadful challenge: debugging their inconsistent behavior across minor
variations of the prompt. We therefore introduce two metrics for classification
tasks, namely sensitivity and consistency, which are complementary to task
performance. First, sensitivity measures changes of predictions across
rephrasings of the prompt, and does not require access to ground truth labels.
Instead, consistency measures how predictions vary across rephrasings for
elements of the same class. We perform an empirical comparison of these metrics
on text classification tasks, using them as guideline for understanding failure
modes of the LLM. Our hope is that sensitivity and consistency will be powerful
allies in automatic prompt engineering frameworks to obtain LLMs that balance
robustness with performance.

使用两个度量标准（敏感性和一致性）来评估大型语言模型在分类任务中的性能，并希望它们能成为自动提示工程框架中寻求平衡鲁棒性与性能的有力工具。