By simply composing prompts, developers can prototype novel generative
applications with Large Language Models (LLMs). To refine prototypes into
products, however, developers must iteratively revise prompts by evaluating
outputs to diagnose weaknesses. Formative interviews (N=8) revealed that
developers invest significant effort in manually evaluating outputs as they
assess context-specific and subjective criteria. We present EvalLM, an
interactive system for iteratively refining prompts by evaluating multiple
outputs on user-defined criteria. By describing criteria in natural language,
users can employ the system's LLM-based evaluator to get an overview of where
prompts excel or fail, and improve these based on the evaluator's feedback. A
comparative study (N=12) showed that EvalLM, when compared to manual
evaluation, helped participants compose more diverse criteria, examine twice as
many outputs, and reach satisfactory prompts with 59% fewer revisions. Beyond
prompts, our work can be extended to augment model evaluation and alignment in
specific application contexts.

通过评估多个用户定义的标准，使用自然语言描述系统的评估器提供的反馈，从而使开发人员能够通过迭代改进引导语。与手动评估相比，EvalLM 有助于参与者构建更多样化的标准，检查两倍数量的输出，并在减少 59% 的修订次数后获得令人满意的引导语。该研究可扩展到模型评估和特定应用环境中的对齐。