When answering complex questions, large language models (LLMs) may produce
answers that do not satisfy all criteria of the question. While existing
self-evaluation techniques aim to detect if such answers are correct, these
techniques are unable to determine which criteria of the question are satisfied
by the generated answers. To address this issue, we propose answer-based claim
decomposition (ABCD), a prompting strategy that decomposes questions into a
series of true/false claims that can be used to verify which criteria of the
input question an answer satisfies. Using the decomposed ABCD claims, we
perform fine-grained self-evaluation. Through preliminary experiments on three
datasets, including a newly-collected challenge dataset ObscureQA, we find that
GPT-3.5 has some ability to determine to what extent its answer satisfies the
criteria of the input question, and can give insights into the errors and
knowledge gaps of the model.

本文介绍了一种针对大型语言模型生成的答案自我评估技术，即 ABCD，该技术通过将复杂问题分解为一系列真假命题来验证模型生成的答案是否满足所有的问题标准， 并进行精细化评估，实验证明该技术可以帮助发现模型的错误和知识盲区。