The attribution of question answering is to provide citations for supporting
generated statements, and has attracted wide research attention. The current
methods for automatically evaluating the attribution, which are often based on
Large Language Models (LLMs), are still inadequate, particularly in recognizing
subtle differences between attributions, and complex relationships between
citations and statements. To compare these attribution evaluation methods and
develop new ones, we introduce a set of fine-grained categories (i.e.,
supportive, insufficient, contradictory and irrelevant) for measuring the
attribution, and develop a Complex Attributed Question Answering (CAQA)
benchmark by leveraging knowledge graphs (KGs) for automatically generating
attributions of different categories to question-answer pairs. Our analysis
reveals that existing evaluators perform poorly under fine-grained attribution
settings and exhibit weaknesses in complex citation-statement reasoning. Our
CAQA benchmark, validated with human annotations, emerges as a promising tool
for selecting and developing LLM attribution evaluators.

使用知识图谱为问题 - 答案对自动生成不同类别的归因，并引入细分类型（支持性、不足、矛盾和不相关）来衡量归因，评估现有评估方法在细致归因设置下的表现较差，使用人工注释验证的 CAQA 基准为选择和开发 LLM 归因评估器提供了有前景的工具。