Traditionally, discriminative models have been the predominant choice for
tasks like document classification and information extraction. These models
make predictions that fall into a limited number of predefined classes,
facilitating a binary true or false evaluation and enabling the direct
calculation of metrics such as the F1 score. However, recent advancements in
generative large language models (GLLMs) have prompted a shift in the field due
to their enhanced zero-shot capabilities, which eliminate the need for a
downstream dataset and computationally expensive fine-tuning. However,
evaluating GLLMs presents a challenge as the binary true or false evaluation
used for discriminative models is not applicable to the predictions made by
GLLMs. This paper introduces a new metric for generative models called ANLS*
for evaluating a wide variety of tasks, including information extraction and
classification tasks. The ANLS* metric extends existing ANLS metrics as a
drop-in-replacement and is still compatible with previously reported ANLS
scores. An evaluation of 7 different datasets and 3 different GLLMs using the
ANLS* metric is also provided, demonstrating the importance of the proposed
metric. We also benchmark a novel approach to generate prompts for documents,
called SFT, against other prompting techniques such as LATIN. In 15 out of 21
cases, SFT outperforms other techniques and improves the state-of-the-art,
sometimes by as much as $15$ percentage points.
Sources are available at this https URL

该论文介绍了一种用于评估生成模型的新度量指标 ANLS*，扩展了现有的 ANLS 度量，适用于信息提取和分类等各种任务，并通过评估 7 个不同数据集和 3 个不同的生成模型，证明了该度量指标的重要性。此外，还对一种新型的文档提示生成技术 SFT 进行了评估，并与其他提示技术（如 LATIN）进行了比较。在 21 个案例中，SFT 在 15 个案例中超过其他技术，提升了最新技术水平，有时可高达 15 个百分点。