Modern generative search engines enhance the reliability of large language
model (LLM) responses by providing cited evidence. However, evaluating the
answer's attribution, i.e., whether every claim within the generated responses
is fully supported by its cited evidence, remains an open problem. This
verification, traditionally dependent on costly human evaluation, underscores
the urgent need for automatic attribution evaluation methods. To bridge the gap
in the absence of standardized benchmarks for these methods, we present
AttributionBench, a comprehensive benchmark compiled from various existing
attribution datasets. Our extensive experiments on AttributionBench reveal the
challenges of automatic attribution evaluation, even for state-of-the-art LLMs.
Specifically, our findings show that even a fine-tuned GPT-3.5 only achieves
around 80% macro-F1 under a binary classification formulation. A detailed
analysis of more than 300 error cases indicates that a majority of failures
stem from the model's inability to process nuanced information, and the
discrepancy between the information the model has access to and that human
annotators do.

通过提供引用的证据，现代生成式搜索引擎增强了大型语言模型（LLM）响应的可靠性。为了弥补缺乏这些方法的标准化基准的差距，我们提出了 AttributionBench，这是一个由各种现有归因数据集编制的综合基准。我们在 AttributionBench 上进行了大量实验，揭示了即使对于最先进的 LLM 也存在自动归因评估的挑战。

自动归因评估的难度有多大？

AttributionBench: How Hard is Automatic Attribution Evaluation?

In online advertising, the Internet users may be exposed to a sequence of
different ad campaigns, i.e., display ads, search, or referrals from multiple
channels, before led up to any final sales conversion and transaction. For both
campaigners and publishers, it is fundamentally critical to estimate the
contribution from ad campaign touch-points during the customer journey
(conversion funnel) and assign the right credit to the right ad exposure
accordingly. However, the existing research on the multi-touch attribution
problem lacks a principled way of utilizing the users' pre-conversion actions
(i.e., clicks), and quite often fails to model the sequential patterns among
the touch points from a user's behavior data. To make it worse, the current
industry practice is merely employing a set of arbitrary rules as the
attribution model, e.g., the popular last-touch model assigns 100% credit to
the final touch-point regardless of actual attributions. In this paper, we
propose a Dual-attention Recurrent Neural Network (DARNN) for the multi-touch
attribution problem. It learns the attribution values through an attention
mechanism directly from the conversion estimation objective. To achieve this,
we utilize sequence-to-sequence prediction for user clicks, and combine both
post-view and post-click attribution patterns together for the final conversion
estimation. To quantitatively benchmark attribution models, we also propose a
novel yet practical attribution evaluation scheme through the proxy of budget
allocation (under the estimated attributions) over ad channels. The
experimental results on two real datasets demonstrate the significant
performance gains of our attribution model against the state of the art.

本文基于 Dual-attention Recurrent Neural Network 模型，提出了一种关于多点触控补偿，序列预测和预期分配方面的新模型。并且，通过预算分配评估了该模型，证明了其在性能方面优于现有模型。