Large Language Models (LLMs) often exhibit positional bias in long-context settings, under-attending to information in the middle of inputs. We investigate the presence of this bias in long-form summarization, its impact on faithfulness, and various techniques to mitigate this bias. To consistently evaluate faithfulness, we first compile a benchmark of eight human-annotated long-form summarization datasets and perform a meta-evaluation of faithfulness metrics. We show that LLM-based faithfulness metrics, though effective with full-context inputs, remain sensitive to document order, indicating positional bias. Analyzing LLM-generated summaries across six datasets, we find a "U-shaped" trend in faithfulness, where LLMs faithfully summarize the beginning and end of documents but neglect middle content. Perturbing document order similarly reveals models are less faithful when important documents are placed in the middle of the input. We find that this behavior is partly due to shifting focus with context length: as context increases, summaries become less faithful, but beyond a certain length, faithfulness improves as the model focuses on the end. Finally, we experiment with different generation techniques to reduce positional bias and find that prompting techniques effectively direct model attention to specific positions, whereas more sophisticated approaches offer limited improvements. Our data and code are available in https://github.com/meetdavidwan/longformfact.

本研究探讨了大语言模型在长文本摘要中存在的位置信息偏差及其对忠实性的影响。我们首次建立了一个八个人工注释的长文本摘要数据集，分析了不同生成技术对减少该偏差的效果，发现简单的提示技术更能有效引导模型关注特定位置。我们的研究结果表明，模型在摘要中忠实性较高的部分主要集中在文档的开始和结束部分，而中间内容则被忽视，这一发现对改善长文本摘要质量具有重要意义。

长文本摘要中的忠实性位置偏差研究