Text summarization models have typically focused on optimizing aspects of
quality such as fluency, relevance, and coherence, particularly in the context
of news articles. However, summarization models are increasingly being used to
summarize diverse sources of text, such as social media data, that encompass a
wide demographic user base. It is thus crucial to assess not only the quality
of the generated summaries, but also the extent to which they can fairly
represent the opinions of diverse social groups. Position bias, a long-known
issue in news summarization, has received limited attention in the context of
social multi-document summarization. We deeply investigate this phenomenon by
analyzing the effect of group ordering in input documents when summarizing
tweets from three distinct linguistic communities: African-American English,
Hispanic-aligned Language, and White-aligned Language. Our empirical analysis
shows that although the textual quality of the summaries remains consistent
regardless of the input document order, in terms of fairness, the results vary
significantly depending on how the dialect groups are presented in the input
data. Our results suggest that position bias manifests differently in social
multi-document summarization, severely impacting the fairness of summarization
models.

对社交媒体数据进行多源文档摘要的时候，用于优化摘要质量的模型往往侧重于流畅性、相关性和连贯性等方面，但摘要模型对不同社会群体意见的公正性也显得越来越重要。我们深入探讨了位置偏见在社交媒体多源文档摘要中的影响，通过分析输入文档中群组顺序对非裔美国英语、西班牙语及白人英语族群的推文进行摘要。实证分析结果显示，尽管文本摘要的质量在不考虑输入文档顺序时保持稳定，但公正性方面的结果会因方言群体在输入数据中的呈现方式而产生显著差异。我们的结果表明，位置偏见在社交媒体多源文档摘要中呈现不同，严重影响了摘要模型的公正性。

社交多文档摘要中了解位置偏见对公平性的影响

Understanding Position Bias Effects on Fairness in Social Multi-Document  Summarization

Recent studies have found that summaries generated by large language models
(LLMs) are favored by human annotators over the original reference summaries in
commonly used summarization datasets. Therefore, we investigate a new learning
paradigm of text summarization models that considers the LLMs as the reference
or the gold-standard oracle on commonly used summarization datasets such as the
CNN/DailyMail dataset. To examine the standard practices that are aligned with
the new learning setting, we propose a novel training method that is based on
contrastive learning with LLMs as a summarization quality evaluator. For this
reward-based training method, we investigate two different methods of utilizing
LLMs for summary quality evaluation, namely GPTScore and GPTRank. Our
experiments on the CNN/DailyMail dataset demonstrate that smaller summarization
models trained by our proposed method can achieve performance equal to or
surpass that of the reference LLMs, as evaluated by the LLMs themselves. This
underscores the efficacy of our proposed paradigm in enhancing model
performance over the standard maximum likelihood estimation (MLE) training
method, and its efficiency since it only requires a small budget to access the
LLMs. We release the training scripts, model outputs, and LLM-based evaluation
results to facilitate future studies.

本文提出了一种新的学习范式，考虑到 LLMS 是常用摘要数据集中的参考标准，用对比学习和 LLM 作为摘要质量评估器进行摘要训练方法。实验证明，用 GPTScore 和 GPTRank 两种 LLM 计分方式训练出来的较小的摘要模型，其性能可以与参考的 LLM 相媲美，通过访问 LLMS 它只需要很小的预算。