Personalized summarization models cater to individuals' subjective
understanding of saliency, as represented by their reading history and current
topics of attention. Existing personalized text summarizers are primarily
evaluated based on accuracy measures such as BLEU, ROUGE, and METEOR. However,
a recent study argued that accuracy measures are inadequate for evaluating the
degree of personalization of these models and proposed EGISES, the first metric
to evaluate personalized text summaries. It was suggested that accuracy is a
separate aspect and should be evaluated standalone. In this paper, we challenge
the necessity of an accuracy leaderboard, suggesting that relying on
accuracy-based aggregated results might lead to misleading conclusions. To
support this, we delve deeper into EGISES, demonstrating both theoretically and
empirically that it measures the degree of responsiveness, a necessary but not
sufficient condition for degree-of-personalization. We subsequently propose
PerSEval, a novel measure that satisfies the required sufficiency condition.
Based on the benchmarking of ten SOTA summarization models on the PENS dataset,
we empirically establish that -- (i) PerSEval is reliable w.r.t human-judgment
correlation (Pearson's r = 0.73; Spearman's $\rho$ = 0.62; Kendall's $\tau$ =
0.42), (ii) PerSEval has high rank-stability, (iii) PerSEval as a rank-measure
is not entailed by EGISES-based ranking, and (iv) PerSEval can be a standalone
rank-measure without the need of any aggregated ranking.

个性化摘要模型研究了人们对要点的个人理解，通过他们的阅读历史和当前关注的主题进行表示。本文挑战了精度排行榜的必要性，并提出了一个新的度量指标 PerSEval，用于评估个性化文本摘要的程度。然后，通过在 PENS 数据集上对最新的十个摘要模型进行基准测试，我们验证了 PerSEval 的可靠性和可独立使用的特性。