While large language models (LLMs) excel in various natural language tasks in
English, their performance in lower-resourced languages like Hebrew, especially
for generative tasks such as abstractive summarization, remains unclear. The
high morphological richness in Hebrew adds further challenges due to the
ambiguity in sentence comprehension and the complexities in meaning
construction. In this paper, we address this resource and evaluation gap by
introducing HeSum, a novel benchmark specifically designed for abstractive text
summarization in Modern Hebrew. HeSum consists of 10,000 article-summary pairs
sourced from Hebrew news websites written by professionals. Linguistic analysis
confirms HeSum's high abstractness and unique morphological challenges. We show
that HeSum presents distinct difficulties for contemporary state-of-the-art
LLMs, establishing it as a valuable testbed for generative language technology
in Hebrew, and MRLs generative challenges in general.

该论文引入了 HeSum，一个专为现代希伯来语抽象文本摘要而设计的基准测试集，其中包含从专业人士撰写的希伯来语新闻网站中获取的 1 万个文章 - 摘要对。通过语言分析，我们证实了 HeSum 的高度抽象性和独特的形态学挑战。我们展示了 HeSum 对于当代最先进的大型语言模型来说具有明显的困难，从而将其确立为希伯来语的生成语言技术以及 MRL 的生成挑战的有价值的试验平台。