The summarization capabilities of pretrained and large language models (LLMs)
have been widely validated in general areas, but their use in scientific
corpus, which involves complex sentences and specialized knowledge, has been
less assessed. This paper presents conceptual and experimental analyses of
scientific summarization, highlighting the inadequacies of traditional
evaluation methods, such as $n$-gram, embedding comparison, and QA,
particularly in providing explanations, grasping scientific concepts, or
identifying key content. Subsequently, we introduce the Facet-aware Metric
(FM), employing LLMs for advanced semantic matching to evaluate summaries based
on different aspects. This facet-aware approach offers a thorough evaluation of
abstracts by decomposing the evaluation task into simpler subtasks.Recognizing
the absence of an evaluation benchmark in this domain, we curate a Facet-based
scientific summarization Dataset (FD) with facet-level annotations. Our
findings confirm that FM offers a more logical approach to evaluating
scientific summaries. In addition, fine-tuned smaller models can compete with
LLMs in scientific contexts, while LLMs have limitations in learning from
in-context information in scientific domains. This suggests an area for future
enhancement of LLMs.

本研究分析和评估了预训练的大型语言模型在科学文献摘要中的应用，并介绍了 Facet-aware Metric 评估方法以及 Facet-based 科学摘要数据集的构建。研究表明，Facet-aware Metric 提供了一种更加合理的科学摘要评估方法，而在科学领域，经过精细调整的较小模型可以与大型语言模型竞争，但大型语言模型在学习科学领域中的上下文信息方面存在局限性，需要进一步改进。