Novelty, akin to gene mutation in evolution, opens possibilities for
scientific advancement. Despite peer review being the gold standard for
evaluating novelty in scholarly communication and resource allocation, the vast
volume of submissions necessitates an automated measure of scientific novelty.
Adopting a perspective that views novelty as the atypical combination of
existing knowledge, we introduce an information-theoretic measure of novelty in
scholarly publications. This measure is quantified by the degree of `surprise'
perceived by a language model that represents the distribution of scientific
discourse. The proposed measure is accompanied by face and construct validity
evidence; the former demonstrates correspondence to scientific common sense,
and the latter is endorsed through alignments with novelty evaluations from a
select panel of domain experts. Additionally, characterized by its
interpretability, fine granularity, and accessibility, this measure addresses
gaps prevalent in existing methods. We believe this measure holds great
potential to benefit editors, stakeholders, and policymakers, and it provides a
confident lens for examining the relationship between novelty and scientific
dynamics such as creativity, interdisciplinarity, scientific advances, and
more.

通过引入信息论的新颖度量，该研究介绍了一种自动化的科学新颖度量方式，并通过面向科学常识的准确性以及领域专家的新奇性评估的结果，验证了该度量方法的有效性。该新颖度量具有可解释性、细粒度和易接近性，有望对编辑、利益相关者和政策制定者产生重要影响，并为研究新颖性与创造力、跨学科研究、科学进展等科学动态之间的关系提供了自信的观察视角。

学术出版物基于内容的新颖度衡量方法：概念验证

A Content-Based Novelty Measure for Scholarly Publications: A Proof of  Concept

The task of multi-document summarization (MDS) aims at models that, given
multiple documents as input, are able to generate a summary that combines
disperse information, originally spread across these documents. Accordingly, it
is expected that both reference summaries in MDS datasets, as well as system
summaries, would indeed be based on such dispersed information. In this paper,
we argue for quantifying and assessing this expectation. To that end, we
propose an automated measure for evaluating the degree to which a summary is
``disperse'', in the sense of the number of source documents needed to cover
its content. We apply our measure to empirically analyze several popular MDS
datasets, with respect to their reference summaries, as well as the output of
state-of-the-art systems. Our results show that certain MDS datasets barely
require combining information from multiple documents, where a single document
often covers the full summary content. Overall, we advocate using our metric
for assessing and improving the degree to which summarization datasets require
combining multi-document information, and similarly how summarization models
actually meet this challenge. Our code is available in
this https URL.

本文介绍一种用于评估多文档摘要生成模型及数据集中数据分散程度的自动化度量方法，并检验几个流行的多文档摘要数据集，结果表明某些数据集没有利用多文档信息，提出应用该度量方法评估数据集和模型的性能。

多篇文档摘要如何 “多”?

How "Multi" is Multi-Document Summarization?

For some images, descriptions written by multiple people are consistent with
each other. But for other images, descriptions across people vary considerably.
In other words, some images are specific $-$ they elicit consistent
descriptions from different people $-$ while other images are ambiguous.
Applications involving images and text can benefit from an understanding of
which images are specific and which ones are ambiguous. For instance, consider
text-based image retrieval. If a query description is moderately similar to the
caption (or reference description) of an ambiguous image, that query may be
considered a decent match to the image. But if the image is very specific, a
moderate similarity between the query and the reference description may not be
sufficient to retrieve the image.
In this paper, we introduce the notion of image specificity. We present two
mechanisms to measure specificity given multiple descriptions of an image: an
automated measure and a measure that relies on human judgement. We analyze
image specificity with respect to image content and properties to better
understand what makes an image specific. We then train models to automatically
predict the specificity of an image from image features alone without requiring
textual descriptions of the image. Finally, we show that modeling image
specificity leads to improvements in a text-based image retrieval application.

本研究介绍了图像的特异性概念，提出了自动化和人工评估测量图像特异性的两种机制，分析了图像内容和属性对其特异性的影响，通过训练模型自动预测图像的特异性，并展示其在文本图像检索中的应用。