Large language models (LLMs) outperform information retrieval techniques for
downstream knowledge-intensive tasks when being prompted to generate world
knowledge. However, community concerns abound regarding the factuality and
potential implications of using this uncensored knowledge. In light of this, we
introduce CONNER, a COmpreheNsive kNowledge Evaluation fRamework, designed to
systematically and automatically evaluate generated knowledge from six
important perspectives -- Factuality, Relevance, Coherence, Informativeness,
Helpfulness and Validity. We conduct an extensive empirical analysis of the
generated knowledge from three different types of LLMs on two widely studied
knowledge-intensive tasks, i.e., open-domain question answering and
knowledge-grounded dialogue. Surprisingly, our study reveals that the
factuality of generated knowledge, even if lower, does not significantly hinder
downstream tasks. Instead, the relevance and coherence of the outputs are more
important than small factual mistakes. Further, we show how to use CONNER to
improve knowledge-intensive tasks by designing two strategies: Prompt
Engineering and Knowledge Selection. Our evaluation code and LLM-generated
knowledge with human annotations will be released to facilitate future
research.

通过 CONNER 对大型语言模型在知识密集型任务中生成的知识进行综合评估，发现事实性的小错误并不会显著影响下游任务，因此相关性和连贯性比事实性更为重要。此外，研究还提出了通过 Prompt Engineering 和 Knowledge Selection 来改进知识密集型任务的方法。

超越事实性：大型语言模型作为知识生成器的综合评估

Beyond Factuality: A Comprehensive Evaluation of Large Language Models  as Knowledge Generators

Named entity recognition (NER) is a fundamental part of extracting
information from documents in biomedical applications. A notable advantage of
NER is its consistency in extracting biomedical entities in a document context.
Although existing document NER models show consistent predictions, they still
do not meet our expectations. We investigated whether the adjectives and
prepositions within an entity cause a low label consistency, which results in
inconsistent predictions. In this paper, we present our method, ConNER, which
enhances the label dependency of modifiers (e.g., adjectives and prepositions)
to achieve higher label agreement. ConNER refines the draft labels of the
modifiers to improve the output representations of biomedical entities. The
effectiveness of our method is demonstrated on four popular biomedical NER
datasets; in particular, its efficacy is proved on two datasets with 7.5-8.6%
absolute improvements in the F1 score. We interpret that our ConNER method is
effective on datasets that have intrinsically low label consistency. In the
qualitative analysis, we demonstrate how our approach makes the NER model
generate consistent predictions. Our code and resources are available at
this https URL.

本文介绍了一种名为 ConNER 的方法，通过提高修饰词（如形容词和介词）的标签依赖性来实现更高的标签一致性，从而改善生物医学实体的输出表示。该方法在四个流行的生物医学 NER 数据集上证明了其有效性，并在两个数据集上实现了 7.5-8.6％ 的绝对改进。