Large language models (LLMs) have shown remarkable achievements in natural
language processing tasks, producing high-quality outputs. However, LLMs still
exhibit limitations, including the generation of factually incorrect
information. In safety-critical applications, it is important to assess the
confidence of LLM-generated content to make informed decisions. Retrieval
Augmented Language Models (RALMs) is relatively a new area of research in NLP.
RALMs offer potential benefits for scientific NLP tasks, as retrieved
documents, can serve as evidence to support model-generated content. This
inclusion of evidence enhances trustworthiness, as users can verify and explore
the retrieved documents to validate model outputs. Quantifying uncertainty in
RALM generations further improves trustworthiness, with retrieved text and
confidence scores contributing to a comprehensive and reliable model for
scientific applications. However, there is limited to no research on UQ for
RALMs, particularly in scientific contexts. This study aims to address this gap
by conducting a comprehensive evaluation of UQ in RALMs, focusing on scientific
tasks. This research investigates how uncertainty scores vary when scientific
knowledge is incorporated as pretraining and retrieval data and explores the
relationship between uncertainty scores and the accuracy of model-generated
outputs. We observe that an existing RALM finetuned with scientific knowledge
as the retrieval data tends to be more confident in generating predictions
compared to the model pretrained only with scientific knowledge. We also found
that RALMs are overconfident in their predictions, making inaccurate
predictions more confidently than accurate ones. Scientific knowledge provided
either as pretraining or retrieval corpus does not help alleviate this issue.
We released our code, data and dashboards at this https URL

通过在科学任务中综合评估检索增强的语言模型中的不确定性量化，本研究旨在填补检索增强语言模型不确定性量化方面的研究空白，发现检索语料库中包含的科学知识无法解决模型对预测结果过度自信的问题。

科学领域中基于检索增强语言模型的不确定性量化的实证评估

Empirical evaluation of Uncertainty Quantification in  Retrieval-Augmented Language Models for Science

We present a reality check on large language models and inspect the promise
of retrieval augmented language models in comparison. Such language models are
semi-parametric, where models integrate model parameters and knowledge from
external data sources to make their predictions, as opposed to the parametric
nature of vanilla large language models. We give initial experimental findings
that semi-parametric architectures can be enhanced with views, a query
analyzer/planner, and provenance to make a significantly more powerful system
for question answering in terms of accuracy and efficiency, and potentially for
other NLP tasks

该研究探讨了大型语言模型的实际效果，并对比了检索增强语言模型的潜力。研究发现，与普通的大型语言模型不同，半参数语言模型将来自外部数据源的模型参数和知识结合起来进行预测。该模型通过视图、查询分析器 / 规划器和来源信息的创新应用使问题回答系统在精度和效率方面显著增强，可能还可用于其他 NLP 任务。

重塑检索增强语言模型以回答查询

Reimagining Retrieval Augmented Language Models for Answering Queries

Retrieval augmented language models have recently become the standard for
knowledge intensive tasks. Rather than relying purely on latent semantics
within the parameters of large neural models, these methods enlist a
semi-parametric memory to encode an index of knowledge for the model to
retrieve over. Most prior work has employed text passages as the unit of
knowledge, which has high coverage at the cost of interpretability,
controllability, and efficiency. The opposite properties arise in other methods
which have instead relied on knowledge base (KB) facts. At the same time, more
recent work has demonstrated the effectiveness of storing and retrieving from
an index of Q-A pairs derived from text \citep{lewis2021paq}. This approach
yields a high coverage knowledge representation that maintains KB-like
properties due to its representations being more atomic units of information.
In this work we push this line of research further by proposing a
question-answer augmented encoder-decoder model and accompanying pretraining
strategy. This yields an end-to-end system that not only outperforms prior QA
retrieval methods on single-hop QA tasks but also enables compositional
reasoning, as demonstrated by strong performance on two multi-hop QA datasets.
Together, these methods improve the ability to interpret and control the model
while narrowing the performance gap with passage retrieval systems.

本文提出了一种以问答对为基础的编码 - 解码模型，通过该模型的预训练策略，能够在单跳及多跳问题上达到强大性能，并进一步提高了对模型的解释和控制能力，缩小了与段落检索系统之间的性能差距。