Retrieval-based language models (LMs) have demonstrated improved
interpretability, factuality, and adaptability compared to their parametric
counterparts, by incorporating retrieved text from external datastores. While
it is well known that parametric models are prone to leaking private data, it
remains unclear how the addition of a retrieval datastore impacts model
privacy. In this work, we present the first study of privacy risks in
retrieval-based LMs, particularly $k$NN-LMs. Our goal is to explore the optimal
design and training procedure in domains where privacy is of concern, aiming to
strike a balance between utility and privacy. Crucially, we find that $k$NN-LMs
are more susceptible to leaking private information from their private
datastore than parametric models. We further explore mitigations of privacy
risks. When privacy information is targeted and readily detected in the text,
we find that a simple sanitization step would completely eliminate the risks,
while decoupling query and key encoders achieves an even better utility-privacy
trade-off. Otherwise, we consider strategies of mixing public and private data
in both datastore and encoder training. While these methods offer modest
improvements, they leave considerable room for future work. Together, our
findings provide insights for practitioners to better understand and mitigate
privacy risks in retrieval-based LMs. Our code is available at:
this https URL .

本文研究检测基于 k 最近邻的检索型语言模型对隐私的风险，探索隐私安全的最佳设计和培训程序以在效用和隐私之间取得平衡，并提供了隐私风险缓解策略。

检索式语言模型的隐私影响

Privacy Implications of Retrieval-Based Language Models

Retrieval-based language models are increasingly employed in
question-answering tasks. These models search in a corpus of documents for
relevant information instead of having all factual knowledge stored in its
parameters, thereby enhancing efficiency, transparency, and adaptability. We
develop the first Norwegian retrieval-based model by adapting the REALM
framework and evaluating it on various tasks. After training, we also separate
the language model, which we call the reader, from the retriever components,
and show that this can be fine-tuned on a range of downstream tasks. Results
show that retrieval augmented language modeling improves the reader's
performance on extractive question-answering, suggesting that this type of
training improves language models' general ability to use context and that this
does not happen at the expense of other abilities such as part-of-speech
tagging, dependency parsing, named entity recognition, and lemmatization. Code,
trained models, and data are made publicly available.

利用 REALM 框架进行适应性评估，开发了第一个挪威基于检索的语言模型，证明了检索增强语言建模可提高读者在摘要问答上的表现，并表明这种类型的训练不会以其他功能为代价。

BRENT: 双向检索增强挪威变压器

BRENT: Bidirectional Retrieval Enhanced Norwegian Transformer

Retrieval-based language models (R-LM) model the probability of natural
language text by combining a standard language model (LM) with examples
retrieved from an external datastore at test time. While effective, a major
bottleneck of using these models in practice is the computationally costly
datastore search, which can be performed as frequently as every time step. In
this paper, we present RetoMaton - retrieval automaton - which approximates the
datastore search, based on (1) saving pointers between consecutive datastore
entries, and (2) clustering of entries into "states". This effectively results
in a weighted finite automaton built on top of the datastore, instead of
representing the datastore as a flat list. The creation of the automaton is
unsupervised, and a RetoMaton can be constructed from any text collection:
either the original training corpus or from another domain. Traversing this
automaton at inference time, in parallel to the LM inference, reduces its
perplexity by up to 1.85, or alternatively saves up to 83% of the nearest
neighbor searches over $k$NN-LM (Khandelwal et al., 2020) without hurting
perplexity. Our code and trained models are available at
this https URL .

本文介绍了一种名为 RetoMaton 的检索自动机，它可以在测试时间对检索的时间成本进行近似处理，从而构建了一个基于加权有限自动机的数据存储库，通过在推理时间与 LM 推理并行遍历自动机，可以将其困惑度降低高达 1.85，或节省高达 83% 的最近邻搜索，而不会降低推理的准确性。

使用自动机增强的检索进行神经符号语言建模

Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval

Automatically inducing high quality knowledge graphs from a given collection
of documents still remains a challenging problem in AI. One way to make headway
for this problem is through advancements in a related task known as slot
filling. In this task, given an entity query in form of [Entity, Slot, ?], a
system is asked to fill the slot by generating or extracting the missing value
exploiting evidence extracted from relevant passage(s) in the given document
collection. The recent works in the field try to solve this task in an
end-to-end fashion using retrieval-based language models. In this paper, we
present a novel approach to zero-shot slot filling that extends dense passage
retrieval with hard negatives and robust training procedures for retrieval
augmented generation models. Our model reports large improvements on both T-REx
and zsRE slot filling datasets, improving both passage retrieval and slot value
generation, and ranking at the top-1 position in the KILT leaderboard.
Moreover, we demonstrate the robustness of our system showing its domain
adaptation capability on a new variant of the TACRED dataset for slot filling,
through a combination of zero/few-shot learning. We release the source code and
pre-trained models.

本文提出了一种使用 hard negatives 和强健的训练程序增强检索生成模型而进行零样本槽填充的新方法，该方法在 T-REx 和 zsRE 槽填充数据集上都有较大的改进，并在 KILT 排行榜上排名首位。此外，通过零 / 少样本学习的组合，我们在 TACRED 数据集的新变体上演示了我们系统的适应能力，并发布了源代码和预训练模型。