Retrieval-based augmentations that aim to incorporate knowledge from an
external database into language models have achieved great success in various
knowledge-intensive (KI) tasks, such as question-answering and text generation.
However, integrating retrievals in non-knowledge-intensive (NKI) tasks, such as
text classification, is still challenging. Existing works focus on
concatenating retrievals to inputs as context to form the prompt-based inputs.
Unfortunately, such methods require language models to have the capability to
handle long texts. Besides, inferring such concatenated data would also consume
a significant amount of computational resources.
To solve these challenges, we propose \textbf{ReFusion} in this paper, a
computation-efficient \textbf{Re}trieval representation \textbf{Fusion} with
neural architecture search. The main idea is to directly fuse the retrieval
representations into the language models. Specifically, we first propose an
online retrieval module that retrieves representations of similar sentences.
Then, we present a retrieval fusion module including two effective ranking
schemes, i.e., reranker-based scheme and ordered-mask-based scheme, to fuse the
retrieval representations with hidden states. Furthermore, we use Neural
Architecture Search (NAS) to seek the optimal fusion structure across different
layers. Finally, we conduct comprehensive experiments, and the results
demonstrate our ReFusion can achieve superior and robust performance on various
NKI tasks.

在本文中，我们提出了一种计算高效的检索表示融合方法 ReFusion，通过神经架构搜索，直接将检索表示融入到语言模型中，来解决将外部数据库中的知识纳入到非知识密集型任务中的挑战。实验证明，我们的 ReFusion 在各种非知识密集型任务上实现了卓越且鲁棒的性能。

提升自然语言理解的计算高效检索表示融合

Improving Natural Language Understanding with Computation-Efficient  Retrieval Representation Fusion

Retrieval-augmented methods have received increasing attention to support
downstream tasks by leveraging useful information from external resources.
Recent studies mainly focus on exploring retrieval to solve knowledge-intensive
(KI) tasks. However, the potential of retrieval for most
non-knowledge-intensive (NKI) tasks remains under-explored. There are two main
challenges to leveraging retrieval-augmented methods for NKI tasks: 1) the
demand for diverse relevance score functions and 2) the dilemma between
training cost and task performance. To address these challenges, we propose a
two-stage framework for NKI tasks, named PGRA. In the first stage, we adopt a
task-agnostic retriever to build a shared static index and select candidate
evidence efficiently. In the second stage, we design a prompt-guided reranker
to rerank the nearest evidence according to task-specific relevance for the
reader. Experimental results show that PGRA outperforms other state-of-the-art
retrieval-augmented methods. Our analyses further investigate the influence
factors to model performance and demonstrate the generality of PGRA. Codes are
available at this https URL

该研究提出了一种名为 PGRA 的两阶段框架，旨在解决利用检索增强方法提高大多数非知识密集型任务表现所面临的两个主要挑战：多样的相关性评分函数和训练成本与任务性能之间的两难境地。PGRA 通过采用面向任务的检索器建立共享的静态索引并高效选择候选证据，并设计了一种基于提示的重新排序算法来根据读者的任务特定相关性重新排序最近的证据，实验结果表明，PGRA 优于其他最先进的检索增强方法。