Retrieval augmented models show promise in enhancing traditional language
models by improving their contextual understanding, integrating private data,
and reducing hallucination. However, the processing time required for retrieval
augmented large language models poses a challenge when applying them to tasks
that require real-time responses, such as composition assistance.
To overcome this limitation, we propose the Hybrid Retrieval-Augmented
Generation (HybridRAG) framework that leverages a hybrid setting that combines
both client and cloud models. HybridRAG incorporates retrieval-augmented memory
generated asynchronously by a Large Language Model (LLM) in the cloud. By
integrating this retrieval augmented memory, the client model acquires the
capability to generate highly effective responses, benefiting from the LLM's
capabilities. Furthermore, through asynchronous memory integration, the client
model is capable of delivering real-time responses to user requests without the
need to wait for memory synchronization from the cloud. Our experiments on
Wikitext and Pile subsets show that HybridRAG achieves lower latency than a
cloud-based retrieval-augmented LLM, while outperforming client-only models in
utility.

利用混合检索增强生成（HybridRAG）框架，在需要实时响应的任务（如作文帮助）中，将客户端模型与云端模型相结合，通过异步生成的检索增强记忆使客户端模型具备生成高效响应的能力，并实现实时响应无需等待云端内存同步，从而克服了检索增强大型语言模型处理时间长的局限性。实验证明，HybridRAG 在延迟方面优于基于云的检索增强大型语言模型，并在效用方面优于仅具备客户端的模型。

混合检索增强生成用于实时作曲协助

Hybrid Retrieval-Augmented Generation for Real-time Composition  Assistance

Retrieval augmented models are becoming increasingly popular for computer
vision tasks after their recent success in NLP problems. The goal is to enhance
the recognition capabilities of the model by retrieving similar examples for
the visual input from an external memory set. In this work, we introduce an
attention-based memory module, which learns the importance of each retrieved
example from the memory. Compared to existing approaches, our method removes
the influence of the irrelevant retrieved examples, and retains those that are
beneficial to the input query. We also thoroughly study various ways of
constructing the memory dataset. Our experiments show the benefit of using a
massive-scale memory dataset of 1B image-text pairs, and demonstrate the
performance of different memory representations. We evaluate our method in
three different classification tasks, namely long-tailed recognition, learning
with noisy labels, and fine-grained classification, and show that it achieves
state-of-the-art accuracies in ImageNet-LT, Places-LT and Webvision datasets.

研究使用检索增强模型来提高计算机视觉任务的识别能力，引入了一种基于注意力机制的记忆模块，学习从外部记忆集合中检索出的每个例子的重要性，证明了使用 10 亿个图像 - 文本对的大规模记忆数据集的好处，并在 ImageNet-LT，Places-LT 和 Webvision 数据集中取得了最先进的准确性。

从全网图片 - 文本数据中检索来提高图像识别

Improving Image Recognition by Retrieving from Web-Scale Image-Text Data

Large language models have shown impressive few-shot results on a wide range
of tasks. However, when knowledge is key for such results, as is the case for
tasks such as question answering and fact checking, massive parameter counts to
store knowledge seem to be needed. Retrieval augmented models are known to
excel at knowledge intensive tasks without the need for as many parameters, but
it is unclear whether they work in few-shot settings. In this work we present
Atlas, a carefully designed and pre-trained retrieval augmented language model
able to learn knowledge intensive tasks with very few training examples. We
perform evaluations on a wide range of tasks, including MMLU, KILT and
NaturalQuestions, and study the impact of the content of the document index,
showing that it can easily be updated. Notably, Atlas reaches over 42% accuracy
on Natural Questions using only 64 examples, outperforming a 540B parameters
model by 3% despite having 50x fewer parameters.

研究表明，在大多数任务中，大规模语言模型在少量样本上具有出色的表现，然而，当知识是关键时，如问题回答和事实检查等任务，需要使用大量参数存储知识。本文提出了 Atlas，一种精心设计和预训练的检索增强语言模型，能够利用极少的训练样本学习知识密集型任务。我们在多种任务中进行了评估，包括 MMLU、KILT 和自然问题，并研究了文本索引的内容对模型效果的影响，表明它可以很容易地进行更新。值得注意的是，Atlas 仅使用 64 个例子就能在自然问题上达到 42% 以上的精度，在性能上优于具有 50 倍参数的 540B 参数模型 3% 以上。