Retrieval-augmented language models (RaLM) have demonstrated the potential to
solve knowledge-intensive natural language processing (NLP) tasks by combining
a non-parametric knowledge base with a parametric language model. Instead of
fine-tuning a fully parametric model, RaLM excels at its low-cost adaptation to
the latest data and better source attribution mechanisms. Among various RaLM
approaches, iterative RaLM delivers a better generation quality due to a more
frequent interaction between the retriever and the language model. Despite the
benefits, iterative RaLM usually encounters high overheads due to the frequent
retrieval step. To this end, we propose RaLMSpec, a speculation-inspired
framework that provides generic speed-up over iterative RaLM while preserving
the same model outputs through speculative retrieval and batched verification.
By further incorporating prefetching, optimal speculation stride scheduler, and
asynchronous verification, RaLMSpec can automatically exploit the acceleration
potential to the fullest. For naive iterative RaLM serving, extensive
evaluations over three language models on four downstream QA datasets
demonstrate that RaLMSpec can achieve a speed-up ratio of 1.75-2.39x,
1.04-1.39x, and 1.31-1.77x when the retriever is an exact dense retriever,
approximate dense retriever, and sparse retriever respectively compared with
the baseline. For KNN-LM serving, RaLMSpec can achieve a speed-up ratio up to
7.59x and 2.45x when the retriever is an exact dense retriever and approximate
dense retriever, respectively, compared with the baseline.

通过猜测的检索和批量验证，RaLMSpec 提供了对迭代式 RaLM 的加速，保持了相同的模型输出。在多个评估上，RaLMSpec 相对于基准模型可以实现 1.75-2.39 倍的加速比。