Recent advances in large language models have demonstrated remarkable effectiveness in information retrieval (IR) tasks. While many neural IR systems encode queries and documents into single-vector representations, multi-vector models elevate the retrieval quality by producing multi-vector representations and facilitating similarity searches at the granularity of individual tokens. However, these models significantly amplify memory and storage requirements for retrieval indices by an order of magnitude. This escalation in index size renders the scalability of multi-vector IR models progressively challenging due to their substantial memory demands. We introduce Embedding from Storage Pipelined Network (ESPN) where we offload the entire re-ranking embedding tables to SSDs and reduce the memory requirements by 5-16x. We design a software prefetcher with hit rates exceeding 90%, improving SSD based retrieval up to 6.4x, and demonstrate that we can maintain near memory levels of query latency even for large query batch sizes.

最近大规模语言模型在信息检索任务中显示出了显著的有效性。然而，多向量模型在检索索引的存储和内存方面需要更大的需求，这使得多向量IR模型的可扩展性逐渐面临挑战。本文介绍了一种名为ESPN的存储引入的嵌入式网络模型，通过将整个重排序嵌入表存储在SSD中并减少内存需求5-16倍。我们设计了一个软件预取器，其命中率超过90%，从而将基于SSD的检索性能提高了最高达6.4倍，并且在大查询批处理的情况下能够保持接近内存级别的查询延迟。

ESPN：高效内存多向量信息检索