Current state-of-the-art document retrieval solutions mainly follow an
index-retrieve paradigm, where the index is hard to be directly optimized for
the final retrieval target. In this paper, we aim to show that an end-to-end
deep neural network unifying training and indexing stages can significantly
improve the recall performance of traditional methods. To this end, we propose
Neural Corpus Indexer (NCI), a sequence-to-sequence network that generates
relevant document identifiers directly for a designated query. To optimize the
recall performance of NCI, we invent a prefix-aware weight-adaptive decoder
architecture, and leverage tailored techniques including query generation,
semantic document identifiers, and consistency-based regularization. Empirical
studies demonstrated the superiority of NCI on two commonly used academic
benchmarks, achieving +21.4% and +16.8% relative enhancement for Recall@1 on
NQ320k dataset and R-Precision on TriviaQA dataset, respectively, compared to
the best baseline method.

本文研究介绍了一种基于神经网络领域的全端到端文档检索方法，通过采用特定的技术，包括查询生成、语义文档标识符和基于一致性的规则化，并引入前缀感知、权重适应解码器架构，NCI（神经语料库索引器）可以直接生成相关文档标识符以实现课题回忆的最佳表现。经实证研究，NCI 在两个常用的学术基准测试中，相对于最佳基线方法，在 NQ320k 数据集的 Recall@1 上实现了 + 21.4% 的相对增强，在 TriviaQA 数据集的 R-Precision 上实现了 + 16.8% 的相对增强。