Neural document rerankers are extremely effective in terms of accuracy.
However, the best models require dedicated hardware for serving, which is
costly and often not feasible. To avoid this serving-time requirement, we
present a method of capturing up to 86% of the gains of a Transformer
cross-attention model with a lexicalized scoring function that only requires
10-6% of the Transformer's FLOPs per document and can be served using commodity
CPUs. When combined with a BM25 retriever, this approach matches the quality of
a state-of-the art dual encoder retriever, that still requires an accelerator
for query encoding. We introduce NAIL (Non-Autoregressive Indexing with
Language models) as a model architecture that is compatible with recent
encoder-decoder and decoder-only large language models, such as T5, GPT-3 and
PaLM. This model architecture can leverage existing pre-trained checkpoints and
can be fine-tuned for efficiently constructing document representations that do
not require neural processing of queries.

通过使用新型 scoring function 和结合 BM25 retriever 的方式，本研究证明可以使用传统 CPU 有效地构建文档排名。同时，NAIL 成为了一种可以利用预训练模型的、适用于构建文档表示的新型模型架构。