Embedding-based Retrieval Models (ERMs) have emerged as a promising framework
for large-scale text retrieval problems due to powerful large language models.
Nevertheless, fine-tuning ERMs to reach state-of-the-art results can be
expensive due to the extreme scale of data as well as the complexity of
multi-stages pipelines (e.g., pre-training, fine-tuning, distillation). In this
work, we propose the PEFA framework, namely ParamEter-Free Adapters, for fast
tuning of ERMs without any backward pass in the optimization. At index building
stage, PEFA equips the ERM with a non-parametric k-nearest neighbor (kNN)
component. At inference stage, PEFA performs a convex combination of two
scoring functions, one from the ERM and the other from the kNN. Based on the
neighborhood definition, PEFA framework induces two realizations, namely
PEFA-XL (i.e., extra large) using double ANN indices and PEFA-XS (i.e., extra
small) using a single ANN index. Empirically, PEFA achieves significant
improvement on two retrieval applications. For document retrieval, regarding
Recall@100 metric, PEFA improves not only pre-trained ERMs on Trivia-QA by an
average of 13.2%, but also fine-tuned ERMs on NQ-320K by an average of 5.5%,
respectively. For product search, PEFA improves the Recall@100 of the
fine-tuned ERMs by an average of 5.3% and 14.5%, for PEFA-XS and PEFA-XL,
respectively. Our code is available at this https URL
amzn/pecos/tree/mainline/examples/pefa-wsdm24

基于嵌入的检索模型 (ERMs) 在大规模文本检索问题中作为一个有前途的框架出现，但是由于数据的规模庞大以及多阶段流水线（如预训练、微调、蒸馏）的复杂性，调整 ERMs 达到最先进的结果成本较高。在这篇论文中，我们提出了 PEFA 框架，即 ParamEter-Free Adapters，用于快速调整 ERMs 而无需进行后向传递的优化。在索引构建阶段，PEFA 为 ERM 添加了一个非参数的 k 近邻 (kNN) 组件。在推理阶段，PEFA 对两个得分函数进行凸组合，一个来自 ERM，另一个来自 kNN。基于邻域定义，PEFA 框架产生了两个实现，即 PEFA-XL (即额外大) 和 PEFA-XS (即额外小)，分别使用双 ANN 索引和单 ANN 索引。凭经验证明，PEFA 在两个检索应用中取得了显著的改进。对于文档检索，关于 Recall@100 指标，PEFA 改进了对 Trivia-QA 的预训练 ERMs 平均为 13.2%，对 NQ-320K 的微调 ERMs 平均为 5.5%。对于产品搜索，PEFA-XS 和 PEFA-XL 分别平均提高了微调 ERMs 的 Recall@100 指标 5.3% 和 14.5%。我们的代码可以在此 https URL 找到：amzn/pecos/tree/mainline/examples/pefa-wsdm24。