The computing cost of transformer self-attention often necessitates breaking
long documents to fit in pretrained models in document ranking tasks. In this
paper, we design Query-Directed Sparse attention that induces IR-axiomatic
structures in transformer self-attention. Our model, QDS-Transformer, enforces
the principle properties desired in ranking: local contextualization,
hierarchical representation, and query-oriented proximity matching, while it
also enjoys efficiency from sparsity. Experiments on one fully supervised and
three few-shot TREC document ranking benchmarks demonstrate the consistent and
robust advantage of QDS-Transformer over previous approaches, as they either
retrofit long documents into BERT or use sparse attention without emphasizing
IR principles. We further quantify the computing complexity and demonstrates
that our sparse attention with TVM implementation is twice more efficient than
the fully-connected self-attention. All source codes, trained model, and
predictions of this work are available at
this https URL

本文介绍了一种名为 QDS-Transformer 的算法，它在 Transformer 自注意力机制的基础上，设计了稀疏的查询导向注意力机制，以在文档排序任务中实现本地化背景、分层表示和查询定向的邻近匹配，同时还具有稀疏性和计算效率。实验证明，QDS-Transformer 模型在全监督和少样本 TREC 排名基准中具有稳定和强大的优势。