Transformer-based Cross-Encoders achieve state-of-the-art effectiveness in
text retrieval. However, Cross-Encoders based on large transformer models (such
as BERT or T5) are computationally expensive and allow for scoring only a small
number of documents within a reasonably small latency window. However, keeping
search latencies low is important for user satisfaction and energy usage. In
this paper, we show that weaker shallow transformer models (i.e., transformers
with a limited number of layers) actually perform better than full-scale models
when constrained to these practical low-latency settings since they can
estimate the relevance of more documents in the same time budget. We further
show that shallow transformers may benefit from the generalized Binary
Cross-Entropy (gBCE) training scheme, which has recently demonstrated success
for recommendation tasks. Our experiments with TREC Deep Learning passage
ranking query sets demonstrate significant improvements in shallow and
full-scale models in low-latency scenarios. For example, when the latency limit
is 25ms per query, MonoBERT-Large (a cross-encoder based on a full-scale BERT
model) is only able to achieve NDCG@10 of 0.431 on TREC DL 2019, while
TinyBERT-gBCE (a cross-encoder based on TinyBERT trained with gBCE) reaches
NDCG@10 of 0.652, a +51% gain over MonoBERT-Large. We also show that shallow
Cross-Encoders are effective even when used without a GPU (e.g., with CPU
inference, NDCG@10 decreases only by 3% compared to GPU inference with 50ms
latency), which makes Cross-Encoders practical to run even without specialized
hardware acceleration.

我们展示了在低延迟情境下，基于较浅的变形器模型（即具有有限层数的变形器）的交叉编码器实际上比全规模模型表现更好，因为它们可以在相同的时间预算内估计更多文档的相关性。我们进一步展示，较浅的变形器模型可能受益于目前成功应用于推荐任务的广义二进制交叉熵 (gBCE) 训练方案。我们在 TREC 深度学习通过排序查询集上的实验中，展示了低延迟环境下较浅和全规模模型的显着改进。