In the era of large language models, applying techniques such as Retrieval
Augmented Generation can better address Open-Domain Question-Answering
problems. Due to constraints including model sizes and computing resources, the
length of context is often limited, and it becomes challenging to empower the
model to cover overlong contexts while answering questions from open domains.
This paper proposes a general and convenient method to covering longer contexts
in Open-Domain Question-Answering tasks. It leverages a small encoder language
model that effectively encodes contexts, and the encoding applies
cross-attention with origin inputs. With our method, the origin language models
can cover several times longer contexts while keeping the computing
requirements close to the baseline. Our experiments demonstrate that after
fine-tuning, there is improved performance across two held-in datasets, four
held-out datasets, and also in two In Context Learning settings.

在大语言模型的时代，应用检索增强生成等技术可以更好地解决开放领域问题回答问题。本文提出了一种通用方便的方法来覆盖更长的上下文，以在开放领域的问答任务中应用。我们的实验证明，在精调后，在两个保留数据集、四个留存数据集和两个上下文学习设置中，性能得到了提高。