Answers to the same question may change depending on the extra-linguistic
contexts (when and where the question was asked). To study this challenge, we
introduce SituatedQA, an open-retrieval QA dataset where systems must produce
the correct answer to a question given the temporal or geographical context. To
construct SituatedQA, we first identify such questions in existing QA datasets.
We find that a significant proportion of information seeking questions have
context-dependent answers (e.g., roughly 16.5% of NQ-Open). For such
context-dependent questions, we then crowdsource alternative contexts and their
corresponding answers. Our study shows that existing models struggle with
producing answers that are frequently updated or from uncommon locations. We
further quantify how existing models, which are trained on data collected in
the past, fail to generalize to answering questions asked in the present, even
when provided with an updated evidence corpus (a roughly 15 point drop in
accuracy). Our analysis suggests that open-retrieval QA benchmarks should
incorporate extra-linguistic context to stay relevant globally and in the
future. Our data, code, and datasheet are available at
this https URL .

本研究提出了一个开放式检索问答数据集 SituatedQA，旨在研究在不同时间和地点提出相同问题的语境下，答案是否会发生变化。通过构建该数据集并对现有模型进行实验，发现模型对于更新频繁或少见地点的答案产生困难，同时模型对于过去采集的数据无法在新问答中适应，因此建议将 extra-linguistic context 纳入开放式检索 QA 评估标准中。