Time is an important dimension in our physical world. Lots of facts can
evolve with respect to time. For example, the U.S. President might change every
four years. Therefore, it is important to consider the time dimension and
empower the existing QA models to reason over time. However, the existing QA
datasets contain rather few time-sensitive questions, hence not suitable for
diagnosing or benchmarking the model's temporal reasoning capability. In order
to promote research in this direction, we propose to construct a time-sensitive
QA dataset. The dataset is constructed by 1) mining time-evolving facts from
WikiData and aligning them to their corresponding Wikipedia page, 2) employing
crowd workers to verify and calibrate these noisy facts, 3) generating
question-answer pairs based on the annotated time-sensitive facts. Our dataset
poses challenges in the aspect of both temporal understanding and temporal
reasoning. We evaluate different SoTA long-document QA systems like BigBird and
FiD on our dataset. The best-performing model FiD can only achieve 46\%
accuracy, still far behind the human performance of 87\%. We demonstrate that
these models are still lacking the ability to perform consistent temporal
reasoning. Therefore, we believe that our dataset could serve as a benchmark to
develop NLP models more sensitive to temporal shifts. The dataset and code are
released in~https://github.com/wenhuchen/Time-Sensitive-QA.

本文提出了构建一种基于时间敏感的问答数据集，用于评估长文档 QA 系统的时间推理能力，针对现有的 QA 数据集缺乏时序问题的问题，展示出当前现有的 SoTA QA 系统（如 BigBird 和 FiD）对于时间推理的能力仍不完备，在此基础上提出将该数据集作为开发更加敏感于时间推理的 NLP 模型的基准测试。