Considering the widespread use of mobile and voice search, answer passage
retrieval for non-factoid questions plays a critical role in modern information
retrieval systems. Despite the importance of the task, the community still
feels the significant lack of large-scale non-factoid question answering
collections with real questions and comprehensive relevance judgments. In this
paper, we develop and release a collection of 2,626 open-domain non-factoid
questions from a diverse set of categories. The dataset, called ANTIQUE,
contains 34,011 manual relevance annotations. The questions were asked by real
users in a community question answering service, i.e., Yahoo! Answers.
Relevance judgments for all the answers to each question were collected through
crowdsourcing. To facilitate further research, we also include a brief analysis
of the data as well as baseline results on both classical and recently
developed neural IR models.

本文介绍了从雅虎问答社区收集的 2,626 个开放域非事实类问题以及 34,011 个评分注释的 ANTIQUE 数据集，旨在为进一步的研究提供基础，并在经典和最近开发的神经信息检索模型上提供基准结果。

ANTIQUE: 非事实型问题回答基准

ANTIQUE: A Non-Factoid Question Answering Benchmark

With the rise in mobile and voice search, answer passage retrieval acts as a
critical component of an effective information retrieval system for open domain
question answering. Currently, there are no comparable collections that address
non-factoid question answering within larger documents while simultaneously
providing enough examples sufficient to train a deep neural network. In this
paper, we introduce a new Wikipedia based collection specific for non-factoid
answer passage retrieval containing thousands of questions with annotated
answers and show benchmark results on a variety of state of the art neural
architectures and retrieval models. The experimental results demonstrate the
unique challenges presented by answer passage retrieval within topically
relevant documents for future research.

本文介绍了一个新的基于维基百科的、特定于非事实性答案段落检索的集合，其中包含成千上万个带注释的答案和问题，展示了各种先进神经架构和检索模型的基准结果，并展示了未来研究在主题相关文件中答案段落检索所面临的独特挑战。