Given questions regarding some prototypical situation such as Name something
that people usually do before they leave the house for work? a human can easily
answer them via acquired experiences. There can be multiple right answers for
such questions, with some more common for a situation than others. This paper
introduces a new question answering dataset for training and evaluating common
sense reasoning capabilities of artificial intelligence systems in such
prototypical situations. The training set is gathered from an existing set of
questions played in a long-running international game show FAMILY- FEUD. The
hidden evaluation set is created by gathering answers for each question from
100 crowd-workers. We also propose a generative evaluation task where a model
has to output a ranked list of answers, ideally covering all prototypical
answers for a question. After presenting multiple competitive baseline models,
we find that human performance still exceeds model scores on all evaluation
metrics with a meaningful gap, supporting the challenging nature of the task.

本文介绍了一个新的问答数据集，用于训练和评估人工智能系统在典型情况下的常识推理能力，数据集采用了家庭智慧问答秀中的问题集合，模型评估使用了生成式评估任务，并在多个基线模型中提出了挑战性的性能评估。结果表明，人类表现仍然超过模型成绩，支持任务的挑战性。