Recent efforts in natural language processing (NLP) commonsense reasoning
research have yielded a considerable number of new datasets and benchmarks.
However, most of these datasets formulate commonsense reasoning challenges in
artificial scenarios that are not reflective of the tasks which real-world NLP
systems are designed to solve. In this work, we present CRoW, a
manually-curated, multi-task benchmark that evaluates the ability of models to
apply commonsense reasoning in the context of six real-world NLP tasks. CRoW is
constructed using a multi-stage data collection pipeline that rewrites examples
from existing datasets using commonsense-violating perturbations. We use CRoW
to study how NLP systems perform across different dimensions of commonsense
knowledge, such as physical, temporal, and social reasoning. We find a
significant performance gap when NLP systems are evaluated on CRoW compared to
humans, showcasing that commonsense reasoning is far from being solved in
real-world task settings. We make our dataset and leaderboard available to the
research community at this https URL

近期在自然语言处理 (NLP) 常识推理研究领域，已经取得了大量新的数据集和基准（benchmark）。然而，这些数据集中的大部分在人工场景中构建常识推理挑战，不反映真实世界 NLP 系统所设计用于解决的任务。本文提出了 CRoW，这是一个手动筛选的多任务基准，用于评估模型在六个真实世界 NLP 任务中应用常识推理的能力。CRoW 使用多阶段数据收集流程构建，通过常识违规扰动重写现有数据集的样例。我们使用 CRoW 研究 NLP 系统在常识知识的不同维度（如物理、时间和社交推理）上的表现。我们发现，在 CRoW 上评估 NLP 系统与人类相比存在显著的性能差距，这表明在真实任务环境中，常识推理远未解决。我们将数据集和排行榜提供给研究社区。