Commonsense reasoning is intuitive for humans but has been a long-term
challenge for artificial intelligence (AI). Recent advancements in pretrained
language models have shown promising results on several commonsense benchmark
datasets. However, the reliability and comprehensiveness of these benchmarks
towards assessing model's commonsense reasoning ability remains unclear. To
this end, we introduce a new commonsense reasoning benchmark dataset comprising
natural language true/false statements, with each sample paired with its
complementary counterpart, resulting in 4k sentence pairs. We propose a
pairwise accuracy metric to reliably measure an agent's ability to perform
commonsense reasoning over a given situation. The dataset is crowdsourced and
enhanced with an adversarial model-in-the-loop setup to incentivize challenging
samples. To facilitate a systematic analysis of commonsense capabilities, we
design our dataset along the dimensions of knowledge domains, reasoning
scenarios and numeracy. Experimental results demonstrate that our strongest
baseline (UnifiedQA-3B), after fine-tuning, achieves ~71% standard accuracy and
~51% pairwise accuracy, well below human performance (~95% for both metrics).
The dataset is available at this https URL

本文旨在探讨预训练语言模型在常识推理数据集中的可靠性和全面性，并提出了一个新的包含自然语言真 / 假语句对的常识推理基准数据集，通过对不同维度的领域的知识、推理场景以及数学的设计来促进常识能力的系统分析，并设计了成对精度度量标准来可靠地衡量代理的常识推理能力。实验结果表明，我们最强的基线模型在微调后达到了约 71％的标准准确性和约 51％的成对准确性，远低于人类表现。