Programming machines with commonsense reasoning (CSR) abilities is a
longstanding challenge in the Artificial Intelligence community. Current CSR
benchmarks use multiple-choice (and in relatively fewer cases, generative)
question-answering instances to evaluate machine commonsense. Recent progress
in transformer-based language representation models suggest that considerable
progress has been made on existing benchmarks. However, although tens of CSR
benchmarks currently exist, and are growing, it is not evident that the full
suite of commonsense capabilities have been systematically evaluated.
Furthermore, there are doubts about whether language models are 'fitting' to a
benchmark dataset's training partition by picking up on subtle, but normatively
irrelevant (at least for CSR), statistical features to achieve good performance
on the testing partition. To address these challenges, we propose a benchmark
called Theoretically-Grounded Commonsense Reasoning (TG-CSR) that is also based
on discriminative question answering, but with questions designed to evaluate
diverse aspects of commonsense, such as space, time, and world states. TG-CSR
is based on a subset of commonsense categories first proposed as a viable
theory of commonsense by Gordon and Hobbs. The benchmark is also designed to be
few-shot (and in the future, zero-shot), with only a few training and
validation examples provided. This report discusses the structure and
construction of the benchmark. Preliminary results suggest that the benchmark
is challenging even for advanced language representation models designed for
discriminative CSR question answering tasks.
Benchmark access and leaderboard:
this https URL Benchmark website:
this https URL

本文提出了一个名为 TG-CSR 的具有理论基础的通识常识推理基准，该基准旨在评估通识常识的各种方面，并基于少量的训练和验证示例进行学习。通过初步的测试结果表明，该基准对于设计用于推理性 CSR 问题答案的先进语言表示模型来说是具有挑战性的。