We present $\texttt{ACCORD}$, a framework and benchmark suite for
disentangling the commonsense grounding and reasoning abilities of large
language models (LLMs) through controlled, multi-hop counterfactuals.
$\texttt{ACCORD}$ introduces formal elements to commonsense reasoning to
explicitly control and quantify reasoning complexity beyond the typical 1 or 2
hops. Uniquely, $\texttt{ACCORD}$ can automatically generate benchmarks of
arbitrary reasoning complexity, and so it scales with future LLM improvements.
Benchmarking state-of-the-art LLMs -- including GPT-4o (2024-05-13),
Llama-3-70B-Instruct, and Mixtral-8x22B-Instruct-v0.1 -- shows performance
degrading to random chance with only moderate scaling, leaving substantial
headroom for improvement. We release a leaderboard of the benchmark suite
tested in this work, as well as code for automatically generating more complex
benchmarks.

通过受控的多跳反事实，我们提出了一个框架和基准套件 $	exttt {ACCORD}$，用于解开大规模语言模型（LLMs）的常识基础和推理能力。通过引入形式化元素到常识推理中，$	exttt {ACCORD}$ 可以明确控制和量化推理复杂性，超越典型的 1 或 2 跳。基准测试最先进的 LLMs（包括 GPT-4o（2024-05-13）、Llama-3-70B-Instruct 和 Mixtral-8x22B-Instruct-v0.1）显示性能仅随着适度扩展而降至随机机会，为改进留下了充足的余地。我们公布了此工作中测试的基准套件的排行榜，以及用于自动生成更复杂基准测试的代码。