Numerical reasoning is often important to accurately understand the world.
Recently, several format-specific datasets have been proposed, such as
numerical reasoning in the settings of Natural Language Inference (NLI),
Reading Comprehension (RC), and Question Answering (QA). Several
format-specific models and architectures in response to those datasets have
also been proposed. However, there exists a strong need for a benchmark which
can evaluate the abilities of models, in performing question format independent
numerical reasoning, as (i) the numerical reasoning capabilities we want to
teach are not controlled by question formats, (ii) for numerical reasoning
technology to have the best possible application, it must be able to process
language and reason in a way that is not exclusive to a single format, task,
dataset or domain. In pursuit of this goal, we introduce NUMBERGAME, a
multifaceted benchmark to evaluate model performance across numerical reasoning
tasks of eight diverse formats. We add four existing question types in our
compilation. Two of the new types we add are about questions that require
external numerical knowledge, commonsense knowledge and domain knowledge. For
building a more practical numerical reasoning system, NUMBERGAME demands four
capabilities beyond numerical reasoning: (i) detecting question format directly
from data (ii) finding intermediate common format to which every format can be
converted (iii) incorporating commonsense knowledge (iv) handling data
imbalance across formats. We build several baselines, including a new model
based on knowledge hunting using a cheatsheet. However, all baselines perform
poorly in contrast to the human baselines, indicating the hardness of our
benchmark. Our work takes forward the recent progress in generic system
development, demonstrating the scope of these under-explored tasks.

介绍了一个多方面的基准测试 NUMBERGAME 来评估八种不同形式的数值推理任务的模型绩效，并需具有探测数据中的问题格式，查找中间常见格式，加入常识知识和解决不同格式数据不平衡的能力。同时，构建了多个基线模型，包括一个基于作弊纸的知识狩猎模型。但所有基线模型都表现较差，说明了该基准测试的难度。

朝向与问题格式无关的数值推理：一套必备任务

Towards Question Format Independent Numerical Reasoning: A Set of  Prerequisite Tasks

We introduce an automatic system that achieves state-of-the-art results on
the Winograd Schema Challenge (WSC), a common sense reasoning task that
requires diverse, complex forms of inference and knowledge. Our method uses a
knowledge hunting module to gather text from the web, which serves as evidence
for candidate problem resolutions. Given an input problem, our system generates
relevant queries to send to a search engine, then extracts and classifies
knowledge from the returned results and weighs them to make a resolution. Our
approach improves F1 performance on the full WSC by 0.21 over the previous best
and represents the first system to exceed 0.5 F1. We further demonstrate that
the approach is competitive on the Choice of Plausible Alternatives (COPA)
task, which suggests that it is generally applicable.

本文介绍了一个自动系统，通过知识搜寻模块从网络上获取证据，利用搜索引擎返回的结果进行知识抽取和分类，对 Winograd Schema Challenge (WSC) 等常识推理任务进行求解，相比最佳方法提高了 0.21 的 F1 值，在 COPA 任务中也具有竞争力。