While pre-trained language models achieve impressive performance on various
NLP benchmarks, they still struggle with tasks that require numerical
reasoning. Recent advances in improving numerical reasoning are mostly achieved
using very large language models that contain billions of parameters and are
not accessible to everyone. In addition, numerical reasoning is measured using
a single score on existing datasets. As a result, we do not have a clear
understanding of the strengths and shortcomings of existing models on different
numerical reasoning aspects and therefore, potential ways to improve them apart
from scaling them up. Inspired by CheckList (Ribeiro et al., 2020), we
introduce a multi-view evaluation set for numerical reasoning in English,
called FERMAT. Instead of reporting a single score on a whole dataset, FERMAT
evaluates models on various key numerical reasoning aspects such as number
understanding, mathematical operations, and training dependency. Apart from
providing a comprehensive evaluation of models on different numerical reasoning
aspects, FERMAT enables a systematic and automated generation of an arbitrarily
large training or evaluation set for each aspect.The datasets and codes are
publicly available to generate further multi-view data for ulterior tasks and
languages.

文章介绍了 FERMAT（英语数值推理的多维视角评估集），它通过对数值理解、数学操作和训练依赖等关键数值推理方面进行评估，提供了关于已有模型在不同数值推理方面的优势和劣势的全面评估，并能够系统化自动生成任意大规模的训练或评估集。