The problem of designing NLP solvers for math word problems (MWP) has seen
sustained research activity and steady gains in the test accuracy. Since
existing solvers achieve high performance on the benchmark datasets for
elementary level MWPs containing one-unknown arithmetic word problems, such
problems are often considered "solved" with the bulk of research attention
moving to more complex MWPs. In this paper, we restrict our attention to
English MWPs taught in grades four and lower. We provide strong evidence that
the existing MWP solvers rely on shallow heuristics to achieve high performance
on the benchmark datasets. To this end, we show that MWP solvers that do not
have access to the question asked in the MWP can still solve a large fraction
of MWPs. Similarly, models that treat MWPs as bag-of-words can also achieve
surprisingly high accuracy. Further, we introduce a challenge dataset, SVAMP,
created by applying carefully chosen variations over examples sampled from
existing datasets. The best accuracy achieved by state-of-the-art models is
substantially lower on SVAMP, thus showing that much remains to be done even
for the simplest of the MWPs.

该研究分析了现有的 NLP 求解器在解决低年级英语数学问题时的表现，指出现有求解器主要依赖于表面浅显的启发式策略。同时，研究提出了一个挑战数据集 SVAMP，并证明当前最优模型的表现还有很大的提升空间。