Fuzzy reasoning is vital due to the frequent use of imprecise information in
daily contexts. However, the ability of current large language models (LLMs) to
handle such reasoning remains largely uncharted. In this paper, we introduce a
new benchmark, FRoG, for fuzzy reasoning, featuring real-world mathematical
word problems that incorporate generalized quantifiers. Our experimental
findings reveal that fuzzy reasoning continues to pose significant challenges
for LLMs. Moreover, we find that existing methods designed to enhance reasoning
do not consistently improve performance in tasks involving fuzzy logic.
Additionally, our results show an inverse scaling effect in the performance of
LLMs on FRoG. Interestingly, we also demonstrate that strong mathematical
reasoning skills are not necessarily indicative of success on our benchmark.

模糊推理在日常环境中对于使用不精确信息至关重要，本文引入了一个新的基准 FRoG，用于对模糊推理进行评估，实验结果表明大型语言模型在处理模糊推理方面仍面临重大挑战，并且现有的增强推理方法在涉及模糊逻辑的任务中并不一致地提高性能。此外，我们的结果显示了大型语言模型在 FRoG 上的逆比例缩放效应，有趣的是，我们还证明了强大的数学推理技能并不一定意味着在我们的基准测试中取得成功。

FRoG：对大型语言模型中广义量词的模糊推理进行评估

FRoG: Evaluating Fuzzy Reasoning of Generalized Quantifiers in Large  Language Models

From the latter half of the last decade, there has been a growing interest in
developing algorithms for automatically solving mathematical word problems
(MWP). It is a challenging and unique task that demands blending surface level
text pattern recognition with mathematical reasoning. In spite of extensive
research, we are still miles away from building robust representations of
elementary math word problems and effective solutions for the general task. In
this paper, we critically examine the various models that have been developed
for solving word problems, their pros and cons and the challenges ahead. In the
last two years, a lot of deep learning models have recorded competing results
on benchmark datasets, making a critical and conceptual analysis of literature
highly useful at this juncture. We take a step back and analyse why, in spite
of this abundance in scholarly interest, the predominantly used experiment and
dataset designs continue to be a stumbling block. From the vantage point of
having analyzed the literature closely, we also endeavour to provide a road-map
for future math word problem research.

本研究论文从近十年开始，对自动解决数学单词问题的算法进行了批判性评估和未来研究规划，主要关键词为数学单词问题、算法、深度学习模型、数据集设计和研究。

自然语言处理模型在初等数学中为何犯错？ 基于深度学习的解决数学应用题的调查

Why are NLP Models Fumbling at Elementary Math? A Survey of Deep Learning based Word Problem Solvers

Solving mathematical word problems (MWPs) automatically is challenging,
primarily due to the semantic gap between human-readable words and
machine-understandable logics. Despite the long history dated back to the1960s,
MWPs have regained intensive attention in the past few years with the
advancement of Artificial Intelligence (AI). Solving MWPs successfully is
considered as a milestone towards general AI. Many systems have claimed
promising results in self-crafted and small-scale datasets. However, when
applied on large and diverse datasets, none of the proposed methods in the
literature achieves high precision, revealing that current MWP solvers still
have much room for improvement. This motivated us to present a comprehensive
survey to deliver a clear and complete picture of automatic math problem
solvers. In this survey, we emphasize on algebraic word problems, summarize
their extracted features and proposed techniques to bridge the semantic gap and
compare their performance in the publicly accessible datasets. We also cover
automatic solvers for other types of math problems such as geometric problems
that require the understanding of diagrams. Finally, we identify several
emerging research directions for the readers with interests in MWPs.

该研究探讨了自动解决数学问题的挑战，包括语言和逻辑之间的语义鸿沟，综述了将人工智能用于代数和几何问题的主要技术及其性能，并探讨未来的研究方向。