In many Natural Language Processing applications, neural networks have been
found to fail to generalize on out-of-distribution examples. In particular,
several recent semantic parsing datasets have put forward important limitations
of neural networks in cases where compositional generalization is required. In
this work, we extend a neural graph-based semantic parsing framework in several
ways to alleviate this issue. Notably, we propose: (1) the introduction of a
supertagging step with valency constraints, expressed as an integer linear
program; (2) a reduction of the graph prediction problem to the maximum
matching problem; (3) the design of an incremental early-stopping training
strategy to prevent overfitting. Experimentally, our approach significantly
improves results on examples that require structural generalization in the COGS
dataset, a known challenging benchmark for compositional generalization.
Overall, our results confirm that structural constraints are important for
generalization in semantic parsing.

在许多自然语言处理应用中，神经网络在处理超出分布范围的示例时被发现无法进行泛化。本文通过多种方法对神经图形语义解析框架进行扩展以缓解这个问题，实验结果表明结构性约束对语义解析中的泛化是重要的。

COGS 中的结构性泛化：超标记化几乎是你所需的全部

Structural generalization in COGS: Supertagging is (almost) all you need

Whether Transformers can learn to apply symbolic rules and generalise to
out-of-distribution examples is an open research question. In this paper, we
devise a data generation method for producing intricate mathematical
derivations, and systematically perturb them with respect to syntax, structure,
and semantics. Our task-agnostic approach generates equations, annotations, and
inter-equation dependencies, employing symbolic algebra for scalable data
production and augmentation. We then instantiate a general experimental
framework on next-equation prediction, assessing systematic mathematical
reasoning and generalisation of Transformer encoders on a total of 200K
examples. The experiments reveal that perturbations heavily affect performance
and can reduce F1 scores of $97\%$ to below $17\%$, suggesting that inference
is dominated by surface-level patterns unrelated to a deeper understanding of
mathematical operators. These findings underscore the importance of rigorous,
large-scale evaluation frameworks for revealing fundamental limitations of
existing models.

使用数据生成方法和符号代数，对变换器编码器的数学推理和泛化进行了 200,000 个例子的超大规模实验，并回答了变换器是否可以学习应用符号规则并推广到超出分布范围的例子。结果表明，扰动严重影响性能，并且可以将 F1 分数降至低于 17％，这表明推理主要由与数学运算的深层理解无关的表层模式所支配。