Compositional generalization benchmarks seek to assess whether models can
accurately compute meanings for novel sentences, but operationalize this in
terms of logical form (LF) prediction. This raises the concern that
semantically irrelevant details of the chosen LFs could shape model
performance. We argue that this concern is realized for the COGS benchmark (Kim
and Linzen, 2020). COGS poses generalization splits that appear impossible for
present-day models, which could be taken as an indictment of those models.
However, we show that the negative results trace to incidental features of COGS
LFs. Converting these LFs to semantically equivalent ones and factoring out
capabilities unrelated to semantic interpretation, we find that even baseline
models get traction. A recent variable-free translation of COGS LFs suggests
similar conclusions, but we observe this format is not semantically equivalent;
it is incapable of accurately representing some COGS meanings. These findings
inform our proposal for ReCOGS, a modified version of COGS that comes closer to
assessing the target semantic capabilities while remaining very challenging.
Overall, our results reaffirm the importance of compositional generalization
and careful benchmark task design.

文章讨论了 COGS 基准测试对句子含义计算的评估方式以及其可能固有的局限性，并提出了一种更具挑战性的改进版本 ReCOGS，以更准确地评估模型的语义能力。

ReCOGS：逻辑形式的附带细节如何 overshadow 语义解释的评估

ReCOGS: How Incidental Details of a Logical Form Overshadow an Evaluation of Semantic Interpretation

If two sentences have the same meaning, it should follow that they are
equivalent in their inferential properties, i.e., each sentence should
textually entail the other. However, many paraphrase datasets currently in
widespread use rely on a sense of paraphrase based on word overlap and syntax.
Can we teach them instead to identify paraphrases in a way that draws on the
inferential properties of the sentences, and is not over-reliant on lexical and
syntactic similarities of a sentence pair? We apply the adversarial paradigm to
this question, and introduce a new adversarial method of dataset creation for
paraphrase identification: the Adversarial Paraphrasing Task (APT), which asks
participants to generate semantically equivalent (in the sense of mutually
implicative) but lexically and syntactically disparate paraphrases. These
sentence pairs can then be used both to test paraphrase identification models
(which get barely random accuracy) and then improve their performance. To
accelerate dataset generation, we explore automation of APT using T5, and show
that the resulting dataset also improves accuracy. We discuss implications for
paraphrase detection and release our dataset in the hope of making paraphrase
detection models better able to detect sentence-level meaning equivalence.

通过对抗方法提出了一个新的数据集创建方式 —— 对抗性同义句生成任务（Adversarial Paraphrasing Task， APT），以更好地检测句级别的意义相等，从而加速数据集生成并提高同义句识别模型的性能。