Many NLP datasets have been found to contain shortcuts: simple decision rules
that achieve surprisingly high accuracy. However, it is difficult to discover
shortcuts automatically. Prior work on automatic shortcut detection has focused
on enumerating features like unigrams or bigrams, which can find only low-level
shortcuts, or relied on post-hoc model interpretability methods like saliency
maps, which reveal qualitative patterns without a clear statistical
interpretation. In this work, we propose to use probabilistic grammars to
characterize and discover shortcuts in NLP datasets. Specifically, we use a
context-free grammar to model patterns in sentence classification datasets and
use a synchronous context-free grammar to model datasets involving sentence
pairs. The resulting grammars reveal interesting shortcut features in a number
of datasets, including both simple and high-level features, and automatically
identify groups of test examples on which conventional classifiers fail.
Finally, we show that the features we discover can be used to generate
diagnostic contrast examples and incorporated into standard robust optimization
methods to improve worst-group accuracy.

本文提出使用概率语法学习发掘 NLP 数据集中的 shortcut，同时也展示这些 features 能够帮助分类器提高最坏组的准确性。

使用语法归纳寻找数据集快捷方式

Finding Dataset Shortcuts with Grammar Induction

Virtual assistants make use of automatic speech recognition (ASR) to help
users answer entity-centric queries. However, spoken entity recognition is a
difficult problem, due to the large number of frequently-changing named
entities. In addition, resources available for recognition are constrained when
ASR is performed on-device.
In this work, we investigate the use of probabilistic grammars as language
models within the finite-state transducer (FST) framework. We introduce a
deterministic approximation to probabilistic grammars that avoids the explicit
expansion of non-terminals at model creation time, integrates directly with the
FST framework, and is complementary to n-gram models.
We obtain a 10% relative word error rate improvement on long tail entity
queries compared to when a similarly-sized n-gram model is used without our
method.

本文通过在有限状态转换器框架内使用概率语法，提出了一种对虚拟助手的语音识别有着较好性能的方法，适用于实体识别问题。使用该方法相较于相同大小的 n-gram 模型，长尾实体查询的相对单词错误率提高了 10%。

基于实体中心查询语言模型的节省空间表示

Space-Efficient Representation of Entity-centric Query Language Models

We describe a framework for inducing probabilistic grammars from corpora of
positive samples. First, samples are {\em incorporated} by adding ad-hoc rules
to a working grammar; subsequently, elements of the model (such as states or
nonterminals) are {\em merged} to achieve generalization and a more compact
representation. The choice of what to merge and when to stop is governed by the
Bayesian posterior probability of the grammar given the data, which formalizes
a trade-off between a close fit to the data and a default preference for
simpler models (`Occam's Razor'). The general scheme is illustrated using three
types of probabilistic grammars: Hidden Markov models, class-based $n$-grams,
and stochastic context-free grammars.

该研究提供一种产生概率语法的框架，通过对正例语料库进行舍选规则加入工作语法，利用贝叶斯后验概率来决定合并元素的时机，从而得到更通用且更紧凑的概率语法模型。本研究运用了三种类型的概率语法模型：隐马尔可夫模型，基于类的 n 元模型，和随机上下文无关文法。