Lattices are compact representations that encode multiple hypotheses, such as
speech recognition results or different word segmentations. It is shown that
encoding lattices as opposed to 1-best results generated by automatic speech
recognizer (ASR) boosts the performance of spoken language understanding (SLU).
Recently, pretrained language models with the transformer architecture have
achieved the state-of-the-art results on natural language understanding, but
their ability of encoding lattices has not been explored. Therefore, this paper
aims at adapting pretrained transformers to lattice inputs in order to perform
understanding tasks specifically for spoken language. Our experiments on the
benchmark ATIS dataset show that fine-tuning pretrained transformers with
lattice inputs yields clear improvement over fine-tuning with 1-best results.
Further evaluation demonstrates the effectiveness of our methods under
different acoustic conditions. Our code is available at
this https URL

本篇论文旨在将 Transformer 预训练模型适应于格输入以执行口语理解任务，并在 ATIS 基准数据集上进行了实验，结果表明，将预训练的 Transformers fine-tuning 用于格输入比用于 1-best 结果有了显著改进，证明了方法的有效性。

将预训练 Transformer 适应于格上，以进行口语语言理解

Adapting Pretrained Transformer to Lattices for Spoken Language  Understanding

Task-agnostic forms of data augmentation have proven widely effective in
computer vision, even on pretrained models. In NLP similar results are reported
most commonly for low data regimes, non-pretrained models, or situationally for
pretrained models. In this paper we ask how effective these techniques really
are when applied to pretrained transformers. Using two popular varieties of
task-agnostic data augmentation (not tailored to any particular task), Easy
Data Augmentation (Wei and Zou, 2019) and Back-Translation (Sennrichet al.,
2015), we conduct a systematic examination of their effects across 5
classification tasks, 6 datasets, and 3 variants of modern pretrained
transformers, including BERT, XLNet, and RoBERTa. We observe a negative result,
finding that techniques which previously reported strong improvements for
non-pretrained models fail to consistently improve performance for pretrained
transformers, even when training data is limited. We hope this empirical
analysis helps inform practitioners where data augmentation techniques may
confer improvements.

本文通过对 BERT、XLNet 和 RoBERTa 共 3 个方面、5 个分类任务和 6 个数据集上应用非针对特定任务的数据增强技术（easy data augmentation 和 back-translation）的实验研究，发现之前针对非预训练模型的技术并未能有效地提高预训练模型的性能表现，从而提醒实践者在进行数据增强时需谨慎。

预训练 Transformer 的任务无关数据增强效果如何？

How Effective is Task-Agnostic Data Augmentation for Pretrained  Transformers?

Although pretrained Transformers such as BERT achieve high accuracy on
in-distribution examples, do they generalize to new distributions? We
systematically measure out-of-distribution (OOD) generalization for seven NLP
datasets by constructing a new robustness benchmark with realistic distribution
shifts. We measure the generalization of previous models including bag-of-words
models, ConvNets, and LSTMs, and we show that pretrained Transformers'
performance declines are substantially smaller. Pretrained transformers are
also more effective at detecting anomalous or OOD examples, while many previous
models are frequently worse than chance. We examine which factors affect
robustness, finding that larger models are not necessarily more robust,
distillation can be harmful, and more diverse pretraining data can enhance
robustness. Finally, we show where future work can improve OOD robustness.

本文通过构建具有现实分布变化的新的鲁棒性基准，系统地测量了七个 NLP 数据集的预训练 Transformers 的 OOD 泛化能力。研究结果表明，Pretrained Transformers 的性能下降显著较小，在检测异常或 OOD 示例方面更加有效，同时发现更多样化的预训练数据可以增强模型的鲁棒性。