In a spoken dialogue system, an NLU model is preceded by a speech recognition
system that can deteriorate the performance of natural language understanding.
This paper proposes a method for investigating the impact of speech recognition
errors on the performance of natural language understanding models. The
proposed method combines the back transcription procedure with a fine-grained
technique for categorizing the errors that affect the performance of NLU
models. The method relies on the usage of synthesized speech for NLU
evaluation. We show that the use of synthesized speech in place of audio
recording does not change the outcomes of the presented technique in a
significant way.

通过使用合成语音进行 NLU 模型评估，本文提出了一种方法来研究语音识别错误对自然语言理解模型性能的影响，并结合细粒度的错误分类技术。结果表明，使用合成语音代替音频记录在很大程度上不改变所提出的技术的结果。

自然语言理解模型对语音识别错误的鲁棒性评估的一种方法

Back Transcription as a Method for Evaluating Robustness of Natural  Language Understanding Models to Speech Recognition Errors

In this article, we use probing to investigate phenomena that occur during
fine-tuning and knowledge distillation of a BERT-based natural language
understanding (NLU) model. Our ultimate purpose was to use probing to better
understand practical production problems and consequently to build better NLU
models. We designed experiments to see how fine-tuning changes the linguistic
capabilities of BERT, what the optimal size of the fine-tuning dataset is, and
what amount of information is contained in a distilled NLU based on a tiny
Transformer. The results of the experiments show that the probing paradigm in
its current form is not well suited to answer such questions. Structural, Edge
and Conditional probes do not take into account how easy it is to decode probed
information. Consequently, we conclude that quantification of information
decodability is critical for many practical applications of the probing
paradigm.

本论文通过 probing 调查 fine-tuning 和 knowledge distillation 过程中，BERT 基础的自然语言理解（NLU）模型出现的现象。实验结果表明，当前形式的 probing 范式不适合回答这些问题，因此，信息可解码的量化是探测范式在许多实际应用中至关重要的。

能否利用探测来更好地理解 BERT NLU 的微调和知识蒸馏？

Can We Use Probing to Better Understand Fine-tuning and Knowledge Distillation of the BERT NLU?

Natural language understanding (NLU) converts sentences into structured
semantic forms. The paucity of annotated training samples is still a
fundamental challenge of NLU. To solve this data sparsity problem, previous
work based on semi-supervised learning mainly focuses on exploiting unlabeled
sentences. In this work, we introduce a dual task of NLU, semantic-to-sentence
generation (SSG), and propose a new framework for semi-supervised NLU with the
corresponding dual model. The framework is composed of dual pseudo-labeling and
dual learning method, which enables an NLU model to make full use of data
(labeled and unlabeled) through a closed-loop of the primal and dual tasks. By
incorporating the dual task, the framework can exploit pure semantic forms as
well as unlabeled sentences, and further improve the NLU and SSG models
iteratively in the closed-loop. The proposed approaches are evaluated on two
public datasets (ATIS and SNIPS). Experiments in the semi-supervised setting
show that our methods can outperform various baselines significantly, and
extensive ablation studies are conducted to verify the effectiveness of our
framework. Finally, our method can also achieve the state-of-the-art
performance on the two datasets in the supervised setting. Our code is
available at https://github.com/rhythmcao/slu-dual-learning.git.

通过引入一个新的 NLU 双重任务 —— 语义到句子生成 (SSG)，并提出一种新的半监督 NLU 框架，其中包括对应的双重模型，以解决之前半监督学习面临的标注样本稀缺问题。该框架通过主任务与对偶任务之间的闭环，使得 NLU 模型可以充分利用数据（标记和无标记），并在每次迭代中逐步提高 NLU 和 SSG 模型的性能，在 ATIS 和 SNIPS 两个公共数据集上的实验结果表明，该方法明显优于基线，而且在监督设置下也可以达到最先进水平。