As Automatic Speech Recognition (ASR) models become ever more pervasive, it
is important to ensure that they make reliable predictions under corruptions
present in the physical and digital world. We propose Speech Robust Bench
(SRB), a comprehensive benchmark for evaluating the robustness of ASR models to
diverse corruptions. SRB is composed of 69 input perturbations which are
intended to simulate various corruptions that ASR models may encounter in the
physical and digital world. We use SRB to evaluate the robustness of several
state-of-the-art ASR models and observe that model size and certain modeling
choices such as discrete representations, and self-training appear to be
conducive to robustness. We extend this analysis to measure the robustness of
ASR models on data from various demographic subgroups, namely English and
Spanish speakers, and males and females, and observed noticeable disparities in
the model's robustness across subgroups. We believe that SRB will facilitate
future research towards robust ASR models, by making it easier to conduct
comprehensive and comparable robustness evaluations.

自动语音识别模型在物理世界和数字世界中的多样化损坏下进行可靠预测的鲁棒性评估是至关重要的。我们提出了一个全面的基准测试工具 Speech Robust Bench (SRB) 来评估 ASR 模型在各种损坏条件下的鲁棒性。通过使用 SRB 评估顶尖 ASR 模型的鲁棒性，我们观察到模型大小、离散表示以及自我训练等建模选择对于鲁棒性有积极影响。我们还进一步测量 ASR 模型在不同社会群体（如英语和西班牙语使用者，男性和女性）的数据上的鲁棒性，并观察到鲁棒性在不同群体之间存在明显差异。我们相信 SRB 将有助于推动鲁棒 ASR 模型的未来研究，使全面且可比较的鲁棒性评估更加容易进行。

语音鲁棒性基准测评：一个语音识别鲁棒性评估基准

Speech Robust Bench: A Robustness Benchmark For Speech Recognition

This paper seeks to develop a deeper understanding of the fundamental
properties of neural text generations models. The study of artifacts that
emerge in machine generated text as a result of modeling choices is a nascent
research area. Previously, the extent and degree to which these artifacts
surface in generated text has not been well studied. In the spirit of better
understanding generative text models and their artifacts, we propose the new
task of distinguishing which of several variants of a given model generated a
piece of text, and we conduct an extensive suite of diagnostic tests to observe
whether modeling choices (e.g., sampling methods, top-$k$ probabilities, model
architectures, etc.) leave detectable artifacts in the text they generate. Our
key finding, which is backed by a rigorous set of experiments, is that such
artifacts are present and that different modeling choices can be inferred by
observing the generated text alone. This suggests that neural text generators
may be more sensitive to various modeling choices than previously thought.

本文提出了一项新任务，即区分生成文本的几个变种中的哪一个，并进行了一系列诊断测试，以观察建模选择（例如取样方法，top-k 概率，模型架构等）是否在生成的文本中留下可检测的痕迹。我们的主要发现是，这些痕迹存在，并且不同的建模选择可以通过观察生成的文本来推断出来，这表明神经文本生成器可能比以前想象的更敏感于各种建模选择。

神经文本生成模型配置反向工程

Reverse Engineering Configurations of Neural Text Generation Models

When building machine learning models that operate on source code, several
decisions have to be made to model source-code vocabulary. These decisions can
have a large impact: some can lead to not being able to train models at all,
others significantly affect performance, particularly for Neural Language
Models. Yet, these decisions are not often fully described. This paper lists
important modeling choices for source code vocabulary, and explores their
impact on the resulting vocabulary on a large-scale corpus of 14,436 projects.
We show that a subset of decisions have decisive characteristics, allowing to
train accurate Neural Language Models quickly on a large corpus of 10,106
projects.

本研究探讨了对源代码词汇进行建模时需要做出的关键决策，研究了这些决策对 14,436 个项目的大规模语料库所产生的词汇的影响，并表明其中的一些决策具有决定性特征，允许在大规模的 10,106 个项目的语料库上快速训练准确的神经语言模型。