A cornerstone in AI research has been the creation and adoption of
standardized training and test datasets to earmark the progress of
state-of-the-art models. A particularly successful example is the GLUE dataset
for training and evaluating Natural Language Understanding (NLU) models for
English. The large body of research around self-supervised BERT-based language
models revolved around performance improvements on NLU tasks in GLUE. To
evaluate language models in other languages, several language-specific GLUE
datasets were created. The area of speech language understanding (SLU) has
followed a similar trajectory. The success of large self-supervised models such
as wav2vec2 enable creation of speech models with relatively easy to access
unlabelled data. These models can then be evaluated on SLU tasks, such as the
SUPERB benchmark. In this work, we extend this to Indic languages by releasing
the IndicSUPERB benchmark. Specifically, we make the following three
contributions. (i) We collect Kathbath containing 1,684 hours of labelled
speech data across 12 Indian languages from 1,218 contributors located in 203
districts in India. (ii) Using Kathbath, we create benchmarks across 6 speech
tasks: Automatic Speech Recognition, Speaker Verification, Speaker
Identification (mono/multi), Language Identification, Query By Example, and
Keyword Spotting for 12 languages. (iii) On the released benchmarks, we train
and evaluate different self-supervised models alongside a commonly used
baseline FBANK. We show that language-specific fine-tuned models are more
accurate than baseline on most of the tasks, including a large gap of 76\% for
the Language Identification task. However, for speaker identification,
self-supervised models trained on large datasets demonstrate an advantage. We
hope IndicSUPERB contributes to the progress of developing speech language
understanding models for Indian languages.

该篇论文主要介绍了自然语言理解和语音语言理解中使用的 GLUE 数据集和 SUPERB 基准集，同时提供了 IndicSUPERB 基准集，为印度语音语言理解模型的开发做出贡献。该论文探讨了使用自我监督模型在各种印度语言中进行的训练和评估，证明了 Self-supervised 模型在语言识别任务中的优势。

IndicSUPERB：印度语言语音处理通用性能基准

IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian languages

Transfer learning (TL) in natural language processing (NLP) has seen a surge
of interest in recent years, as pre-trained models have shown an impressive
ability to transfer to novel tasks. Three main strategies have emerged for
making use of multiple supervised datasets during fine-tuning: training on an
intermediate task before training on the target task (STILTs), using multi-task
learning (MTL) to train jointly on a supplementary task and the target task
(pairwise MTL), or simply using MTL to train jointly on all available datasets
(MTL-ALL). In this work, we compare all three TL methods in a comprehensive
analysis on the GLUE dataset suite. We find that there is a simple heuristic
for when to use one of these techniques over the other: pairwise MTL is better
than STILTs when the target task has fewer instances than the supporting task
and vice versa. We show that this holds true in more than 92% of applicable
cases on the GLUE dataset and validate this hypothesis with experiments varying
dataset size. The simplicity and effectiveness of this heuristic is surprising
and warrants additional exploration by the TL community. Furthermore, we find
that MTL-ALL is worse than the pairwise methods in almost every case. We hope
this study will aid others as they choose between TL methods for NLP tasks.

本文研究比较了三种自然语言处理中的迁移学习方法，即 STILTs、MTL 以及 MTL-ALL 的表现。结果显示，MTL-ALL 的性能不如其他两种方法，而在目标任务比较少的情况下，性能较好的是采用 pairwise MTL 方法。

使用多任务学习还是中间微调进行预训练编码器转移学习的决策

When to Use Multi-Task Learning vs Intermediate Fine-Tuning for Pre-Trained Encoder Transfer Learning

The performance of fine-tuning pre-trained language models largely depends on
the hyperparameter configuration. In this paper, we investigate the performance
of modern hyperparameter optimization methods (HPO) on fine-tuning pre-trained
language models. First, we study and report three HPO algorithms' performances
on fine-tuning two state-of-the-art language models on the GLUE dataset. We
find that using the same time budget, HPO often fails to outperform grid search
due to two reasons: insufficient time budget and overfitting. We propose two
general strategies and an experimental procedure to systematically troubleshoot
HPO's failure cases. By applying the procedure, we observe that HPO can succeed
with more appropriate settings in the search space and time budget; however, in
certain cases overfitting remains. Finally, we make suggestions for future
work. Our implementation can be found in
this https URL

本文研究了模型微调和超参数优化的关系，并提出了两种一般策略以及一个实验流程，以系统地解决超参数优化在微调预训练语言模型时的失败案例。通过实验，我们发现在更适当的搜索空间和时间预算设置下，超参数优化可以取得成功，但在某些情况下，过拟合仍然存在。