We introduce the Dutch Model Benchmark: DUMB. The benchmark includes a
diverse set of datasets for low-, medium- and high-resource tasks. The total
set of eight tasks include three tasks that were previously not available in
Dutch. Instead of relying on a mean score across tasks, we propose Relative
Error Reduction (RER), which compares the DUMB performance of models to a
strong baseline which can be referred to in the future even when assessing
different sets of models. Through a comparison of 14 pre-trained models (mono-
and multi-lingual, of varying sizes), we assess the internal consistency of the
benchmark tasks, as well as the factors that likely enable high performance.
Our results indicate that current Dutch monolingual models under-perform and
suggest training larger Dutch models with other architectures and pre-training
objectives. At present, the highest performance is achieved by DeBERTaV3
(large), XLM-R (large) and mDeBERTaV3 (base). In addition to highlighting best
strategies for training larger Dutch models, DUMB will foster further research
on Dutch. A public leaderboard is available at this https URL

我们引入了荷兰模型基准: DUMB。该基准测试包括一组多样化的低、中、高资源任务数据集。通过比较 14 个预先训练的模型 (单语言和多语言，并且大小不同)，我们评估了基准任务的内部一致性，以及可能导致高性能的因素。我们的结果表明，当前的荷兰单语言模型表现不佳，并建议使用其他架构和预训练目标训练更大的荷兰模型。目前，最高性能是由 DeBERTaV3（大型）、XLM-R（大型）和 mDeBERTaV3（基础）实现的。除了突出训练较大的荷兰语模型的最佳策略外，DUMB 还将促进对荷兰语言研究的进一步开展。

DUMB: 用于荷兰模型智能评估的基准测试

DUMB: A Benchmark for Smart Evaluation of Dutch Models

Transformer-based acoustic modeling has achieved great suc-cess for both
hybrid and sequence-to-sequence speech recogni-tion. However, it requires
access to the full sequence, and thecomputational cost grows quadratically with
respect to the in-put sequence length. These factors limit its adoption for
stream-ing applications. In this work, we proposed a novel augmentedmemory
self-attention, which attends on a short segment of theinput sequence and a
bank of memories. The memory bankstores the embedding information for all the
processed seg-ments. On the librispeech benchmark, our proposed
methodoutperforms all the existing streamable transformer methods bya large
margin and achieved over 15% relative error reduction,compared with the widely
used LC-BLSTM baseline. Our find-ings are also confirmed on some large internal
datasets.

本文提出了一种新颖的增强记忆自注意力机制，用于 Transformer 语音识别中的流式应用，相对于现有的可流式 Transformer 方法，减小了计算量并在 Librispeech 基准测试中实现了超过 15% 的相对误差降低。