We introduce the Dutch Model Benchmark: DUMB. The benchmark includes a
diverse set of datasets for low-, medium- and high-resource tasks. The total
set of eight tasks include three tasks that were previously not available in
Dutch. Instead of relying on a mean score across tasks, we propose Relative
Error Reduction (RER), which compares the DUMB performance of models to a
strong baseline which can be referred to in the future even when assessing
different sets of models. Through a comparison of 14 pre-trained models (mono-
and multi-lingual, of varying sizes), we assess the internal consistency of the
benchmark tasks, as well as the factors that likely enable high performance.
Our results indicate that current Dutch monolingual models under-perform and
suggest training larger Dutch models with other architectures and pre-training
objectives. At present, the highest performance is achieved by DeBERTaV3
(large), XLM-R (large) and mDeBERTaV3 (base). In addition to highlighting best
strategies for training larger Dutch models, DUMB will foster further research
on Dutch. A public leaderboard is available at this https URL

我们引入了荷兰模型基准: DUMB。该基准测试包括一组多样化的低、中、高资源任务数据集。通过比较 14 个预先训练的模型 (单语言和多语言，并且大小不同)，我们评估了基准任务的内部一致性，以及可能导致高性能的因素。我们的结果表明，当前的荷兰单语言模型表现不佳，并建议使用其他架构和预训练目标训练更大的荷兰模型。目前，最高性能是由 DeBERTaV3（大型）、XLM-R（大型）和 mDeBERTaV3（基础）实现的。除了突出训练较大的荷兰语模型的最佳策略外，DUMB 还将促进对荷兰语言研究的进一步开展。