In the last five years, the rise of the self-attentional Transformer-based
architectures led to state-of-the-art performances over many natural language
tasks. Although these approaches are increasingly popular, they require large
amounts of data and computational resources. There is still a substantial need
for benchmarking methodologies ever upwards on under-resourced languages in
data-scarce application conditions. Most pre-trained language models were
massively studied using the English language and only a few of them were
evaluated on French. In this paper, we propose a unified benchmark, focused on
evaluating models quality and their ecological impact on two well-known French
spoken language understanding tasks. Especially we benchmark thirteen
well-established Transformer-based models on the two available spoken language
understanding tasks for French: MEDIA and ATIS-FR. Within this framework, we
show that compact models can reach comparable results to bigger ones while
their ecological impact is considerably lower. However, this assumption is
nuanced and depends on the considered compression method.

本研究采用新的基准测试方法，重点评估了 13 种 Transformer 模型在两个已有的法语口语理解任务中的质量和其生态影响，并证明了压缩模型可以取得较大模型相似的结果，同时具有较小的生态环境影响。