Self-supervised learning (SSL) has recently allowed leveraging large datasets of unlabeled speech signals to reach impressive performance on speech tasks using only small amounts of annotated data. The high number of proposed approaches fostered the need and rise of extended benchmarks that evaluate their performance on a set of downstream tasks exploring various aspects of the speech signal. However, and while the number of considered tasks has been growing, most rely upon a single decoding architecture that maps the frozen SSL representations to the downstream labels. This work investigates the robustness of such benchmarking results to changes in the decoder architecture. Interestingly, it appears that varying the architecture of the downstream decoder leads to significant variations in the leaderboards of most tasks. Concerningly, our study reveals that benchmarking using limited decoders may cause a counterproductive increase in the sizes of the developed SSL models.

本文研究了自我监督学习在语音任务中的应用，特别是其在下游任务中的性能表现和解码器架构的影响。结果发现，使用不同的解码器架构可能会导致表现结果的显著变化，使用局限的解码器也可能会适得其反地增加SSL模型的大小。

语音自监督表示基准测试：我们做得对吗？