Self-supervised learning (SSL) speech models, which can serve as powerful
upstream models to extract meaningful speech representations, have achieved
unprecedented success in speech representation learning. However, their
effectiveness on non-speech datasets is relatively less explored. In this work,
we propose an ensemble framework, with a combination of ensemble techniques, to
fuse SSL speech models' embeddings. Extensive experiments on speech and
non-speech audio datasets are conducted to investigate the representation
abilities of our ensemble method and its single constituent model. Ablation
studies are carried out to evaluate the performances of different ensemble
techniques, such as feature averaging and concatenation. All experiments are
conducted during NeurIPS 2021 HEAR Challenge as a standard evaluation pipeline
provided by competition officials. Results demonstrate SSL speech models'
strong abilities on various non-speech tasks, while we also note that they fail
to deal with fine-grained music tasks, such as pitch classification and note
onset detection. In addition, feature ensemble is shown to have great potential
on producing more holistic representations, as our proposed framework generally
surpasses state-of-the-art SSL speech/audio models and has superior performance
on various datasets compared with other teams in HEAR Challenge. Our code is
available at this https URL --
NTU-GURA.

本研究提出融合自监督学习语音模型嵌入的集成框架，旨在探究其在音频和非语音任务中的表示能力，实验证明该框架普遍优于当前最先进的自监督学习语音 / 音频模型，特别在面对细粒度音乐任务时也表现出强大的能力。

自监督语音模型在音频表示方面的功效

The Efficacy of Self-Supervised Speech Models for Audio Representations

Transformer-based methods have achieved impressive image restoration
performance due to their capacities to model long-range dependency compared to
CNN-based methods. However, advances like SwinIR adopts the window-based and
local attention strategy to balance the performance and computational overhead,
which restricts employing large receptive fields to capture global information
and establish long dependencies in the early layers. To further improve the
efficiency of capturing global information, in this work, we propose SwinFIR to
extend SwinIR by replacing Fast Fourier Convolution (FFC) components, which
have the image-wide receptive field. We also revisit other advanced techniques,
i.e, data augmentation, pre-training, and feature ensemble to improve the
effect of image reconstruction. And our feature ensemble method enables the
performance of the model to be considerably enhanced without increasing the
training and testing time. We applied our algorithm on multiple popular
large-scale benchmarks and achieved state-of-the-art performance comparing to
the existing methods. For example, our SwinFIR achieves the PSNR of 32.83 dB on
Manga109 dataset, which is 0.8 dB higher than the state-of-the-art SwinIR
method.

本文介绍了一种基于 Transformer 的方法，用于图像恢复，提出了一种名为 SwinFIR 的扩展方法，旨在提高捕捉全局信息的效率，同时改进了数据增强、预训练、特征集成等高级技术，使模型的性能得到显着提高，并在多个大型基准测试上取得了最先进的性能。