Deep neural networks have been applied to audio spectrograms for respiratory
sound classification. Existing models often treat the spectrogram as a
synthetic image while overlooking its physical characteristics. In this paper,
a Multi-View Spectrogram Transformer (MVST) is proposed to embed different
views of time-frequency characteristics into the vision transformer.
Specifically, the proposed MVST splits the mel-spectrogram into different sized
patches, representing the multi-view acoustic elements of a respiratory sound.
These patches and positional embeddings are then fed into transformer encoders
to extract the attentional information among patches through a self-attention
mechanism. Finally, a gated fusion scheme is designed to automatically weigh
the multi-view features to highlight the best one in a specific scenario.
Experimental results on the ICBHI dataset demonstrate that the proposed MVST
significantly outperforms state-of-the-art methods for classifying respiratory
sounds.

提出了一种 Multi-View Spectrogram Transformer (MVST) 模型，将深度神经网络应用于呼吸音频谱图的分类，通过将 mel-spectrogram 分为不同大小的 patches，使用 transformer encoders 提取 patches 之间的注意力信息，并设计了一种门控融合机制来加强多视图特征，在呼吸音分类任务上明显优于现有最先进方法。