Transformers have drawn attention in the MIR field for their remarkable
performance shown in natural language processing and computer vision. However,
prior works in the audio processing domain mostly use Transformer as a temporal
feature aggregator that acts similar to RNNs. In this paper, we propose
SpecTNT, a Transformer-based architecture to model both spectral and temporal
sequences of an input time-frequency representation. Specifically, we introduce
a novel variant of the Transformer-in-Transformer (TNT) architecture. In each
SpecTNT block, a spectral Transformer extracts frequency-related features into
the frequency class token (FCT) for each frame. Later, the FCTs are linearly
projected and added to the temporal embeddings (TEs), which aggregate useful
information from the FCTs. Then, a temporal Transformer processes the TEs to
exchange information across the time axis. By stacking the SpecTNT blocks, we
build the SpecTNT model to learn the representation for music signals. In
experiments, SpecTNT demonstrates state-of-the-art performance in music tagging
and vocal melody extraction, and shows competitive performance for chord
recognition. The effectiveness of SpecTNT and other design choices are further
examined through ablation studies.

本文介绍了一种基于 Transformer 网络的音频处理系统 SpecTNT，使用自己提出的 Transformer-in-Transformer 架构实现了同时建模时域和频域信息的效果，在乐曲分析的任务上达到了当时最好的性能。