In recent years, a great deal of attention has been paid to the Transformer
network for speech recognition tasks due to its excellent model performance.
However, the Transformer network always involves heavy computation and large
number of parameters, causing serious deployment problems in devices with
limited computation sources or storage memory. In this paper, a new lightweight
model called Sim-T has been proposed to expand the generality of the
Transformer model. Under the help of the newly developed multiplexing
technique, the Sim-T can efficiently compress the model with negligible
sacrifice on its performance. To be more precise, the proposed technique
includes two parts, that are, module weight multiplexing and attention score
multiplexing. Moreover, a novel decoder structure has been proposed to
facilitate the attention score multiplexing. Extensive experiments have been
conducted to validate the effectiveness of Sim-T. In Aishell-1 dataset, when
the proposed Sim-T is 48% parameter less than the baseline Transformer, 0.4%
CER improvement can be obtained. Alternatively, 69% parameter reduction can be
achieved if the Sim-T gives the same performance as the baseline Transformer.
With regard to the HKUST and WSJ eval92 datasets, CER and WER will be improved
by 0.3% and 0.2%, respectively, when parameters in Sim-T are 40% less than the
baseline Transformer.

本文提出了一种新型轻量级模型 Sim-T，通过分别运用模块权重多路复用和注意力分数多路复用两种技术，实现了对 Transformer 模型的有效压缩，并验证了其在语音识别任务上的高效性和优越性。