Memory constraint of always-on devices is one of the major concerns when
deploying speech processing models on these devices. While larger models
trained with sufficiently large amount of data generally perform better, making
them fit in the device memory is a demanding challenge. In this paper, we aim
to reduce model size by reparameterizing model weights across Transformer
encoder layers and assuming a special weight composition and structure. More
specifically, inspired by ResNet and the more recent LoRA work, we propose an
approach named ResidualTransformer, where each weight matrix in a Transformer
layer comprises 1) a shared full-rank component with its adjacent layers, and
2) a unique low-rank component to itself. The low-rank matrices only account
for a small amount of model size increase. In addition, we add diagonal weight
matrices to improve modeling capacity of the low-rank matrices. Experiments of
our 10k-hour speech recognition and speech translation tasks show that the
Transformer encoder size can be reduced by ~3X with very slight performance
degradation.

通过重新参数化模型权重，将 Transformer 编码器层中的模型权重缩小，可以降低模型大小，减轻设备内存的压力，并在语音处理任务上取得了显著的性能改进。